Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add shared client performance disclaimer #149

Merged
merged 4 commits into from
Jul 18, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion lib/broadway_kafka/producer.ex
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ defmodule BroadwayKafka.Producer do

* `:shared_client` - Optional. When false, it starts one client per producer.
When true, it starts a single shared client across all producers (which may reduce
memory/resource usage). Default is `false`.
memory/resource usage). May cause severe performance degradation, see
["Shared Client Performance"](#module-shared-client-performance) for details. Default is `false`.

* `:group_config` - Optional. A list of options used to configure the group
coordinator. See the ["Group config options"](#module-group-config-options) section below for a list of all available
Expand Down Expand Up @@ -215,6 +216,19 @@ defmodule BroadwayKafka.Producer do
* `[:broadway_kafka, :assignments_revoked, :start | :stop | :exception]` spans -
these events are emitted in "span style" when receiving assignments revoked call from consumer group coordinator
See `:telemetry.span/3`.

## Shared Client Performance

Enable shared client may drastically decrease performance. This happens because mutiple producers may block
each other waiting for the client response since the connection is hidden inside a process it becomes a
bottleneck.
oliveigah marked this conversation as resolved.
Show resolved Hide resolved

This is more likely to be an issue if the producers on your pipeline are fetching message from multiple topics
and specially if there are very low traffic topics in the mix because of batch wait times.
oliveigah marked this conversation as resolved.
Show resolved Hide resolved

In summary to mitigate this you can split your topics between multiple pipelines, but notice that this will
increase the resource usage as well creating one new client/connection for each pipeline effectively diminishing
the shared_client resource usage gains.
oliveigah marked this conversation as resolved.
Show resolved Hide resolved
"""

use GenStage
Expand Down
Loading