Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add shared client performance disclaimer #149

Merged
merged 4 commits into from
Jul 18, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion lib/broadway_kafka/producer.ex
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ defmodule BroadwayKafka.Producer do

* `:shared_client` - Optional. When false, it starts one client per producer.
When true, it starts a single shared client across all producers (which may reduce
memory/resource usage). Default is `false`.
memory/resource usage). May cause severe performance degradation, see
["Shared Client Performance"](#module-shared-client-performance) for details. Default is `false`.

* `:group_config` - Optional. A list of options used to configure the group
coordinator. See the ["Group config options"](#module-group-config-options) section below for a list of all available
Expand Down Expand Up @@ -215,6 +216,19 @@ defmodule BroadwayKafka.Producer do
* `[:broadway_kafka, :assignments_revoked, :start | :stop | :exception]` spans -
these events are emitted in "span style" when receiving assignments revoked call from consumer group coordinator
See `:telemetry.span/3`.

## Shared Client Performance

Enabling shared client may drastically decrease performance. Since connection is handled by a single process,
producers may block each other waiting for the client response.

This is more likely to be an issue if the producers on your pipeline are fetching message from
multiple topics and specially if there are very low traffic topics, which may block on batch wait times.

To mitigate this, you can split your topics between multiple pipelines, but notice that this will
increase the resource usage as well. By creating one new client/connection for each pipeline,
you effectively diminishing the `shared_client` resource usage gains. So make sure to measure
if you enable this option.
"""

use GenStage
Expand Down
Loading