Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Describing Kafka consumer behaviour when consuming from multiple topics #3931

Merged
merged 9 commits into from
Jan 11, 2024
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,40 @@ auth:
secretStore: <SECRET_STORE_NAME>
```

## Consuming from multiple topics
KrylixZA marked this conversation as resolved.
Show resolved Hide resolved

When consuming from multiple topics using a single pub/sub component, there is no guarantee about how the consumers in your consumer group are balanced across the topic partitions. For instance, if you are subscribing to two topics with 10 partitions per topic and you have 20 replicas of your service consuming from the two topics, then there is no guarantee that 10 will be assigned to the first topic and 10 to the second topic. It could be possible that more than 10 are assigned to the first topic and subsequently less than 10 assigned to the second topic. This can result in idle consumers listening to the first topic and over-extended consumers on the second topic, or vice versa. This same beheaviour can be observed when using auto-scalers such as HPA or KEDA.
msfussell marked this conversation as resolved.
Show resolved Hide resolved

If you run into this particular issue, it is recommended that you configure a single pub/sub component per topic with uniquely defined consumer groups per component. This guarantees that all replicas of your service are fully allocated to the unique consumer group, where each consumer group targets one specific topic.

For example, you may define two Dapr components with the following configuration:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub-topic-one
spec:
type: pubsub.kafka
version: v1
metadata:
- name: consumerGroup
value: "{appID}-topic-one"
```

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub-topic-two
spec:
type: pubsub.kafka
version: v1
metadata:
- name: consumerGroup
value: "{appID}-topic-two"
```

## Sending and receiving multiple messages

Apache Kafka component supports sending and receiving multiple messages in a single operation using the bulk Pub/sub API.
Expand Down
Loading