kafka2: Could not connect to leader for partition

[Matiasmct]

Hi all,

We're randomly having an issue with kafka2 plugin when connection to Amazon MSK, where fluentd pods running on kubernetes throw the following error:

`2023-03-15 19:33:01 +0000 [error]: #0 Could not connect to leader for partition logging.logstash.events/14: no partition 14 in topic logging.logstash.events

2023-03-15 19:33:03 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2023-03-15 19:33:04 +0000 chunk="5f6f5693c3a1a0e003defdaa34611a36" error_class=Kafka::DeliveryFailed error="Failed to send messages to logging.logstash.events/12, logging.logstash.events/13, logging.logstash.events/14"`

We have only 12 partitions in this topic, and for some reason fluentd is looking for partition 14, which doesn't exist. Has anyone come across this issue?

[Matiasmct]

Any updates on this?