Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka timeouts now throw an exception instead of marking the batch complete #106

Open
wants to merge 5 commits into
base: 3.10
Choose a base branch
from

Conversation

nikidimi
Copy link

KafkaWriterFilter will now throw an RuntimeException instead of failing silently (only with a log message) and marking the batch as complete.

Calling get() on the Future is necessary because producer.close() doesn't do anything on timeout.

This is the result after this patch:

`[data-116] - NetworkClient - [Producer clientId=SymmetricDS] Connection to node -1 could not be established. Broker may not be available.
[data-116] - NetworkClient - [Producer clientId=SymmetricDS] Connection to node -1 could not be established. Broker may not be available.
[data-116] - NetworkClient - [Producer clientId=SymmetricDS] Connection to node -1 could not be established. Broker may not be available.
[data-116] - NetworkClient - [Producer clientId=SymmetricDS] Connection to node -1 could not be established. Broker may not be available.
[data-116] - NetworkClient - [Producer clientId=SymmetricDS] Connection to node -1 could not be established. Broker may not be available.
[data-116] - NetworkClient - [Producer clientId=SymmetricDS] Connection to node -1 could not be established. Broker may not be available.
[data-116] - NetworkClient - [Producer clientId=SymmetricDS] Connection to node -1 could not be established. Broker may not be available.
[data-116] - IDatabaseWriterFilter - Unable to write batch to Kafka batch-000-143
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
...
[data-116] - ManageIncomingBatchListener - Failed to load batch 000-143
java.lang.RuntimeException: Unable to write batch to Kafka batch-000-143
...
[data-116] - DataLoaderService - 1 data and 1 batches loaded during push request from 166:000:000. There were 1 batches in error
[data-166] - AcknowledgeService - The outgoing batch 001-143 failed: Failed to update metadata after 60000 ms.
[data-116] - IncomingBatchService - Retrying batch 000-143

`

@rudiejd
Copy link
Contributor

rudiejd commented Jul 28, 2021

Could you please rebase this to 3.12? Thank you

@rudiejd rudiejd added the waiting on author waiting on fix/reply from author label Jul 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
waiting on author waiting on fix/reply from author
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants