Performance Optimisation of fluentD for opensearch transactions #4473
sankar-kumar
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
In my scenario, I have 80000 messages per second and I have to store in elasticsearch/opensearch which will run continuously for months.
Setup: 3 nodes(12 CPU, 128GB RAM) deployed in same ESXI host. opensearch is interconnected between these nodes.
Actual Process: 1 node will receive the docs in the form of UDP packets and distribute it with other 2 nodes via forward plugin to store it in opensearch.
ISSUE we are facing is, while storing it in opensearch in few minutes we are seeing slow flush log thrushold warning and read timed out. it is taking several retry attempts to store the documents in opensearch and some of the events are dropped in pipeline.
Questions:
Beta Was this translation helpful? Give feedback.
All reactions