[Q&A] 400 - Rejected by Elasticsearch errors

[jalsanishant]

What is a problem?

I cannot see any buffers being created in fluentdpod. But I can see Flows, Outputs good configured. It's same & working in Dev/QA but this buffers are not working in Prod.

Describe the configuration of Fluentd

we have cattle-logging-system namespace with rancher-logging-fluentd pod in it which draws logs to elastic from the app deployment in different namespace in same cluster. But I can see logs in elastic for a different deployment(database deployment) in same namespace to that of app deployment.

Please let me know where to check or if any misconfiguration that leading to 400 errros or if any other details needed from me so I can provide those. Thanks in advance!

Describe the logs of Fluentd

I'm seeing 400 errros.

2024-10-01 13:11:18 +0000 [warn]: #0 send an error event to https://github.com/error: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.containers.ddb.log" time=2024-10-01 13:11:15.329969734 +0000

---

Environment

- Fluentd version: fluentd:v1.11.5
- Operating system:NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.13.4
PRETTY_NAME="Alpine Linux v3.13"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
- Kernel version: 5.4.17-2136.330.7.1.el7uek.x86_64