"Incorrect Index" error handling

[vinzent]

Hi

Users are allowed to set splunk.com/index annotations on pods. So it is expected that sometimes they are producing typos in the index name. This leads to the issue that the splunk server repsonds with:

{"text":"Incorrect index","code":7,"invalid-event-number":1}

It points to the root cause, the incorrect index, but doesn't mention what the incorrect name is. It would be very useful to have the name of the incorrect index to narrow down the faulty pod.

Currently I need to list all pods, create a unique list of the splunk.com/index annotations, ask the splunk admin to send me a list of valid indexes and then compare.

How do you handle such cases?

Full error logged by splunk otel collector:

2023-07-12T11:23:04.085Z        error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Permanent error: \"HTTP/1.1 400 Bad Request\\r\\nContent-Length: 60\\r\\nConnection: Keep-Alive\\r\\nContent-Type: application/json; charset=UTF-8\\r\\nDate: Thu, 12 Jul 2023 11:23:04 GMT\\r\\nServer: Splunkd\\r\\nVary: Authorization\\r\\nX-Content-Type-Options: nosniff\\r\\nX-Frame-Options: SAMEORIGIN\\r\\n\\r\\n{\\\"text\\\":\\\"Incorrect index\\\",\\\"code\\\":7,\\\"invalid-event-number\\\":1}\"", "dropped_items": 1}