one of druid supervisor going into unhealthy state frequently

[shridhar95]

Hi Team,
We have been observed in diksha druid that one of the supervisor going into the unhealthy state frequently. if we suspend and resume it, it will start running.
we have checked the log and found the following logs,
{"task":"index_kafka_telemetry-events-syncts_a243276ab4df7cf_fjgihpoj","status":{"id":"index_kafka_telemetry-events-syncts_a243276ab4df7cf_fjgihpoj","groupId":"index_kafka_telemetry-events-syncts","type":"index_kafka","createdTime":"2023-11-29T05:30:37.303Z","queueInsertionTime":"1970-01-01T00:00:00.000Z","statusCode":"FAILED","status":"FAILED","runnerStatusCode":"WAITING","duration":-1,"location":{"host":null,"port":-1,"tlsPort":-1},"dataSource":"telemetry-events-syncts","errorMsg":"Task [index_kafka_telemetry-events-syncts_a243276ab4df7cf_fjgihpoj] failed to stop in a timely manne..."}}

we kindly requesting you to please help us

[manjudr]

@shridhar95 - The provided log information is hard to analyse. Please provide additional information on the following questions.

1. Any middle manager restarts? If so, could you please share details on the associated error?

2. What is the daily data ingestion volume?

3. What is the task duration value of the ingestion task?

4. What is the allocated memory for the middle manager? How many peons are configured and memory information peons.

5. How many supervisor tasks are currently operational?

6. If you see any error logs in the ingestion task id index_kafka_telemetry-events-syncts_a243276ab4df7cf_fjgihpoj log file, Please share if you find