Resolve kafka consumer dropped traces errors in Datadog

[robrap]

I noticed that the error in logs for
"failed to send, dropping 1 traces to intake at unix:///var/run/datadog/apm.socket/v0.5/traces after 3 retries" seems to be hitting our kafka consumers. It may be hitting some other workers, but not sure if we just have inconsistent naming.

I'm wondering if this has anything to do with the long-running infinite loop on the consumers, and if we need to clean up the trace, like we clean up the db connection, etc.?

A/C:

• Confirm with DD Support that this is actually important to fix and doesn't just represent an expected level of failed trace sends
  • They should also be able to give us some debugging pointers
• Either document as wontfix, or fix it
  • Confirm fix (on or after Sept 13th)
  • Communicate fix for non-edxapp workers.
  • Confirm if edxapp workers get fixed in Nov 2024 (SRE believes this to be fixed)
  • Maybe update thread once edxapp workers have been fixed.

[robrap]

Created ticket: https://help.datadoghq.com/hc/en-us/requests/1805067

[robrap]

After thinking about how to respond to the DD ticket, and looking at logs more closely, I decided to open an SRE ticket for further investigation: https://2u-internal.atlassian.net/browse/GSRE-1988.

[robrap]

I tried looking in AWS a bit, but this really needs SRE support for now. Marking as blocked and I'll check in on the GSRE ticket in a few weeks.

[robrap]

[update] Blocked on the GSRE ticket which was picked up on Aug 22, but no comments were added yet.

[robrap]

I will confirm that this has actually gone away.

[robrap]

• The fix was deployed on Sept 5th @ 10:00am ET, and the GSRE ticket was closed.
• I'd like to leave this blocked for at least a week (Sept 13th), and then confirm that the original search confirms that the issue has gone away. So far so good, but it has only been a day.

Todo:

• Confirm fix.
• Communicate, in case anyone was aware.

[robrap]

• I confirmed that this was mostly fixed, but still seems to be an issue for edxapp workers. I added a comment on https://2u-internal.atlassian.net/browse/GSRE-1988?focusedCommentId=5176626
• Will reconfirm after I hear back from SRE.
• I can send a note for the rest of the services.

[robrap]

Slack announce of initial fix.

[robrap]

Note: Original GSRE comment on Sept 26 did not have a response yet. Posted a reminder today on Oct 2.

[robrap]

SRE added a fix for edxapp workers on the morning of Oct 4. There were 2 week gaps between issues, so I'll leave this in blocked and wait until Nov to confirm.

[robrap]

Another spike on Oct-10, but this was for the k8s servers.

• Nov is still a good time to confirm the edxapp workers fix.
• But, we need to check in on the GSRE ticket to see if there will be a new change to review.

[robrap]

The Oct-10 spike was because the Datadog agent was restarted for other purposes (SRE working on log parsing issues).