fix(llmobs): recreate writer on fork

[Yun-Kim]

This PR ensures that the LLMObsSpanWriter is correctly recreated and restarted on a forked process.

Previously, on a process fork we were not recreating/restarting the writer worker correctly. This resulted in situations with using celery/gunicorn where LLMObs spans were created, but never submitted as the forked worker process did not have a running writer thread.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

releasenotes/notes/fix-llmobs-forked-writer-257b993bcf131af8.yaml       @DataDog/apm-python
ddtrace/llmobs/__init__.py                                              @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_writer.py                                               @DataDog/ml-observability
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: yunkim/llmobs-fix-fork-writer
Commit report: ef8ddcd
Test service: dd-trace-py

✅ 0 Failed, 2624 Passed, 38250 Skipped, 28m 14.8s Total Time

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-08-16 22:27:28

Comparing candidate commit 2fffeb3 in PR branch yunkim/llmobs-fix-fork-writer with baseline commit fc1209e in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 353 metrics, 47 unstable metrics.

[P403n1x87]

May I suggest a change similar to #10247 to automatically recreate the periodic thread on fork (usage example in #10251)?

[Yun-Kim]

May I suggest a change similar to #10247 to automatically recreate the periodic thread on fork (usage example in #10251)?

Hi @P403n1x87, I have a couple questions about Awakeable periodic services (and what that would mean for LLMObsSpanWriter in this case). This is a bit of an urgent fix so I'd rather move forward with this for now, but I'll 100% follow up with you about this and am happy to make the refactor to use your utility class.

[github-actions]

The backport to 2.10 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.10 2.10
# Navigate to the new working tree
cd .worktrees/backport-2.10
# Create a new branch
git switch --create backport-10249-to-2.10
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 64b58044cf94110e7ddf97b656575b6707ee2a3f
# Push it to GitHub
git push --set-upstream origin backport-10249-to-2.10
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.10

Then, create a pull request where the base branch is 2.10 and the compare/head branch is backport-10249-to-2.10.