Profiling

[seemk]

Description

Add support for CPU profiling.

Brings in new environment variables:

• SPLUNK_PROFILER_ENABLED - disabled by default
• SPLUNK_PROFILER_LOGS_ENDPOINT - defaults to http://localhost:4317
• SPLUNK_PROFILER_CALL_STACK_INTERVAL - thread sampling (stacktrace taking) interval in milliseconds, defaults to 1000
• SPLUNK_PROFILER_INCLUDE_INTERNAL_STACKS - whether to include stacktraces from internal profiler threads (sampler thread and logs processor/exporter), disabled by default.

The profiler / sampling works from a different thread, where call stacks for all threads are periodically sampled. These are then serialized to protobuf (pprof format), gzipped and encoded as base64. The base64 content are sent to collector as OTel logs. For the exporting we can leverage the OTel logs SDK, which makes it quite convenient.

To correlate active span contexts with sampled stacktraces, opentelemetry.context.{attach,detach} are monkeypatched, to store a mapping of thread ID -> span context. This mapping is available for the sampler thread.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

• Tested manually
• Added automated tests

Checklist:

• Changelogs have been updated
• Unit tests have been added/updated
• Documentation has been updated

[nstawski]

@seemk please rebase this pull against main

[nstawski]

@seemk also the lint job is failing. Looks like just an extra newline

[tsloughter-splunk]

Is there a design doc or can you describe how it is working?

Not knowing much about this the one thing that caught my eye when trying to review is the global batch log processor, but I assume that is the best solution? :)

But do other profilers use their own log processor/exporters instead of using one that might already be created by the application for logs?

[seemk]

Is there a design doc or can you describe how it is working?

Not knowing much about this the one thing that caught my eye when trying to review is the global batch log processor, but I assume that is the best solution? :)

But do other profilers use their own log processor/exporters instead of using one that might already be created by the application for logs?

We went over the workings during a call, but I'll write a summary into the PR description.

As for the global batch processor, I just used it for convenience to get the export pipeline set up. Other distros (Node) didn't have the logs SDK available when profiling was implemented and had to have a custom exporter for the logs protos.

[pmcollins]

Thanks for doing this Siim. A few questions/comments below.

[seemk]

Is there a design doc or can you describe how it is working?

Not knowing much about this the one thing that caught my eye when trying to review is the global batch log processor, but I assume that is the best solution? :)

But do other profilers use their own log processor/exporters instead of using one that might already be created by the application for logs?

Added a small description how the "profiler" operates, it turned out to be pretty small in Python.

[seemk]

Added filtering of internal threads (enabled by default) and stop_profiling

[seemk]

• Added reinitialization of the profiler thread on forks (for forking servers such as gunicorn).
• Added LoggerProvider instead of working with the batch processor directly. The logger provider has exit handlers, where it will shut down the exporting pipeline.

[pmcollins]

I'm wondering if we can modify how we access the Profiler global variable, such that its global state is accessed once by a global function, but all other access is via function arguments.

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[pmcollins]

I have read the CLA Document and I hereby sign the CLA

[pmcollins]

Looks great, thanks for taking my feedback.

[seemk]

@pmcollins Any outstanding issues you want addressed?

[pmcollins]

@pmcollins Any outstanding issues you want addressed?

LGTM.