chore(lib-injection): add early injection exits based on list of denied executables

[emmettbutler]

This change adds functionality to the Single Step Instrumentation logic that makes it exit early when sys.argv[0:1] contains an entry included in the list of denied executables.

Testing:

$ docker run --rm --volume /home/emmett/git/ddtrace:/opt/datadog-packages/ddtrace -it amazonlinux:2023 bash
bash-5.2# DD_API_KEY=a DD_SITE="datadoghq.com" DD_APM_INSTRUMENTATION_ENABLED=host DD_APM_INSTRUMENTATION_LIBRARIES=java:1,python:2,js:5,dotnet:2,ruby:2 bash -c "$(curl -L https://install.datadoghq.com/scripts/
install_script_agent7.sh)"
bash-5.2# cd  /opt/datadog-packages/datadog-apm-library-python/2.11.3
bash-5.2# ln -s /opt/datadog-packages/ddtrace/lib-injection/sitecustomize.py sitecustomize.py
bash-5.2# ln -s /opt/datadog-packages/ddtrace/lib-injection/denied_executables.txt denied_executables.txt  
# ... checkout this branch...
bash-5.2# DD_TRACE_DEBUG=1 DD_APM_INSTRUMENTATION_DEBUG=true yum update                                   
                                                                                                         
glibc detected, version: glibc:2.34                                                                                                                                                                               debug mode enabled                       
waiting...                                                                                                                                                                                                        
executable Path: /usr/bin/python3.9                                                                                                                                                                               
calling go processor                                                                                                                                                                                              {"level":"debug","ts":1725566018.4266808,"caller":"process/main.go:94","msg":"starting process"}                                                                                                                  
{"level":"debug","ts":1725566018.4267485,"caller":"preload_go/inject.go:29","msg":"arguments","inFilename":"/usr/bin/python3.9","libc":"glibc:2.34","envs":["DD_APM_INSTRUMENTATION_DEBUG=true","DD_INSTRUMENT_SER
VICE_WITH_APM=false","DD_TRACE_DEBUG=1"]}                                                                                                                                                                         
{"level":"debug","ts":1725566018.4267921,"caller":"preload_go/parser.go:164","msg":"using USM algorithm for service name"}                                           
{"level":"debug","ts":1725566018.4271073,"caller":"preload_go/parser.go:174","msg":"usm name","service_name":"bin","additional_services":[]}                         
{"level":"debug","ts":1725566018.4272437,"caller":"preload_go/parser.go:176","msg":"props values","props":{"Env":"","ServiceProps":{"ConfigKey":"bin","DDName":"bin","Names":null},"Version":"","ProcessProps":{"Path":"/usr/bin/python3.9","Args":["/usr/bin/python3","/usr/bin/yum","update"]},"ContainerProps":{"Labels":null,"Name":"","ShortName":"","Tag":""},"ServiceNameGenerated":true}}
{"level":"debug","ts":1725566018.4273496,"caller":"preload_go/process.go:245","msg":"python processor matched"}                                                                                                   {"level":"debug","ts":1725566018.4274027,"caller":"preload_go/process.go:99","msg":"using oci path","path":"/opt/datadog-packages/datadog-apm-library-python/2.11.3"}                                             {"level":"debug","ts":1725566018.427624,"caller":"preload_go/process.go:99","msg":"using oci path","path":"/opt/datadog-packages/datadog-apm-library-python/2.11.3"}                                              {"level":"debug","ts":1725566018.4276772,"caller":"preload_go/process.go:99","msg":"using oci path","path":"/opt/datadog-packages/datadog-apm-library-python/2.11.3"}    
{"level":"debug","ts":1725566018.4277923,"caller":"preload_go/inject.go:92","msg":"results","envs":["DD_TAGS=_dd.injection.mode:host","DD_RUNTIME_METRICS_ENABLED=true","DD_LOGS_INJECTION=true","DD_INJECTION_ENABLED=tracer,service_name","DD_TRACE_ENABLED=true","DD_SERVICE=bin","DD_TELEMETRY_FORWARDER_PATH=/opt/datadog-packages/datadog-apm-inject/0.18.0/inject/process"]}                                                 {"level":"debug","ts":1725566018.4281154,"caller":"telemetry/client.go:189","msg":"request json","request":"{\"api_version\":\"v1\",\"request_type\":\"generate-metrics\",\"tracer_time\":1725566018,\"runtime_id\":\"3254ba8e-7e40-48a2-94f8-9f89cb5939c6\",\"seq_id\":1,\"application\":{\"service_name\":\"bin\",\"env\":\"unknown\",\"tracer_version\":\"2.11.3\",\"language_name\":\"python\",\"language_version\":\"unknown\"},\"host\":{\"hostname\":\"87db14cf1e9d\",\"os\":\"Amazon Linux\",\"os_version\":\"2023\",\"architecture\":\"arm64\"},\"payload\":{\"namespace\":\"tracers\",\"lib_language\":\"python\",\"lib_version\":\"2.11.3\"
,\"series\":[{\"metric\":\"inject.success\",\"points\":[[1725566018,1]],\"type\":\"count\",\"tags\":[\"injector_version:0.18.0\",\"platform:host\"],\"common\":true}]}}"}
{"level":"error","ts":1725566018.428645,"caller":"preload_go/process.go:198","msg":"error flushing telemetry","error":"Post \"http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry\": dial tcp 127.0.0.1:8126
: connect: connection refused","stacktrace":"github.com/datadog/preload_go.CommandLineEvaluator.FlushTelemetry\n\t/go/src/github.com/DataDog/auto_inject/preload_go/process.go:198\ngithub.com/datadog/preload_go.
CommandLineEvaluator.Process.func1.1\n\t/go/src/github.com/DataDog/auto_inject/preload_go/process.go:227\ngithub.com/datadog/preload_go.InjectCmd.func1\n\t/go/src/github.com/DataDog/auto_inject/preload_go/injec
t.go:76\ngithub.com/datadog/preload_go.InjectCmd\n\t/go/src/github.com/DataDog/auto_inject/preload_go/inject.go:99\nmain.main\n\t/go/src/github.com/DataDog/auto_inject/preload_go/cmd/process/main.go:108\nruntim
e.main\n\t/usr/local/go/src/runtime/proc.go:267"}                                                                                                                                                                 
{"level":"debug","ts":1725566018.4287698,"caller":"process/main.go:125","msg":"exiting process"}       
Operation took 9ms                                                                                                                                                                                                
done waiting!                                                                                                                                                                                                     [2024-09-05 19:53:38] [INFO] datadog.autoinstrumentation(pid: 3466): user-installed ddtrace not found, configuring application to use injection site-packages
[2024-09-05 19:53:38] [DEBUG] datadog.autoinstrumentation(pid: 3466): detected platform manylinux2014
[2024-09-05 19:53:38] [DEBUG] datadog.autoinstrumentation(pid: 3466): ddtrace_pkgs path is '/opt/datadog-packages/datadog-apm-library-python/2.11.3/ddtrace_pkgs'
[2024-09-05 19:53:38] [DEBUG] datadog.autoinstrumentation(pid: 3466): ddtrace_pkgs contents: ['shared']
[2024-09-05 19:53:38] [DEBUG] datadog.autoinstrumentation(pid: 3466): Found incompatible executable: /usr/bin/yum.
[2024-09-05 19:53:38] [DEBUG] datadog.autoinstrumentation(pid: 3466): Aborting dd-trace-py instrumentation.
[2024-09-05 19:53:38] [DEBUG] datadog.autoinstrumentation(pid: 3466): maybe sending telemetry to /opt/datadog-packages/datadog-apm-inject/0.18.0/inject/process
[2024-09-05 19:53:38] [DEBUG] datadog.autoinstrumentation(pid: 3466): not sending telemetry: TELEMETRY_ENABLED=False
Last metadata expiration check: 1:31:10 ago on Thu Sep  5 18:22:28 2024.
Dependencies resolved.
Nothing to do.
Complete!

# ... checkout main
bash-5.2# DD_TRACE_DEBUG=1 DD_APM_INSTRUMENTATION_DEBUG=true yum update

glibc detected, version: glibc:2.34
debug mode enabled
waiting...
executable Path: /usr/bin/python3.9
calling go processor
{"level":"debug","ts":1725566087.33635,"caller":"process/main.go:94","msg":"starting process"}
{"level":"debug","ts":1725566087.3364048,"caller":"preload_go/inject.go:29","msg":"arguments","inFilename":"/usr/bin/python3.9","libc":"glibc:2.34","envs":["DD_APM_INSTRUMENTATION_DEBUG=true","DD_INSTRUMENT_SERVICE_WITH_APM=false","DD_TRACE_DEBUG=1"]}
{"level":"debug","ts":1725566087.3364666,"caller":"preload_go/parser.go:164","msg":"using USM algorithm for service name"}
{"level":"debug","ts":1725566087.3368323,"caller":"preload_go/parser.go:174","msg":"usm name","service_name":"bin","additional_services":[]}
{"level":"debug","ts":1725566087.3369246,"caller":"preload_go/parser.go:176","msg":"props values","props":{"Env":"","ServiceProps":{"ConfigKey":"bin","DDName":"bin","Names":null},"Version":"","ProcessProps":{"Path":"/usr/bin/python3.9","Args":["/usr/bin/python3","/usr/bin/yum","update"]},"ContainerProps":{"Labels":null,"Name":"","ShortName":"","Tag":""},"ServiceNameGenerated":true}}
{"level":"debug","ts":1725566087.3370175,"caller":"preload_go/process.go:245","msg":"python processor matched"}
{"level":"debug","ts":1725566087.3371184,"caller":"preload_go/process.go:99","msg":"using oci path","path":"/opt/datadog-packages/datadog-apm-library-python/2.11.3"}
{"level":"debug","ts":1725566087.3371856,"caller":"preload_go/process.go:99","msg":"using oci path","path":"/opt/datadog-packages/datadog-apm-library-python/2.11.3"}
{"level":"debug","ts":1725566087.3373294,"caller":"preload_go/process.go:99","msg":"using oci path","path":"/opt/datadog-packages/datadog-apm-library-python/2.11.3"}
{"level":"debug","ts":1725566087.337516,"caller":"preload_go/inject.go:92","msg":"results","envs":["DD_SERVICE=bin","DD_RUNTIME_METRICS_ENABLED=true","DD_INJECTION_ENABLED=tracer,service_name","DD_LOGS_INJECTION=true","DD_TRACE_ENABLED=true","DD_TAGS=_dd.injection.mode:host","DD_TELEMETRY_FORWARDER_PATH=/opt/datadog-packages/datadog-apm-inject/0.18.0/inject/process"]}
{"level":"debug","ts":1725566087.3377264,"caller":"telemetry/client.go:189","msg":"request json","request":"{\"api_version\":\"v1\",\"request_type\":\"generate-metrics\",\"tracer_time\":1725566087,\"runtime_id\":\"0de12728-02e4-4f9b-8d16-054667b17066\",\"seq_id\":1,\"application\":{\"service_name\":\"bin\",\"env\":\"unknown\",\"tracer_version\":\"2.11.3\",\"language_name\":\"python\",\"language_version\":\"unknown\"},\"host\":{\"hostname\":\"87db14cf1e9d\",\"os\":\"Amazon Linux\",\"os_version\":\"2023\",\"architecture\":\"arm64\"},\"payload\":{\"namespace\":\"tracers\",\"lib_language\":\"python\",\"lib_version\":\"2.11.3\",\"series\":[{\"metric\":\"inject.success\",\"points\":[[1725566087,1]],\"type\":\"count\",\"tags\":[\"injector_version:0.18.0\",\"platform:host\"],\"common\":true}]}}"}
{"level":"error","ts":1725566087.3385413,"caller":"preload_go/process.go:198","msg":"error flushing telemetry","error":"Post \"http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry\": dial tcp 127.0.0.1:8126: connect: connection refused","stacktrace":"github.com/datadog/preload_go.CommandLineEvaluator.FlushTelemetry\n\t/go/src/github.com/DataDog/auto_inject/preload_go/process.go:198\ngithub.com/datadog/preload_go.CommandLineEvaluator.Process.func1.1\n\t/go/src/github.com/DataDog/auto_inject/preload_go/process.go:227\ngithub.com/datadog/preload_go.InjectCmd.func1\n\t/go/src/github.com/DataDog/auto_inject/preload_go/inject.go:76\ngithub.com/datadog/preload_go.InjectCmd\n\t/go/src/github.com/DataDog/auto_inject/preload_go/inject.go:99\nmain.main\n\t/go/src/github.com/DataDog/auto_inject/preload_go/cmd/process/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"}
{"level":"debug","ts":1725566087.3385959,"caller":"process/main.go:125","msg":"exiting process"}
Operation took 13ms
done waiting!
[2024-09-05 19:54:47] [INFO] datadog.autoinstrumentation(pid: 3493): user-installed ddtrace not found, configuring application to use injection site-packages
[2024-09-05 19:54:47] [DEBUG] datadog.autoinstrumentation(pid: 3493): detected platform manylinux2014
[2024-09-05 19:54:47] [DEBUG] datadog.autoinstrumentation(pid: 3493): ddtrace_pkgs path is '/opt/datadog-packages/datadog-apm-library-python/2.11.3/ddtrace_pkgs'
[2024-09-05 19:54:47] [DEBUG] datadog.autoinstrumentation(pid: 3493): ddtrace_pkgs contents: ['shared']
[2024-09-05 19:54:47] [DEBUG] datadog.autoinstrumentation(pid: 3493): site-packages path is '/opt/datadog-packages/datadog-apm-library-python/2.11.3/ddtrace_pkgs/site-packages-ddtrace-py3.9-manylinux2014'
[2024-09-05 19:54:47] [ERROR] datadog.autoinstrumentation(pid: 3493): ddtrace site-packages not found in '/opt/datadog-packages/datadog-apm-library-python/2.11.3/ddtrace_pkgs/site-packages-ddtrace-py3.9-manylinux2014', aborting
Last metadata expiration check: 1:32:19 ago on Thu Sep  5 18:22:28 2024.
Dependencies resolved.
Nothing to do.
Complete!

Observe that the first yum output aborts injection because of the executable name and the second does not.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

lib-injection/denied_executables.txt                                    @DataDog/apm-core-python
lib-injection/sitecustomize.py                                          @DataDog/apm-core-python

[brettlangdon]

How should we test this? 🤔

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-09-05 20:38:47

Comparing candidate commit 3ad2907 in PR branch emmett.butler/denied-executables with baseline commit c905ee4 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 353 metrics, 47 unstable metrics.

[brettlangdon]

lgtm, where should tests for this live? do we have existing guardrails tests that we can validate some of the items in the denylist get rejected?

[emmettbutler]

As far as I know there aren't any automated tests for this list at the moment. We'll probably need to test manually on a custom-built box.

[sanchda]

Added some comments for discussion. Overall this is a really great idea.

[sanchda]

Added some comments for discussion. Overall this is a really great idea.

[sanchda]

As far as I know there aren't any automated tests for this list at the moment. We'll probably need to test manually on a custom-built box.

Equivalently, could we possibly inject scripts (which do something like checking whether ddtrace is loaded, idk) at denied
locations (like yum) into directories on the $PATH and verify that they do not get instrumented? Like, we know that almost all of the scripts we want to avoid are shebang scripts just to python3 or env. Might be an easier way of getting off the ground for now.

[sanchda]

Looks good to me, thank you