chore(ci): collect coverage data

[gnufede]

This PR makes sure we run the tests with coverage collection when the "itr:noskip" keyword is added to the commit message, or if the branch is not a PR (main, 2.x, etc).

Also, it removes the usage of the coverage parameter in the yaml config, simply uploading the coverage files if they exist.

Checklist

• Change(s) are motivated and described in the PR description
• Testing strategy is described if automated tests are not included in the PR
• Risks are described (performance impact, potential for breakage, maintainability)
• Change is maintainable (easy to change, telemetry, documentation)
• Library release note guidelines are followed or label changelog/no-changelog is set
• Documentation is included (in-code, generated user docs, public corp docs)
• Backport labels are set (if applicable)
• If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

• Title is accurate
• All changes are related to the pull request's stated goal
• Description motivates each change
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Change is maintainable (easy to change, telemetry, documentation)
• Release note makes sense to a user of the library
• Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: gnufede/send-coverage-itr-noskip
Commit report: d1f2405
Test service: dd-trace-py

✅ 0 Failed, 8215 Passed, 165192 Skipped, 3h 30m 11.7s Total duration (8h 7m 29.97s time saved)

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-04-19 11:12:29

Comparing candidate commit 2d55c0a in PR branch gnufede/send-coverage-itr-noskip with baseline commit 01d5e01 in branch main.

Found 3 performance improvements and 3 performance regressions! Performance is the same for 195 metrics, 9 unstable metrics.

scenario:flasksimple-tracer

• 🟥 execution_time [+217.579µs; +264.696µs] or [+3.458%; +4.207%]

scenario:httppropagationextract-datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_match

• 🟩 max_rss_usage [-775.289KB; -599.329KB] or [-3.542%; -2.738%]

scenario:httppropagationextract-wsgi_large_valid_headers_all

• 🟩 max_rss_usage [-720.966KB; -511.520KB] or [-3.306%; -2.346%]

scenario:httppropagationinject-with_all

• 🟥 max_rss_usage [+726.406KB; +782.970KB] or [+3.430%; +3.697%]

scenario:httppropagationinject-with_dd_origin

• 🟥 max_rss_usage [+0.977MB; +1.025MB] or [+4.672%; +4.902%]

scenario:sethttpmeta-obfuscation-regular-case-implicit-query

• 🟩 max_rss_usage [-789.085KB; -591.267KB] or [-3.525%; -2.641%]

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 5.67%. Comparing base (5376eb5) to head (c4073e6).
Report is 11 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #8951       +/-   ##
===========================================
- Coverage   72.80%    5.67%   -67.13%     
===========================================
  Files        1247     1217       -30     
  Lines      116866   115198     -1668     
===========================================
- Hits        85083     6542    -78541     
- Misses      31783   108656    +76873     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[brettlangdon]

I think the only thing really changed from before, if we have a test suite which doesn't generate coverage data, then previously the job would fail (on the persist_to_workspace) where now it will succeed.

I think that is ok, it means we probably need to keep an eye on coverage per-PR to ensure we something massive doesn't get missed/not run (in which case we need to go fix it)