Skip to content

Commit

Permalink
ci: update info output tests to be less brittle [backport 2.7] (#9129)
Browse files Browse the repository at this point in the history 
Backport 081bc1a from #9124 to 2.7.

In response to:
https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/60346/workflows/45933e89-0290-461b-9d35-d59d73dcbdd4/jobs/3790097

The tests currently depend a bit too much on configs not changing and
having an exact string match. With this change we skip testing certain
more unpredictable configs being in the output and change to checking
for each one individually.

## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Co-authored-by: Zachary Groves <[email protected]>
  • Loading branch information
@github-actions @ZStriker19
github-actions[bot] and ZStriker19 authored Apr 30, 2024
1 parent 92079bc commit 0a07421
Showing 1 changed file with 55 additions and 80 deletions.
135 changes: 55 additions & 80 deletions tests/commands/test_runner.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -334,49 +334,36 @@ def test_info_no_configs():
)
p.wait()
stdout = p.stdout.read()
assert (stdout) == (
b"""\x1b[94m\x1b[1mTracer Configurations:\x1b[0m
Tracer enabled: True
Application Security enabled: False
Remote Configuration enabled: False
IAST enabled (experimental): False
Debug logging: False
Writing traces to: http://localhost:8126
Agent error: Agent not reachable at http://localhost:8126. """
+ b"""Exception raised: [Errno 99] Cannot assign requested address\n"""
b""" App Analytics enabled(deprecated): False
Log injection enabled: False
Health metrics enabled: False
Priority sampling enabled: True
Partial flushing enabled: True
Partial flush minimum number of spans: 300
WAF timeout: 5.0 msecs
\x1b[92m\x1b[1mTagging:\x1b[0m
DD Service: None
DD Env: None
DD Version: None
Global Tags: None
Tracer Tags: None
\x1b[96m\x1b[1mSummary\x1b[0m"""
b"""\n\n\x1b[91mERROR: It looks like you have an agent error: 'Agent not reachable at http://localhost:8126."""
b""" Exception raised: [Errno 99] Cannot assign requested address'\n"""
b""" If you're experiencing a connection error, please """
b"""make sure you've followed the setup for your particular environment so that the tracer and Datadog """
b"""agent are configured properly to connect, and that the Datadog agent is running:"""
b""" https://ddtrace.readthedocs.io/en/stable/troubleshooting.html"""
b"""#failed-to-send-traces-connectionrefusederror"""
b"""\nIf your issue is not a connection error then please reach out to support for further assistance:"""
b""" https://docs.datadoghq.com/help/\x1b[0m"""
b"""\n\n\x1b[93mWARNING SERVICE NOT SET: It is recommended that a service tag be set for all traced """
b"""applications. For more information please see"""
b""" https://ddtrace.readthedocs.io/en/stable/troubleshooting.html\x1b[0m"""
b"""\n\n\x1b[93mWARNING ENV NOT SET: It is recommended that an env tag be set for all traced applications. """
b"""For more information please see https://ddtrace.readthedocs.io/en/stable/troubleshooting.html\x1b[0m"""
b"""\n\n\x1b[93mWARNING VERSION NOT SET: """
b"""It is recommended that a version tag be set for all traced applications. """
b"""For more information please see https://ddtrace.readthedocs.io/en/stable/troubleshooting.html\x1b[0m\n"""
)
# checks most of the output but some pieces are removed due to the dynamic nature of the output
expected_strings = [
b"\x1b[1mTracer Configurations:\x1b[0m",
b"Tracer enabled: True",
b"Application Security enabled: False",
b"Remote Configuration enabled: False",
b"Debug logging: False",
b"App Analytics enabled(deprecated): False",
b"Log injection enabled: False",
b"Health metrics enabled: False",
b"Priority sampling enabled: True",
b"Partial flushing enabled: True",
b"Partial flush minimum number of spans: 300",
b"WAF timeout: 5.0 msecs",
b"Tagging:",
b"DD Service: None",
b"DD Env: None",
b"DD Version: None",
b"Global Tags: None",
b"Tracer Tags: None",
b"Summary",
b"WARNING SERVICE NOT SET: It is recommended that a service tag be set for all traced applications.",
b"For more information please see https://ddtrace.readthedocs.io/en/stable/troubleshooting.html\x1b[0m",
b"WARNING ENV NOT SET: It is recommended that an env tag be set for all traced applications. For more",
b"information please see https://ddtrace.readthedocs.io/en/stable/troubleshooting.html\x1b[0m",
b"WARNING VERSION NOT SET: It is recommended that a version tag be set for all traced applications.",
b"For more information please see https://ddtrace.readthedocs.io/en/stable/troubleshooting.html\x1b[0m",
]
for expected in expected_strings:
assert expected in stdout, f"Expected string not found in output: {expected.decode()}"

assert p.returncode == 0

Expand Down Expand Up @@ -405,43 +392,31 @@ def test_info_w_configs():

p.wait()
stdout = p.stdout.read()
assert (
(stdout)
== b"""\x1b[94m\x1b[1mTracer Configurations:\x1b[0m
Tracer enabled: True
Application Security enabled: True
Remote Configuration enabled: True
IAST enabled (experimental): True
Debug logging: True
Writing traces to: http://168.212.226.204:8126
Agent error: Agent not reachable at http://168.212.226.204:8126. Exception raised: timed out
App Analytics enabled(deprecated): False
Log injection enabled: True
Health metrics enabled: False
Priority sampling enabled: True
Partial flushing enabled: True
Partial flush minimum number of spans: 1000
WAF timeout: 5.0 msecs
\x1b[92m\x1b[1mTagging:\x1b[0m
DD Service: tester
DD Env: dev
DD Version: 0.45
Global Tags: None
Tracer Tags: None
\x1b[96m\x1b[1mSummary\x1b[0m"""
b"""\n\n\x1b[91mERROR: It looks like you have an agent error: """
b"""'Agent not reachable at http://168.212.226.204:8126. """
b"""Exception raised: timed out'\n If you're experiencing a connection error, """
b"""please make sure you've followed the """
b"""setup for your particular environment so that the tracer and """
b"""Datadog agent are configured properly to connect,"""
b""" and that the Datadog agent is running:"""
b""" https://ddtrace.readthedocs.io/en/stable/troubleshooting.html#failed-to-send-traces-"""
b"""connectionrefusederror\n"""
b"""If your issue is not a connection error then please reach out to support for further assistance: """
b"""https://docs.datadoghq.com/help/\x1b[0m\n"""
)
# checks most of the output but some pieces are removed due to the dynamic nature of the output
expected_strings = [
b"1mTracer Configurations:\x1b[0m",
b"Tracer enabled: True",
b"Remote Configuration enabled: True",
b"IAST enabled (experimental)",
b"Debug logging: True",
b"App Analytics enabled(deprecated): False",
b"Log injection enabled: True",
b"Health metrics enabled: False",
b"Priority sampling enabled: True",
b"Partial flushing enabled: True",
b"Partial flush minimum number of spans: 1000",
b"WAF timeout: 5.0 msecs",
b"Tagging:",
b"DD Service: tester",
b"DD Env: dev",
b"DD Version: 0.45",
b"Global Tags: None",
b"Tracer Tags: None",
b"m\x1b[1mSummary\x1b[0m",
]

for expected in expected_strings:
assert expected in stdout, f"Expected string not found in output: {expected.decode()}"

assert p.returncode == 0

Expand Down

0 comments on commit 0a07421

Please sign in to comment.