Local and remote debugging of avocado tests

[pevogam]

Hi all, I guess you might already be aware of that but I recently checked for a way to debug python tests run via Avocado (which can still be useful for some more complex settings where isolation tests are not sufficiently precise at the time and so on). I ended up using debugpy which is a great tool implementing the Debug Adapter Protocol for Python like so

PYTHONPATH=:. python3 -m debugpy --log-to "." --listen 0.0.0.0:5678 --wait-for-client /usr/bin/avocado run --max-parallel-tasks=1 selftests/isolation/test_cartesian_graph.py:CartesianGraphTest.test_trace_setup_swarm

(give or take some SSH port forwarding since the python unit tests actually run inside a container) but interestingly could only stop at a breakpoint if I add it manually inside the python code of the unit test:

    def test_trace_setup_swarm(self):
        """Test a multi-object test run where the workers will run multiple tests reusing their own local swarm setup."""
        ...
        import debugpy
        debugpy.listen(("0.0.0.0", 5678))
        print("Waiting for debugger attach")
        debugpy.wait_for_client()
        debugpy.breakpoint()
        ...

which is done only in code and the invocation remains a regular avocado test running invocation without involving debugpy. This prompted me to investigate the logs (dumped in . due to the extra setting above) and find out that the break point request was sent just fine if I used VS code as a client front end

0.19s - Process SetBreakpointsRequest: {
    "arguments": {
        "breakpoints": [
            {
                "line": 2198
            }
        ],
        "lines": [
            2198
        ],
        "source": {
            "name": "test_cartesian_graph.py",
            "path": "/home/pevogam/Programs/6.libraries/avocado-i2n/selftests/isolation/test_cartesian_graph.py"
        },
        "sourceModified": false
    },
    "command": "setBreakpoints",
    "seq": 6,
    "type": "request"
}

with a success status followed by information on various spawned processes but specifically for the unit test subprocess:

0.01s - Patching args: ['avocado-runner-avocado-instrumented', 'task-run', '-i', '1-selftests/isolation/test_cartesian_graph.py:CartesianGraphTest.test_trace_setup_swarm', '-j', '7a4e8ea927b863d995ee2701a409551c3bdd7245', '-k', 'avocado-instrumented', '-u', 'selftests/isolation/test_cartesian_graph.py:CartesianGraphTest.test_trace_setup_swarm', '-c', '{"run.test_parameters": [], "datadir.paths.cache_dirs": ["/root/avocado/data/cache"], "core.show": {"__encoded_set__": ["app"]}, "job.output.loglevel": "DEBUG", "job.run.store_logging_stream": []}', 'tags=json:{}', 'output_dir=/root/avocado/job-results/job-2024-01-12T16.01-7a4e8ea/test-results/1-selftests_isolation_test_cartesian_graph.py_CartesianGraphTest.test_trace_setup_swarm', '-s', '/var/tmp/avocado_fws_4wr1/.status_server.sock']
0.00s - Process is not python, returning.

This informed me that the python unit tests are run as INSTRUMENTED tests and it seems these are not easily recognized as python processes so the debugging could also be injected inside those subprocesses. I then confirmed this is the case by running the unit test via regular python invocation:

python3 -m debugpy --listen 0.0.0.0:5678 --wait-for-client selftests/isolation/test_cartesian_graph.py CartesianGraphTest.test_trace_setup_swarm

which stopped at all requested break points and performed just fine. So all in all I wanted to see what is your opinion about supporting debugging in Avocado, especially debugging via such a flexible protocol? Perhaps there is some simple change we can do so that the subprocesses for the unit tests are recognized as python processes and such debugging might become fully possible? Or do you suggest another way to debug code via the unit/isolation tests or perhaps believe that debugging should only be possible with integration tests?