Improve coverage.py hand-off to cocotb

[alexforencich]

I have been doing a lot of experimenting with coverage.py recently to try to figure out the correct way to configure it when using cocotb-test. The main setup that I use for CI looks a little like this:

1. Tox sets up the python environment and calls pytest
2. pytest uses pytest-cov to run coverage.py and pytest-xdist to run tests in parallel on multiple CPUs (and possibly pytest-split to distribute across multiple nodes on github actions)
3. pytest uses cocotb-test to run cocotb testbenches ("outer tests")
4. cocotb-test manages building the testbench and starting the simulator
5. the simulator invokes the python interpreter internally to run cocotb
6. cocotb runs coverage.py internally
7. cocotb uses its own test runner to run the tests for the testbench ("inner tests")

Initially, tox was calling pytest with a bunch of command line parameters to configure coverage. For some reason, this worked with coverage.py 5.5, but broke with coverage.py 6.0. TBH, this seems to be one of those how-did-that-ever-work-correctly sort of situations. Initially I thought there might have been some sort of a bug in coverage itself, so I opened up an issue here while working things through: nedbat/coveragepy#1540 . But, at this point I think coverage is working as expected and there are some integration issues to sort out.

There are several problems with the original setup: there is no hand-off of coverage.py configuration when cocotb-test starts up the simulator, coverage running in cocotb simply runs with the default settings. This worked until coverage 6.0, which changed some stuff relating to determining which paths would be traced. Seems like the main change was better exclusion of "third-party" code. Since tox was installing the extension, it was being detected as third-party code and hence was being excluded.

One workaround is to set usedevelop in tox so that it installs the extension in editable mode and hence it is not detected as "third-party" code. But it seems like a better solution could be to make sure that coverage running inside of cocotb has the correct configuration.

There are three main problems that I ran in to:

1. Command line arguments to pytest don't get passed into sub-processes, aside from the ones internal to pytest-xdist. pytest-cov must be doing something specific to support pytest-xdist, but with the current setup, there is no way for these parameters to get passed through to cocotb. So, they must go in the config file directly (tox.ini, setup.cfg, etc.). It would be nice to be able to pass these options through, but it looks like maybe this would require changes to cocotb to pass these through to coverage.
2. coverage running inside cocotb cannot find the config file without help, due to running in a different working directory. Sure, we can simply specify COVERAGE_RCFILE in tox.ini/setup.cfg, but this could be a place where cocotb-test could improve things, perhaps by taking the config file used by pytest-cov and setting COVERAGE_RCFILE automatically based on that. And possibly also setting COVERAGE automatically.
3. coverage running inside cocotb doesn't remap paths correctly, since it is running in a different working directory. Not sure if there is much we can do about that at the moment as it doesn't seem like coverage has any configuration options to help with this, and the final combine step does seem to remap things correctly after adding the proper path entries.

[alexforencich]

Actually, maybe this will be prohibitively difficult due to limitations in coverage configuration. When passing the same config file to both the "outer" and "inner" coverage instances, things don't work very well due to the fact that the coverage instances run with different working directories, so all of the paths are wrong. I have not been able to figure out how to get the actual test code covered, aside from using default settings for the "inner" coverage instance and using usedevelop in tox. I think either coverage itself is going to have to support some additional options for path matching and such, or individual coverage config files are going to have to be written out somehow for each test, neither of which are particularly good options. While using usedevelop with default coverage settings feels like a bit of a fragile hack, it seems like it may be the most straightforward option.