Improved CLI configuration options

[davidlenfesty]

I'm trying to integrate cocotb-test into a verification GUI I'm writing, and would like better options for runtime management of tests.

Specifically, I need to support two use cases:

• When a testcase fails, I want to automatically re-run it with the same options (i.e. the same seed) and with tracing enabled, so I can save this trace for later analysis.
• I need to run coverage and regression separately from within my tool.

I'm not sure what this looks like. I'm leaning towards environment variables overriding or adding arguments to the various cocotb options, similar to what WAVES does, but I'm not sure if that's the preferred direction. FWIW, I only need to access RANDOM_SEED, WAVES, and EXTRA_ARGS, for this use case.

I do know how I would go about to add some external tooling to pull in extra environment variables, however that would be a bit of a usability issue, as if a test function wasn't properly annotated it would show up in the list of tests, but wouldn't function in the expected ways. Also it would require changes to existing tests, which is not preferred.

[themperek]

I do not think I know what you are training to do. Something like this: https://github.com/tfio/pytest-gui

Not sure if you develop your own runtime 1) or you want to use cocotb-test 2)?

For option 1) you need to use the environment internally because this is what cocotb uses and just add integration https://docs.cocotb.org/en/stable/custom_flows.html. How this works for the user is up to you.

For option 2) I would not mind adding control for RANDOM_SEED and EXTRA_ARGS similar to WAVES.

[davidlenfesty]

Yes, something similar to that but geared towards HDL, largely with the two extra use cases I mentioned.

I would like to use cocotb-test for this (option 2), so that we also get the extra infrastructure from pytest. Adding RANDOM_SEED and EXTRA_ARGS would be perfect for this, thank you!