-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cocotb-test sim_build directory location #114
Comments
What exactly? I would think it should be ok if it is the same design all the time? This would be potentially also desirable to compile sources once.
Yes. This is a bug.
I am also not sure how to do this better. |
The problem is that each directory tests a different module, and all of the sim_build directories are called sim_build_[datawidth] or something, so all of the testbenches for datawidth 8 for many completely different modules end up pointed at the same directory, and basically it's a crapshoot which one actually runs correctly (well, if any of them run correctly). |
This looks like a bit special case but it still should work (for Icarus)? The design is compiled based on cocotb-test/cocotb_test/simulator.py Line 305 in 76dc1d6
|
eh, maybe that would work, but that's pretty hackish and I'm planning on using other simulators as well, particularly verilator. I think it's a bug that the build files get dumped in whatever directory you happen to run py.test from. The issue is that py.test will recurse into subdirectories looking for tests, but then execute them with the working directory of wherever py.test was run from. In my case, I want to be able to run py.test within one subdirectory to run all of the tests for one module, or go up one level and run py.test to run the tests for all of the modules. My ideal solution would be to do the same thing as the libs directory and make it relative to |
I can do this immediately. And you can do
I have tried something similar already. cocotb-test/cocotb_test/run.py Lines 24 to 28 in f0be378
I found it tricky because it is hard to cover all the uses cases (the user has to handle this now). We want also to run without a pytest or with customized simulations classes etc ... If you have some good idea let me know. |
When using Pytest in parallel, I've been doing this to avoid colliding sim_build directory:
The string of .replace is quite hacky but without it I get directory names that were not so friendly for some tool I can't remember. |
well, that's a rather nice idea. Right now I am doing stuff like this for each module:
but if request.node.name includes the parameter values, then it would be a more general solution. The problem I am running in to, though, is that the build directories from several different python test scripts are getting dumped into the same directory. I'm not sure if request.node.name would be unique across different scripts or not. At any rate, what I need to do is ensure sim_build ends up in the same directory as the python script, no matter what the working directory happens to be. |
I'm not too familiar with pytest but there might be a way to get the calling script name or directory directly? The issue I was having with the method above is each simulation would have to recompile the verilog. Many time the 'parameters' are only affecting the tb and do not mandate a recompilation. I was wondering if there shouldn't be a split between the directory holding the compiled output, and the directory where we dump simulation outputs. |
In my case the parameters are all module parameters, as such the module needs to be recompiled for each set of parameters. |
So #115 means absolute paths work for sim_build at least as a stop-gap, now the question is what to do with relative paths. Is using the working directory the best method? Is there a clean way to get pytest to change the directory when recursively running tests? Are there other options worth considering? Right now I am experimenting with assigning sim_build like so:
Seems to be working so far. |
@alexforencich Not sure if can make everyone happy ;-( My use-case is normally to have one design and multiple tests in multiple directories and compile once. One can make their own |
Yeah, true. My current use-case is testing libraries that contain a bunch of modules. One folder and one testbench python file per module, using cocotb-test to step through a decent set of parameters. Currently, I am running in to some pytest test collection issues due to some duplicate python file names, however - some of the myhdl testbenches have the same names as the cocotb testbenches. pytest recommends adding |
Add module path to cocotb-test/cocotb_test/simulator.py Line 38 in 00d2b9a
|
interesting idea, just tried setting |
Heh. So I think It should work with Anyhow, I should document |
I uploaded some of the tests I've been working on here, why don't you experiment with it and see if you can figure out what the correct solution is: https://github.com/alexforencich/cocotbext-axi/tree/master/tests I tried setting both python_search and work_dir, setting python_search did nothing while setting work_dir seemed to enable cocotb to find the module in question. Not sure why work_dir would be legacy, it looks like that's used to set the working directory of the simulator process. If that's legacy, then how do you set the working directory for the simulator process? |
So if I do The only drawback using If you make sim_build = os.path.join(tests_dir,
"sim_build", request.node.name.replace('[', '-').replace(']', '')) BTW:
You are right. Was thinking of default to Would you like to set up some CI? |
Oh, it needs to be a list. That makes sense, I'll swap that out then and leave work_dir alone. And yes, I am definitely using xdist, it makes running hundreds of tests bearable on my EPYC 7742 CPU. I will definitely set up CI on that repo at some point, I have travis-ci set up on some of my other repos. If you have any input on that, I would be happy to take any suggestions. At least the last time I checked, travis-ci has an execution time limit of like an hour or something, but some of my tests run for 2 hours or more (lots of nested loops). Also, I think work_dir may default to sim_build if it's not otherwise specified. |
Do you think we can close this? |
I mean, with the sim_build absolute path bug fixed, setting that (and python_search) manually from |
As I alluded to in a comment above I think it would be good to have an option to differentiate between the directory where stuff is compiled to, and the directory where simulation output are stored. They could be made the same by default. There are two types of pytest parametrization possible. One that affects the dut (like changing top level dut parameters, requiring a recompile) and one where tb/dit modes are parameterized. The latter doesn't need recompilation while outputs from each sim should be in a separate directory. |
Well, and there are also multiple simulators, maybe it would make sense to include the simulator in the build directory name so that running the same tests with multiple simulators don't result in conflicts, especially in a CI context. |
I just ran in to an issue with cocotb-test that needs to be addressed. I have a directory structure that looks something like this
If I run one of the makefiles or if I run py.test from within test_mod1 or test_mod2, everything works fine, and the sim_build_* directories are created inside the test_mod1/test_mod2 directories alongside the test scripts. However, if I run py.test from the tb directory that's up one level, all of the sim_build directories are created there and there are all sorts of conflicts.
The problem is the first line of
Simulator.__init__
:There are two main issues here: first, if somebody provides an absolute path for sim_build, things will break, so detecting that and preserving an absolute path would be a good idea. Second, using os.getcwd like that is causing the sim_build directories from the tests in the subdirectories to all end up in the same location. I'm not sure what the best solution would be for that. One option is to do the same thing as the lib directory:
There may be other options as well, perhaps there is a way to set up pytest to change the cwd. However, I do not like passing any command line arguments to pytest - everything should run correctly if pytest is run with no arguments.
The text was updated successfully, but these errors were encountered: