Use env for repro tests, improve repro names #3430
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
charleskawczynski
requested review from
trontrytel,
Sbozzolo,
szy21 and
haakon-e
charleskawczynski
force-pushed
the
ck/rm_reproducibility_test_flag
branch
from
50f4bb2
to
6acb6d1
Compare
|
What's the issue with the 4 GPU job? |
Sbozzolo
approved these changes
Nov 11, 2024
Not sure yet, still needs fixes.. |
charleskawczynski
force-pushed
the
ck/rm_reproducibility_test_flag
branch
from
6acb6d1
to
dad1b1b
Compare
|
I incremented the reference counter because I think it's far easier to "reset" our comparable references than to support comparing data exported from HDF5 formats (which we now use to support reproducibility tests for GPU jobs) with NC files (which we previously used). |
charleskawczynski
force-pushed
the
ck/rm_reproducibility_test_flag
branch
from
710c7e6
to
aed070d
Compare
Fixes for gpu repro tests, auto-compare all state variables Improve error message, increment ref counter Fix zero_dict calls Fix dict init Fixes to zero_dict Improve debug info
charleskawczynski
force-pushed
the
ck/rm_reproducibility_test_flag
branch
from
2d6a653
to
f802f8b
Compare
|
The part that I was getting stuck on was that the "print new MSE tables" job was not finding any mse json files, but that's because they never got moved over (because we hadn't set the |
|
If the restart job gets tripped up again, then I'll manually merge. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR:
reproducibility_testflag in the yaml config, and we instead use an environment variable. I don't think experiments should explicitly need to worry about this, and there's no way for users to leverage this feature locally anyway, so an env flag seems more appropriate.