-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use multiprocessing.Pool.map()
for parallelism
#182
Use multiprocessing.Pool.map()
for parallelism
#182
Conversation
Codecov Report
@@ Coverage Diff @@
## main #182 +/- ##
==========================================
+ Coverage 87.10% 88.37% +1.26%
==========================================
Files 25 25
Lines 1613 1591 -22
==========================================
+ Hits 1405 1406 +1
+ Misses 208 185 -23
|
We currently do not exit with non-zero if an uncaught exception occurs within a multiprocessing child process. `multiprocessing.Pool.map()` will kill all processes if any one process throws an exception and then re-raise that exception in the parent process. Fixes #122
66043f8
to
068e0bb
Compare
Integration testsTest
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the behaviour when one of the comparison task fails, i.e. when the files are different?
@ccarouge the behaviour should still be the same but I've run a case that fails the comparison: #!/bin/bash
bench_example_dir='bench_example_test_failed_bitwise_comparison'
rm -rf $bench_example_dir
git clone [email protected]:CABLE-LSM/bench_example.git $bench_example_dir
cd $bench_example_dir
git reset --hard 6287539e96fc8ef36dc578201fbf9847314147fb
cat > config.yaml << EOL
project: tm70
experiment: AU-Tum
realisations:
- path: trunk
- path: trunk
name: trunk-patch
patch:
cable:
cable_user:
SSNOW_POTEV: P-M
modules: [
intel-compiler/2021.1.1,
netcdf/4.7.4,
openmpi/4.1.0
]
EOL
benchcab run -v Output from PBS job:
|
We currently do not exit with non-zero if an uncaught exception occurs within a multiprocessing child process.
multiprocessing.Pool.map()
will kill all processes if any one process throws an exception and then re-raise that exception in the parent process.Fixes #122