Skip to content
This repository has been archived by the owner on Oct 23, 2020. It is now read-only.

Ocean irreproducible on Cori-KNL/Intel #1519

Open
philipwjones opened this issue Mar 21, 2018 · 4 comments
Open

Ocean irreproducible on Cori-KNL/Intel #1519

philipwjones opened this issue Mar 21, 2018 · 4 comments
Assignees

Comments

@philipwjones
Copy link
Contributor

During a QU240 test case on Cori-KNL, MPAS ocean model was found to be irreproducible in a pure MPI configuration when built using Intel at -O2 or higher. Namely, two identical 8-core jobs submitted one after the other generated non-b4b differences in the output file.

srun -n 8 ./ocean_model
mv output.nc orig.nc
srun -n 8 .ocean_model
nccompare output.nc orig.nc finds significant differences but should be b4b

@philipwjones
Copy link
Contributor Author

Further diagnosis by @greenrongreen pointed to a data alignment issue. Am testing and will submit a PR soon with a fix to compiler options.

@philipwjones
Copy link
Contributor Author

Added -fp-model consistent flag fixes this and is in PR #1520
Can also fix using -align array64byte but above flag appears more general.

@philipwjones
Copy link
Contributor Author

Replaced by
MPAS-Dev/MPAS-Model#37
that addresses issue above and reports in new repo

@philipwjones
Copy link
Contributor Author

Sorry, meant to close related PR not issue

mark-petersen added a commit to MPAS-Dev/MPAS-Model that referenced this issue May 14, 2018
added -align array64byte flag to optimized Intel builds to fix
reproducibility issues on some machines (KNL)
also replaced deprecated -FR flag with -free to denote free-form source

B4B
Tested on Cori, Edison at NERSC

Replaces previous PR in MPAS-Dev/MPAS
MPAS-Dev/MPAS#1520
though see discussion therein.

Fixes previously reported bug in MPAS-Dev/MPAS
MPAS-Dev/MPAS#1519
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant