Understand ESMPy summary - possible issues

[theurich]

Currently we are running automated ESMPy testing on some of the Cheyenne combos.

The esmpy-test.log shows output like this:

find . -name "*ESMF_LogFile*" -exec rm -f {} \; || :
find . -name "*.report.json" -exec rm -rf {} \; || :
find . -name "*.test" -exec rm -rf {} \; || :
bash src/esmpy/test/test_all.bash "mpirun "
/glade/scratch/theurich/ESMF-Nightly-Testing/gfortran_10.1.0_mpt_O_develop/esmpy_venv/lib/python3.7/site-packages/esmpy/interface/loadESMF.py:83: VersionWarning: ESMF installation version 8.5.0 beta snapshot, ESMPy version 8.5.0b0
  esmfversion, constants._ESMPY_VERSION), VersionWarning)
Testing ESMPy 8.5.0b0
mpirun -np 1 python3 -m pytest -vs --json-report --json-report-summary > esmpy8.5.0b0-petx1.test 2>&1
mpirun -np 4 python3 -m pytest -vs --json-report --json-report-summary > esmpy8.5.0b0-petx4.test 2>&1
mpirun -np 6 python3 -m pytest -vs --json-report --json-report-summary > esmpy8.5.0b0-petx6.test 2>&1
Report is in esmpy8.5.0b0-petx6.test

When looking into the files that capture the output as per above, e.g. esmpy8.5.0b0-petx1.test, esmpy8.5.0b0-petx4.test, or esmpy8.5.0b0-petx6.test, I find this:

report saved to: .report.json
====== 118 passed, 4 skipped, 9 xfailed, 14 warnings in 248.44s (0:04:08) ======

However, the summary for ESMPy that results from it is this:

esmpy tests:        PASS 296 FAIL 0

It is not at all clear to me what is going on here. Is the summary correct, should there be failures reported in the summary. How does the PASS/FAIL count in the summary relate to the actual tests that are run?

[billsacks]

For a run of the ESMPy testing where all tests passed, this seems to be working correctly as: The PASS count in the summary gives the total number of PASSed tests across the three runs (petx1, petx4 and petx6) and the FAIL count is 0. However, things seem to be working wrong when there are failed tests. I'm going to close this issue and open a new one (#95 ) about problems when there are failed ESMPy tests.

[billsacks]

(@theurich feel free to reopen if you don't feel like this addressed your question sufficiently.)