Cross section mismatch in pp_tt012j (P2_gu_ttxgu) in CI tmad tests - reset_cumulative_variable was called twice in fortran and once in cudacpp

[valassi]

There is a cross section mismatch in pp_tt012j in CI tmad tests.

This only appears after fixing the rotxxx crash (which otherwise hides it).
I have seen this in the CI for PR #857, where volatile is added to fix rotxxx.
https://github.com/madgraph5/madgraph4gpu/actions/runs/9694817881/job/26753418805

*** (2-none) Compare MADEVENT_CPP xQUICK xsec to MADEVENT_FORTRAN xsec ***

ERROR! xsec from fortran (0.82047507505698292) and cpp (0.67821465194602271) differ by more than 3E-13 (0.1733878669026967)

[valassi]

I forgot to mention: the CI now runs tmad tests in each P* subprocess. The issue appears in P2_gu_ttxgu. (Other subprocesses succeed).

[valassi]

I am having a look at this.

I confirm that the CI fails in P2_gu_ttxgu. This is a recent run in PR #934 where the pptt012j tests are the only pending issues
https://github.com/madgraph5/madgraph4gpu/actions/runs/10045773913/job/27763863225?pr=934

*******************************************************************************
*** tmad_test pp_tt012j.mad (P2_gu_ttxgu)
*******************************************************************************

Testing in /home/runner/work/madgraph4gpu/madgraph4gpu/epochX/cudacpp/pp_tt012j.mad/SubProcesses/P2_gu_ttxgu

*** (1) EXECUTE MADEVENT_FORTRAN (create results.dat) ***
 [OPENMPTH] omp_get_max_threads/nproc = 1/4
 [NGOODHEL] ngoodhel/ncomb = 32
32/64
64
 [XSECTION] VECSIZE_USED = 32
 [XSECTION] MultiChannel = TRUE
 [XSECTION] Configuration = 1
 [XSECTION] ChannelId = 1
 [XSECTION] Cross section = 0.8205 [0.82047507505698292] fbridge_mode=0
 [UNWEIGHT] Wrote 5 events (found 37 events)
 [COUNTERS] PROGRAM TOTAL          :    0.0631s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.0497s
 [COUNTERS] Fortran MEs      ( 1 ) :    0.0134s for      128 events => throughput is 9.54E+03 events/s

*** (1) EXECUTE MADEVENT_FORTRAN xQUICK (create events.lhe) ***
 [OPENMPTH] omp_get_max_threads/nproc = 1/4
 [NGOODHEL] ngoodhel/ncomb = 32
32/64
64
 [XSECTION] VECSIZE_USED = 32
 [XSECTION] MultiChannel = TRUE
 [XSECTION] Configuration = 1
 [XSECTION] ChannelId = 1
 [XSECTION] Cross section = 0.8205 [0.82047507505698292] fbridge_mode=0
 [UNWEIGHT] Wrote 7 events (found 36 events)
 [COUNTERS] PROGRAM TOTAL          :    0.0660s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.0514s
 [COUNTERS] Fortran MEs      ( 1 ) :    0.0145s for      128 events => throughput is 8.81E+03 events/s

*** (2-none) EXECUTE MADEVENT_CPP xQUICK (create events.lhe) ***
 [OPENMPTH] omp_get_max_threads/nproc = 1/4
 [NGOODHEL] ngoodhel/ncomb = 32/64
 [XSECTION] VECSIZE_USED = 32
 [XSECTION] MultiChannel = TRUE
 [XSECTION] Configuration = 1
 [XSECTION] ChannelId = 1
 [XSECTION] Cross section = 0.6782 [0.67821465194602271] fbridge_mode=1
 [UNWEIGHT] Wrote 4 events (found 26 events)
 [COUNTERS] PROGRAM TOTAL          :    0.0589s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.0519s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.0070s for       96 events => throughput is 1.38E+04 events/s

*** (2-none) Compare MADEVENT_CPP xQUICK xsec to MADEVENT_FORTRAN xsec ***

ERROR! xsec from fortran (0.82047507505698292) and cpp (0.67821465194602271) differ by more than 3E-13 (0.1733878669026967)

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[testsuite_oneprocess.sh] tmad_test (pp_tt012j.mad) finished with status=1 (NOT OK) at Mon Jul 22 18:02:30 UTC 2024
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Error: Process completed with exit code 1.

[roiser]

Hi @valassi , very good, I also started looking into it this morning, but please go ahead.

FYI I just noticed a small issue with the CI scripts, not sure why it appears only now? I.e. when I have nvcc in my PATH then, when running the CI "by hand" from the command line, it stops working at some stage. IIRC at the tput_test step, because it looks for a cuda build which hasn't been done. I believe I notice this the first time. Taking out the nvcc location from the PATH then it builds and runs fine of course (for CPU).

[valassi]

FYI I just noticed a small issue with the CI scripts, not sure why it appears only now? I.e. when I have nvcc in my PATH then, when running the CI "by hand" from the command line, it stops working at some stage. IIRC at the tput_test step, because it looks for a cuda build which hasn't been done. I believe I notice this the first time. Taking out the nvcc location from the PATH then it builds and runs fine of course (for CPU).

I made some changes to some scripts to exclude ggttggg builds on HIP due to #933. But this should only appear in my own branches eg PR #934 (and I think I did not even push it yet). So I do not understand what you refer to. If you have a concrete example and reproducer please open a ticket.

[valassi]

I think this can be closed again. I added a patch in PR #935

• function reset_cumulative_variable is called once per mirror
• note that helicities are still called once in cudacpp, and twice in fortran (this can be improved, but let's assume there are no inconsistencies)

[valassi]

PS reset_cumulative_variable issues are related to #486 and oliviermattelaer/mg5amc_test#22

[valassi]

I reopen this, just to mark the change of strategy

• this will not be fixed by my WIP add pp_tt to repo (plus obsolete fixes for bug 872 via reset_cumulative_variable, now fixed by Olivier via a single helicity filter) #935
• instead it will be fixed by Olivier's fix patch to be compatible with mg5 branch gpucpp_goodhel #955 (with minor extensions neded in Complete PR 955 for goodhel (resync to master, regenerate code) #986)