WIP: refactor makefiles (additions to Stephan's and Joorgen's PRs) #797
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ff so that it can be digested by both clang13 and clang16 Using Stephan's previous commits, codegen (for ggtt.mad) was failing clang-format checks on itscrd90. Now codegen succeeds completely on itscrd90, but fails on itscrd80 because patchMad.sh fails.
…patch.P1 and patch.common from scratch (This is an old commit from 9 Dec but the description is still valid on 14 Dec) ./CODEGEN/generateAndCompare.sh gg_tt --mad --nopatch sed -i 's/DEFAULT_F2PY_COMPILER=f2py3.*/DEFAULT_F2PY_COMPILER=f2py3/' gg_tt.mad/Source/make_opts git diff --no-ext-diff -R gg_tt.mad/Source/make_opts gg_tt.mad/Source/makefile gg_tt.mad/Source/dsample.f gg_tt.mad/Source/genps.inc gg_tt.mad/SubProcesses/makefile > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common git diff --no-ext-diff -R gg_tt.mad/bin/internal/banner.py gg_tt.mad/bin/internal/gen_ximprove.py gg_tt.mad/bin/internal/madevent_interface.py >> CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common git diff --no-ext-diff -R gg_tt.mad/SubProcesses/P1_gg_ttx/auto_dsig1.f gg_tt.mad/SubProcesses/P1_gg_ttx/driver.f gg_tt.mad/SubProcesses/P1_gg_ttx/matrix1.f > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.P1 git checkout gg_tt.mad NB1: In the instructions above, I changed the order of files in the patches (I moved make_opts to the beginning), to minimise differences in patch.common with respect to Stephan's. NB2: In another branch mch (PR madgraph5#796) I had decided to remove make_opts from patch.common: this is clearly no longer possible now Before this commit, i.e. using Stephan's patch.P1 and patch.common, this was happening: - code generation succeeds on itscrd90, fails on itscrd80 (patchMad.sh fails on the latter) - executing './CODEGEN/generateAndCompare.sh gg_tt --mad --nopatch' shows that the only difference between itscrd80 and itscrd90 is f2py (f2py3.9 on the former, f2py on the latter) After this commit, however: - code generation fails on itscrd90 (patchMad.sh fails, probably due to f2py issues)
…2py patching from generateAndCompare.sh to patchMad.sh (This is an old commit from 9 Dec but the description is still valid on 14 Dec) Before this commit, codegen was now failing on itscrd90 as it was previously failing on itscrd80 (patchMd.sh fails due to f2py) After this commit, codegen is succeeding again on itscrd90, and now also on itscrd80 (fixing the issue in PR madgraph5#753)
valassi
force-pushed
the
refactorMakefile
branch
2 times, most recently
from
5473f65
to
fbcf4a4
Compare
valassi
force-pushed
the
refactorMakefile
branch
from
fbcf4a4
to
5473f65
Compare
|
@hageboeck I have now moved to your latest 629d5aa and am adding my previous patches on top one by one. I will go on until this CI is fixed, at least. As previously observed, at this stage codegen succeeds on itscrd80 and itscrd90 but not in the CI. I will again add my sorting magic to make everything reproducible and allow all codegen to succeed. Stephan then I will ask you to cherry pick or merge these changes (and stop force pushing please at that point) |
….sh in the CI (it works instead in itscrd90/80): reorder make_opts_variables before patching - however codegen now fails on itscrd90 (This is an old commit from 9 Dec but the description is still valid on 14 Dec)
… keep the special handling of make_opts also in --nopatch mode (and will regenerate patch.common) (This is an old commit from 9 Dec but the description is still valid on 14 Dec)
(This is an old commit from 9 Dec but the description is still valid on 14 Dec - patch.common is regenerated on 14 Dec anyway) ./CODEGEN/generateAndCompare.sh gg_tt --mad --nopatch sed -i 's/DEFAULT_F2PY_COMPILER=f2py3.*/DEFAULT_F2PY_COMPILER=f2py3/' gg_tt.mad/Source/make_opts git diff --no-ext-diff -R gg_tt.mad/Source/make_opts gg_tt.mad/Source/makefile gg_tt.mad/Source/dsample.f gg_tt.mad/Source/genps.inc gg_tt.mad/SubProcesses/makefile > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common git diff --no-ext-diff -R gg_tt.mad/bin/internal/banner.py gg_tt.mad/bin/internal/gen_ximprove.py gg_tt.mad/bin/internal/madevent_interface.py >> CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common git diff --no-ext-diff -R gg_tt.mad/SubProcesses/P1_gg_ttx/auto_dsig1.f gg_tt.mad/SubProcesses/P1_gg_ttx/driver.f gg_tt.mad/SubProcesses/P1_gg_ttx/matrix1.f > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.P1 git checkout gg_tt.mad Now code generation succeeds again on itscrd90 and also on itscrd80 - and is expected to succeed also in the CI
|
The CI codegen was indeed failing, I have now again addded the commits necessary to fix that. Now codegen succeeds in the CI (for mad) but the build fails because 'make avxall' does not exist. I will also change that. First however I will also make other minor changes to disable ccache for fortran. |
…for fortran as it is not supported (This is an old commit from 12 Dec but the description is still valid on 14 Dec)
(This is an old commit from 12 Dec but the description is still valid on 14 Dec - patch.common is regenerated on 14 Dec anyway) ./CODEGEN/generateAndCompare.sh gg_tt --mad --nopatch sed -i 's/DEFAULT_F2PY_COMPILER=f2py3.*/DEFAULT_F2PY_COMPILER=f2py3/' gg_tt.mad/Source/make_opts git diff --no-ext-diff -R gg_tt.mad/Source/make_opts gg_tt.mad/Source/makefile gg_tt.mad/Source/dsample.f gg_tt.mad/Source/genps.inc gg_tt.mad/SubProcesses/makefile > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common git diff --no-ext-diff -R gg_tt.mad/bin/internal/banner.py gg_tt.mad/bin/internal/gen_ximprove.py gg_tt.mad/bin/internal/madevent_interface.py >> CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common git diff --no-ext-diff -R gg_tt.mad/SubProcesses/P1_gg_ttx/auto_dsig1.f gg_tt.mad/SubProcesses/P1_gg_ttx/driver.f gg_tt.mad/SubProcesses/P1_gg_ttx/matrix1.f > CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.P1 git checkout gg_tt.mad I need to regenerate all .mad files (.sa generation still needs to be fixed)
…---sa flags (This is an old commit from 9 Dec but the description is still valid on 14 Dec)
./CODEGEN/allGenerateAndCompare.sh --mad
…ild.* directory (without bash nullglob, this expands to literal build.* otherwise)
…r.mod in Source/makefile
Without this fix, the following situation could take place:
1. When make is interrupted by ctrl-c, DiscreteSampler.o exists but discretesampler.mod does not exist
make[1]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gg_tt.mad/Source'
AVX=none
RNDGEN=hasCurand
gfortran -I. -fPIC -w -ffixed-line-length-132 -O3 -ffast-math -fbounds-check -c StringCast.f -o StringCast.o
gfortran -I. -fPIC -w -ffixed-line-length-132 -O3 -ffast-math -fbounds-check -c DiscreteSampler.f -o DiscreteSampler.o
gfortran -I. -fPIC -w -ffixed-line-length-132 -O3 -ffast-math -fbounds-check -c dsample.f -o dsample.o
^Cmake: *** [../../Source/.timestamp_guard] Deleting file '../../Source/discretesampler.mod'
make: *** [makefile:65: ../../Source/.timestamp_guard] Interrupt
2. When make (or make -j) is then attempted, discretesampler.mod does not exist and cannot be recreated
make[1]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gg_tt.mad/Source'
AVX=none
RNDGEN=hasCurand
gfortran -I. -fPIC -w -ffixed-line-length-132 -O3 -ffast-math -fbounds-check -c dsample.f -o dsample.o
dsample.f:1090:10:
1090 | use DiscreteSampler
| 1
Fatal Error: Cannot open module file ‘discretesampler.mod’ for reading at (1): No such file or directory
compilation terminated.
…all makefile targets
This reverts commit 06ec67a.
…END" This reverts commit eaa3bfb.
valassi
added a commit
to valassi/madgraph4gpu
that referenced
this pull request
Dec 15, 2023
…ster's version instead of Jorgen's for those 5 lines) NB1: currently, Jorgen's version differs from upstream/master only in three CODEGEN files git diff --name-only upstream/master epochX/cudacpp/CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/MG5aMC_patches/PROD/patch.common epochX/cudacpp/CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/madgraph/iolibs/template_files/gpu/cudacpp.mk epochX/cudacpp/CODEGEN/PLUGIN/CUDACPP_SA_OUTPUT/madgraph/iolibs/template_files/gpu/cudacpp_src.mk NB2: code generation for ggtt.mad currently fails on itscrd90 (patchMad..sh fails for makefile - same as seen in PR madgraph5#797?)
valassi
added a commit
to valassi/madgraph4gpu
that referenced
this pull request
Dec 17, 2023
…ral fixes to ensure make_opts is reproducible This is the combination of three successive patches I had first used in PR madgraph5#797 - keep the special handling of make_opts also in --nopatch mode - reorder make_opts_variables before patching - move back f2py patching from generateAndCompare.sh to patchMad.sh
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a WIP PR about refactoring makefiles.
It contains a few additions to Stephan's PR #753 (itself including Joorgen's PR #775 IIUC).
I find it easier to have it as a separate PR so that I can trigger the CI.
Specifically, the first issue I wanted to fix is that 96f9ba7 fails the CI in PR #753.
Later todo: