Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reorganizye scripts directory, simplity creation of TWTarballs #8727

Open
belforte opened this issue Sep 30, 2024 · 2 comments
Open

reorganizye scripts directory, simplity creation of TWTarballs #8727

belforte opened this issue Sep 30, 2024 · 2 comments

Comments

@belforte
Copy link
Member

belforte commented Sep 30, 2024

in current scripts directory we mix files uses for different purposes, which forces us to keep explicit lists of file names in buildTWTarballs.sh

Instead we can structure scripts so that there is a top directory for "things which need to go in the scheduler's SPOOL_DIR" and "things which need to go in the grid job directory"

That will simplify both tarball creations and use and will be safer against adding, renaming files.

See #8699 (comment)

@belforte
Copy link
Member Author

I paste here text from #8699 (comment) for convenience

add two subdirectories in CRABServer/scripts

scripts
├── dagman
│   └── task_process
└── job_wrapper

so that in

cp -r "${CRABSERVERDIR}/scripts"/{TweakPSet.py,CMSRunAnalysis.py,task_process} .
cp "${CRABSERVERDIR}/src/python"/{ServerUtilities.py,RucioUtils.py,CMSGroupMapper.py,RESTInteractions.py} .
echo "Making TaskManagerRun tarball"
tar zcf "${RUNTIME_WORKDIR}/TaskManagerRun.tar.gz" CRAB3.zip TweakPSet.py CMSRunAnalysis.py task_process ServerUtilities.py RucioUtils.py CMSGroupMapper.py RESTInteractions.py || exit 4
echo "Making CMSRunAnalysis tarball"
tar zcf "${RUNTIME_WORKDIR}/CMSRunAnalysis.tar.gz" WMCore.zip TweakPSet.py CMSRunAnalysis.py ServerUtilities.py CMSGroupMapper.py RESTInteractions.py || exit 4

we only copy well named directories around, like done now for task_process

and also move expanding of job_wrapper tarball

tar xmf CMSRunAnalysis.tar.gz || exit 10042

to the top wrapper script between these lines

echo "======== PROXY INFORMATION FINISH at $(TZ=GMT date) ========"
echo "======== CMSRunAnalysis.sh at $(TZ=GMT date) STARTING ========"

@belforte
Copy link
Member Author

better do right after #6544 "while memory is fresh" so that I can capture all changes in a documentation #8728

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant