Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove pslot dir from local archive folder on CI case completion #2961

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ pipeline {
sh(script: " rm -f ${error_file}")
try {
sh(script: "${HOMEgfs}/ci/scripts/run-check_ci.sh ${CUSTOM_WORKSPACE} ${pslot} ${system}")
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh cleanup_experiment ${CUSTOM_WORKSPACE}/RUNTESTS/EXPDIR/${pslot}")
} catch (Exception error_experment) {
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh cancel_batch_jobs ${pslot}")
ws(CUSTOM_WORKSPACE) {
Expand Down
3 changes: 1 addition & 2 deletions ci/scripts/check_ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,7 @@ for pr in ${pr_list}; do
fi
if [[ "${rocoto_state}" == "DONE" ]]; then
#Remove Experment cases that completed successfully
rm -Rf "${pslot_dir}"
rm -Rf "${pr_dir}/RUNTESTS/COMROOT/${pslot}"
"${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh" cleanup_experiment "${pslot_dir}"
rm -f "${output_ci_single}"
# echo "\`\`\`" > "${output_ci_single}"
DATE=$(date +'%D %r')
Expand Down
22 changes: 22 additions & 0 deletions ci/scripts/utils/ci_utils.sh
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Oct 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few other directories that contain data, though none are very large:

${ARCHIVEDIR}/metplus_data/by_VSDB/{grid2grid,grid2obs,precip}/*/*/${pslot}
${HOME}/.rocoto/<rocoto version>/${pslot}  # These are typically empty directories on success with just a log and .lock file otherwise

EDIT: changed ${HOMEDIR} to ${HOME} for the rocoto directory.

Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,25 @@ function publish_logs() {
fi
echo "${URL}"
}

function cleanup_experiment() {
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
# cleanup_experiment function
# This function takes a directory path of an EXPDIR and
# gets HOMEDIR value from config.base to
# clean up the archived files on disk.

local PSLOT_PATH
local pslot
local HOMEDIR
local ARCHIVEDIR

PSLOT_PATH="$1"
pslot=$(basename "${PSLOT_PATH}")
HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]"' || true) || true
eval HOMEDIR="${HOMEDIR}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this eval doing? Isn't it already defined on the prior line? eval should be avoided if at all possible.

ARCHIVEDIR="${HOMEDIR}/archive"

rm -Rf "${ARCHIVEDIR:?}/${pslot}"
rm -Rf "${PSLOT_PATH}"
rm -Rf "${PSLOT_PATH}/../COMROOT/${pslot}"
}
Loading