Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove pslot dir from local archive folder on CI case completion #2961

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from
Open

Remove pslot dir from local archive folder on CI case completion #2961

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
1c159d8
added a cleanup_experment funciton to ci_utils that cleans local arch…
TerryMcGuinness-NOAA Sep 24, 2024
e7aa3ba
Merge branch 'NOAA-EMC:develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Sep 25, 2024
463655a
moved rm for EXPDIR into cleanup_expermiment function
TerryMcGuinness-NOAA Sep 25, 2024
b511367
Merge branch 'ci_clean_loc_arch' of github.com:TerrenceMcGuinness-NOA…
TerryMcGuinness-NOAA Sep 25, 2024
40c96b2
fixed bracket sytax error in ci_utils and missnamed function call in …
TerryMcGuinness-NOAA Sep 26, 2024
54ce216
fixed SC2155 in bash utils for cleanup
TerryMcGuinness-NOAA Sep 26, 2024
cacde8c
fixed SC2115 for var expation in path for clean experiment util
TerryMcGuinness-NOAA Sep 26, 2024
c8a10a9
needed the OR truch for var assigment via actions in bash
TerryMcGuinness-NOAA Sep 26, 2024
7c93f4b
fixed the evaluation of HOMDIR so that internal strification of env v…
TerryMcGuinness-NOAA Sep 27, 2024
bb92a39
Merge branch 'develop' into ci_clean_loc_arch
TerrenceMcGuinness-NOAA Sep 27, 2024
bfcbba5
still needed archive tagged back onto HOMEDIR
TerryMcGuinness-NOAA Sep 27, 2024
69fc51b
Merge branch 'ci_clean_loc_arch' of github.com:TerrenceMcGuinness-NOA…
TerryMcGuinness-NOAA Sep 27, 2024
49b4489
new linee at aend of ci_utils
TerryMcGuinness-NOAA Sep 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ pipeline {
sh(script: " rm -f ${error_file}")
try {
sh(script: "${HOMEgfs}/ci/scripts/run-check_ci.sh ${CUSTOM_WORKSPACE} ${pslot} ${system}")
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh clean_experiment ${CUSTOM_WORKSPACE}/RUNTESTS/EXPDIR/${pslot}")
} catch (Exception error_experment) {
sh(script: "${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh cancel_batch_jobs ${pslot}")
ws(CUSTOM_WORKSPACE) {
Expand Down
3 changes: 1 addition & 2 deletions ci/scripts/check_ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,7 @@ for pr in ${pr_list}; do
fi
if [[ "${rocoto_state}" == "DONE" ]]; then
#Remove Experment cases that completed successfully
rm -Rf "${pslot_dir}"
rm -Rf "${pr_dir}/RUNTESTS/COMROOT/${pslot}"
"${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh" cleanup_experiment "${pslot_dir}"
rm -f "${output_ci_single}"
# echo "\`\`\`" > "${output_ci_single}"
DATE=$(date +'%D %r')
Expand Down
16 changes: 16 additions & 0 deletions ci/scripts/utils/ci_utils.sh
View file
Open in desktop
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Oct 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few other directories that contain data, though none are very large:

${ARCHIVEDIR}/metplus_data/by_VSDB/{grid2grid,grid2obs,precip}/*/*/${pslot}
${HOME}/.rocoto/<rocoto version>/${pslot}  # These are typically empty directories on success with just a log and .lock file otherwise

EDIT: changed ${HOMEDIR} to ${HOME} for the rocoto directory.

Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,19 @@
fi
echo "${URL}"
}

function cleanup_experiment() {
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
# cleanup_experiment function
# This function takes a directory path of an EXPDIR and
# gets HOMEDIR value from config.base to
# clean up the archived files on disk.

local PSLOT_PATH="$1"
local pslot=$(basename "${PSLOT_PATH}")
Fixed Show fixed Hide fixed
local HOMEDIR=$(grep 'export HOMEDIR=' "${PSLOT_PATH}/config.base" | cut -d'=' -f2 | tr -d '[:space:]')
Fixed Show fixed Hide fixed
local ARCHIVEDIR="${HOMEDIR}/archive"

rm -rf "${ARCHIVEDIR}/${pslot}"
Fixed Show fixed Hide fixed
rm -Rf "${PSLOT_PATH}"
rm -Rf "${PSLOT_PATH}/../COMROOT/${pslot}"

Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
Loading