Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide an option to use the docker state saving stuff #10

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Provide an option to use the docker state saving stuff #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e96ffc1
Update session_config.sh
Edwardius Nov 7, 2024
83bb99f
Update session_config.sh
Edwardius Nov 7, 2024
8c0e2bd
Update asd_interactive_job.slurm
Edwardius Nov 7, 2024
e3d0bf5
Update asd_interactive_job.slurm
Edwardius Nov 7, 2024
024d7be
Update srun_interactive_job_template.sh
Edwardius Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions session_config.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@ export VRAM=0 # How much GPU VRAM you want [in MiB]
# SLURM tooling configuration
export UPDATE_WATO_ASD_TOOLING=1 # Set to 0 if you don't want to update ASD tooling on remote hosts
export SAVE_DOCKER_STATE_ON_EXIT=0 # Set to 1 if you want to save docker state on exit
export CLEAN_SAVED_DOCKER_STATE=0 # Set to 1 to clean your docker state, do this is your docker is corrupted or too large
5 changes: 5 additions & 0 deletions slurm_job_templates/asd_interactive_job.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ save_docker_state() {
trap - INT TERM
}

if [[ "$CLEAN_SAVED_DOCKER_STATE" -eq 1 ]] && [[ -f "$DOCKER_STATE_PATH" ]]; then
rm -rf "$DOCKER_SESSION_DIR"
echo -e "Docker state directory cleaned."
fi

if [[ -f "$DOCKER_STATE_PATH" ]]; then
echo "Saved docker state found. Loading..."
zstd --fast=3 -d "$DOCKER_STATE_PATH" --stdout | pv | rootlesskit tar --blocking-factor=64 -xpf - -C /
Expand Down
1 change: 1 addition & 0 deletions slurm_job_templates/srun_interactive_job_template.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
sed -i 's/\r$//' $REMOTE_SLURM_JOB_SCRIPT

export SAVE_DOCKER_STATE_ON_EXIT=${SAVE_DOCKER_STATE_ON_EXIT}
export CLEAN_SAVED_DOCKER_STATE=${CLEAN_SAVED_DOCKER_STATE}

# Run srun command
/opt/slurm/bin/srun --cpus-per-task=${NUMBER_OF_CPUS} --mem=${MEMORY} \
Expand Down