-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prune unused artifacts from non-static builds #59
base: main
Are you sure you want to change the base?
Changes from all commits
2669fb9
4abe19d
fb51e90
c20a97d
a0fcdf2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,155 @@ | ||
#!/usr/bin/env bash | ||
|
||
set -eu | ||
|
||
|
||
filter_out_dirs() { | ||
list="$1" | ||
exclude_list="$2" | ||
|
||
for item in $exclude_list; do | ||
dir=$item | ||
|
||
while [ "$dir" != "/" ]; do | ||
list=$(echo "$list" | grep -xv $dir) | ||
|
||
dir=$(dirname $dir) | ||
done | ||
done | ||
|
||
echo "$list" | ||
} | ||
|
||
find_elf_executables() { | ||
targets=$@ | ||
|
||
while read -r executable; do | ||
read -r -N 4 trailer < "$executable" | ||
|
||
# Output only ELF binaries | ||
if [ "$trailer" = $'\x7fELF' ]; then | ||
echo $executable | ||
fi | ||
done < <(find $targets -type f -executable) | ||
Comment on lines
+26
to
+33
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We must loop this way here because they might be file names with spaces. Indeed, MCA is such a representative of such scenario. Because of that, I wonder if I haven't assumed to much elsewhere regarding file names. Also, would this deserve a comment or is this clear enough? |
||
} | ||
|
||
find_shared_libs() { | ||
libs=$(find_elf_executables $@) | ||
|
||
echo "$libs" | grep -E "*.so(.[0-9]+)*$" | sort -u | ||
} | ||
|
||
find_linked_libraries() { | ||
executables=$(find_elf_executables $@) | ||
|
||
# Depend on the glibc-specific behavior of supporting multiple executables | ||
# to be queried at once | ||
linked=$(ldd $executables 2>/dev/null | grep '=>') | ||
|
||
# We grep out not found libraries, since they cannot be kept if we don't | ||
# know where they are. | ||
# | ||
# Final binary may be actually runnable, since rpath of another binary may | ||
# pull those not found libraries | ||
found="$(echo "$linked" | grep -v "not found")" | ||
|
||
# Get their full path | ||
libs=$(echo "$found"| cut -d'>' -f 2 | cut -d' ' -f 2) | ||
Comment on lines
+56
to
+57
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I feel like there is a better way to do this. I though abort |
||
|
||
echo "$libs" | sort -u | ||
} | ||
|
||
get_used_epics_modules() { | ||
epics_libs=$(find_linked_libraries $@) | ||
modules=$(get_all_epics_modules) | ||
|
||
unused_modules=$(filter_out_dirs "$modules" "$epics_libs") | ||
|
||
filter_out_dirs "$modules" "$unused_modules" | ||
} | ||
|
||
get_all_epics_modules() { | ||
release_defs=$(grep = ${EPICS_RELEASE_FILE} | cut -d'=' -f 2) | ||
|
||
echo "$release_defs" | grep $EPICS_MODULES_PATH | ||
} | ||
|
||
remove_static_libs() { | ||
for target; do | ||
libs=$(find $target -type f -name *.a) | ||
|
||
if [ -n "$libs" ]; then | ||
size=$(du -hsc $libs | tail -n 1 | cut -f 1) | ||
|
||
echo "Removing static libraries from $target ($size)" | ||
ericonr marked this conversation as resolved.
Show resolved
Hide resolved
|
||
rm -f $libs | ||
fi | ||
done | ||
} | ||
|
||
remove_unused_shared_libs() { | ||
used_libs=$(find_linked_libraries $@) | ||
remove_libs=$(find_shared_libs /opt /usr/local) | ||
|
||
for lib in $used_libs; do | ||
remove_libs=$(echo "$remove_libs" | grep -vx $lib) | ||
done | ||
|
||
for lib in $remove_libs; do | ||
size=$(du -hs $lib | cut -f 1) | ||
|
||
echo "Removing shared library '$lib' ($size)" | ||
rm -f ${lib%.so*}.so* | ||
done | ||
} | ||
|
||
prune_module_dirs() { | ||
module=$1 | ||
|
||
keep_paths=" | ||
$(find_shared_libs $module) | ||
$(find $module -type f -regex ".*\.\(db\|template\|req\)" -printf "%h\n" | sort -u) | ||
" | ||
|
||
while read -r candidate; do | ||
[ -d $candidate ] || continue | ||
|
||
if [[ ! $keep_paths =~ "$candidate".* ]]; then | ||
size=$(du -hs $candidate | cut -f 1) | ||
|
||
printf "Removing directory '$candidate' ($size)...\n" | ||
rm -rf $candidate | ||
fi | ||
done < <(find $module -type d) | ||
} | ||
|
||
clean_up_epics_modules() { | ||
targets=$@ | ||
|
||
unused_modules=$(get_all_epics_modules) | ||
used_modules=$(get_used_epics_modules $targets) | ||
|
||
keep_dirs="$targets $used_modules" | ||
|
||
for module in $(filter_out_dirs "$unused_modules" "$keep_dirs"); do | ||
# if we already removed it because of its top-level repository or | ||
# because it is an IOC, move on to the next. | ||
[ ! -d $module ] && continue | ||
|
||
size=$(du -hs $module | cut -f 1) | ||
|
||
echo "Removing module '$module' ($size)..." | ||
rm -rf $module | ||
done | ||
Comment on lines
+134
to
+143
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This loop will misbehave if we turn out having modules with empty spaces. I kept this as a loop with such requirement because we mostly control such paths. Let me know what you think about this. |
||
|
||
for dir in $(filter_out_dirs "$used_modules" "$targets"); do | ||
echo "Pruning module '$dir'..." | ||
prune_module_dirs $dir | ||
done | ||
|
||
prune_module_dirs $EPICS_BASE_PATH | ||
} | ||
|
||
clean_up_epics_modules $@ | ||
remove_static_libs /opt | ||
remove_unused_shared_libs $@ |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,6 +8,6 @@ services: | |
labels: | ||
org.opencontainers.image.source: https://github.com/cnpem/epics-in-docker | ||
args: | ||
REPONAME: mca | ||
REPONAME: epics/modules/mca | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To be honest, I don't like very much the idea of reusing I did so because it was easier to test other things first. Let me know what you think. |
||
RUNDIR: /opt/epics/modules/mca/iocBoot/iocAmptek | ||
RUNTIME_PACKAGES: libpcap0.8 libnet1 libusb-1.0-0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added this last so that image cache is not invalidated every time for nothing. It can be moved to be side-by-side with
lnls-run
after the script ready to be merged.