Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamp CI with dependency and Python caching for efficient installs #3112

Open
wants to merge 40 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
7509341
Add basic Python dependency caching
johvincau Jul 6, 2024
1524cf7
Test NJOY install script refactor for specific install location
johvincau Jul 16, 2024
1bc8572
Temporarily turn off testing suite
johvincau Jul 23, 2024
ac06042
Refactor mcpl install script
johvincau Jul 24, 2024
d7c1bdc
Add commit hashing and dependency caching for testing
johvincau Jul 24, 2024
d168428
Adjust MCPL branch
johvincau Jul 24, 2024
b572a0e
Add MCPL build to path
johvincau Jul 24, 2024
d2dec2d
Return to github workspace after reinstalling makes
johvincau Jul 26, 2024
dd2f4fb
Re-add support for DAGMC caching
johvincau Jul 26, 2024
9a170cd
Reenable testing and fix DAGMC hashing
johvincau Jul 26, 2024
67a4740
Add cache support for ncrystal and temporarily take out testing
johvincau Jul 29, 2024
afd5144
Re-add testing and minor tweaks for ncrystal
johvincau Jul 29, 2024
8349c8a
Add cache support for libmesh
johvincau Jul 30, 2024
41e8f3d
Test removal of make install
johvincau Jul 30, 2024
b38e1fa
Fix mcpl install dir and remove testing again
johvincau Jul 30, 2024
8dc462c
Add mcpl prefix path to gha-install.py
johvincau Jul 30, 2024
8877edb
Cache vectfit and temporarily remove testing
johvincau Jul 31, 2024
83d41de
Handle case for successful vectfit restore
johvincau Jul 31, 2024
57b6388
Try renaming scheme for job matrix
johvincau Aug 1, 2024
62c31aa
Extract job matrix from json
johvincau Aug 5, 2024
1e7de1e
Add in setup-job and try JSON input/output scheme
johvincau Aug 5, 2024
faa4883
Clean up comments and prepare for PR
johvincau Aug 5, 2024
f43fed5
Minor tweaks and clean up trash
johvincau Aug 6, 2024
a5b5a75
Remove mcpl build dir flag
johvincau Aug 14, 2024
2264f3e
Remove nonfunctional mcpl_cmake_path and -DMCPL_DIR cmake flags
johvincau Aug 14, 2024
cd4d977
Fix libmesh/vectfit install commands in gha-install.sh
johvincau Aug 14, 2024
36d0ded
Improve gha-caches.sh hash commands
johvincau Aug 14, 2024
f107643
Clarify GHA 'setup' step name
johvincau Aug 14, 2024
048feeb
Remove build dirs after NJOY and vectfit installs
johvincau Aug 14, 2024
652c57c
Try removing NJOY2016 tests dir after install
johvincau Aug 14, 2024
6e9b941
Fix vectfit in gha-install.sh
johvincau Aug 14, 2024
a35fa53
Re-add NJOY2016 tests dir
johvincau Aug 14, 2024
dd78966
Revert vectfit changes in gha-install.sh
johvincau Aug 15, 2024
b639169
Revert all changes to NJOY and vectfit caches to fix install issues
johvincau Aug 15, 2024
1897a46
Re-try removal of njoy-specific testing suite
johvincau Aug 15, 2024
6a1d1c0
Remove njoy build directory after successful build to trim cache
johvincau Aug 16, 2024
6ef06d3
Retain NJOY build directory for cache
johvincau Aug 16, 2024
d7714ce
Properly add NJOY dir to path and remove build dir
johvincau Aug 26, 2024
53e9353
Remove Python-version-dependency caches
johvincau Aug 26, 2024
f7f32bf
NJOY2016/bin should be on PATH
paulromano Aug 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 79 additions & 48 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,54 +21,28 @@ env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

jobs:
setup:
runs-on: ubuntu-22.04
outputs:
matrix: ${{ steps.read-json.outputs.matrix }}

steps:
- uses: actions/checkout@v4

- name: Read JSON
id: read-json
johvincau marked this conversation as resolved.
Show resolved Hide resolved
shell: bash
run: |
echo "matrix<<EOF" >> $GITHUB_OUTPUT
cat $GITHUB_WORKSPACE/tools/ci/matrix.json >> $GITHUB_OUTPUT

main:
runs-on: ubuntu-22.04
needs: setup
strategy:
matrix:
python-version: ["3.10"]
mpi: [n, y]
omp: [n, y]
dagmc: [n]
ncrystal: [n]
libmesh: [n]
event: [n]
vectfit: [n]

include:
- python-version: "3.11"
omp: n
mpi: n
- python-version: "3.12"
omp: n
mpi: n
- dagmc: y
python-version: "3.10"
mpi: y
omp: y
- ncrystal: y
python-version: "3.10"
mpi: n
omp: n
- libmesh: y
python-version: "3.10"
mpi: y
omp: y
- libmesh: y
python-version: "3.10"
mpi: n
omp: y
- event: y
python-version: "3.10"
omp: y
mpi: n
- vectfit: y
python-version: "3.10"
omp: n
mpi: y
name: "Python ${{ matrix.python-version }} (omp=${{ matrix.omp }},
mpi=${{ matrix.mpi }}, dagmc=${{ matrix.dagmc }}, ncrystal=${{ matrix.ncrystal }},
libmesh=${{ matrix.libmesh }}, event=${{ matrix.event }}
vectfit=${{ matrix.vectfit }})"
matrix: "${{ fromJSON( needs.setup.outputs.matrix ) }}"

name: "${{ matrix.job-name }}"

env:
MPI: ${{ matrix.mpi }}
Expand All @@ -94,6 +68,15 @@ jobs:
with:
python-version: ${{ matrix.python-version }}

- name: Restore Python cache
uses: actions/cache@v4
with:
path: ${{ env.pythonLocation }}
key: py-${{ matrix.python-version }}-(omp=${{ matrix.omp }}
mpi=${{ matrix.mpi }} dagmc=${{ matrix.dagmc }} ncrystal=${{ matrix.ncrystal }}
libmesh=${{ matrix.libmesh }} event=${{ matrix.event }}
vectfit=${{ matrix.vectfit }})

- name: Environment Variables
run: |
echo "DAGMC_ROOT=$HOME/DAGMC"
Expand Down Expand Up @@ -125,19 +108,67 @@ jobs:
if: ${{ matrix.vectfit == 'y' }}
run: sudo apt install -y libblas-dev liblapack-dev

- name: Hash dependencies to check caches
run: |
$GITHUB_WORKSPACE/tools/ci/gha-caches.sh

- name: Restore NJOY2016
uses: actions/cache@v4
with:
path: ~/NJOY2016
key: NJOY2016-${{ env.NJOY_HASH }}

- name: Restore DAGMC
if: ${{ matrix.dagmc == 'y' }}
uses: actions/cache@v4
with:
path: |
~/DAGMC
~/MOAB
key: DAGMC-${{ env.DAGMC_HASH }}/MOAB-${{ env.MOAB_HASH }}

- name: Restore NCrystal
if: ${{ matrix.ncrystal == 'y' }}
uses: actions/cache@v4
with:
path: |
~/ncrystal_src
~/ncrystal_bld
~/ncrystal_inst
key: NCrystal-${{ env.NC_HASH }}

- name: Restore vectfit
if: ${{ matrix.vectfit == 'y' }}
uses: actions/cache@v4
with:
path: |
~/pybind11
~/xtl
~/xtensor-python
~/xtensor-blas
~/vectfit
key: vectfit-${{ env.VF_HASH }}/pybind-${{ env.PYBIND_HASH }}

- name: Restore libmesh
if: ${{ matrix.libmesh == 'y' }}
uses: actions/cache@v4
with:
path: |
~/LIBMESH
key: libmesh-mpi-${{matrix.mpi}}-${{ env.NC_HASH }}

- name: install
shell: bash
run: |
echo "$HOME/NJOY2016/build" >> $GITHUB_PATH
$GITHUB_WORKSPACE/tools/ci/gha-install.sh

- name: cache-xs
- name: Restore endf-b-vii.1 cache
uses: actions/cache@v4
with:
path: |
~/nndc_hdf5
~/endf-b-vii.1
key: ${{ runner.os }}-build-xs-cache
key: endf-b-vii.1-cache

- name: before
shell: bash
Expand Down
31 changes: 31 additions & 0 deletions tools/ci/gha-caches.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/bash
set -ex

echo "NJOY_HASH=$(git ls-remote https://github.com/njoy/NJOY2016.git | cut -f 1 |\
head -c 15)" >> $GITHUB_ENV
johvincau marked this conversation as resolved.
Show resolved Hide resolved

if [[ $DAGMC = 'y' ]]; then
echo "MOAB_HASH=$(git ls-remote https://bitbucket.org/fathomteam/moab.git \
--branch Version5.1.0 | cut -f 1 | head -c 15)" >> $GITHUB_ENV

echo "DAGMC_HASH=$(git ls-remote https://github.com/svalinn/dagmc.git |\
cut -f 1 | head -c 15)" >> $GITHUB_ENV
fi

if [[ $NCRYSTAL = 'y' ]]; then
echo "NC_HASH=$(git ls-remote https://github.com/mctools/ncrystal \
--branch develop | cut -f 1 | head -c 15)" >> $GITHUB_ENV
fi

if [[ $VECTFIT = 'y' ]]; then
echo "PYBIND_HASH=$(git ls-remote https://github.com/pybind/pybind11 \
--branch master | cut -f 1 | head -c 15)" >> $GITHUB_ENV

echo "VF_HASH=$(git ls-remote https://github.com/liangjg/vectfit.git |\
cut -f 1 | head -c 15)" >> $GITHUB_ENV
fi

if [[ $LIBMESH = 'y' ]]; then
echo "NC_HASH=$(git ls-remote https://github.com/libmesh/libmesh \
--branch v1.7.1 | cut -f 1 | head -c 15)" >> $GITHUB_ENV
fi
3 changes: 2 additions & 1 deletion tools/ci/gha-install-libmesh.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ if [[ $MPI == 'y' ]]; then
../libmesh/configure --prefix=$HOME/LIBMESH CXX=mpicxx CC=mpicc FC=mpifort F77=mpif77 \
--enable-exodus --disable-netcdf-4 --disable-eigen --disable-lapack
else
../libmesh/configure --prefix=$HOME/LIBMESH --enable-exodus --disable-netcdf-4 --disable-eigen --disable-lapack --disable-mpi
../libmesh/configure --prefix=$HOME/LIBMESH --enable-exodus --disable-netcdf-4 \
--disable-eigen --disable-lapack --disable-mpi
fi
make -j4 install
export LIBMESH_PC=$HOME/LIBMESH/lib/pkgconfig/
Expand Down
9 changes: 7 additions & 2 deletions tools/ci/gha-install-mcpl.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
#!/bin/bash
set -ex

# MCPL variables
MCPL_REPO='https://github.com/mctools/mcpl'

cd $HOME
git clone https://github.com/mctools/mcpl
git clone $MCPL_REPO
cd mcpl
mkdir build && cd build
cmake .. && make 2>/dev/null && sudo make install
cmake .. -DCMAKE_INSTALL_PREFIX=$MCPL_INSTALL_DIR &&
johvincau marked this conversation as resolved.
Show resolved Hide resolved
make 2>/dev/null && sudo make install
1 change: 0 additions & 1 deletion tools/ci/gha-install-ncrystal.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ make install
# yet. If it appears in the future, we should add it here.

# Output the configuration to the log

"${INST_DIR}/bin/ncrystal-config" --setup

# Change environmental variables
Expand Down
10 changes: 8 additions & 2 deletions tools/ci/gha-install-njoy.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
#!/bin/bash
set -ex

# NJOY variables
NJOY_REPO='https://github.com/njoy/NJOY2016'
NJOY_INSTALL_DIR=$HOME/NJOY2016
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be nice to put all the installed dependencies in a single folder, e.g., $HOME/dependencies/NJOY2016. Also, in the case of NJOY (and probably the others), we should just be caching the installed version. Right now, the cache for NJOY is 1 GB when it should be much less.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've trimmed the NJOY repository by ~30% by removing its own NJOY-specific testing suite out of the cache; however, removing the build directory causes issues with the vectfit installation. Aside from NJOY and vectfit, all other of the dependencies at the very least remove their own build directories before caching.

I would like to move all the installed dependencies into a single folder as well; however, I will address that in a future PR since it would take a lot of testing and plenty of commits (and this PR already has enough of those as is!).


cd $HOME
git clone https://github.com/njoy/NJOY2016
git clone $NJOY_REPO
cd NJOY2016
mkdir build && cd build
cmake -Dstatic=on .. && make 2>/dev/null && sudo make install
cmake -Dstatic=on .. -DCMAKE_INSTALL_PREFIX=$NJOY_INSTALL_DIR &&
make 2>/dev/null && sudo make install
2 changes: 2 additions & 0 deletions tools/ci/gha-install.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ def install(omp=False, mpi=False, phdf5=False, dagmc=False, libmesh=False, ncrys

# Build in debug mode by default with support for MCPL
cmake_cmd = ['cmake', '-DCMAKE_BUILD_TYPE=Debug', '-DOPENMC_USE_MCPL=on']
mcpl_cmake_path = os.environ.get('HOME') + '/mcpl/lib/cmake/MCPL'
cmake_cmd.append(f'-DMCPL_DIR={mcpl_cmake_path}')

# Turn off OpenMP if specified
if not omp:
Expand Down
40 changes: 33 additions & 7 deletions tools/ci/gha-install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,56 @@ pip install --upgrade pytest
pip install --upgrade numpy

# Install NJOY 2016
./tools/ci/gha-install-njoy.sh
if [[ ! -d "$HOME/NJOY2016" ]]; then
./tools/ci/gha-install-njoy.sh
fi
echo "$HOME/NJOY2016/build" >> $GITHUB_PATH

# Install DAGMC if needed
if [[ $DAGMC = 'y' ]]; then
./tools/ci/gha-install-dagmc.sh
if [ ! -d "$HOME/DAGMC" ] || [ ! -d "$HOME/MOAB" ]
then
./tools/ci/gha-install-dagmc.sh
fi
fi

# Install NCrystal if needed
if [[ $NCRYSTAL = 'y' ]]; then
./tools/ci/gha-install-ncrystal.sh
if [ ! -d "$HOME/ncrystal_bld" ]
then
./tools/ci/gha-install-ncrystal.sh
else
cd $HOME/ncrystal_bld
eval $( "$HOME/ncrystal_inst/bin/ncrystal-config" --setup )
cd $GITHUB_WORKSPACE
fi
fi

# Install vectfit for WMP generation if needed
if [[ $VECTFIT = 'y' ]]; then
./tools/ci/gha-install-vectfit.sh
if [[ $VECTFIT = 'y' ]]; then
if [ ! -d "$HOME/vectfit" ]
then
./tools/ci/gha-install-vectfit.sh
fi
fi

# Install libMesh if needed
if [[ $LIBMESH = 'y' ]]; then
./tools/ci/gha-install-libmesh.sh
if [ ! -d "$HOME/libmesh" ]
then
./tools/ci/gha-install-libmesh.sh
else
pip install $HOME/pybind11
pip install wheel
pip install --no-build-isolation ./vectfit
johvincau marked this conversation as resolved.
Show resolved Hide resolved
fi
fi

# Install MCPL
./tools/ci/gha-install-mcpl.sh
if [[ ! -d "$HOME/mcpl" ]]; then
./tools/ci/gha-install-mcpl.sh
fi


# For MPI configurations, make sure mpi4py and h5py are built against the
# correct version of MPI
Expand Down
Loading