Create common python script for gathering images (#1038)

* Create get-all-images.py

* requirements.txt

* Use the get-all-images.py in airgap tests

* Use get-all-images.py from scanning action

Signed-off-by: Kimonas Sotirchos <[email protected]>

* remove get-all-images.sh

Signed-off-by: Kimonas Sotirchos <[email protected]>

* Address review comments

---------

Signed-off-by: Kimonas Sotirchos <[email protected]>

.github/workflows/scan-images.yaml

@@ -13,58 +13,52 @@ jobs:
         # specfy location of bundle(s) to be scanned
         bundle:
           - releases/1.8/stable/kubeflow
-          - releases/1.9/stable/kubeflow
+          - releases/1.9/stable
           - releases/latest/edge
-    runs-on: ubuntu-20.04
+    runs-on: [self-hosted, linux, X64, jammy, large]
     steps:
-      # Ideally we'd use self-hosted runners, but this effort is still not stable
-      # This action will remove unused software (dotnet, haskell, android libs, codeql,
-      # and docker images) from the GH runner, which will liberate around 60 GB of storage
-      # distributed in 40GB for root and around 20 for a mnt point.
-      - name: Maximise GH runner space
-        uses: easimon/maximize-build-space@v7
-        with:
-          root-reserve-mb: 29696
-          remove-dotnet: 'true'
-          remove-haskell: 'true'
-          remove-android: 'true'
-          remove-codeql: 'true'
-          remove-docker-images: 'true'
       - name: Checkout
         uses: actions/checkout@v3
         with:
           fetch-depth: 0
+
       - name: Setup tools
         id: setup
         run: |
           sudo snap install yq
           echo "date=$(date '+%Y-%m-%d-%H-%M-%S')" >> $GITHUB_OUTPUT
+
       - name: Checkout kubeflow-ci
         uses: actions/checkout@v3
         with:
           repository: canonical/kubeflow-ci.git
           sparse-checkout: scripts/images/
           ref: main
           path: kubeflow-ci
+
       - name: Get images
         id: images
         run: |
           BUNDLE="${{ matrix.bundle }}"
           BUNDLE_SPLIT=(${BUNDLE//\// })
           RELEASE=${BUNDLE_SPLIT[1]}
           RISK=${BUNDLE_SPLIT[2]}
-          IMAGES=$(./kubeflow-ci/scripts/images/get-all-images.sh ${{ matrix.bundle }}/bundle.yaml ${RELEASE}-${RISK})
-          echo "$IMAGES" > ./image_list.txt
+
+          pip3 install -r scripts/requirements.txt
+          python3 scripts/get-all-images.py ${{ matrix.bundle }}/bundle.yaml > image_list.txt
           echo "Image list:"
           cat ./image_list.txt
           echo "release_risk=${RELEASE}-${RISK}" >> $GITHUB_OUTPUT
+
       - name: Scan images
         run: |
           ./kubeflow-ci/scripts/images/scan-images.sh ./image_list.txt
           ./kubeflow-ci/scripts/images/get-summary.py --report-path ./trivy-reports --print-header > scan-summary-${{ steps.setup.outputs.date}}-${{ steps.images.outputs.release_risk }}.csv
+
       - name: Prepare artifacts
         run: |
           tar zcvf trivy-reports-${{ steps.setup.outputs.date}}-${{ steps.images.outputs.release_risk }}-${{ strategy.job-index }}.tar.gz ./trivy-reports
+
       - name: Upload Trivy reports
         uses: actions/upload-artifact@v3
         with:

scripts/README.md

@@ -0,0 +1,26 @@
+# Utility Script
+
+This directory contains helper scripts for Charmed Kubeflow, during CI and not only.
+
+## Gather images used by a bundle
+
+You can get a list of all the OCI images used by the bundle by running the following command:
+```bash
+pip3 install -r scritps/requirements.txt
+
+python3 scripts/get-all-images.py \
+    --append-images tests/airgapped/ckf-1.8-testing-images.txt \
+    releases/1.8/stable/kubeflow/bundle.yaml \
+    > images-all.txt
+```
+
+The script will gather the images in the following way:
+1. For each `application` in the provided `bundle.yaml` file:
+2. detect if it's owned by us or another team (by looking at the `_github_dependency_repo_name` and such metadata)
+3. clone its repo, by looking at `_github_repo_name` and such metadata
+4. If owned by another team: only parse it's `metadata.yaml` and look for `oci-resources`
+5. If owned by us: run the `tools/get-images.sh` script the repo **must** have
+6. If a repo does not have `tools/get-images.sh` (i.e. kubeflow-roles) then the script should skip the repo
+7. If the `get-images.sh` script either fails (return code non zero) or has error logs then the script should **fail**
+8. Aggregate the outputs of all `get-images.sh` scripts to one output
+9. If user passed an argument `--append-images` then the script will amend a list of images we need for airgap testing

scripts/airgapped/README.md

@@ -7,12 +7,13 @@ to create airgap artifacts or via our testing scripts.
 We'll document some use-case scenarios here for the different scripts.
 
 ## Prerequisites
+NOTE: All the commands are expected to be run from the root directory of the repo
 
 To use the scripts in this directory you'll need to install a couple of Python
 and Ubuntu packages on the host machine, driving the test (not the LXC machine
 that will contain the airgapped environment).
 ```
-pip3 install -r requirements.txt
+pip3 install -r scripts/airgapped/requirements.txt
 sudo apt install pigz
 sudo snap install docker
 sudo snap install yq
@@ -32,7 +33,10 @@ This script makes the following assumptions:
    the images for that repo
 
 ```bash
-./scripts/airgapped/get-all-images.sh releases/1.7/stable/kubeflow/bundle.yaml > images.txt
+python3 scripts/get-all-images.py \
+    --append-images=tests/airgapped/ckf-1.8-testing-images.txt \
+    releases/1.8/stable/kubeflow/bundle.yaml \
+    > images.txt
 ```
 
 ## Pull images to docker cache

scripts/airgapped/requirements.txt

@@ -1,4 +1,3 @@
 docker
-#FIXME: remove requests pin when https://github.com/docker/docker-py/issues/3256 is solved 
-requests<2.32.0
+requests
 PyYAML