tests, run-spread: install snapd snap during classic system prepare (c…

…anonical#14294) * tests/lib/prepare: install snapd snap for classic systems * run-spread, tests: split out a helper to build the dev build Signed-off-by: Maciej Borzecki <[email protected]> * tests/lib/state: save snapd state instead of reset Signed-off-by: Maciej Borzecki <[email protected]> * tests/lib/prepare: keep a copy of repacked core snap Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/snapd-snap-transition: fix the test to install core before transitioning Signed-off-by: Maciej Borzecki <[email protected]> * tests/lib/state: also save core state Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/nfs-support: check snap-confine profile for internal parser Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/exitcodes: fix the test when using snapd snap Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/apparmor-batch-reload: account for apparmor_parser from snapd snap Signed-off-by: Maciej Borzecki <[email protected]> * fixup! tests/main/snapd-snap-transition: fix the test to install core before transitioning Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/snapd-reexec: adapt to test for snapd in core and snapd snap * tests/main/cifs-home: fix for snapd snapd Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/remove-core: fix for snapd snap Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/snap-userd-reexec: fix for snapd snap Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/snapd-reexec-snapd-snap: fixes for snapd snap Signed-off-by: Maciej Borzecki <[email protected]> * tests: account for prompting supported on >= 22.04 with internal apparmor Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/snap-seccomp-blocks-tty-injection: TODO for a fix Signed-off-by: Maciej Borzecki <[email protected]> * fixup! tests/main/snap-seccomp-blocks-tty-injection: TODO for a fix Signed-off-by: Maciej Borzecki <[email protected]> * tests: keep a copy of repacked core snap under $TESTSTMP Signed-off-by: Maciej Borzecki <[email protected]> * run-spread: slightly smarter handling of snap build Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/mount-ns: update expected mount ns view Signed-off-by: Maciej Borzecki <[email protected]> * spread: bump the storage for google:ubuntu-16.04 nodes Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/snapd-slow-startup: account for additional snap Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/interfaces-many-core-provided: account for snapd snap vendored apparmor Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/apparmor-prompting-snapd-startup: account for vendored apparmor on 22.04 Signed-off-by: Maciej Borzecki <[email protected]> * tests/lib/prepare: build snapd snap on ARM only Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/debug-execution: update to account for snapd snap Signed-off-by: Maciej Borzecki <[email protected]> * tests/lib/prepare: clean up after snapcraft and lxd Signed-off-by: Maciej Borzecki <[email protected]> * tests: review improvements * tests/main/interfaces-snap-interfaces-requests-control: disable on debian-sid Debian Sid kernel seems to support prompting, but it appears to be non-functional. Signed-off-by: Maciej Borzecki <[email protected]> * tests/build-test-snapd-snap: find tweaks Signed-off-by: Maciej Borzecki <[email protected]> * tests/main/exitcodes: check we mocked at least 2 commands Signed-off-by: Maciej Borzecki <[email protected]> --------- Signed-off-by: Maciej Borzecki <[email protected]> Co-authored-by: Maciej Borzecki <[email protected]>
pedronis · Sep 2, 2024 · 60f28bd · 60f28bd
1 parent 4159f95
commit 60f28bd
Show file tree

Hide file tree

Showing 29 changed files with 335 additions and 175 deletions.
diff --git a/run-spread b/run-spread
@@ -1,25 +1,33 @@
 #!/bin/bash
 
-# check dependencies
-if ! snap list snapcraft &>/dev/null; then
-    echo "snapcraft is not installed"
-    exit 1
-fi
+set -e
+
+need_rebuild=1
 
-# Clean the snaps created in previous runs
-rm -rf built-snap test-build
-touch test-build
-mkdir built-snap
+shopt -s nullglob
 
-rm -f snapd_1337.*.snap 
-rm -f snapd_1337.*.snap.keep
+if [ "$NO_REBUILD" = "1" ]; then
+    echo "-- $(date) -- requested no snap rebuild"
+    need_rebuild=0
 
-# Build snapd snap
-snapcraft clean
-snapcraft --use-lxd
-for snap in snapd_1337.*.snap; do
-    mv "${snap}" built-snap/"${snap}.keep"
-done
+    # check if we have any snaps built at all
+    built_snaps=(built-snap/snapd_*.snap.keep)
+    if (( "${#built_snaps[@]}" > 0 )); then
+        echo "-- $(date) -- found prebuilt snapd snaps:"
+        for s in "${built_snaps[@]}"; do
+            echo "--   $s"
+        done
+    else
+        echo "-- $(date) -- no prebuilt snaps found"
+        need_rebuild=1
+    fi
+fi
+
+if [ "$need_rebuild" = 1 ]; then
+    echo "-- $(date) -- rebuilding snapd snap"
+    ./tests/build-test-snapd-snap
+    echo "-- $(date) -- snapd snap rebuild complete"
+fi
 
 # Run spread
 SPREAD_USE_PREBUILT_SNAPD_SNAP=true spread "$@"
diff --git a/spread.yaml b/spread.yaml
@@ -123,7 +123,7 @@ backends:
                   workers: 6
             - ubuntu-16.04-64:
                   workers: 8
-                  storage: 12G
+                  storage: 15G
             - ubuntu-18.04-64:
                   storage: 15G
                   workers: 8

diff --git a/tests/build-test-snapd-snap b/tests/build-test-snapd-snap
@@ -0,0 +1,31 @@
+#!/bin/bash -e
+
+shopt -s nullglob
+
+# check dependencies
+if ! snap list snapcraft >/dev/null; then
+    echo "snapcraft is not installed"
+    exit 1
+fi
+
+# Clean the snaps created in previous runs
+rm -rfv built-snap
+
+find . -name 'snapd_1337.*.snap' -delete -print
+find . -name 'snapd_1337.*.snap.keep' -delete -print
+
+touch test-build
+mkdir -p built-snap
+
+# Build snapd snap
+if [ -z "$SNAPCRAFT_NO_CLEAN" ]; then
+    snapcraft --verbose clean
+fi
+
+snapcraft --verbose
+
+for snap_file in snapd_1337.*.snap; do
+    mv "${snap_file}" built-snap/"${snap_file}.keep"
+done
+
+rm -fv test-build
diff --git a/tests/lib/prepare.sh b/tests/lib/prepare.sh
@@ -154,7 +154,12 @@ setup_experimental_features() {
     fi
 }
 
+# update_core_snap_for_classic_reexec modifies the core snap for snapd re-exec
+# by injecting binaries from the installed snapd deb built from our modified code.
+# $1: directory where updated core snap should be copied (optional)
 update_core_snap_for_classic_reexec() {
+    local target_dir="${1-}"
+
     # it is possible to disable this to test that snapd (the deb) works
     # fine with whatever is in the core snap
     if [ "$MODIFY_CORE_SNAP_FOR_REEXEC" != "1" ]; then
@@ -230,6 +235,12 @@ update_core_snap_for_classic_reexec() {
     chmod --reference="${snap}.orig" "$snap"
     rm -rf squashfs-root
 
+    # make a copy for later use
+    if [ -n "$target_dir" ]; then
+        mkdir -p "$target_dir"
+        cp -av "$snap" "$target_dir/"
+    fi
+
     # Now mount the new core snap, first discarding the old mount namespace
     snapd.tool exec snap-discard-ns core
     mount "$snap" "$core"
@@ -386,6 +397,22 @@ prepare_classic() {
         fi
     done
 
+    # Install snapd snap to ensure re-exec to snapd snap instead of snapd in core.
+    # This also prevents snapd from automatically installing snapd snap as
+    # prerequisite for installing any non-base snap introduced in PR#14173.
+    if snap list snapd ; then
+	    snap snap info snapd
+	    echo "Error: not expecting snapd snap to be installed"
+	    exit 1
+    else
+	    build_dir="$WORK_DIR/snapd_snap_for_classic"
+	    rm -rf "$build_dir"
+	    mkdir -p "$build_dir"
+	    build_snapd_snap "$build_dir"
+	    snap install --dangerous "$build_dir/"snapd_*.snap
+    fi
+    snap list snapd
+
     setup_systemd_snapd_overrides
 
     if [ "$REMOTE_STORE" = staging ]; then
@@ -424,7 +451,8 @@ prepare_classic() {
         snap list | grep core
 
         systemctl stop snapd.{service,socket}
-        update_core_snap_for_classic_reexec
+        # repack and also make a side copy of the core snap
+        update_core_snap_for_classic_reexec "$TESTSTMP/core_snap"
         systemctl start snapd.{service,socket}
 
         prepare_reexec_override
@@ -472,6 +500,10 @@ cleanup_snapcraft() {
     snap remove --purge lxd || true
     "$TESTSTOOLS"/lxd-state undo-mount-changes
     snap remove --purge snapcraft || true
+    # TODO there should be some smarter cleanup helper which removes all snaps
+    # in the right order
+    # base snap of both lxd and snapcraft
+    snap remove --purge core22 || true
 }
 
 run_snapcraft() {
@@ -502,8 +534,20 @@ build_snapd_snap() {
                     cp "${PROJECT_PATH}/built-snap"/snapd_1337.*.snap.keep "${snapd_snap_cache}/snapd_from_ci.snap"
                 fi
             else
+                # This is not reliable across classic releases so only allow on
+                # ARM variants as a special case since we cannot cross build
+                # snapd snap for ARM right now
+                case "$SPREAD_SYSTEM" in
+                    *-arm-*)
+                        ;;
+                    *)
+                        echo "system $SPREAD_SYSTEM should use a prebuilt snapd snapd"
+                        exit 1
+                        ;;
+                esac
                 [ -d "${TARGET}" ] || mkdir -p "${TARGET}"
                 chmod -R go+r "${PROJECT_PATH}/tests"
+                # TODO: run_snapcraft does not currently guarantee or check the required version for building snapd
                 run_snapcraft --use-lxd --verbosity quiet --output="snapd_from_snapcraft.snap"
                 mv "${PROJECT_PATH}"/snapd_from_snapcraft.snap "${snapd_snap_cache}"
             fi

diff --git a/tests/lib/state.sh b/tests/lib/state.sh
@@ -51,13 +51,16 @@ save_snapd_state() {
         done
         snapd_env="/etc/environment"
         snapd_service_env=$(ls -d /etc/systemd/system/snapd.*.d || true)
-        snap_confine_profiles="$(ls /etc/apparmor.d/snap.core.* || true)"
+        snap_confine_profiles="$(ls /etc/apparmor.d/snap.snapd.* || true)"
 
         # shellcheck disable=SC2086
         tar cf "$SNAPD_STATE_FILE" \
             /var/lib/snapd \
             /var/cache/snapd \
             "$SNAP_MOUNT_DIR" \
+            /etc/systemd/system/"$escaped_snap_mount_dir"-*snapd*.mount \
+            /etc/systemd/system/snapd.mounts.target.wants/"$escaped_snap_mount_dir"-*snapd*.mount \
+            /etc/systemd/system/multi-user.target.wants/"$escaped_snap_mount_dir"-*snapd*.mount \
             /etc/systemd/system/"$escaped_snap_mount_dir"-*core*.mount \
             /etc/systemd/system/snapd.mounts.target.wants/"$escaped_snap_mount_dir"-*core*.mount \
             /etc/systemd/system/multi-user.target.wants/"$escaped_snap_mount_dir"-*core*.mount \

diff --git a/tests/main/apparmor-batch-reload/task.yaml b/tests/main/apparmor-batch-reload/task.yaml
@@ -41,8 +41,10 @@ execute: |
     echo "Update system key"
     printf '{"version":1}' > /var/lib/snapd/system-key
 
-    echo "Replace apparmor parser with a broken one"
-    cp -f bin/apparmor_parser.fake /sbin/apparmor_parser
+    # snapd will execute the vendored apparmor-parser from the snapd snap
+    echo "Replace apparmor parser from the snapd snap with a broken one"
+    mount -o bind bin/apparmor_parser.fake /snap/snapd/current/usr/lib/snapd/apparmor_parser
+    tests.cleanup defer umount /snap/snapd/current/usr/lib/snapd/apparmor_parser
 
     # remove all apparmor cached profiles so we can check they are recreated
     rm -f $AA_CACHE/snap* $AA_CACHE/*/snap*

diff --git a/tests/main/apparmor-prompting-flag-restart/task.yaml b/tests/main/apparmor-prompting-flag-restart/task.yaml
@@ -106,9 +106,9 @@ execute: |
     fi
 
     echo "Enable prompting via snap client where possible"
-    if os.query is-core || os.query is-ubuntu-lt 24.04; then
+    if os.query is-core || os.query is-ubuntu-lt 22.04; then
         # prompting is disabled on Ubuntu Core
-        # TODO on releases < 24.04 we need the snapd snap for testing
+        # on releases < 22.04 the kernel does not support prompting
         not snap set system experimental.apparmor-prompting=true >& err.out
         if os.query is-core ; then
             MATCH "cannot enable prompting feature as it is not supported on Ubuntu Core systems" < err.out

diff --git a/tests/main/apparmor-prompting-snapd-startup/task.yaml b/tests/main/apparmor-prompting-snapd-startup/task.yaml
@@ -24,8 +24,8 @@ execute: |
     echo '{"rules":[{"id":"0000000000000002","timestamp":"2004-10-20T14:05:08.901174186-05:00","user":1000,"snap":"shellcheck","interface":"home","constraints":{"path-pattern":"/home/test/Projects/**","permissions":["read"]},"outcome":"allow","lifespan":"forever","expiration":"0001-01-01T00:00:00Z"},{"id":"0000000000000003","timestamp":"2004-10-20T16:47:32.138415627-05:00","user":1000,"snap":"firefox","interface":"home","constraints":{"path-pattern":"/home/test/Downloads/**","permissions":["read","write"]},"outcome":"allow","lifespan":"timespan","expiration":"2005-04-08T00:00:00Z"}]}' | tee "$RULES_PATH"
 
     # Prompting is disabled everywhere but the Ubuntu systems
-    # TODO: on Ubuntu releases < 24.04 we need the snapd snap for testing
-    if ! os.query is-ubuntu || os.query is-ubuntu-lt 24.04 || os.query is-core ; then
+    # on releases < 22.04 the kernel does not support prompting
+    if ! os.query is-ubuntu || os.query is-ubuntu-lt 22.04 || os.query is-core ; then
         not snap set system experimental.apparmor-prompting=true >& err.out
         if os.query is-core; then
             # there is a more specific error on Ubuntu Core

diff --git a/tests/main/cifs-home/task.yaml b/tests/main/cifs-home/task.yaml
@@ -57,7 +57,9 @@ prepare: |
     # Later on, restart snapd and ensure that nfs/cifs workaround is gone.
     # This cleanup handler is registered before we mount the cifs file system.
     if [ "$(snap debug confinement)" = strict ]; then
-        tests.cleanup defer test ! -e /var/lib/snapd/apparmor/snap-confine/nfs-support
+        # we're testing on Ubuntu where we know that reexec is active and we use
+        # an internal apparmor userspace stack
+        tests.cleanup defer test ! -e /var/lib/snapd/apparmor/snap-confine.internal/nfs-support
     fi
     tests.cleanup defer systemctl restart snapd.service
     tests.cleanup defer systemctl reset-failed snapd.service snapd.socket
@@ -87,7 +89,7 @@ prepare: |
     systemctl reset-failed snapd.service snapd.socket
     systemctl restart snapd.service
     if [ "$(snap debug confinement)" = strict ]; then
-        MATCH 'network inet,' < /var/lib/snapd/apparmor/snap-confine/nfs-support
+        MATCH 'network inet,' < /var/lib/snapd/apparmor/snap-confine.internal/nfs-support
         MATCH 'network inet,' < /var/lib/snapd/apparmor/profiles/snap.test-snapd-sh.with-home-plug
     fi
 

diff --git a/tests/main/debug-execution/task.yaml b/tests/main/debug-execution/task.yaml
@@ -74,9 +74,7 @@ execute: |
             MATCH 'is-reexec-enabled: true' < snap-default.out
             MATCH 'is-reexec-explicitly-enabled: false' < snap-default.out
             MATCH 'is-reexecd: true' < snap-default.out
-            # TODO: once snapd snap lands the output wlll be different:
-            # MATCH 'self-exe: /snap/snapd/.*/usr/bin/snap' < snap-default.out
-            MATCH 'self-exe: /snap/core/.*/usr/bin/snap' < snap-default.out
+            MATCH 'self-exe: /snap/snapd/.*/usr/bin/snap' < snap-default.out
 
             echo "Checking without reeexec scenario"
             MATCH 'distro-supports-reexec: true' < snap-no-reexec.out
@@ -86,25 +84,20 @@ execute: |
             MATCH 'self-exe: /usr/bin/snap' < snap-no-reexec.out
 
             echo "Checking AppArmor"
-            # TODO: once snapd snap lands, this will use internal parser
-            # MATCH 'apparmor-parser: /snap/snapd/.*/usr/lib/snapd/apparmor_parser' < snap-apparmor-default.out
-            # MATCH 'internal: true' < snap-apparmor-default.out
+            MATCH 'apparmor-parser: /snap/snapd/.*/usr/lib/snapd/apparmor_parser' < snap-apparmor-default.out
+            MATCH 'apparmor-parser-command: /snap/snapd/.*/apparmor_parser --config-file /snap/snapd/.*/usr/lib/snapd/apparmor/parser.conf --base /snap/snapd/.*/usr/lib/snapd/apparmor\.d --policy-features /snap/snapd/.*/usr/lib/snapd/apparmor\.d/abi/4\.0' < snap-apparmor-default.out
+            MATCH 'internal: true' < snap-apparmor-default.out
+
             if os.query is-xenial || os.query is-bionic; then
                 # Ubuntu < 20.04 does not have usr-merge
-                MATCH 'apparmor-parser: /sbin/apparmor_parser' < snap-apparmor-default.out
-                MATCH 'internal: false' < snap-apparmor-default.out
                 MATCH 'apparmor-parser: /sbin/apparmor_parser' < snap-apparmor-no-reexec.out
                 MATCH 'internal: false' < snap-apparmor-no-reexec.out
             else
-                MATCH 'apparmor-parser: /usr/sbin/apparmor_parser' < snap-apparmor-default.out
-                MATCH 'internal: false' < snap-apparmor-default.out
                 MATCH 'apparmor-parser: /usr/sbin/apparmor_parser' < snap-apparmor-no-reexec.out
                 MATCH 'internal: false' < snap-apparmor-no-reexec.out
             fi
 
-            # TODO: once snapd snap lands, this will use snap-update-ns from
-            # snapd snap
-            MATCH 'snap-update-ns: /snap/core/.*/usr/lib/snapd/snap-update-ns' < snap-uns-default.out
+            MATCH 'snap-update-ns: /snap/snapd/.*/usr/lib/snapd/snap-update-ns' < snap-uns-default.out
             ;;
         *)
             echo "Checking default scenario"
@@ -134,12 +127,12 @@ execute: |
                 opensuse-*)
                     # snap mount dir is /snap
                     MATCH 'is-reexecd: true' < snap-yes-reexec.out
-                    MATCH 'self-exe: /snap/core/.*/usr/bin/snap' < snap-yes-reexec.out
+                    MATCH 'self-exe: /snap/snapd/.*/usr/bin/snap' < snap-yes-reexec.out
                     ;;
                 amazon-linux-*)
                     # has /snap -> /var/lib/snapd symlink
                     MATCH 'is-reexecd: true' < snap-yes-reexec.out
-                    MATCH 'self-exe: /var/lib/snapd/snap/core/.*/usr/bin/snap' < snap-yes-reexec.out
+                    MATCH 'self-exe: /var/lib/snapd/snap/snapd/.*/usr/bin/snap' < snap-yes-reexec.out
                     ;;
                 *)
                     echo "unexpected distro $SPREAD_SYSTEM"

diff --git a/tests/main/exitcodes/task.yaml b/tests/main/exitcodes/task.yaml
@@ -26,10 +26,20 @@ execute: |
     test "$RET" -eq 64
 
     echo "snap command with broken mksquashfs returns exit code 20"
-    for b in /usr/bin/mksquashfs /snap/core/current/usr/bin/mksquashfs; do
-        mount -o bind /bin/false "$b"
-        tests.cleanup defer umount "$b"
+    command_found=0
+    for b in /usr/bin/mksquashfs /snap/core/current/usr/bin/mksquashfs /snap/snapd/current/usr/bin/mksquashfs; do
+        if [ -f "$b" ]; then
+            command_found=$((command_found + 1))
+            mount -o bind /bin/false "$b"
+            tests.cleanup defer umount "$b"
+        fi
     done
+    # make sure we found at least one of the commands
+    if (( command_found < 2 )); then
+        echo "should have mocked at least 2 commands"
+        exit 1
+    fi
+
     set +e
     snap pack "$TESTSLIB/snaps/test-snapd-sh"
     RET=$?

diff --git a/tests/main/interfaces-many-core-provided/task.yaml b/tests/main/interfaces-many-core-provided/task.yaml
@@ -91,9 +91,9 @@ execute: |
             continue
         fi
 
-        if [ "$plug_iface" = "$CONSUMER_SNAP:qualcomm-ipc-router" ] && ( os.query is-trusty || os.query is-xenial || os.query is-core16) ; then
-            # the qualcomm-ipc-router interface is known not to work on xenial,
-            # just check that it cannot be connected and move on
+        if [ "$plug_iface" = "$CONSUMER_SNAP:qualcomm-ipc-router" ] && ( os.query is-trusty || os.query is-core16) ; then
+            # the qualcomm-ipc-router interface is known not to work on UC16
+            # without snapd, just check that it cannot be connected and move on
             snap connect "$plug_iface" "$slot_iface" 2>&1 | MATCH "cannot connect plug on system without qipcrtr socket support"
             continue
         fi

diff --git a/tests/main/interfaces-requests-activates-handlers/task.yaml b/tests/main/interfaces-requests-activates-handlers/task.yaml
@@ -19,9 +19,9 @@ restore: |
 
 execute: |
     echo "Enable prompting via snap client where possible"
-    if os.query is-core || os.query is-ubuntu-lt 24.04; then
+    if os.query is-core || os.query is-ubuntu-lt 22.04; then
         # prompting is disabled on Ubuntu Core
-        # TODO on releases < 24.04 we need the snapd snap for testing
+        # on releases < 22.04 we need the kernel does not support prompting
         not snap set system experimental.apparmor-prompting=true >& err.out
         if os.query is-core ; then
             MATCH "cannot enable prompting feature as it is not supported on Ubuntu Core systems" < err.out

diff --git a/tests/main/interfaces-snap-interfaces-requests-control/task.yaml b/tests/main/interfaces-snap-interfaces-requests-control/task.yaml
@@ -10,6 +10,10 @@ details: |
       # TODO: - /v2/interfaces/requests/prompts: to receive and reply to request prompts
       # TODO: - /v2/interfaces/requests/rules: to view and manage request rules
 
+systems:
+    # debian-sid: prompting is supposedly supported by the kernel, but doesn't work
+    - -debian-sid-*
+
 environment:
     # not all terminals support UTF-8, but Python tries to be smart and attempts
     # to guess the encoding as if the output would go to the terminal, but in
@@ -40,8 +44,8 @@ execute: |
 
     echo "Ensure AppArmor Prompting experimental feature can be enabled where possible"
     # prompting is disabled everywhere but the Ubuntu systems
-    # TODO on Ubuntu releases < 24.04 we need the snapd snap for testing
-    if ! os.query is-ubuntu || os.query is-ubuntu-lt 24.04 || os.query is-core ; then
+    # on Ubuntu releases < 22.04 the kernel does not support prompting
+    if ! os.query is-ubuntu || os.query is-ubuntu-lt 22.04 || os.query is-core ; then
         not snap set system experimental.apparmor-prompting=true >& err.out
         if os.query is-core; then
             # there is a more specific error on Ubuntu Core