ci(e2e): run nightly and release-tag e2e on the container lane

Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.

Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.

DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit da28ea7ee8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

.github/workflows/e2e-tag.yaml

@@ -11,8 +11,10 @@ name: E2E Release Tag
 # This lane validates published tags only; draft and promote-time validation are
 # explicitly out of scope because release checks run at RC publication time.
 # Prerelease tags use the matching digest-pinned staging branch, while stable
-# tags use their pinned merge tree directly. In both cases the nocloud disk
-# comes from the published release identified by the same tag.
+# tags use their pinned merge tree directly. The nodes run as Talos containers
+# booted from the upstream image, so the release's nocloud disk is required to
+# exist but is not what the suite runs on. It is still required because users
+# install from it, and promote-rc.yaml checks it for an rc only.
 
 on:
   workflow_call:
@@ -37,8 +39,8 @@ concurrency:
   group: e2e-tag-${{ inputs.tag }}
   cancel-in-progress: false
 
-# Listing published releases, reading Git refs, downloading a release asset, and
-# checking out the pinned tree require only repository contents read access.
+# Listing published releases, reading Git refs and checking out the pinned tree
+# require only repository contents read access.
 # Draft releases are not part of this lane's contract. The public GHCR images
 # referenced by that tree need no package token.
 #
@@ -57,7 +59,6 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 10
     outputs:
-      disk_id: ${{ steps.resolve.outputs.disk_id }}
       tree_ref: ${{ steps.resolve.outputs.tree_ref }}
     steps:
       - name: Validate tag and resolve release
@@ -123,16 +124,15 @@ jobs:
               throw error;
             }
 
-            core.setOutput('disk_id', String(disk.id));
             core.setOutput('tree_ref', treeRef);
             core.info(`Resolved ${tag}: tree '${treeRef}', disk asset ${disk.id}`);
 
   e2e:
     name: E2E ${{ inputs.tag }} (full suite)
     needs: resolve
-    # The sandbox boots three 8-vCPU / 24-GiB QEMU guests. This runner leaves
-    # enough host headroom for QEMU, networking, containerd, and the runner
-    # agent while the full platform installs across all three guests.
+    # The sandbox runs three 8-vCPU / 24-GiB Talos containers. This runner
+    # leaves enough host headroom for networking, containerd, and the runner
+    # agent while the full platform installs across all three nodes.
     runs-on: oracle-vm-32cpu-128gb-x86-64
     timeout-minutes: 180
     permissions:
@@ -148,45 +148,74 @@ jobs:
           fetch-depth: 1
           persist-credentials: false
 
-      - name: Download disk asset from release
-        env:
-          DISK_ID: ${{ needs.resolve.outputs.disk_id }}
-          GH_TOKEN: ${{ github.token }}
-        run: |
-          mkdir -p _out/assets
-          curl --fail-with-body --silent --show-error --location \
-            -H "Authorization: Bearer ${GH_TOKEN}" \
-            -H "Accept: application/octet-stream" \
-            -o _out/assets/nocloud-amd64.raw.xz \
-            "https://api.github.com/repos/${GITHUB_REPOSITORY}/releases/assets/${DISK_ID}"
-
       - name: Set sandbox ID
         env:
           TAG: ${{ inputs.tag }}
         run: echo "SANDBOX_NAME=cozy-e2e-sandbox-$(echo "${GITHUB_REPOSITORY}:${GITHUB_WORKFLOW}:${TAG}" | sha256sum | cut -c1-10)" >> "$GITHUB_ENV"
 
+      # tags.yaml calls the e2e-tag.yaml from the tag's own commit, so this can
+      # only fire on a dispatch from a newer ref against an older tag.
+      - name: Require a container-lane tree
+        env:
+          TAG: ${{ inputs.tag }}
+        run: |
+          if [ ! -f hack/e2e-compose.yaml ]; then
+            echo "::error title=e2e-tag::${TAG} predates the container lane; dispatch e2e-tag.yaml with --ref ${TAG} to run it on the lane it shipped with"
+            exit 1
+          fi
+
       - name: Prepare workspace
         run: |
           rm -rf "/tmp/$SANDBOX_NAME"
           cp -r "${{ github.workspace }}" "/tmp/$SANDBOX_NAME"
 
-      # This is the only retried stage: Talos image unpacking, QEMU startup, and
-      # sandbox networking are pure runner infrastructure, not product logic.
+      # Talos cannot load kernel modules in container mode, so the nodes get
+      # exactly what this host has loaded. Why each module, and why failing
+      # here is better than an hour later: the same step in pull-requests.yaml.
+      - name: Prepare container-lane host
+        run: |
+          set -eu
+          for m in openvswitch zfs; do
+            grep -q "^${m} " /proc/modules && continue
+            if [ "${m}" = zfs ]; then
+              sudo apt-get update -qq
+              sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
+            fi
+            sudo modprobe "${m}" || echo "${m}: modprobe failed"
+          done
+          missing=''
+          for m in openvswitch zfs; do
+            grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
+          done
+          if [ -n "${missing}" ]; then
+            echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
+            exit 1
+          fi
+
+      # This is the only retried stage: Talos container startup and sandbox
+      # networking are pure runner infrastructure, not product logic.
       - name: Prepare environment
         run: |
           cd "/tmp/$SANDBOX_NAME"
           attempt=0
-          until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
+          until make SANDBOX_NAME="$SANDBOX_NAME" prepare-env-container; do
             attempt=$((attempt + 1))
             if [ $attempt -ge 3 ]; then
               echo "Attempt $attempt failed, exiting..."
               exit 1
             fi
             echo "Attempt $attempt failed, retrying..."
+            make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
           done
           echo "Prepare environment completed after $((attempt + 1)) attempts"
 
       - name: Install Cozystack into sandbox
+        env:
+          # No ARP VIP in container mode: .11 is a plain control-plane node.
+          COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
+          # The nodes share the runner kernel and cannot load DRBD.
+          COZY_LINSTOR_DRBD_ENABLED: "false"
+          COZY_E2E_STORAGE_CLASS: local
         run: |
           cd "/tmp/$SANDBOX_NAME"
           if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
@@ -216,6 +245,8 @@ jobs:
       # CHAINSAW_SUITES is the testing Makefile's full-suite mode.
       - name: Run E2E tests (full suite)
         id: e2e_tests
+        env:
+          COZY_E2E_STORAGE_CLASS: local
         run: |
           cd "/tmp/$SANDBOX_NAME"
           if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES=""; then
@@ -358,13 +389,17 @@ jobs:
             Resume from inside: `breakpoint resume`. Otherwise the breakpoint
             exits 10 minutes after the last SSH session disconnects.
 
+      # The compose project owns the nodes and the zpools outlive it, so both
+      # halves go, in this order.
       - name: Tear down sandbox
         if: always()
-        run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
+        run: |
+          make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
+          make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete || true
 
       - name: Remove workspace
         if: always()
-        run: rm -rf "/tmp/$SANDBOX_NAME"
+        run: rm -rf "/tmp/${SANDBOX_NAME:?}"
 
       - name: Summarize outcome
         if: always()

.github/workflows/nightly.yaml

@@ -10,7 +10,8 @@ name: Nightly
 # cozy-installer chart are re-published on GHCR so the whole install closure is
 # self-contained there. The only thing genuinely built is the Talos nocloud disk
 # (assembled from upstream profiles, independent of the registry), published as a
-# GHCR OCI artifact so a nightly is installable on real hardware, not just e2e.
+# GHCR OCI artifact so a nightly is installable on real hardware. The nightly
+# e2e does not boot it.
 #
 # No GitHub release and no api/apps/v1alpha1 Go-module tag are created — a
 # nightly is an installable artifact set on GHCR, nothing more. retention.yaml
@@ -216,13 +217,18 @@ jobs:
           oras tag "${DST_REGISTRY}/cozystack-nocloud:${VERSION}" nightly
           echo "Published cozystack-nocloud:${VERSION} (+ :nightly) to GHCR."
 
-  # ── Full e2e against the published GHCR closure (chart + images + disk).
+  # ── Full e2e against the published GHCR closure (chart + images), on the
+  #    same Talos-container substrate as the PR lanes. The nodes boot the
+  #    upstream Talos image, so the nocloud disk built above is published but
+  #    not exercised here, and e2e does not wait for it.
   e2e:
     name: E2E (full suite)
-    needs: [mirror, build-disk]
-    # Same sandbox as the other e2e lanes: three nodes at 8 vCPU each commit
-    # 24 vCPU for the whole run, so the runner class has to be strictly larger
-    # to leave the host its own cycles. pull-requests.yaml has the full note.
+    needs: [mirror]
+    # Same sandbox as the other e2e lanes: three nodes at 8 vCPU / 24 GiB each
+    # (hack/e2e-compose.yaml) commit 24 vCPU for the whole run, so the runner
+    # class has to be strictly larger, leaving the host its own cycles for
+    # containerd and the runner agent. hack/e2e-container-up.sh refuses a
+    # runner that is not.
     runs-on: oracle-vm-32cpu-128gb-x86-64
     timeout-minutes: 180
     permissions:
@@ -235,17 +241,15 @@ jobs:
           ref: ${{ needs.mirror.outputs.sha }}
           fetch-depth: 1
           persist-credentials: false
-      - name: Set up toolchain (flux, oras)
+      - name: Set up toolchain (flux)
         run: |
           command -v flux >/dev/null || curl -fsSL https://fluxcd.io/install.sh | sudo bash
-          command -v oras >/dev/null || { curl -fsSL https://github.com/oras-project/oras/releases/download/v1.2.0/oras_1.2.0_linux_amd64.tar.gz | sudo tar -xz -C /usr/local/bin oras; }
       - name: Login to GHCR
         env:
           GHCR_USER:  ${{ github.actor }}
           GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           echo "$GHCR_TOKEN" | docker login "${DST_REGISTRY%%/*}" -u "$GHCR_USER" --password-stdin
-          echo "$GHCR_TOKEN" | oras login  "${DST_REGISTRY%%/*}" -u "$GHCR_USER" --password-stdin
 
       # Overlay the GHCR-rewritten tree (operator image, .e2e.image and every
       # component now point at GHCR), then pin the installer chart to the GHCR
@@ -264,17 +268,32 @@ jobs:
           yq -i ".cozystackOperator.platformSourceUrl = \"oci://${DST_REGISTRY}/cozystack-packages\"" packages/core/installer/values.yaml
           yq -i ".cozystackOperator.platformSourceRef = \"digest=${PACKAGES_DIGEST}\"" packages/core/installer/values.yaml
 
-      - name: Fetch nocloud disk from GHCR
-        env:
-          VERSION: ${{ needs.mirror.outputs.version }}
-        run: |
-          set -eu
-          mkdir -p _out/assets
-          oras pull "${DST_REGISTRY}/cozystack-nocloud:${VERSION}" --output _out/assets
-
       - name: Set sandbox ID
         run: echo "SANDBOX_NAME=cozy-e2e-sandbox-$(echo "${GITHUB_REPOSITORY}:${GITHUB_WORKFLOW}:${GITHUB_REF}" | sha256sum | cut -c1-10)" >> "$GITHUB_ENV"
 
+      # Talos cannot load kernel modules in container mode, so the nodes get
+      # exactly what this host has loaded. Why each module, and why failing
+      # here is better than an hour later: the same step in pull-requests.yaml.
+      - name: Prepare container-lane host
+        run: |
+          set -eu
+          for m in openvswitch zfs; do
+            grep -q "^${m} " /proc/modules && continue
+            if [ "${m}" = zfs ]; then
+              sudo apt-get update -qq
+              sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
+            fi
+            sudo modprobe "${m}" || echo "${m}: modprobe failed"
+          done
+          missing=''
+          for m in openvswitch zfs; do
+            grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
+          done
+          if [ -n "${missing}" ]; then
+            echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
+            exit 1
+          fi
+
       - name: Prepare workspace
         run: |
           rm -rf "/tmp/$SANDBOX_NAME"
@@ -284,13 +303,20 @@ jobs:
         run: |
           cd "/tmp/$SANDBOX_NAME"
           attempt=0
-          until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
+          until make SANDBOX_NAME="$SANDBOX_NAME" prepare-env-container; do
             attempt=$((attempt + 1))
             if [ $attempt -ge 3 ]; then echo "Attempt $attempt failed, exiting..."; exit 1; fi
             echo "Attempt $attempt failed, retrying..."
+            make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
           done
 
       - name: Install Cozystack into sandbox
+        env:
+          # No ARP VIP in container mode: .11 is a plain control-plane node.
+          COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
+          # The nodes share the runner kernel and cannot load DRBD.
+          COZY_LINSTOR_DRBD_ENABLED: "false"
+          COZY_E2E_STORAGE_CLASS: local
         run: |
           cd "/tmp/$SANDBOX_NAME"
           if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
@@ -316,6 +342,8 @@ jobs:
       # Always the FULL app suite — a nightly has no diff to scope against, so
       # Test Impact Analysis does not apply (empty CHAINSAW_SUITES = whole suite).
       - name: Run E2E tests (full suite)
+        env:
+          COZY_E2E_STORAGE_CLASS: local
         run: |
           cd "/tmp/$SANDBOX_NAME"
           if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES=""; then
@@ -360,9 +388,13 @@ jobs:
           path: /tmp/${{ env.SANDBOX_NAME }}/_out/cozyreport.tgz
           if-no-files-found: warn
 
+      # The compose project owns the nodes and the zpools outlive it, so both
+      # halves go, in this order.
       - name: Tear down sandbox
         if: always()
-        run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
+        run: |
+          make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
+          make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete || true
 
       - name: Remove workspace
         if: always()

.github/workflows/pull-requests.yaml

@@ -611,7 +611,7 @@ jobs:
       # nocloud disk asset and export its id; the container lane boots no disk,
       # so nothing in this workflow reads that id any more and resolving it only
       # added a way to fail a release PR over an asset it does not use. The
-      # asset is still resolved where it is still consumed, in e2e-tag.yaml.
+      # asset's presence is still checked in e2e-tag.yaml and promote-rc.yaml.
       - name: Verify draft release exists (release PR)
         if: contains(github.event.pull_request.labels.*.name, 'release')
         uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
@@ -631,15 +631,10 @@ jobs:
             }
 
 
-  # In-tree e2e, on Talos nodes running as CONTAINERS. The QEMU substrate this
-  # lane used to boot is off the per-PR path only: it is still wired into
-  # e2e-tag.yaml, which runs `make prepare-env` -> `make -C packages/core/testing
-  # prepare-cluster`. That target and hack/e2e-prepare-cluster.bats are
-  # therefore live code, not leftovers, and deleting them takes every
-  # release-candidate run with them. They carry the coverage containers cannot:
-  # DRBD, and with it the `replicated` StorageClass, its Immediate binding mode
-  # and the tenant StorageClass propagation that only applies to
-  # remotely-accessible classes.
+  # In-tree e2e, on Talos nodes running as CONTAINERS, like every other e2e
+  # lane. What containers cannot carry has no lane at all: DRBD, and with it
+  # the `replicated` StorageClass, its Immediate binding mode and the tenant
+  # StorageClass propagation that only applies to remotely-accessible classes.
   e2e:
     name: "E2E Tests"
     # hack/e2e-compose.yaml gives each of the three nodes `cpus: 8` and