From 186547d835ccfd2ea3714a43196c43a6c73ef092 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 2 Sep 2026 12:52:52 +0500 Subject: [PATCH] test(e2e): run the merge-gating lanes on Talos containers The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos containers instead of QEMU guests, so a tenant worker sits at L2 rather than L3. Measured in CI on commits where both substrates ran the same tree: whole-job wall clock down 24-49 minutes, essentially all of it in Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to 466-809s and stop depending on the runner's vgif flag. The node-join soft-red gate goes with the substrate, which is what it was for. It existed because nested virtualisation on the shared runners degraded far enough that a worker registering in minutes elsewhere could miss any deadline this test could afford; #3513 established that the discriminator was the host's kvm_amd vgif flag, and removing a nesting level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from all four workflows, and a missed deadline is an ordinary red again. The ::warning annotation survives, because the reason a run went red should be legible without opening the log, and so does the 124-versus-everything -else test, because "the workers were slow" and "the wait never ran" send a reader to different places. Done by swapping the substrate inside the existing e2e jobs rather than by promoting a separate container job. Those jobs carry more than a substrate -- the GitHub App token, the report-overrun guard, the images list, the SSH breakpoint -- and promoting a job that never had them would have dropped four features without saying so. Three properties of container mode drive the design, and each fails quietly rather than loudly. machine.kernel.modules is a silent no-op -- kernel_module_spec.go returns early on ModeContainer with no error and no event -- so the host loads openvswitch and zfs before the nodes start and the workflow step asserts both rather than trusting them; a missing module surfaces an hour later looking like a CNI or a storage regression. Docker gives a privileged container its own tmpfs /dev seeded once at creation, so /dev/kvm works but ZFS zvols created later never appear and linstor-csi fails ControllerPublishVolume; the compose file binds the host devtmpfs, which is also why talosctl cluster create docker can never carry LINSTOR. And there is no ARP VIP, so 192.168.123.11 stands in for the QEMU lane's .10. The QEMU machinery stays in the tree and keeps running: nightly.yaml and e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated StorageClass, its Immediate binding mode, the tenant StorageClass propagation that only applies to remotely-accessible classes, and the Cozystack Talos node image with its extensions all keep nightly and release-candidate coverage. None of them is exercised per pull request any more. That is the trade. Nothing on the PR path builds or downloads a nocloud disk now, so build-talos drops the talos-nocloud step and the talos-image artifact. build-talos itself stays: finalize consumes its digest fragment. One property this does not fix, recorded beside the runner class because it will be read off a red run eventually: a container node reports the HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters that with kubelet systemReserved. That corrects the scheduler and not what a pod reads from /proc, so a workload sizing itself from /proc/cpuinfo or /proc/meminfo is configured for the whole runner. Also carried here because the same files carry them: the ghcr.io pull-through mirror removal (see #4007 -- whichever lands first makes the other a no-op), the authoritative HelmRelease readiness gate, the backup-credential preflight, the prepull guard against an image-less container fragment, and folding the two standalone OIDC suites into the latest-version tenant cluster. Signed-off-by: Myasnikov Daniil --- .github/workflows/e2e-fork.yaml | 94 +- .github/workflows/e2e-tag.yaml | 5 +- .github/workflows/nightly.yaml | 5 +- .github/workflows/pull-requests.yaml | 209 +-- Makefile | 8 +- .../backups/clickhouse/03-create-bucket.sh | 4 + examples/backups/mariadb/run-all.sh | 4 + examples/backups/mongodb/run-all.sh | 4 + examples/backups/postgres/run-all.sh | 4 + hack/backup-access-preflight.bats | 163 ++ hack/container-lane-capacity_test.bats | 273 +++ .../_lib/backup-access-preflight.sh | 127 ++ hack/e2e-chainsaw/_lib/ghcr-mirror.sh | 713 -------- hack/e2e-chainsaw/_lib/run-kubernetes.sh | 847 ++++++--- .../clickhouse/chainsaw-test.yaml | 8 +- .../foundationdb/chainsaw-test.yaml | 49 +- .../kubernetes-latest/chainsaw-test.yaml | 19 +- .../chainsaw-test.yaml | 117 -- .../kubernetes-oidc-byo.yaml | 48 - .../kubernetes-oidc-system/chainsaw-test.yaml | 123 -- .../kubernetes-oidc-system.yaml | 33 - .../kubernetes-previous/chainsaw-test.yaml | 12 +- hack/e2e-chainsaw/mariadb/chainsaw-test.yaml | 2 +- hack/e2e-chainsaw/mongodb/chainsaw-test.yaml | 2 +- hack/e2e-chainsaw/postgres/chainsaw-test.yaml | 2 +- hack/e2e-chainsaw/vminstance/vmdisk-vmi.yaml | 2 +- hack/e2e-chainsaw/vminstance/vmdisk.yaml | 2 +- hack/e2e-compose.yaml | 143 ++ hack/e2e-container-up.sh | 307 ++++ hack/e2e-ghcr-mirror.yaml | 195 --- hack/e2e-install-cozystack.bats | 174 +- hack/e2e-node-join-soft-red.sh | 146 -- hack/e2e-platform-packages.sh | 83 + hack/e2e-post-install-prep.sh | 135 +- hack/e2e-prepare-cluster-container.bats | 149 ++ hack/e2e-prepull-images.sh | 15 + hack/e2e-talos-image-cache.yaml | 32 +- hack/e2e-wait-helmreleases.sh | 92 + hack/ghcr-mirror_test.bats | 1509 ----------------- hack/helmrelease-readiness.bats | 127 ++ hack/linstor-storage-mode_test.bats | 313 ++++ hack/node-join-soft-red_test.bats | 675 -------- hack/platform-packages_test.bats | 111 ++ hack/run-kubernetes-cpu-throttle_test.bats | 12 +- hack/run-kubernetes-drain_test.bats | 35 +- hack/run-kubernetes-join-timing_test.bats | 88 + hack/run-kubernetes-node-join_test.bats | 142 +- hack/run-kubernetes-oidc_test.bats | 124 ++ hack/run-kubernetes-serial-console_test.bats | 24 +- ...run-kubernetes-talos-diagnostics_test.bats | 2 +- hack/run-kubernetes-talos-spec_test.bats | 82 +- hack/sandbox-cidr-disjoint.bats | 2 +- hack/select-e2e.sh | 2 +- hack/select-e2e_test.bats | 15 +- hack/talos-image-cache_test.bats | 8 + packages/core/testing/Makefile | 108 +- 56 files changed, 3481 insertions(+), 4248 deletions(-) create mode 100644 hack/backup-access-preflight.bats create mode 100644 hack/container-lane-capacity_test.bats create mode 100644 hack/e2e-chainsaw/_lib/backup-access-preflight.sh delete mode 100644 hack/e2e-chainsaw/_lib/ghcr-mirror.sh delete mode 100644 hack/e2e-chainsaw/kubernetes-oidc-customconfig/chainsaw-test.yaml delete mode 100644 hack/e2e-chainsaw/kubernetes-oidc-customconfig/kubernetes-oidc-byo.yaml delete mode 100644 hack/e2e-chainsaw/kubernetes-oidc-system/chainsaw-test.yaml delete mode 100644 hack/e2e-chainsaw/kubernetes-oidc-system/kubernetes-oidc-system.yaml create mode 100644 hack/e2e-compose.yaml create mode 100755 hack/e2e-container-up.sh delete mode 100644 hack/e2e-ghcr-mirror.yaml delete mode 100755 hack/e2e-node-join-soft-red.sh create mode 100755 hack/e2e-platform-packages.sh create mode 100644 hack/e2e-prepare-cluster-container.bats create mode 100644 hack/e2e-wait-helmreleases.sh delete mode 100644 hack/ghcr-mirror_test.bats create mode 100644 hack/helmrelease-readiness.bats create mode 100644 hack/linstor-storage-mode_test.bats delete mode 100644 hack/node-join-soft-red_test.bats create mode 100644 hack/platform-packages_test.bats create mode 100644 hack/run-kubernetes-join-timing_test.bats create mode 100644 hack/run-kubernetes-oidc_test.bats diff --git a/.github/workflows/e2e-fork.yaml b/.github/workflows/e2e-fork.yaml index a21542ed8..620fa4f8b 100644 --- a/.github/workflows/e2e-fork.yaml +++ b/.github/workflows/e2e-fork.yaml @@ -540,17 +540,12 @@ jobs: name: E2E (fork run) needs: [resolve, publish] if: ${{ needs.resolve.outputs.should_run == 'true' }} - # What the gate decided, promoted to a job output because the job's RESULT - # can no longer answer it: a red the lane tolerated finishes `success`, so - # the status this lane reports would otherwise call that run a pass. - outputs: - soft_red: ${{ steps.e2e_tests.outputs.soft_red }} + # Talos nodes as containers, the same substrate pull-requests.yaml uses; the + # per-node 8 CPU / 24 GiB in hack/e2e-compose.yaml is why the class is this + # one. See the `e2e` job there for the full sizing note. runs-on: oracle-vm-32cpu-128gb-x86-64 # Sized above what the tenant-Kubernetes pair can spend on a failing run, and - # held to one figure across every lane that runs the suite. That bound, the - # measured run the rest of the margin comes from, and what neither of them - # covers are all in hack/node-join-soft-red_test.bats rather than restated - # here in four workflows that would then drift apart. + # held to one figure across every lane that runs the suite. timeout-minutes: 215 permissions: contents: read @@ -582,13 +577,6 @@ jobs: - name: Set up toolchain (yq) run: | command -v yq >/dev/null || { sudo curl -sSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq; } - - name: Download Talos image - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 - with: - name: talos-image - path: _out/assets - github-token: ${{ secrets.GITHUB_TOKEN }} - run-id: ${{ needs.resolve.outputs.run_id }} - name: Download pr.patch uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: @@ -606,15 +594,50 @@ jobs: run: | yq -i ".cozystackOperator.platformSourceUrl = \"oci://${REGISTRY}/cozystack-packages\"" packages/core/installer/values.yaml yq -i ".cozystackOperator.platformSourceRef = \"digest=${PACKAGES_DIGEST}\"" packages/core/installer/values.yaml + # Mandatory, and up front: the nodes take their kernel from this host and + # Talos cannot load modules in container mode, so a module missing here is + # missing for the whole run and surfaces later as a CNI or storage failure. + # See the same step in pull-requests.yaml for the full rationale. + - name: Prepare container-lane host + run: | + set -eu + for m in openvswitch zfs; do + if grep -q "^${m} " /proc/modules; then + echo "${m}: already loaded" + continue + fi + if [ "${m}" = zfs ]; then + sudo apt-get update -qq + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux + fi + sudo modprobe "${m}" || echo "${m}: modprobe failed" + done + missing='' + for m in openvswitch zfs; do + grep -q "^${m} " /proc/modules || missing="${missing} ${m}" + done + if [ -n "${missing}" ]; then + echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}" + echo "This is a host problem, not a product failure: openvswitch backs the" >&2 + echo "kube-ovn dataplane and zfs backs every LINSTOR pool, and container-mode" >&2 + echo "Talos cannot load either one itself." >&2 + exit 1 + fi + grep -E '^(openvswitch|zfs) ' /proc/modules - name: Prepare environment run: | attempt=0 - until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do + until make SANDBOX_NAME=$SANDBOX_NAME prepare-env-container; do attempt=$((attempt + 1)) if [ $attempt -ge 3 ]; then echo "Attempt $attempt failed, exiting..."; exit 1; fi echo "Attempt $attempt failed, retrying..." + make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true done - name: Install Cozystack into sandbox + env: + COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443 + COZY_LINSTOR_DRBD_ENABLED: "false" + COZY_E2E_STORAGE_CLASS: local run: | if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then echo "❌ Install failed (no retry — diagnostics below)" @@ -674,6 +697,10 @@ jobs: id: e2e_tests if: ${{ steps.select.outputs.skip != 'true' }} env: + # Must match the install step above: installing on `local` while + # templating the Chainsaw fixtures with `replicated` fails deep inside + # a suite instead of at the mismatch. + COZY_E2E_STORAGE_CLASS: local SELECTED_APPS: ${{ steps.select.outputs.apps }} FULL_E2E: ${{ needs.resolve.outputs.full_e2e }} run: | @@ -699,19 +726,6 @@ jobs: kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1 ' || true echo "::endgroup::" - # One failure the lane does not block on, and only one: the tenant - # node-join deadline. The test itself stays red -- Chainsaw ran its - # whole catch above and the report records the failure -- and this - # decides only whether the job does. It blocks on anything it cannot - # attribute to that deadline, including a report it could not read. - if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then - # Recorded because this exit makes the job succeed, and the status - # this lane reports back to the pull request is derived from that - # result: without this the required check would say the suite - # passed on a run where it failed. - echo "soft_red=true" >>"$GITHUB_OUTPUT" - exit 0 - fi exit 1 fi echo "✅ All selected E2E tests passed" @@ -739,9 +753,15 @@ jobs: name: cozyreport path: _out/cozyreport.tgz if-no-files-found: warn + # Both halves, and in this order: the compose project owns the nodes and + # the zpools outlive it. - name: Tear down sandbox if: always() - run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true + run: | + cleanup_rc=0 + make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$? + make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true + exit "$cleanup_rc" # Flip the pending "E2E Tests" commit status posted by `resolve` to its # terminal state. Runs on failure too, so a failed publish or e2e turns the @@ -769,22 +789,14 @@ jobs: env: PUBLISH_RESULT: ${{ needs.publish.result }} E2E_RESULT: ${{ needs.e2e.result }} - E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }} HEAD_SHA: ${{ needs.resolve.outputs.head_sha }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} with: script: | - const { PUBLISH_RESULT, E2E_RESULT, E2E_SOFT_RED, HEAD_SHA, RUN_URL } = process.env; + const { PUBLISH_RESULT, E2E_RESULT, HEAD_SHA, RUN_URL } = process.env; const { owner, repo } = context.repo; let state, description; - if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') { - // The suite did NOT pass: the tenant node-join deadline expired and - // the lane decided not to block on it. `success` is the decision; - // this text is the only place a reader standing in front of the - // merge button learns what it was a decision about. - state = 'success'; - description = 'fork e2e soft: node-join deadline missed, suite failed, merge not blocked'; - } else if (E2E_RESULT === 'success') { + if (E2E_RESULT === 'success') { state = 'success'; description = 'fork e2e passed (privileged run)'; } else if (PUBLISH_RESULT !== 'success') { diff --git a/.github/workflows/e2e-tag.yaml b/.github/workflows/e2e-tag.yaml index c58dcc93f..0b045388f 100644 --- a/.github/workflows/e2e-tag.yaml +++ b/.github/workflows/e2e-tag.yaml @@ -135,10 +135,7 @@ jobs: # agent while the full platform installs across all three guests. runs-on: oracle-vm-32cpu-128gb-x86-64 # Sized above what the tenant-Kubernetes pair can spend on a failing run, and - # held to one figure across every lane that runs the suite. That bound, the - # measured run the rest of the margin comes from, and what neither of them - # covers are all in hack/node-join-soft-red_test.bats rather than restated - # here in four workflows that would then drift apart. + # held to one figure across every lane that runs the suite. timeout-minutes: 215 permissions: contents: read diff --git a/.github/workflows/nightly.yaml b/.github/workflows/nightly.yaml index fb7cfdb86..1af6590cc 100644 --- a/.github/workflows/nightly.yaml +++ b/.github/workflows/nightly.yaml @@ -228,10 +228,7 @@ jobs: # hack/sandbox-runner-headroom.bats. runs-on: oracle-vm-32cpu-128gb-x86-64 # Sized above what the tenant-Kubernetes pair can spend on a failing run, and - # held to one figure across every lane that runs the suite. That bound, the - # measured run the rest of the margin comes from, and what neither of them - # covers are all in hack/node-join-soft-red_test.bats rather than restated - # here in four workflows that would then drift apart. + # held to one figure across every lane that runs the suite. timeout-minutes: 215 permissions: contents: read diff --git a/.github/workflows/pull-requests.yaml b/.github/workflows/pull-requests.yaml index 55d1e28c9..b982c1f70 100644 --- a/.github/workflows/pull-requests.yaml +++ b/.github/workflows/pull-requests.yaml @@ -502,8 +502,6 @@ jobs: PUBLISH_FLOATING: '0' # Fork PR: export to OCI archive instead of pushing (see build job). OCI_EXPORT_DIR: ${{ github.event.pull_request.head.repo.fork && format('{0}/_out/oci', github.workspace) || '' }} - - name: Build Talos nocloud disk - run: make -C packages/core/talos talos-nocloud - name: Capture Talos patch fragment run: | mkdir -p _out/fragments @@ -523,12 +521,6 @@ jobs: name: pr-oci-talos path: _out/oci/*.oci.tar if-no-files-found: error - - name: Upload Talos image - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 - with: - name: talos-image - path: _out/assets/nocloud-amd64.raw.xz - if-no-files-found: error # Reassemble the unified pr.patch from every fragment, then build the installer # LAST: `make -C packages/core/installer image` runs `flux push artifact @@ -811,7 +803,15 @@ jobs: core.setOutput('disk_id', diskId); - # In-tree e2e for SAME-REPO PRs. Deliberately NOT named "E2E Tests": the + # In-tree e2e for SAME-REPO PRs, on Talos nodes running as CONTAINERS. The + # QEMU substrate this lane used to boot is not wired into any workflow now; + # `make -C packages/core/testing prepare-cluster` and hack/e2e-prepare-cluster.bats + # stay in the tree for the trimmed nightly lane that will carry the coverage + # containers cannot: DRBD, and with it the `replicated` StorageClass, its + # Immediate binding mode and the tenant StorageClass propagation that only + # applies to remotely-accessible classes. + # + # Deliberately NOT named "E2E Tests": the # required "E2E Tests" context is a commit status posted by the `e2e-report` # job (non-fork) or by e2e-fork.yaml (fork). A required *job* that skips # counts as PASSED by branch protection, which is exactly how fork PRs used @@ -821,39 +821,34 @@ jobs: # workflow_run instead. e2e: name: "E2E (in-tree)" - # What the gate decided, promoted to a job output because the job's RESULT - # can no longer answer it: a red the lane tolerated finishes `success`, so - # every consumer downstream that reads the result alone would report this run - # as one where the suite passed. - outputs: - soft_red: ${{ steps.e2e_tests.outputs.soft_red }} - # The sandbox boots three QEMU guests (hack/e2e-prepare-cluster.bats), each - # with `-smp 8 -m 24576`, so the guests alone demand 24 vCPU and 72 GiB. - # On the 24-vCPU runner that is a 1:1 CPU commit with zero host headroom for - # QEMU's own emulation/IO threads, vhost-net, the host iptables MASQUERADE - # every guest image pull traverses, containerd, the kernel and the runner - # agent. During "Install Cozystack" (all platform operators + image pulls - # across three nodes at once) the guest vCPUs contend with those host threads - # and lose cycles to CPU steal, so in-guest kubelet/apiserver readiness and - # liveness probes miss their deadlines ("context deadline exceeded"). A - # HelmRelease then flips InProgress -> Failed and its dependents cascade to - # DependencyNotReady — on a different component each run, the signature of - # resource thrash rather than a product bug. The 32-vCPU / 128 GiB class - # keeps the guests' 24 vCPU / 72 GiB unchanged (no slower install) while - # leaving 8 vCPU (25%) and 56 GiB of host headroom, which removes the - # oversubscription that starves the probes. + # hack/e2e-compose.yaml gives each of the three nodes `cpus: 8` and + # `mem_limit: 24576m`, matching what the QEMU guests were given, so the nodes + # still demand 24 vCPU and 72 GiB between them. The same headroom argument + # therefore still decides the class: on a 24-vCPU runner that is a 1:1 commit + # with nothing left for containerd, the kernel, the host iptables MASQUERADE + # every image pull traverses and the runner agent, and during "Install + # Cozystack" the nodes contend with those and lose cycles. Readiness and + # liveness probes then miss their deadlines, a HelmRelease flips InProgress -> + # Failed and its dependents cascade to DependencyNotReady -- on a different + # component each run, the signature of resource thrash rather than a product + # bug. The 32-vCPU / 128 GiB class leaves 8 vCPU (25%) and 56 GiB of headroom + # and removes that oversubscription. + # + # One thing this class does NOT fix, and it is worth knowing before reading a + # red run: a container node reports the HOST's CPU and memory as its capacity, + # and hack/e2e-container-up.sh counterbalances that with kubelet + # systemReserved so the scheduler sees 8 CPU / 24 GiB. That corrects the + # scheduler and not what a pod reads from /proc, so a workload that sizes + # itself from /proc/cpuinfo or /proc/meminfo is configured for the whole host. runs-on: oracle-vm-32cpu-128gb-x86-64 # Sized above what the tenant-Kubernetes pair can spend on a failing run, and - # held to one figure across every lane that runs the suite. That bound, the - # measured run the rest of the margin comes from, and what neither of them - # covers are all in hack/node-join-soft-red_test.bats rather than restated - # here in four workflows that would then drift apart. + # held to one figure across every lane that runs the suite. timeout-minutes: 215 permissions: contents: read packages: read checks: write - needs: ["finalize", "build-talos", "resolve_assets", "verify-release-candidate"] + needs: ["finalize", "resolve_assets", "verify-release-candidate"] # Regular PRs still enter through finalize. A release-labelled promote PR # enters through resolve_assets only when a maintainer explicitly adds # full-e2e; without that label the immutable rc's already-green suite is not @@ -877,17 +872,6 @@ jobs: ) steps: - - name: Generate GitHub App token - if: contains(github.event.pull_request.labels.*.name, 'release') - id: app-token - uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1 - with: - app-id: ${{ secrets.COZYSTACK_CI_APP_ID }} - private-key: ${{ secrets.COZYSTACK_CI_PRIVATE_KEY }} - owner: cozystack - repositories: ${{ github.event.repository.name }} - permission-contents: read - # ▸ Checkout and prepare the codebase # fetch-depth: 0 is required by the "Select E2E tests" step so that # `git diff origin/${BASE_REF}...HEAD` can resolve the base ref. @@ -896,14 +880,9 @@ jobs: with: fetch-depth: 0 - # ▸ Regular PR path – download the Talos disk from build-talos and pr.patch from finalize - - name: "Download Talos image (regular PR)" - if: "!contains(github.event.pull_request.labels.*.name, 'release')" - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 - with: - name: talos-image - path: _out/assets - + # ▸ Regular PR path – pr.patch from finalize. No Talos disk: the nodes boot + # from ghcr.io/siderolabs/talos directly (hack/e2e-compose.yaml), so this + # lane consumes no nocloud asset and waits on no Talos build. - name: Download PR patch if: "!contains(github.event.pull_request.labels.*.name, 'release')" uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 @@ -916,36 +895,85 @@ jobs: run: | git apply _out/assets/pr.patch - # ▸ Release PR path – fetch artefacts from the corresponding draft release - - name: Download assets from draft release (release PR) - if: contains(github.event.pull_request.labels.*.name, 'release') - run: | - mkdir -p _out/assets - curl -sSL -H "Authorization: token ${APP_TOKEN}" -H "Accept: application/octet-stream" \ - -o _out/assets/nocloud-amd64.raw.xz \ - "https://api.github.com/repos/${GITHUB_REPOSITORY}/releases/assets/${{ needs.resolve_assets.outputs.disk_id }}" - env: - APP_TOKEN: ${{ steps.app-token.outputs.token }} - - name: Set sandbox ID run: echo "SANDBOX_NAME=cozy-e2e-sandbox-$(echo "${GITHUB_REPOSITORY}:${GITHUB_WORKFLOW}:${GITHUB_REF}" | sha256sum | cut -c1-10)" >> $GITHUB_ENV - # ▸ Prepare environment + # ▸ Host kernel modules. MANDATORY and up front, because nothing later can + # supply them: the nodes take their kernel from this host, and Talos + # cannot load modules in container mode -- `machine.kernel.modules` is a + # silent no-op there, `kernel_module_spec.go` returns early on + # ModeContainer with no error and no event. + # + # A module missing here is missing for every node for the whole run, and + # it surfaces in disguise: no `openvswitch` and kube-ovn comes up with no + # dataplane, no `zfs` and every LINSTOR pool creation fails. Both read as + # a CNI or a storage regression an hour later. So this loads both and + # then asserts both, and a failure here names the host rather than the + # product. + - name: Prepare container-lane host + run: | + set -eu + echo "::group::Loading kernel modules required by the container lane" + for m in openvswitch zfs; do + if grep -q "^${m} " /proc/modules; then + echo "${m}: already loaded" + continue + fi + if [ "${m}" = zfs ]; then + sudo apt-get update -qq + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux + fi + echo "${m}: loading" + sudo modprobe "${m}" || echo "${m}: modprobe failed" + done + echo "::endgroup::" + missing='' + for m in openvswitch zfs; do + grep -q "^${m} " /proc/modules || missing="${missing} ${m}" + done + if [ -n "${missing}" ]; then + echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}" + echo "This is a host problem, not a product failure. The e2e nodes run as" >&2 + echo "containers on this runner's kernel and Talos cannot load modules in" >&2 + echo "container mode, so what is not here now is unavailable for the whole" >&2 + echo "run: openvswitch backs the kube-ovn dataplane, zfs backs every" >&2 + echo "LINSTOR pool. Failing here rather than an hour later is the point." >&2 + exit 1 + fi + grep -E '^(openvswitch|zfs) ' /proc/modules + + # ▸ Prepare environment. Two make targets rather than one: the nodes are + # siblings on the runner's docker daemon (the /dev and /run/udev binds in + # hack/e2e-compose.yaml must resolve against the real host devtmpfs, which + # is what lets this lane carry LINSTOR at all), so the host-side half + # needs a docker socket the sandbox does not have. - name: Prepare environment run: | attempt=0 - until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do + until make SANDBOX_NAME=$SANDBOX_NAME prepare-env-container; do attempt=$((attempt + 1)) if [ $attempt -ge 3 ]; then echo "Attempt $attempt failed, exiting..." exit 1 fi echo "Attempt $attempt failed, retrying..." + make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true done echo "Prepare environment completed after $((attempt + 1)) attempts" # ▸ Install Cozystack - name: Install Cozystack into sandbox + env: + # This lane has no ARP VIP: .11 is a plain control-plane node standing + # in for the QEMU lane's 192.168.123.10. + COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443 + # Talos containers share the runner kernel and cannot load DRBD, so the + # harness suppresses DRBD-dependent chart content, registers the host's + # pre-created zpools and keeps LINSTOR's local ZFS path. + COZY_LINSTOR_DRBD_ENABLED: "false" + # Root monitoring is the one install-time workload whose chart selects + # `replicated` explicitly instead of inheriting the default class. + COZY_E2E_STORAGE_CLASS: local run: | if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then echo "❌ Install failed (no retry — see diagnostics below)" @@ -1016,6 +1044,10 @@ jobs: id: e2e_tests if: ${{ steps.select.outputs.skip != 'true' }} env: + # No DRBD-backed class exists here. Chainsaw templates the storage + # fixtures with `local`, and the Kubernetes helper omits only the + # remote-propagation and RWX assertions while running the same suites. + COZY_E2E_STORAGE_CLASS: local SELECTED_APPS: ${{ steps.select.outputs.apps }} FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }} run: | @@ -1048,20 +1080,6 @@ jobs: kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1 ' || true echo "::endgroup::" - # One failure the lane does not block on, and only one: the tenant - # node-join deadline. The test itself stays red -- Chainsaw ran its - # whole catch above and the report records the failure -- and this - # decides only whether the job does. It blocks on anything it cannot - # attribute to that deadline, including a report it could not read. - if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then - # The breakpoint below opens on `failure()`, which this exit is - # about to make false. Record the decision so it can open on this - # red too: the sandbox is still up at that point, and a join that - # never finished is the failure an interactive session is worth - # most on. - echo "soft_red=true" >>"$GITHUB_OUTPUT" - exit 0 - fi exit 1 fi echo "✅ All selected E2E tests passed" @@ -1173,14 +1191,13 @@ jobs: # forks cannot reach the self-hosted rendezvous server. - name: Breakpoint on E2E failure if: | - (failure() || steps.e2e_tests.outputs.soft_red == 'true') && + failure() && vars.BREAKPOINT_ENDPOINT != '' && (contains(github.event.pull_request.labels.*.name, 'debug') || contains(github.event.pull_request.labels.*.name, 'release')) - # A step that fails here fails an otherwise-successful job, which on the - # softened path would let a debugging tool overturn a lane that had - # decided not to block. Same reason `Collect report` above carries it: - # what the job REPORTS must not depend on what a helper did. + # A step that fails here would fail the job on its own account. Same + # reason `Collect report` above carries it: what the job REPORTS must not + # depend on what a helper did. continue-on-error: true # cozystack/breakpoint-action v2-cozy.1 # mode: pause-idle defaults: grace-period=20m, idle-timeout=10m @@ -1208,9 +1225,17 @@ jobs: exits 10 minutes after the last SSH session disconnects. # ▸ Tear down environment (always runs) + # + # Both halves, and in this order: the compose project owns the nodes and + # the zpools outlive it. A leftover pool makes the next run on a reused + # host adopt storage that already carries a dead cluster's volumes. - name: Tear down sandbox if: always() - run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true + run: | + cleanup_rc=0 + make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$? + make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true + exit "$cleanup_rc" # Report the required "E2E Tests" gate as a commit status rather than relying # on a job named "E2E Tests". A required *job* that is skipped counts @@ -1274,7 +1299,6 @@ jobs: PLAN_RESULT: ${{ needs.plan.result }} CODE: ${{ needs.plan.outputs.code }} E2E_RESULT: ${{ needs.e2e.result }} - E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }} CANDIDATE_RESULT: ${{ needs.verify-release-candidate.result }} IS_RELEASE: ${{ contains(github.event.pull_request.labels.*.name, 'release') }} IS_FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }} @@ -1282,7 +1306,7 @@ jobs: RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} with: script: | - const { PLAN_RESULT, CODE, E2E_RESULT, E2E_SOFT_RED, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env; + const { PLAN_RESULT, CODE, E2E_RESULT, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env; let state, description; if (PLAN_RESULT !== 'success') { state = 'failure'; @@ -1303,13 +1327,6 @@ jobs: // full suite re-run or an admin override. state = 'success'; description = 'release promotion — verified at rc, e2e not repeated (add full-e2e to force a run)'; - } else if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') { - // The suite did NOT pass: the tenant node-join deadline expired and - // the lane decided not to block on it. `success` is the decision; - // this text is the only place a reader standing in front of the - // merge button learns what it was a decision about. - state = 'success'; - description = 'in-tree e2e soft: node-join deadline missed, suite failed, merge not blocked'; } else if (E2E_RESULT === 'success') { state = 'success'; description = 'in-tree e2e passed'; diff --git a/Makefile b/Makefile index 7abecad10..003e100d3 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: manifests assets unit-tests helm-unit-tests bats-unit-tests bats-unit-files-check rd-presets-check migrations-target-check test test-controllers preflight +.PHONY: manifests assets prepare-env prepare-env-container unit-tests helm-unit-tests bats-unit-tests bats-unit-files-check rd-presets-check migrations-target-check test test-controllers preflight include hack/common-envs.mk @@ -200,6 +200,12 @@ prepare-env: make -C packages/core/testing apply make -C packages/core/testing prepare-cluster +# Container-lane sibling of prepare-env. Same sandbox, Talos nodes as containers +# instead of QEMU guests, so no nocloud asset is copied in. +prepare-env-container: + make -C packages/core/testing apply + make -C packages/core/testing prepare-cluster-container + generate: hack/update-codegen.sh diff --git a/examples/backups/clickhouse/03-create-bucket.sh b/examples/backups/clickhouse/03-create-bucket.sh index bcdda9dfa..826287e9c 100755 --- a/examples/backups/clickhouse/03-create-bucket.sh +++ b/examples/backups/clickhouse/03-create-bucket.sh @@ -33,6 +33,10 @@ kubectl -n "$NAMESPACE" wait bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_ # (the "-backup" suffix is the BucketAccessClass name); the BucketInfo Secret # carries the same name. kubectl -n "$NAMESPACE" wait bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-backup" --for=jsonpath='{.status.accessGranted}'=true --timeout=180s +if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then + source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh" + cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-backup" 90 +fi log_substep "Reading bucket coordinates from BucketInfo Secret..." TMP=$(mktemp) diff --git a/examples/backups/mariadb/run-all.sh b/examples/backups/mariadb/run-all.sh index 514695596..c50734fb6 100755 --- a/examples/backups/mariadb/run-all.sh +++ b/examples/backups/mariadb/run-all.sh @@ -54,6 +54,10 @@ wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \ '{.status.bucketReady}' true "$NAMESPACE" 300 wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \ '{.status.accessGranted}' true "$NAMESPACE" 300 +if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then + source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh" + cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-${BUCKET_USER}" 90 +fi log_substep "Reading bucket coordinates from BucketInfo Secret..." TMP=$(mktemp) diff --git a/examples/backups/mongodb/run-all.sh b/examples/backups/mongodb/run-all.sh index 444481e4b..264923022 100755 --- a/examples/backups/mongodb/run-all.sh +++ b/examples/backups/mongodb/run-all.sh @@ -49,6 +49,10 @@ wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \ '{.status.bucketReady}' true "$NAMESPACE" 300 wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \ '{.status.accessGranted}' true "$NAMESPACE" 300 +if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then + source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh" + cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-${BUCKET_USER}" 90 +fi log_substep "Reading bucket coordinates from BucketInfo Secret..." TMP=$(mktemp) diff --git a/examples/backups/postgres/run-all.sh b/examples/backups/postgres/run-all.sh index 31bba526e..07356061f 100755 --- a/examples/backups/postgres/run-all.sh +++ b/examples/backups/postgres/run-all.sh @@ -43,6 +43,10 @@ wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \ '{.status.bucketReady}' true "$NAMESPACE" 300 wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-backup" \ '{.status.accessGranted}' true "$NAMESPACE" 300 +if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then + source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh" + cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-backup" 90 +fi log_substep "Reading bucket coordinates from BucketInfo Secret..." TMP=$(mktemp) diff --git a/hack/backup-access-preflight.bats b/hack/backup-access-preflight.bats new file mode 100644 index 000000000..aa75ce9db --- /dev/null +++ b/hack/backup-access-preflight.bats @@ -0,0 +1,163 @@ +#!/usr/bin/env bats +# Unit tests for the E2E-only S3 access gate that runs before expensive backup +# workloads. All cluster and S3 calls are shell stubs. + +@test "preflight helper remains sourceable by the POSIX test runner" { + helper=hack/e2e-chainsaw/_lib/backup-access-preflight.sh + [ "$(sed -n '1p' "$helper")" = '#!/bin/sh' ] + if grep -nE '^[[:space:]]*(set[[:space:]].*pipefail|local([[:space:]]|$))' "$helper"; then + echo "backup preflight is sourced by /bin/sh and must not contain Bash-only shell state" >&2 + exit 1 + fi + sh -n "$helper" +} + +@test "preflight fails closed when the BucketInfo Secret cannot be read" { + . hack/e2e-chainsaw/_lib/backup-access-preflight.sh + out=$(mktemp) + kubectl() { + case "$*" in + *" get secret bucket-demo-backup "*) return 1 ;; + *) echo "preflight continued after the Secret read failed: $*" >&2; return 1 ;; + esac + } + + if cozy_backup_access_preflight tenant-test bucket-demo-backup 0 >"$out" 2>&1; then + echo "failed BucketInfo Secret read passed the S3 preflight" >&2 + exit 1 + fi + grep -q 'failed to read BucketInfo Secret tenant-test/bucket-demo-backup' "$out" + if grep -q 'lacks access key' "$out"; then + echo "preflight continued into credential parsing after the Secret read failed" >&2 + exit 1 + fi +} + +@test "preflight retries IAM convergence then proves PUT GET compare and DELETE" { + . hack/e2e-chainsaw/_lib/backup-access-preflight.sh + attempts=$(mktemp) + uploaded_source=$(mktemp) + calls=$(mktemp) + printf '0\n' > "$attempts" + + kubectl() { + printf 'kubectl %s\n' "$*" >> "$calls" + case "$*" in + *" get secret bucket-demo-backup "*) + printf '%s' '{"spec":{"secretS3":{"accessKeyID":"access","accessSecretKey":"secret"},"bucketName":"bucket-real"}}' | base64 | tr -d '\n' + ;; + *" port-forward "*) + while :; do command sleep 1; done + ;; + *) echo "unexpected kubectl call: $*" >&2; return 1 ;; + esac + } + timeout() { + shift + if [ "$1" = sh ]; then + return 0 + fi + "$@" + } + sleep() { :; } + mc() { + printf 'mc %s\n' "$*" >> "$calls" + case "$1" in + alias) return 0 ;; + cp) + case "$3" in + backup-preflight/*) + command cp "$(sed -n '1p' "$uploaded_source")" "$4" + ;; + *) + count=$(sed -n '1p' "$attempts") + count=$(( count + 1 )) + printf '%s\n' "$count" > "$attempts" + if [ "$count" -eq 1 ]; then + echo 'AccessDenied: IAM identity not loaded yet' >&2 + return 1 + fi + printf '%s\n' "$3" > "$uploaded_source" + ;; + esac + ;; + rm) return 0 ;; + *) echo "unexpected mc call: $*" >&2; return 1 ;; + esac + } + + cozy_backup_access_preflight tenant-test bucket-demo-backup 5 >/dev/null + [ "$(sed -n '1p' "$attempts")" -eq 2 ] || { + echo "preflight did not retry the not-yet-loaded IAM identity" >&2 + cat "$calls" >&2 + exit 1 + } + [ "$(grep -c '^mc rm ' "$calls")" -eq 1 ] || { + echo "preflight did not delete its successfully verified object exactly once" >&2 + cat "$calls" >&2 + exit 1 + } +} + +@test "preflight fails before backup work when granted credentials never reach S3" { + . hack/e2e-chainsaw/_lib/backup-access-preflight.sh + out=$(mktemp) + kubectl() { + case "$*" in + *" get secret bucket-demo-backup "*) + printf '%s' '{"spec":{"secretS3":{"accessKeyID":"access","accessSecretKey":"secret"},"bucketName":"bucket-real"}}' | base64 | tr -d '\n' + ;; + *" port-forward "*) while :; do command sleep 1; done ;; + *) return 1 ;; + esac + } + timeout() { + shift + if [ "$1" = sh ]; then return 0; fi + "$@" + } + mc() { + case "$1" in + alias) return 0 ;; + cp) echo 'AccessDenied' >&2; return 1 ;; + *) return 0 ;; + esac + } + + if cozy_backup_access_preflight tenant-test bucket-demo-backup 0 >"$out" 2>&1; then + echo "unusable credentials passed the S3 preflight" >&2 + exit 1 + fi + grep -q 'granted but did not pass S3 PUT/GET/DELETE within 0s' "$out" + grep -q 's3-preflight: AccessDenied' "$out" +} + +@test "all active database backup flows gate immediately after accessGranted" { + for file in \ + examples/backups/postgres/run-all.sh \ + examples/backups/mariadb/run-all.sh \ + examples/backups/mongodb/run-all.sh \ + examples/backups/clickhouse/03-create-bucket.sh; do + granted=$(grep -n 'accessGranted' "$file" | head -n 1 | cut -d: -f1) + preflight=$(grep -n 'cozy_backup_access_preflight' "$file" | head -n 1 | cut -d: -f1) + coordinates=$(grep -n 'Reading bucket coordinates' "$file" | head -n 1 | cut -d: -f1) + if [ -z "$granted" ] || [ -z "$preflight" ] || [ -z "$coordinates" ] \ + || [ "$granted" -ge "$preflight" ] || [ "$preflight" -ge "$coordinates" ]; then + echo "$file must run the S3 preflight after accessGranted and before consuming bucket coordinates" >&2 + exit 1 + fi + done +} + +@test "all four Chainsaw backup roundtrips enable the E2E preflight" { + for file in \ + hack/e2e-chainsaw/postgres/chainsaw-test.yaml \ + hack/e2e-chainsaw/mariadb/chainsaw-test.yaml \ + hack/e2e-chainsaw/mongodb/chainsaw-test.yaml \ + hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml; do + grep -q 'COZY_E2E_BACKUP_PREFLIGHT=1' "$file" || { + echo "$file does not enable the backup access preflight" >&2 + exit 1 + } + done +} diff --git a/hack/container-lane-capacity_test.bats b/hack/container-lane-capacity_test.bats new file mode 100644 index 000000000..dee997d32 --- /dev/null +++ b/hack/container-lane-capacity_test.bats @@ -0,0 +1,273 @@ +#!/usr/bin/env bats +# Unit coverage for the scheduler-capacity correction required by Talos +# container nodes. The live BATS suite checks the resulting Node status; this +# file pins the arithmetic and the cross-process plumbing without a cluster. + +HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)" +CONTAINER_UP="$HACK_DIR/e2e-container-up.sh" +E2E_CONTAINER_UP_LIB=true . "$CONTAINER_UP" + +@test "container reservations reduce host capacity to one node limit" { + reservations=$(calculate_node_reservations 32 131897116 8 24576 3) + expected=$(printf '%s\n' 24000m 106731292Ki) + + if [ "$reservations" != "$expected" ]; then + echo "unexpected kubelet reservations: $reservations" >&2 + return 1 + fi +} + +@test "container reservations require aggregate host headroom" { + if calculate_node_reservations 24 131897116 8 24576 3 >/dev/null 2>&1; then + echo "three 8-CPU nodes unexpectedly fit on exactly 24 host CPUs" >&2 + return 1 + fi + if calculate_node_reservations 32 75497472 8 24576 3 >/dev/null 2>&1; then + echo "three 24-GiB nodes unexpectedly fit in exactly 72 GiB host memory" >&2 + return 1 + fi +} + +@test "container storage pools match the QEMU lane capacity" { + if ! grep -Fq 'ZPOOL_SIZE="${ZPOOL_SIZE:-200G}"' "$CONTAINER_UP"; then + echo "container ZFS pools do not provide the QEMU lane's 200G per node" >&2 + return 1 + fi +} + +@test "Kubernetes canonical memory units normalize to KiB" { + for fixture in \ + '25165824Ki:25165824' \ + '24476Mi:25063424' \ + '23Gi:24117248'; do + quantity=${fixture%%:*} + expected=${fixture#*:} + actual=$(kubernetes_memory_to_kib "$quantity") + if [ "$actual" != "$expected" ]; then + echo "$quantity normalized to $actual KiB, expected $expected KiB" >&2 + return 1 + fi + done + + for invalid in 24476 24.0Gi garbage; do + if kubernetes_memory_to_kib "$invalid" >/dev/null 2>&1; then + echo "invalid quantity was accepted: $invalid" >&2 + return 1 + fi + done +} + +@test "container capacity values cross host compose sandbox and live assertion" { + make_command=$(make -n -C packages/core/testing SANDBOX_NAME=test prepare-cluster-container) + + for expected in \ + 'COZY_E2E_NODE_CPUS="8" COZY_E2E_NODE_MEMORY_MIB="24576"' \ + '-e COZY_E2E_NODE_CPUS="8" -e COZY_E2E_NODE_MEMORY_MIB="24576"'; do + if ! printf '%s\n' "$make_command" | grep -Fq -- "$expected"; then + echo "testing Makefile does not forward container capacity: $expected" >&2 + return 1 + fi + done + + if ! grep -Fq 'cpus: ${COZY_E2E_NODE_CPUS:-8}' hack/e2e-compose.yaml \ + || ! grep -Fq 'mem_limit: ${COZY_E2E_NODE_MEMORY_MIB:-24576}m' hack/e2e-compose.yaml; then + echo "compose limits are not sourced from the shared capacity values" >&2 + return 1 + fi + if ! grep -Fq 'cpu: "${COZY_E2E_SYSTEM_RESERVED_CPU}"' "$CONTAINER_UP" \ + || ! grep -Fq 'memory: "${COZY_E2E_SYSTEM_RESERVED_MEMORY}"' "$CONTAINER_UP"; then + echo "generated Talos config does not carry the calculated reservations" >&2 + return 1 + fi + if ! grep -Fq '@test "Node allocatable matches the container limits"' hack/e2e-prepare-cluster-container.bats; then + echo "container bring-up does not assert scheduler-visible capacity" >&2 + return 1 + fi +} + +@test "container compose overrides reach both startup and teardown" { + compose_file=/tmp/cozy-custom-compose.yaml + make_command=$(make -n -C packages/core/testing \ + SANDBOX_NAME=test \ + COMPOSE_PROJECT=cozy-custom \ + COMPOSE_FILE="$compose_file" \ + ZPOOL_BACKING_DIR=/tmp/cozy-custom-zpools \ + prepare-cluster-container \ + delete-cluster-container) + + if ! printf '%s\n' "$make_command" | grep -Fq \ + "COMPOSE_PROJECT=\"cozy-custom\" COMPOSE_FILE=\"$compose_file\""; then + echo "container startup does not receive compose overrides" >&2 + return 1 + fi + if ! printf '%s\n' "$make_command" | grep -Fq \ + "docker compose -p \"cozy-custom\" -f \"$compose_file\" down -v"; then + echo "container teardown does not reuse compose overrides" >&2 + return 1 + fi + if ! printf '%s\n' "$make_command" | grep -Fq \ + 'ZPOOL_BACKING_DIR="/tmp/cozy-custom-zpools"'; then + echo "container startup does not receive the backing-directory override" >&2 + return 1 + fi + for expected in \ + 'zpool destroy "data-srv${n}"' \ + 'rm -f "/tmp/cozy-custom-zpools/srv${n}.img"' \ + 'rmdir "/tmp/cozy-custom-zpools"'; do + if ! printf '%s\n' "$make_command" | grep -Fq "$expected"; then + echo "container teardown does not remove its host storage: $expected" >&2 + return 1 + fi + done + cleanup_line=$(printf '%s\n' "$make_command" | grep -n 'zpool destroy' | cut -d: -f1) + startup_line=$(printf '%s\n' "$make_command" | grep -n 'e2e-container-up.sh' | cut -d: -f1) + if [ -z "$cleanup_line" ] || [ -z "$startup_line" ] || [ "$cleanup_line" -ge "$startup_line" ]; then + echo "container storage cleanup does not run before startup" >&2 + return 1 + fi +} + +@test "container teardown reports compose failure after reclaiming host storage" { + tmp=$(mktemp -d) + mkdir -p "$tmp/bin" "$tmp/zpools" + calls="$tmp/calls" + : >"$calls" + for n in 1 2 3; do + : >"$tmp/zpools/srv${n}.img" + done + + printf '%s\n' \ + '#!/bin/sh' \ + 'printf "docker:%s\n" "$*" >>"$CALLS"' \ + 'exit 23' >"$tmp/bin/docker" + printf '%s\n' \ + '#!/bin/sh' \ + 'printf "zpool:%s\n" "$*" >>"$CALLS"' \ + 'exit 0' >"$tmp/bin/zpool" + printf '%s\n' \ + '#!/bin/sh' \ + 'exec "$@"' >"$tmp/bin/sudo" + chmod +x "$tmp/bin/docker" "$tmp/bin/zpool" "$tmp/bin/sudo" + + if PATH="$tmp/bin:$PATH" CALLS="$calls" make -s -C packages/core/testing \ + COMPOSE_PROJECT=cozy-cleanup-test \ + COMPOSE_FILE="$PWD/hack/e2e-compose.yaml" \ + ZPOOL_BACKING_DIR="$tmp/zpools" \ + delete-cluster-container >"$tmp/output" 2>&1; then + echo "container teardown hid the compose failure" >&2 + return 1 + fi + + grep -Fq 'container compose teardown failed with exit 23' "$tmp/output" + for n in 1 2 3; do + grep -Fq "zpool:destroy data-srv${n}" "$calls" + if [ -e "$tmp/zpools/srv${n}.img" ]; then + echo "backing file srv${n}.img survived failed compose teardown" >&2 + return 1 + fi + done + if [ -d "$tmp/zpools" ]; then + echo "empty backing directory survived failed compose teardown" >&2 + return 1 + fi + rm -r "$tmp" +} + +@test "privileged Talos container image is digest pinned" { + image=$(yq '.["x-node"].image' hack/e2e-compose.yaml) + if ! printf '%s\n' "$image" | grep -Eq '^ghcr\.io/siderolabs/talos:v1\.13\.5@sha256:[0-9a-f]{64}$'; then + echo "mutable or unexpected Talos container image: $image" >&2 + return 1 + fi +} + +@test "container startup does not mask stale compose teardown failure" { + if grep -Eq 'docker compose .* down -v.*\|\| true' "$CONTAINER_UP"; then + echo "container startup can proceed after stale compose teardown fails" >&2 + return 1 + fi + grep -Fq 'die "failed to remove a stale ${COMPOSE_PROJECT} compose project before startup"' "$CONTAINER_UP" +} + +@test "container teardown keeps backing file when zpool destroy fails" { + tmp=$(mktemp -d) + mkdir -p "$tmp/bin" "$tmp/zpools" + calls="$tmp/calls" + : >"$calls" + for n in 1 2 3; do + : >"$tmp/zpools/srv${n}.img" + done + + printf '%s\n' '#!/bin/sh' 'exit 0' >"$tmp/bin/docker" + printf '%s\n' \ + '#!/bin/sh' \ + 'printf "zpool:%s\n" "$*" >>"$CALLS"' \ + 'if [ "$1" = destroy ] && [ "$2" = data-srv2 ]; then exit 29; fi' \ + 'exit 0' >"$tmp/bin/zpool" + printf '%s\n' '#!/bin/sh' 'exec "$@"' >"$tmp/bin/sudo" + chmod +x "$tmp/bin/docker" "$tmp/bin/zpool" "$tmp/bin/sudo" + + if PATH="$tmp/bin:$PATH" CALLS="$calls" make -s -C packages/core/testing \ + COMPOSE_PROJECT=cozy-cleanup-test \ + COMPOSE_FILE="$PWD/hack/e2e-compose.yaml" \ + ZPOOL_BACKING_DIR="$tmp/zpools" \ + delete-cluster-container >"$tmp/output" 2>&1; then + echo "container teardown hid the zpool destroy failure" >&2 + return 1 + fi + + grep -Fq 'failed to destroy zpool data-srv2 (exit 29)' "$tmp/output" + grep -Fq 'keeping '"$tmp/zpools/srv2.img"' because data-srv2 is still imported' "$tmp/output" + if [ ! -e "$tmp/zpools/srv2.img" ]; then + echo "backing file was removed from the still-imported data-srv2 pool" >&2 + return 1 + fi + for n in 1 3; do + if [ -e "$tmp/zpools/srv${n}.img" ]; then + echo "cleanup stopped before removing srv${n}.img" >&2 + return 1 + fi + done + rm -r "$tmp" +} + +@test "container teardown keeps backing files when zpool inventory is unreadable" { + tmp=$(mktemp -d) + mkdir -p "$tmp/bin" "$tmp/zpools" + calls="$tmp/calls" + : >"$calls" + for n in 1 2 3; do + : >"$tmp/zpools/srv${n}.img" + done + + printf '%s\n' '#!/bin/sh' 'exit 0' >"$tmp/bin/docker" + printf '%s\n' \ + '#!/bin/sh' \ + 'printf "zpool:%s\n" "$*" >>"$CALLS"' \ + 'if [ "$1" = list ]; then exit 31; fi' \ + 'exit 0' >"$tmp/bin/zpool" + printf '%s\n' '#!/bin/sh' 'exec "$@"' >"$tmp/bin/sudo" + chmod +x "$tmp/bin/docker" "$tmp/bin/zpool" "$tmp/bin/sudo" + + if PATH="$tmp/bin:$PATH" CALLS="$calls" make -s -C packages/core/testing \ + COMPOSE_PROJECT=cozy-cleanup-test \ + COMPOSE_FILE="$PWD/hack/e2e-compose.yaml" \ + ZPOOL_BACKING_DIR="$tmp/zpools" \ + delete-cluster-container >"$tmp/output" 2>&1; then + echo "container teardown treated an unreadable zpool inventory as absence" >&2 + return 1 + fi + + grep -Fq 'failed to inventory zpools after data-srv1 lookup returned exit 31 (inventory exit 31)' "$tmp/output" + if grep -Fq 'zpool:destroy ' "$calls"; then + echo "container teardown destroyed a pool without a readable inventory" >&2 + return 1 + fi + for n in 1 2 3; do + if [ ! -e "$tmp/zpools/srv${n}.img" ]; then + echo "backing file srv${n}.img was removed without proving its pool absent" >&2 + return 1 + fi + done + rm -r "$tmp" +} diff --git a/hack/e2e-chainsaw/_lib/backup-access-preflight.sh b/hack/e2e-chainsaw/_lib/backup-access-preflight.sh new file mode 100644 index 000000000..508020d1c --- /dev/null +++ b/hack/e2e-chainsaw/_lib/backup-access-preflight.sh @@ -0,0 +1,127 @@ +#!/bin/sh +# E2E-only proof that a freshly granted COSI BucketAccess is already usable by +# the S3 server. accessGranted describes the COSI object state; it does not +# prove that the backing server has reloaded the new IAM identity. + +cozy_backup_access_preflight() ( + set -eu + + namespace="$1" + access_name="$2" + timeout_seconds="${3:-90}" + s3_namespace="${COZY_E2E_S3_NAMESPACE:-tenant-root}" + s3_service="${COZY_E2E_S3_SERVICE:-seaweedfs-s3}" + local_port="${COZY_E2E_S3_PREFLIGHT_PORT:-18333}" + workdir='' + pf_pid='' + uploaded=false + access_key='' + secret_key='' + bucket='' + deadline='' + attempt='' + object='' + + case "${timeout_seconds}:${local_port}" in + *[!0-9:]*) + echo "backup access preflight timeout and local port must be integers" >&2 + return 1 + ;; + esac + + if ! workdir=$(mktemp -d); then + echo "could not create backup access preflight work directory" >&2 + return 1 + fi + # Invoked indirectly by the exit trap below. + # shellcheck disable=SC2329 + cleanup() { + rc=$? + trap - 0 + if [ -n "${pf_pid}" ] && kill -0 "${pf_pid}" 2>/dev/null; then + if ! kill "${pf_pid}" 2>/dev/null; then + echo "WARNING: could not stop S3 preflight port-forward process ${pf_pid}" >&2 + fi + # wait reports the SIGTERM used above as a non-zero child status. That is + # the expected port-forward shutdown, not a preflight failure. + if ! wait "${pf_pid}" 2>/dev/null; then + : + fi + fi + if [ "${uploaded}" = true ]; then + if ! timeout 15 mc rm --insecure "backup-preflight/${bucket}/${object}" >/dev/null 2>&1; then + echo "WARNING: could not remove failed S3 preflight object ${bucket}/${object}" >&2 + fi + fi + rm -rf -- "${workdir}" + exit "${rc}" + } + trap cleanup 0 + + # Keep the producer and decoder as separate commands. This helper is sourced + # by cozytest.sh under /bin/sh, where pipefail is unavailable; a pipeline + # would otherwise let a failed Secret read look successful when base64 exits + # zero on empty input. + if ! kubectl -n "${namespace}" get secret "${access_name}" \ + -o jsonpath='{.data.BucketInfo}' > "${workdir}/bucket-info.b64"; then + echo "failed to read BucketInfo Secret ${namespace}/${access_name}" >&2 + return 1 + fi + if ! base64 -d < "${workdir}/bucket-info.b64" > "${workdir}/bucket-info.json"; then + echo "BucketInfo Secret ${namespace}/${access_name} is not valid base64" >&2 + return 1 + fi + access_key=$(jq -r '.spec.secretS3.accessKeyID // ""' "${workdir}/bucket-info.json") + secret_key=$(jq -r '.spec.secretS3.accessSecretKey // ""' "${workdir}/bucket-info.json") + bucket=$(jq -r '.spec.bucketName // ""' "${workdir}/bucket-info.json") + if [ -z "${access_key}" ] || [ -z "${secret_key}" ] || [ -z "${bucket}" ]; then + echo "BucketInfo Secret ${namespace}/${access_name} lacks access key, secret key, or bucket name" >&2 + return 1 + fi + + echo "Checking that BucketAccess ${namespace}/${access_name} is live in S3 before provisioning the backup workload..." + kubectl -n "${s3_namespace}" port-forward \ + "service/${s3_service}" "${local_port}:8333" >"${workdir}/port-forward.log" 2>&1 & + pf_pid=$! + if ! timeout 30 sh -ec "until nc -z 127.0.0.1 ${local_port}; do sleep 1; done"; then + echo "S3 preflight port-forward did not become ready" >&2 + sed 's/^/ port-forward: /' "${workdir}/port-forward.log" >&2 + return 1 + fi + + export MC_CONFIG_DIR="${workdir}/mc" + if ! mc alias set backup-preflight "https://127.0.0.1:${local_port}" \ + "${access_key}" "${secret_key}" --insecure >/dev/null; then + echo "failed to configure S3 client for BucketAccess ${namespace}/${access_name}" >&2 + return 1 + fi + + object=".cozystack-e2e-preflight/${access_name}-$(date +%s)-$$" + printf 'cozystack backup access preflight: %s\n' "${access_name}" > "${workdir}/source" + deadline=$(( $(date +%s) + timeout_seconds )) + attempt=0 + while :; do + attempt=$(( attempt + 1 )) + : > "${workdir}/mc.log" + if timeout 15 mc cp --insecure "${workdir}/source" \ + "backup-preflight/${bucket}/${object}" >"${workdir}/mc.log" 2>&1; then + uploaded=true + if timeout 15 mc cp --insecure "backup-preflight/${bucket}/${object}" \ + "${workdir}/download" >>"${workdir}/mc.log" 2>&1 \ + && cmp "${workdir}/source" "${workdir}/download" \ + && timeout 15 mc rm --insecure "backup-preflight/${bucket}/${object}" \ + >>"${workdir}/mc.log" 2>&1; then + uploaded=false + echo "BucketAccess ${namespace}/${access_name} passed PUT, GET, compare, and DELETE on attempt ${attempt}." + return 0 + fi + fi + + if [ "$(date +%s)" -ge "${deadline}" ]; then + echo "BucketAccess ${namespace}/${access_name} was granted but did not pass S3 PUT/GET/DELETE within ${timeout_seconds}s" >&2 + sed 's/^/ s3-preflight: /' "${workdir}/mc.log" >&2 + return 1 + fi + sleep 5 + done +) diff --git a/hack/e2e-chainsaw/_lib/ghcr-mirror.sh b/hack/e2e-chainsaw/_lib/ghcr-mirror.sh deleted file mode 100644 index d10ffd64f..000000000 --- a/hack/e2e-chainsaw/_lib/ghcr-mirror.sh +++ /dev/null @@ -1,713 +0,0 @@ -# shellcheck shell=bash -# Shared helper: point tenant Kubernetes e2e worker nodes at the in-sandbox -# ghcr.io pull-through registry (hack/e2e-ghcr-mirror.yaml) when it is up, via the -# chart's `talos.registryMirrors` knob, otherwise emit nothing so workers fall back -# to pulling ghcr.io directly. -# -# Why: tenant worker Talos nodes pull `ghcr.io/siderolabs/kubelet` directly; that -# egress is flaky/rate-limited from the CI runner and the pull times out with a TLS -# handshake timeout, so the kubelet service never starts and no tenant node joins. -# Diagnosed in cozystack/cozystack#3548 (in-guest Talos dmesg), tracked by #3513. -# Same flaky-public-egress class the talos-image-cache (#3231) fixed for the OS image. -# One variant of that flake, not the whole of it: #3513 also records failures that -# lose the budget before Talos ever reaches the kubelet pull, and those this cannot -# touch. -# -# WHAT THE GATE BELOW ESTABLISHES, precisely, because it is easy to overstate: the -# Deployment went Available, and the API accepted the egress allow. Neither observes -# the tenant-side datapath. If the allow selects an identity the worker's pull does -# not actually carry (see assumption 2 in hack/e2e-ghcr-mirror.yaml), workers are -# pointed at a ClusterIP whose SYNs are dropped, and Talos burns a TCP connect -# timeout per pull before falling back to ghcr.io -- slower than not mirroring at -# all, on exactly the runs this exists to protect. So: the fallback paths cannot -# make CI worse, the committed path is only as safe as the assumptions listed -# there. -# -# talos-image-cache.sh gates the equivalent decision on a byte-level 206 from a Pod -# carrying the consumer's own label, which is what turns that assumption into a -# measurement. Doing the same here is the hardening follow-up, and it is also what -# would let the suite assert the mirror served the pull rather than the worker -# quietly falling back. - -GHCR_MIRROR_SVC_URL="${GHCR_MIRROR_SVC_URL:-http://ghcr-mirror.kube-system.svc}" -_GHCR_MIRROR_DECISION_FILE="${_GHCR_MIRROR_DECISION_FILE:-/tmp/e2e-ghcr-mirror-endpoint}" -GHCR_MIRROR_MANIFEST="${GHCR_MIRROR_MANIFEST:-hack/e2e-ghcr-mirror.yaml}" -GHCR_WARM_VERSIONS_FILE="${GHCR_WARM_VERSIONS_FILE:-packages/apps/kubernetes/files/versions.yaml}" -GHCR_WARM_JOB_NAME="${GHCR_WARM_JOB_NAME:-ghcr-mirror-warm}" -GHCR_WARM_REPO="${GHCR_WARM_REPO:-siderolabs/kubelet}" -GHCR_WARM_READY_TIMEOUT="${GHCR_WARM_READY_TIMEOUT:-600}" -# How many minors to warm. Two, because the kubernetes-latest and -# kubernetes-previous suites resolve the top two keys and no other suite spins a -# tenant on a third. Raise it if a suite starts selecting deeper; the cost is -# ~61 MiB of upstream fetch per extra tag on the leg this feature exists for -# (one platform's config plus layers, measured for v1.35.6 against ghcr.io). -GHCR_WARM_TAG_COUNT="${GHCR_WARM_TAG_COUNT:-2}" - -# ghcr_warm_tags: print the kubelet image tags worth pre-fetching, one per line, -# newest minor first. -# -# Read from the chart's own version map rather than a list kept here. The suites -# pick their Kubernetes version out of that same file at runtime, so a copy would -# drift silently on the next version bump: the tenant would boot on a tag nothing -# warmed and the first worker would pay the cold upstream fetch this exists to -# remove, with no signal that the warm-up had missed. -# -# Ordered by minor, highest first, which is the order the suites resolve: they pass -# `keys | sort_by(.) | .[-1]` and `.[-2]` to run_kubernetes_test. Sorting here rather -# than reading the file's own order is not a guard against the file being written -# some other way -- packages/apps/kubernetes/hack/update-versions.sh regenerates it -# through `sort -V -r`, so highest-first is what the generator guarantees and a new -# minor lands at the top, not the bottom. It is so that this list and the suites -# agree by construction instead of by both happening to read the same file the same -# way. The caller warms a prefix of this, so the order decides what gets warmed. -ghcr_warm_tags() { - if ! command -v yq >/dev/null 2>&1; then - echo "yq is not installed, so the kubelet version map could not be read" >&2 - return 1 - fi - yq 'to_entries | sort_by(.key) | reverse | .[].value' \ - "$GHCR_WARM_VERSIONS_FILE" -} - -# ghcr_warm_job_manifest ...: print the warm-up Job, or fail. -# -# Pure string builder, kept separate so it can be unit-tested, same reason -# talos-image-cache.sh keeps its probe overrides out of the caller. -# -# The Job asks the mirror for each tag's index, every manifest the index lists, and -# every blob those manifests reference, discarding the bytes. distribution's proxy -# populates its blobstore on the way past, so the tenant worker's later pull is a -# local cache hit instead of a live fetch across the leg that stalls. -# -# "On the way past" is not free, and the cost is worth stating exactly. In -# distribution 2.8 a local miss reads the blob from upstream TWICE: ServeBlob starts -# `go storeLocal(...)`, which copies remote -> local, and separately runs -# copyContent(ctx, dgst, w) to serve the client; both open remoteStore. There is no -# tee. So a warm-up blob GET crosses the throttled leg twice. -# -# What keeps that from being a net addition is warming ONE platform per tag -- the -# one the workers run. Then the Job fetches the same blobs the first worker would -# have fetched, and the 2x is paid once either way: by the Job now, or by the worker -# later inside its node-Ready budget. Walking both platforms broke exactly this: the -# arm64 half is bytes nobody in the sandbox pulls, so it was a true addition to the -# leg, and the claim that the total was unchanged was false while it was there. -# -# The byte figures below are image sizes, not measurements of the mirror's upstream -# side, so read them as a lower bound on install-time egress. -# -# Selecting linux/amd64 out of the index stays a grep because digest and platform -# share a record: normalise the whitespace, split on record boundaries, take the -# record that names the architecture. Pulling every sha256 out of a document is a grep, -# and a digest that turns out not to be a blob answers 404. That is free on the -# manifest leg, where the walk just moves on; on the blob leg a 404 counts as a miss -# and fails the Job, which is what a single-arch tag would do -- its config and layer -# digests are not manifests, so the walk finds no index to descend. -ghcr_warm_job_manifest() { - local image="$1" tag tags="" - shift - [ "$#" -gt 0 ] || return 1 - # An empty endpoint or repo does not fail anything downstream, which is why it has - # to fail here. The Job would still build, run, exit 0, and print "index - # unavailable" for every tag while the cache stayed cold -- and from inside the - # Pod that is indistinguishable from a mirror that is simply unreachable. - [ -n "$GHCR_MIRROR_SVC_URL" ] || return 1 - [ -n "$GHCR_WARM_REPO" ] || return 1 - for tag in "$@"; do - # Refuse anything that is not a bare version. These tags land in a `for tag in - # ...` inside the container's shell, where a substitution would run as the Job. - # The version map is in-tree today, which is why the guard is cheap to add now - # and expensive to retrofit later. No real kubelet tag is rejected by this. - # Character class first, regex second. `grep -q` succeeds when ANY line of its - # input matches and `$` anchors end-of-LINE, so a value whose first line is a - # bare version passes the pattern while everything after the newline is pasted - # into the heredoc as shell. The earlier list of rejected inputs enumerated ways - # to inject -- $(), backticks, semicolon, space -- and missed this one, because - # it was reasoned about as "more than one token" rather than as what the anchor - # treats as a boundary. This case does not depend on that. - case "$tag" in - *[!v0-9.]*) return 1 ;; - esac - printf '%s' "$tag" | grep -qE '^v[0-9]+\.[0-9]+\.[0-9]+$' || return 1 - tags="${tags}${tags:+ }${tag}" - done - cat <&1); then - # Only the API's own NotFound means the mirror was never applied here. A missing - # kubectl, a bad context and an apiserver blip all fail this read too, and the - # captured message is the only place their cause exists -- the same distinction - # resolve_ghcr_mirror_endpoint documents below. - case "$out" in - *NotFound*) - echo "ghcr-mirror not deployed -- nothing to warm" >&2 ;; - *) - echo "WARNING: could not query the ghcr-mirror Deployment; skipping the warm-up. Reason: ${out:-none reported}" >&2 ;; - esac - return 0 - fi - # Warm only the minors a suite can actually select. The suites resolve `.[-1]` and - # `.[-2]` of the sorted keys, ghcr_warm_tags emits that same order, so a prefix of - # it is exactly their set without restating their expression here. - # - # The whole map would be ~300 MiB at the walk this file ships: five tags at ~61 MiB - # each for the single platform the Job fetches. (634 MiB was the measured figure - # while the walk still covered both platforms of every tag; quoting it against - # one-platform code was the same accounting slip the paragraph above pins on the - # platform walk itself.) It crosses the one leg - # this feature exists because of, the one the access log measures at 40-165 KB/s and - # which answers 500 mid-stream. Warming - # tags nothing selects spends that leg on nothing while the install and the - # pre-pull step are still using it. - # No 2>/dev/null. Command substitution captures stdout only, so ghcr_warm_tags' - # complaint about a missing yq cannot reach $tags in the first place -- redirecting - # it merely deleted the one line that names the cause, leaving the caller to blame - # versions.yaml for a tool that is not installed. - tags=$(ghcr_warm_tags | head -n "$GHCR_WARM_TAG_COUNT") || tags="" - if [ -z "$tags" ]; then - echo "WARNING: could not read kubelet tags from ${GHCR_WARM_VERSIONS_FILE}; skipping the warm-up" >&2 - return 0 - fi - # Reuse the mirror's own image so there is one place to bump. It is NOT already - # cached on the nodes: hack/e2e-ghcr-mirror.yaml keeps it out of the pre-pull set - # on purpose, and the pre-pull step runs after this one anyway. The Job's own wait - # for the mirror to serve is what absorbs that. - if ! image=$(kubectl -n kube-system get deploy ghcr-mirror \ - -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null); then - image="" - fi - if [ -z "$image" ]; then - echo "WARNING: could not read the ghcr-mirror image; skipping the warm-up" >&2 - return 0 - fi - # The image is the only builder input that arrives from the cluster, and it is - # interpolated into the Job YAML. Writing it needs Deployment write access, so this - # is not a trust boundary. The point is narrower: the endpoint and the repository - # are checked and the tags are pattern-matched, so leaving the one input that comes - # from outside unchecked reads as an oversight. Other locally-set values - # (GHCR_WARM_JOB_NAME, GHCR_WARM_READY_TIMEOUT) also reach the script unquoted and - # are equally unchecked -- they are set in this file and nowhere else. - # Reference characters only. - case "$image" in - *[!A-Za-z0-9./:@_-]*) - echo "WARNING: the ghcr-mirror image reads as [$image], which is not an image reference; skipping the warm-up" >&2 - return 0 ;; - esac - # Capture the manifest with stderr NOT folded in. `2>&1` here would put anything - # written to stderr inside the YAML, and under a runner that traces the body (the - # bats runner does) that is the trace itself: the Job then fails to parse for a - # reason that has nothing to do with the Job. The builder communicates through its - # exit status, so there is nothing on stderr worth keeping. - # - # shellcheck disable=SC2086 # tags is a builder-validated whitespace-separated list - if ! manifest=$(ghcr_warm_job_manifest "$image" $tags 2>/dev/null); then - echo "WARNING: refused to build the warm-up Job (a tag is not a plain version); skipping" >&2 - return 0 - fi - if ! out=$(printf '%s\n' "$manifest" | kubectl apply -f - 2>&1); then - echo "WARNING: could not start the ghcr-mirror warm-up; tenant workers may pay a cold fetch. Reason: ${out:-none reported}" >&2 - return 0 - fi - echo "ghcr-mirror warm-up started for: $(printf '%s' "$tags" | tr '\n' ' ')" >&2 - return 0 -} - -# ghcr_registry_mirrors_block: pure string builder. Given a mirror endpoint URL -# ($1, may be empty), print the `registryMirrors:` sub-block for a tenant Kubernetes -# CR `spec.talos` (4-space indented, i.e. nested under ` talos:`), or nothing when -# the endpoint is empty. Kept separate so it can be unit-tested: an indentation or -# quoting slip here would emit an invalid CR and every kubernetes-* test would fail. -ghcr_registry_mirrors_block() { - local endpoint="$1" - [ -n "$endpoint" ] || return 0 - printf ' registryMirrors:\n ghcr.io:\n endpoints:\n - %s\n' "$endpoint" -} - -# _apply_ghcr_mirror_egress_policy: install the CiliumClusterwideNetworkPolicy that -# lets tenant worker VM (virt-launcher) Pods egress to the mirror. It ships in the -# manifest but is applied here, after Cilium's CRDs exist. Idempotent, best-effort. -# -# On failure it PRINTS the reason to stdout for the caller to surface. The causes -# are not all alike -- a missing yq, an apiVersion the cluster does not serve, an -# RBAC denial are permanent, while an apiserver blip is not -- and the exit code -# cannot tell them apart. Discarding the message would leave the caller announcing -# a cause it does not know, in the subsystem this exists to make legible. -_apply_ghcr_mirror_egress_policy() { - if ! command -v yq >/dev/null 2>&1; then - echo "yq is not installed, so the policy could not be extracted from ${GHCR_MIRROR_MANIFEST}" - return 1 - fi - # Braces, not a bare `2>&1 >/dev/null`: the goal is to keep stderr (the reason) - # on stdout for the caller and drop kubectl's success chatter, and the braced - # form says that unambiguously. - { yq 'select(.kind == "CiliumClusterwideNetworkPolicy")' "$GHCR_MIRROR_MANIFEST" \ - | kubectl apply -f - >/dev/null; } 2>&1 -} - -# resolve_ghcr_mirror_endpoint: print the mirror endpoint URL to use, or an empty -# string to signal "pull ghcr.io directly". Resolved once and cached in a /tmp file -# that persists across bats files (same sandbox container), so only the first tenant -# test pays the readiness wait. -resolve_ghcr_mirror_endpoint() { - if [ -f "$_GHCR_MIRROR_DECISION_FILE" ]; then - cat "$_GHCR_MIRROR_DECISION_FILE" - return 0 - fi - local endpoint="" cache=1 out rc avail ready apply_err - # One `get deploy` serves both the existence check and the not-deployed-vs-error - # split: a genuine NotFound is a stable decision worth caching, but any other - # failure (transient apiserver blip) must NOT be cached, or one blip would - # disable the mirror for every later test in the shared sandbox. - # - # Capture inside `if` (not a bare `out=$(...); rc=$?`): the chainsaw suites run - # this script under `sh -c` with `set -eu`, and dash inherits errexit into a - # command substitution, so a non-zero kubectl in a bare assignment would kill - # the script before either fallback branch runs. bash does not inherit it, - # which is why a bare assignment survives locally but not in CI. - if out=$(kubectl -n kube-system get deploy ghcr-mirror 2>&1); then - rc=0 - else - rc=$? - fi - if [ "$rc" -ne 0 ]; then - case "$out" in - # The API's own NotFound only. A bare "not found" also matches a shell - # reporting a missing kubectl and a bad --context, neither of which says - # anything about whether the mirror was applied -- and both would then be - # cached as the stable answer for every later suite in the sandbox. - *NotFound*) - echo "ghcr-mirror not deployed -- tenant workers pull ghcr.io directly" >&2 ;; - *) - echo "WARNING: could not query ghcr-mirror Deployment (transient); not caching, will retry" >&2 - cache=0 ;; - esac - else - # Readiness is decided once, then acted on once. The watch is the cheap way to - # learn it, but `rollout status` also exits non-zero on a broken or aborted - # watch, and its output goes to /dev/null -- so when it fails, ask the API what - # the Deployment's own condition says instead of reading the watch's exit code - # as the answer. - # - # 2m, not longer: the wait is serial with the rest of the 67m e2e step, whose - # budget inventory has to stay above the sum of its legs, and spending more here - # eats the margin the failure-path diagnostics need in order to be collected at - # all. It is already generous -- the Deployment pulls one small digest-pinned - # image and was applied at install time, tens of minutes earlier. - if kubectl -n kube-system rollout status deploy/ghcr-mirror --timeout=2m >/dev/null 2>&1; then - ready=yes - else - if avail=$(kubectl -n kube-system get deploy ghcr-mirror \ - -o jsonpath='{.status.conditions[?(@.type=="Available")].status}' 2>/dev/null); then - : - else - avail="" - fi - case "$avail" in - True) ready=yes ;; - False) ready=no ;; - *) ready=unknown ;; - esac - fi - case "$ready" in - yes) - # Only commit the endpoint once the tenant egress allow is actually in - # place. The mirror is a kube-system ClusterIP that the tenant default-deny - # egress drops without this allow; pointing workers at it when the policy - # did not apply just adds dial-timeout latency before Talos falls back. - # Leaving the endpoint empty is what keeps that cost off the run -- it - # removes the one failure mode this function can see, not every one of them - # (see the header on what the gate does and does not establish). - if apply_err=$(_apply_ghcr_mirror_egress_policy); then - endpoint="$GHCR_MIRROR_SVC_URL" - echo "ghcr-mirror ready and egress allow applied -- tenant workers mirror ghcr.io via ${endpoint}" >&2 - else - # Not cached, for the same reason as the failed query above: a failed - # apply says nothing about whether the next attempt would succeed. The - # reason is printed rather than characterised -- some causes here are - # permanent and retrying will not help, and only the message says which. - echo "WARNING: ghcr-mirror ready but its egress allow failed to apply; not caching, will retry. Reason: ${apply_err:-none reported}" >&2 - cache=0 - fi - ;; - no) - # Cached, and this is a trade rather than an invariant: a Deployment can - # still go Available later on its own -- a kubelet image-pull retry or a - # Pending Pod finally scheduling would both do it, with nothing re-applying - # the manifest. What caching buys is not re-paying the wait in every later - # suite, serial with the rest of the step; what it costs is missing a mirror - # that recovers mid-run. For a best-effort accelerator that trade is worth - # taking, and re-reading the condition cheaply on a cached miss would take - # it back if it ever bites. - echo "WARNING: ghcr-mirror not Available -- tenant workers pull ghcr.io directly" >&2 - ;; - *) - echo "WARNING: could not confirm ghcr-mirror readiness (transient); not caching, will retry" >&2 - cache=0 - ;; - esac - fi - # Cache atomically (temp + mv) so a concurrent reader in the shared sandbox never - # sees a half-written decision file. - if [ "$cache" -eq 1 ]; then - printf '%s' "$endpoint" > "${_GHCR_MIRROR_DECISION_FILE}.tmp" \ - && mv "${_GHCR_MIRROR_DECISION_FILE}.tmp" "$_GHCR_MIRROR_DECISION_FILE" - fi - printf '%s' "$endpoint" -} - -# _ghcr_mirror_bounded_read