admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-10-23 03:27:50 +09:00
Code Issues Packages Projects Releases Wiki Activity

test(e2e): run the merge-gating lanes on Talos containers

Browse Source 
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This commit is contained in:
Myasnikov Daniil committed 2026-09-05 13:19:17 +05:00
1 parent e49633208c
commit 186547d835
56 files changed
+3481 -4248
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/e2e-fork.yaml
+53 -41
View File
@@ -540,17 +540,12 @@ jobs:
name: E2E (fork run)
needs: [resolve, publish]
if: ${{ needs.resolve.outputs.should_run == 'true' }}
# What the gate decided, promoted to a job output because the job's RESULT
# can no longer answer it: a red the lane tolerated finishes `success`, so
# the status this lane reports would otherwise call that run a pass.
outputs:
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
# Talos nodes as containers, the same substrate pull-requests.yaml uses; the
# per-node 8 CPU / 24 GiB in hack/e2e-compose.yaml is why the class is this
# one. See the `e2e` job there for the full sizing note.
runs-on: oracle-vm-32cpu-128gb-x86-64
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
# held to one figure across every lane that runs the suite. That bound, the
# measured run the rest of the margin comes from, and what neither of them
# covers are all in hack/node-join-soft-red_test.bats rather than restated
# here in four workflows that would then drift apart.
# held to one figure across every lane that runs the suite.
timeout-minutes: 215
permissions:
contents: read
@@ -582,13 +577,6 @@ jobs:
- name: Set up toolchain (yq)
run: |
command -v yq >/dev/null || { sudo curl -sSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq; }
- name: Download Talos image
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: talos-image
path: _out/assets
github-token: ${{ secrets.GITHUB_TOKEN }}
run-id: ${{ needs.resolve.outputs.run_id }}
- name: Download pr.patch
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
@@ -606,15 +594,50 @@ jobs:
run: |
yq -i ".cozystackOperator.platformSourceUrl = \"oci://${REGISTRY}/cozystack-packages\"" packages/core/installer/values.yaml
yq -i ".cozystackOperator.platformSourceRef = \"digest=${PACKAGES_DIGEST}\"" packages/core/installer/values.yaml
# Mandatory, and up front: the nodes take their kernel from this host and
# Talos cannot load modules in container mode, so a module missing here is
# missing for the whole run and surfaces later as a CNI or storage failure.
# See the same step in pull-requests.yaml for the full rationale.
- name: Prepare container-lane host
run: |
set -eu
for m in openvswitch zfs; do
if grep -q "^${m} " /proc/modules; then
echo "${m}: already loaded"
continue
fi
if [ "${m}" = zfs ]; then
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
fi
sudo modprobe "${m}" || echo "${m}: modprobe failed"
done
missing=''
for m in openvswitch zfs; do
grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
done
if [ -n "${missing}" ]; then
echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
echo "This is a host problem, not a product failure: openvswitch backs the" >&2
echo "kube-ovn dataplane and zfs backs every LINSTOR pool, and container-mode" >&2
echo "Talos cannot load either one itself." >&2
exit 1
fi
grep -E '^(openvswitch|zfs) ' /proc/modules
- name: Prepare environment
run: |
attempt=0
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env-container; do
attempt=$((attempt + 1))
if [ $attempt -ge 3 ]; then echo "Attempt $attempt failed, exiting..."; exit 1; fi
echo "Attempt $attempt failed, retrying..."
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true
done
- name: Install Cozystack into sandbox
env:
COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
COZY_LINSTOR_DRBD_ENABLED: "false"
COZY_E2E_STORAGE_CLASS: local
run: |
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
echo "❌ Install failed (no retry — diagnostics below)"
@@ -674,6 +697,10 @@ jobs:
id: e2e_tests
if: ${{ steps.select.outputs.skip != 'true' }}
env:
# Must match the install step above: installing on `local` while
# templating the Chainsaw fixtures with `replicated` fails deep inside
# a suite instead of at the mismatch.
COZY_E2E_STORAGE_CLASS: local
SELECTED_APPS: ${{ steps.select.outputs.apps }}
FULL_E2E: ${{ needs.resolve.outputs.full_e2e }}
run: |
@@ -699,19 +726,6 @@ jobs:
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
' || true
echo "::endgroup::"
# One failure the lane does not block on, and only one: the tenant
# node-join deadline. The test itself stays red -- Chainsaw ran its
# whole catch above and the report records the failure -- and this
# decides only whether the job does. It blocks on anything it cannot
# attribute to that deadline, including a report it could not read.
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
# Recorded because this exit makes the job succeed, and the status
# this lane reports back to the pull request is derived from that
# result: without this the required check would say the suite
# passed on a run where it failed.
echo "soft_red=true" >>"$GITHUB_OUTPUT"
exit 0
fi
exit 1
fi
echo "✅ All selected E2E tests passed"
@@ -739,9 +753,15 @@ jobs:
name: cozyreport
path: _out/cozyreport.tgz
if-no-files-found: warn
# Both halves, and in this order: the compose project owns the nodes and
# the zpools outlive it.
- name: Tear down sandbox
if: always()
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
run: |
cleanup_rc=0
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$?
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
exit "$cleanup_rc"
# Flip the pending "E2E Tests" commit status posted by `resolve` to its
# terminal state. Runs on failure too, so a failed publish or e2e turns the
@@ -769,22 +789,14 @@ jobs:
env:
PUBLISH_RESULT: ${{ needs.publish.result }}
E2E_RESULT: ${{ needs.e2e.result }}
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
with:
script: |
const { PUBLISH_RESULT, E2E_RESULT, E2E_SOFT_RED, HEAD_SHA, RUN_URL } = process.env;
const { PUBLISH_RESULT, E2E_RESULT, HEAD_SHA, RUN_URL } = process.env;
const { owner, repo } = context.repo;
let state, description;
if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
// The suite did NOT pass: the tenant node-join deadline expired and
// the lane decided not to block on it. `success` is the decision;
// this text is the only place a reader standing in front of the
// merge button learns what it was a decision about.
state = 'success';
description = 'fork e2e soft: node-join deadline missed, suite failed, merge not blocked';
} else if (E2E_RESULT === 'success') {
if (E2E_RESULT === 'success') {
state = 'success';
description = 'fork e2e passed (privileged run)';
} else if (PUBLISH_RESULT !== 'success') {
.github/workflows/e2e-tag.yaml
+1 -4
View File
@@ -135,10 +135,7 @@ jobs:
# agent while the full platform installs across all three guests.
runs-on: oracle-vm-32cpu-128gb-x86-64
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
# held to one figure across every lane that runs the suite. That bound, the
# measured run the rest of the margin comes from, and what neither of them
# covers are all in hack/node-join-soft-red_test.bats rather than restated
# here in four workflows that would then drift apart.
# held to one figure across every lane that runs the suite.
timeout-minutes: 215
permissions:
contents: read
.github/workflows/nightly.yaml
+1 -4
View File
@@ -228,10 +228,7 @@ jobs:
# hack/sandbox-runner-headroom.bats.
runs-on: oracle-vm-32cpu-128gb-x86-64
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
# held to one figure across every lane that runs the suite. That bound, the
# measured run the rest of the margin comes from, and what neither of them
# covers are all in hack/node-join-soft-red_test.bats rather than restated
# here in four workflows that would then drift apart.
# held to one figure across every lane that runs the suite.
timeout-minutes: 215
permissions:
contents: read
.github/workflows/pull-requests.yaml
+113 -96
View File
@@ -502,8 +502,6 @@ jobs:
PUBLISH_FLOATING: '0'
# Fork PR: export to OCI archive instead of pushing (see build job).
OCI_EXPORT_DIR: ${{ github.event.pull_request.head.repo.fork && format('{0}/_out/oci', github.workspace) || '' }}
- name: Build Talos nocloud disk
run: make -C packages/core/talos talos-nocloud
- name: Capture Talos patch fragment
run: |
mkdir -p _out/fragments
@@ -523,12 +521,6 @@ jobs:
name: pr-oci-talos
path: _out/oci/*.oci.tar
if-no-files-found: error
- name: Upload Talos image
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: talos-image
path: _out/assets/nocloud-amd64.raw.xz
if-no-files-found: error
# Reassemble the unified pr.patch from every fragment, then build the installer
# LAST: `make -C packages/core/installer image` runs `flux push artifact
@@ -811,7 +803,15 @@ jobs:
core.setOutput('disk_id', diskId);
# In-tree e2e for SAME-REPO PRs. Deliberately NOT named "E2E Tests": the
# In-tree e2e for SAME-REPO PRs, on Talos nodes running as CONTAINERS. The
# QEMU substrate this lane used to boot is not wired into any workflow now;
# `make -C packages/core/testing prepare-cluster` and hack/e2e-prepare-cluster.bats
# stay in the tree for the trimmed nightly lane that will carry the coverage
# containers cannot: DRBD, and with it the `replicated` StorageClass, its
# Immediate binding mode and the tenant StorageClass propagation that only
# applies to remotely-accessible classes.
#
# Deliberately NOT named "E2E Tests": the
# required "E2E Tests" context is a commit status posted by the `e2e-report`
# job (non-fork) or by e2e-fork.yaml (fork). A required *job* that skips
# counts as PASSED by branch protection, which is exactly how fork PRs used
@@ -821,39 +821,34 @@ jobs:
# workflow_run instead.
e2e:
name: "E2E (in-tree)"
# What the gate decided, promoted to a job output because the job's RESULT
# can no longer answer it: a red the lane tolerated finishes `success`, so
# every consumer downstream that reads the result alone would report this run
# as one where the suite passed.
outputs:
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
# The sandbox boots three QEMU guests (hack/e2e-prepare-cluster.bats), each
# with `-smp 8 -m 24576`, so the guests alone demand 24 vCPU and 72 GiB.
# On the 24-vCPU runner that is a 1:1 CPU commit with zero host headroom for
# QEMU's own emulation/IO threads, vhost-net, the host iptables MASQUERADE
# every guest image pull traverses, containerd, the kernel and the runner
# agent. During "Install Cozystack" (all platform operators + image pulls
# across three nodes at once) the guest vCPUs contend with those host threads
# and lose cycles to CPU steal, so in-guest kubelet/apiserver readiness and
# liveness probes miss their deadlines ("context deadline exceeded"). A
# HelmRelease then flips InProgress -> Failed and its dependents cascade to
# DependencyNotReady — on a different component each run, the signature of
# resource thrash rather than a product bug. The 32-vCPU / 128 GiB class
# keeps the guests' 24 vCPU / 72 GiB unchanged (no slower install) while
# leaving 8 vCPU (25%) and 56 GiB of host headroom, which removes the
# oversubscription that starves the probes.
# hack/e2e-compose.yaml gives each of the three nodes `cpus: 8` and
# `mem_limit: 24576m`, matching what the QEMU guests were given, so the nodes
# still demand 24 vCPU and 72 GiB between them. The same headroom argument
# therefore still decides the class: on a 24-vCPU runner that is a 1:1 commit
# with nothing left for containerd, the kernel, the host iptables MASQUERADE
# every image pull traverses and the runner agent, and during "Install
# Cozystack" the nodes contend with those and lose cycles. Readiness and
# liveness probes then miss their deadlines, a HelmRelease flips InProgress ->
# Failed and its dependents cascade to DependencyNotReady -- on a different
# component each run, the signature of resource thrash rather than a product
# bug. The 32-vCPU / 128 GiB class leaves 8 vCPU (25%) and 56 GiB of headroom
# and removes that oversubscription.
#
# One thing this class does NOT fix, and it is worth knowing before reading a
# red run: a container node reports the HOST's CPU and memory as its capacity,
# and hack/e2e-container-up.sh counterbalances that with kubelet
# systemReserved so the scheduler sees 8 CPU / 24 GiB. That corrects the
# scheduler and not what a pod reads from /proc, so a workload that sizes
# itself from /proc/cpuinfo or /proc/meminfo is configured for the whole host.
runs-on: oracle-vm-32cpu-128gb-x86-64
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
# held to one figure across every lane that runs the suite. That bound, the
# measured run the rest of the margin comes from, and what neither of them
# covers are all in hack/node-join-soft-red_test.bats rather than restated
# here in four workflows that would then drift apart.
# held to one figure across every lane that runs the suite.
timeout-minutes: 215
permissions:
contents: read
packages: read
checks: write
needs: ["finalize", "build-talos", "resolve_assets", "verify-release-candidate"]
needs: ["finalize", "resolve_assets", "verify-release-candidate"]
# Regular PRs still enter through finalize. A release-labelled promote PR
# enters through resolve_assets only when a maintainer explicitly adds
# full-e2e; without that label the immutable rc's already-green suite is not
@@ -877,17 +872,6 @@ jobs:
)
steps:
- name: Generate GitHub App token
if: contains(github.event.pull_request.labels.*.name, 'release')
id: app-token
uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1
with:
app-id: ${{ secrets.COZYSTACK_CI_APP_ID }}
private-key: ${{ secrets.COZYSTACK_CI_PRIVATE_KEY }}
owner: cozystack
repositories: ${{ github.event.repository.name }}
permission-contents: read
# ▸ Checkout and prepare the codebase
# fetch-depth: 0 is required by the "Select E2E tests" step so that
# `git diff origin/${BASE_REF}...HEAD` can resolve the base ref.
@@ -896,14 +880,9 @@ jobs:
with:
fetch-depth: 0
# ▸ Regular PR path – download the Talos disk from build-talos and pr.patch from finalize
- name: "Download Talos image (regular PR)"
if: "!contains(github.event.pull_request.labels.*.name, 'release')"
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: talos-image
path: _out/assets
# ▸ Regular PR path – pr.patch from finalize. No Talos disk: the nodes boot
# from ghcr.io/siderolabs/talos directly (hack/e2e-compose.yaml), so this
# lane consumes no nocloud asset and waits on no Talos build.
- name: Download PR patch
if: "!contains(github.event.pull_request.labels.*.name, 'release')"
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
@@ -916,36 +895,85 @@ jobs:
run: |
git apply _out/assets/pr.patch
# ▸ Release PR path – fetch artefacts from the corresponding draft release
- name: Download assets from draft release (release PR)
if: contains(github.event.pull_request.labels.*.name, 'release')
run: |
mkdir -p _out/assets
curl -sSL -H "Authorization: token ${APP_TOKEN}" -H "Accept: application/octet-stream" \
-o _out/assets/nocloud-amd64.raw.xz \
"https://api.github.com/repos/${GITHUB_REPOSITORY}/releases/assets/${{ needs.resolve_assets.outputs.disk_id }}"
env:
APP_TOKEN: ${{ steps.app-token.outputs.token }}
- name: Set sandbox ID
run: echo "SANDBOX_NAME=cozy-e2e-sandbox-$(echo "${GITHUB_REPOSITORY}:${GITHUB_WORKFLOW}:${GITHUB_REF}" | sha256sum | cut -c1-10)" >> $GITHUB_ENV
# ▸ Prepare environment
# ▸ Host kernel modules. MANDATORY and up front, because nothing later can
# supply them: the nodes take their kernel from this host, and Talos
# cannot load modules in container mode -- `machine.kernel.modules` is a
# silent no-op there, `kernel_module_spec.go` returns early on
# ModeContainer with no error and no event.
#
# A module missing here is missing for every node for the whole run, and
# it surfaces in disguise: no `openvswitch` and kube-ovn comes up with no
# dataplane, no `zfs` and every LINSTOR pool creation fails. Both read as
# a CNI or a storage regression an hour later. So this loads both and
# then asserts both, and a failure here names the host rather than the
# product.
- name: Prepare container-lane host
run: |
set -eu
echo "::group::Loading kernel modules required by the container lane"
for m in openvswitch zfs; do
if grep -q "^${m} " /proc/modules; then
echo "${m}: already loaded"
continue
fi
if [ "${m}" = zfs ]; then
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
fi
echo "${m}: loading"
sudo modprobe "${m}" || echo "${m}: modprobe failed"
done
echo "::endgroup::"
missing=''
for m in openvswitch zfs; do
grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
done
if [ -n "${missing}" ]; then
echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
echo "This is a host problem, not a product failure. The e2e nodes run as" >&2
echo "containers on this runner's kernel and Talos cannot load modules in" >&2
echo "container mode, so what is not here now is unavailable for the whole" >&2
echo "run: openvswitch backs the kube-ovn dataplane, zfs backs every" >&2
echo "LINSTOR pool. Failing here rather than an hour later is the point." >&2
exit 1
fi
grep -E '^(openvswitch|zfs) ' /proc/modules
# ▸ Prepare environment. Two make targets rather than one: the nodes are
# siblings on the runner's docker daemon (the /dev and /run/udev binds in
# hack/e2e-compose.yaml must resolve against the real host devtmpfs, which
# is what lets this lane carry LINSTOR at all), so the host-side half
# needs a docker socket the sandbox does not have.
- name: Prepare environment
run: |
attempt=0
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env-container; do
attempt=$((attempt + 1))
if [ $attempt -ge 3 ]; then
echo "Attempt $attempt failed, exiting..."
exit 1
fi
echo "Attempt $attempt failed, retrying..."
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true
done
echo "Prepare environment completed after $((attempt + 1)) attempts"
# ▸ Install Cozystack
- name: Install Cozystack into sandbox
env:
# This lane has no ARP VIP: .11 is a plain control-plane node standing
# in for the QEMU lane's 192.168.123.10.
COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
# Talos containers share the runner kernel and cannot load DRBD, so the
# harness suppresses DRBD-dependent chart content, registers the host's
# pre-created zpools and keeps LINSTOR's local ZFS path.
COZY_LINSTOR_DRBD_ENABLED: "false"
# Root monitoring is the one install-time workload whose chart selects
# `replicated` explicitly instead of inheriting the default class.
COZY_E2E_STORAGE_CLASS: local
run: |
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
echo "❌ Install failed (no retry — see diagnostics below)"
@@ -1016,6 +1044,10 @@ jobs:
id: e2e_tests
if: ${{ steps.select.outputs.skip != 'true' }}
env:
# No DRBD-backed class exists here. Chainsaw templates the storage
# fixtures with `local`, and the Kubernetes helper omits only the
# remote-propagation and RWX assertions while running the same suites.
COZY_E2E_STORAGE_CLASS: local
SELECTED_APPS: ${{ steps.select.outputs.apps }}
FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
run: |
@@ -1048,20 +1080,6 @@ jobs:
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
' || true
echo "::endgroup::"
# One failure the lane does not block on, and only one: the tenant
# node-join deadline. The test itself stays red -- Chainsaw ran its
# whole catch above and the report records the failure -- and this
# decides only whether the job does. It blocks on anything it cannot
# attribute to that deadline, including a report it could not read.
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
# The breakpoint below opens on `failure()`, which this exit is
# about to make false. Record the decision so it can open on this
# red too: the sandbox is still up at that point, and a join that
# never finished is the failure an interactive session is worth
# most on.
echo "soft_red=true" >>"$GITHUB_OUTPUT"
exit 0
fi
exit 1
fi
echo "✅ All selected E2E tests passed"
@@ -1173,14 +1191,13 @@ jobs:
# forks cannot reach the self-hosted rendezvous server.
- name: Breakpoint on E2E failure
if: |
(failure() || steps.e2e_tests.outputs.soft_red == 'true') &&
failure() &&
vars.BREAKPOINT_ENDPOINT != '' &&
(contains(github.event.pull_request.labels.*.name, 'debug')
|| contains(github.event.pull_request.labels.*.name, 'release'))
# A step that fails here fails an otherwise-successful job, which on the
# softened path would let a debugging tool overturn a lane that had
# decided not to block. Same reason `Collect report` above carries it:
# what the job REPORTS must not depend on what a helper did.
# A step that fails here would fail the job on its own account. Same
# reason `Collect report` above carries it: what the job REPORTS must not
# depend on what a helper did.
continue-on-error: true
# cozystack/breakpoint-action v2-cozy.1
# mode: pause-idle defaults: grace-period=20m, idle-timeout=10m
@@ -1208,9 +1225,17 @@ jobs:
exits 10 minutes after the last SSH session disconnects.
# ▸ Tear down environment (always runs)
#
# Both halves, and in this order: the compose project owns the nodes and
# the zpools outlive it. A leftover pool makes the next run on a reused
# host adopt storage that already carries a dead cluster's volumes.
- name: Tear down sandbox
if: always()
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
run: |
cleanup_rc=0
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$?
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
exit "$cleanup_rc"
# Report the required "E2E Tests" gate as a commit status rather than relying
# on a job named "E2E Tests". A required *job* that is skipped counts
@@ -1274,7 +1299,6 @@ jobs:
PLAN_RESULT: ${{ needs.plan.result }}
CODE: ${{ needs.plan.outputs.code }}
E2E_RESULT: ${{ needs.e2e.result }}
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
CANDIDATE_RESULT: ${{ needs.verify-release-candidate.result }}
IS_RELEASE: ${{ contains(github.event.pull_request.labels.*.name, 'release') }}
IS_FULL_E2E: ${{ contains(github.event.pull_request.labels.*.name, 'full-e2e') }}
@@ -1282,7 +1306,7 @@ jobs:
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
with:
script: |
const { PLAN_RESULT, CODE, E2E_RESULT, E2E_SOFT_RED, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env;
const { PLAN_RESULT, CODE, E2E_RESULT, CANDIDATE_RESULT, IS_RELEASE, IS_FULL_E2E, HEAD_SHA, RUN_URL } = process.env;
let state, description;
if (PLAN_RESULT !== 'success') {
state = 'failure';
@@ -1303,13 +1327,6 @@ jobs:
// full suite re-run or an admin override.
state = 'success';
description = 'release promotion — verified at rc, e2e not repeated (add full-e2e to force a run)';
} else if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
// The suite did NOT pass: the tenant node-join deadline expired and
// the lane decided not to block on it. `success` is the decision;
// this text is the only place a reader standing in front of the
// merge button learns what it was a decision about.
state = 'success';
description = 'in-tree e2e soft: node-join deadline missed, suite failed, merge not blocked';
} else if (E2E_RESULT === 'success') {
state = 'success';
description = 'in-tree e2e passed';
Makefile
+7 -1
View File
@@ -1,4 +1,4 @@
.PHONY: manifests assets unit-tests helm-unit-tests bats-unit-tests bats-unit-files-check rd-presets-check migrations-target-check test test-controllers preflight
.PHONY: manifests assets prepare-env prepare-env-container unit-tests helm-unit-tests bats-unit-tests bats-unit-files-check rd-presets-check migrations-target-check test test-controllers preflight
include hack/common-envs.mk
@@ -200,6 +200,12 @@ prepare-env:
make -C packages/core/testing apply
make -C packages/core/testing prepare-cluster
# Container-lane sibling of prepare-env. Same sandbox, Talos nodes as containers
# instead of QEMU guests, so no nocloud asset is copied in.
prepare-env-container:
make -C packages/core/testing apply
make -C packages/core/testing prepare-cluster-container
generate:
hack/update-codegen.sh
examples/backups/clickhouse/03-create-bucket.sh
+4
View File
@@ -33,6 +33,10 @@ kubectl -n "$NAMESPACE" wait bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_
# (the "-backup" suffix is the BucketAccessClass name); the BucketInfo Secret
# carries the same name.
kubectl -n "$NAMESPACE" wait bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-backup" --for=jsonpath='{.status.accessGranted}'=true --timeout=180s
if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then
source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh"
cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-backup" 90
fi
log_substep "Reading bucket coordinates from BucketInfo Secret..."
TMP=$(mktemp)
examples/backups/mariadb/run-all.sh
+4
View File
@@ -54,6 +54,10 @@ wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
'{.status.bucketReady}' true "$NAMESPACE" 300
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \
'{.status.accessGranted}' true "$NAMESPACE" 300
if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then
source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh"
cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-${BUCKET_USER}" 90
fi
log_substep "Reading bucket coordinates from BucketInfo Secret..."
TMP=$(mktemp)
examples/backups/mongodb/run-all.sh
+4
View File
@@ -49,6 +49,10 @@ wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
'{.status.bucketReady}' true "$NAMESPACE" 300
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \
'{.status.accessGranted}' true "$NAMESPACE" 300
if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then
source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh"
cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-${BUCKET_USER}" 90
fi
log_substep "Reading bucket coordinates from BucketInfo Secret..."
TMP=$(mktemp)
examples/backups/postgres/run-all.sh
+4
View File
@@ -43,6 +43,10 @@ wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
'{.status.bucketReady}' true "$NAMESPACE" 300
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-backup" \
'{.status.accessGranted}' true "$NAMESPACE" 300
if [[ "${COZY_E2E_BACKUP_PREFLIGHT:-0}" == "1" ]]; then
source "$SCRIPT_DIR/../../../hack/e2e-chainsaw/_lib/backup-access-preflight.sh"
cozy_backup_access_preflight "$NAMESPACE" "bucket-${BUCKET_NAME}-backup" 90
fi
log_substep "Reading bucket coordinates from BucketInfo Secret..."
TMP=$(mktemp)
hack/backup-access-preflight.bats
+163
View File
@@ -0,0 +1,163 @@
#!/usr/bin/env bats
# Unit tests for the E2E-only S3 access gate that runs before expensive backup
# workloads. All cluster and S3 calls are shell stubs.
@test "preflight helper remains sourceable by the POSIX test runner" {
helper=hack/e2e-chainsaw/_lib/backup-access-preflight.sh
[ "$(sed -n '1p' "$helper")" = '#!/bin/sh' ]
if grep -nE '^[[:space:]]*(set[[:space:]].*pipefail|local([[:space:]]|$))' "$helper"; then
echo "backup preflight is sourced by /bin/sh and must not contain Bash-only shell state" >&2
exit 1
fi
sh -n "$helper"
}
@test "preflight fails closed when the BucketInfo Secret cannot be read" {
. hack/e2e-chainsaw/_lib/backup-access-preflight.sh
out=$(mktemp)
kubectl() {
case "$*" in
*" get secret bucket-demo-backup "*) return 1 ;;
*) echo "preflight continued after the Secret read failed: $*" >&2; return 1 ;;
esac
}
if cozy_backup_access_preflight tenant-test bucket-demo-backup 0 >"$out" 2>&1; then
echo "failed BucketInfo Secret read passed the S3 preflight" >&2
exit 1
fi
grep -q 'failed to read BucketInfo Secret tenant-test/bucket-demo-backup' "$out"
if grep -q 'lacks access key' "$out"; then
echo "preflight continued into credential parsing after the Secret read failed" >&2
exit 1
fi
}
@test "preflight retries IAM convergence then proves PUT GET compare and DELETE" {
. hack/e2e-chainsaw/_lib/backup-access-preflight.sh
attempts=$(mktemp)
uploaded_source=$(mktemp)
calls=$(mktemp)
printf '0\n' > "$attempts"
kubectl() {
printf 'kubectl %s\n' "$*" >> "$calls"
case "$*" in
*" get secret bucket-demo-backup "*)
printf '%s' '{"spec":{"secretS3":{"accessKeyID":"access","accessSecretKey":"secret"},"bucketName":"bucket-real"}}' | base64 | tr -d '\n'
;;
*" port-forward "*)
while :; do command sleep 1; done
;;
*) echo "unexpected kubectl call: $*" >&2; return 1 ;;
esac
}
timeout() {
shift
if [ "$1" = sh ]; then
return 0
fi
"$@"
}
sleep() { :; }
mc() {
printf 'mc %s\n' "$*" >> "$calls"
case "$1" in
alias) return 0 ;;
cp)
case "$3" in
backup-preflight/*)
command cp "$(sed -n '1p' "$uploaded_source")" "$4"
;;
*)
count=$(sed -n '1p' "$attempts")
count=$(( count + 1 ))
printf '%s\n' "$count" > "$attempts"
if [ "$count" -eq 1 ]; then
echo 'AccessDenied: IAM identity not loaded yet' >&2
return 1
fi
printf '%s\n' "$3" > "$uploaded_source"
;;
esac
;;
rm) return 0 ;;
*) echo "unexpected mc call: $*" >&2; return 1 ;;
esac
}
cozy_backup_access_preflight tenant-test bucket-demo-backup 5 >/dev/null
[ "$(sed -n '1p' "$attempts")" -eq 2 ] || {
echo "preflight did not retry the not-yet-loaded IAM identity" >&2
cat "$calls" >&2
exit 1
}
[ "$(grep -c '^mc rm ' "$calls")" -eq 1 ] || {
echo "preflight did not delete its successfully verified object exactly once" >&2
cat "$calls" >&2
exit 1
}
}
@test "preflight fails before backup work when granted credentials never reach S3" {
. hack/e2e-chainsaw/_lib/backup-access-preflight.sh
out=$(mktemp)
kubectl() {
case "$*" in
*" get secret bucket-demo-backup "*)
printf '%s' '{"spec":{"secretS3":{"accessKeyID":"access","accessSecretKey":"secret"},"bucketName":"bucket-real"}}' | base64 | tr -d '\n'
;;
*" port-forward "*) while :; do command sleep 1; done ;;
*) return 1 ;;
esac
}
timeout() {
shift
if [ "$1" = sh ]; then return 0; fi
"$@"
}
mc() {
case "$1" in
alias) return 0 ;;
cp) echo 'AccessDenied' >&2; return 1 ;;
*) return 0 ;;
esac
}
if cozy_backup_access_preflight tenant-test bucket-demo-backup 0 >"$out" 2>&1; then
echo "unusable credentials passed the S3 preflight" >&2
exit 1
fi
grep -q 'granted but did not pass S3 PUT/GET/DELETE within 0s' "$out"
grep -q 's3-preflight: AccessDenied' "$out"
}
@test "all active database backup flows gate immediately after accessGranted" {
for file in \
examples/backups/postgres/run-all.sh \
examples/backups/mariadb/run-all.sh \
examples/backups/mongodb/run-all.sh \
examples/backups/clickhouse/03-create-bucket.sh; do
granted=$(grep -n 'accessGranted' "$file" | head -n 1 | cut -d: -f1)
preflight=$(grep -n 'cozy_backup_access_preflight' "$file" | head -n 1 | cut -d: -f1)
coordinates=$(grep -n 'Reading bucket coordinates' "$file" | head -n 1 | cut -d: -f1)
if [ -z "$granted" ] || [ -z "$preflight" ] || [ -z "$coordinates" ] \
|| [ "$granted" -ge "$preflight" ] || [ "$preflight" -ge "$coordinates" ]; then
echo "$file must run the S3 preflight after accessGranted and before consuming bucket coordinates" >&2
exit 1
fi
done
}
@test "all four Chainsaw backup roundtrips enable the E2E preflight" {
for file in \
hack/e2e-chainsaw/postgres/chainsaw-test.yaml \
hack/e2e-chainsaw/mariadb/chainsaw-test.yaml \
hack/e2e-chainsaw/mongodb/chainsaw-test.yaml \
hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml; do
grep -q 'COZY_E2E_BACKUP_PREFLIGHT=1' "$file" || {
echo "$file does not enable the backup access preflight" >&2
exit 1
}
done
}
hack/container-lane-capacity_test.bats
+273
View File
@@ -0,0 +1,273 @@
#!/usr/bin/env bats
# Unit coverage for the scheduler-capacity correction required by Talos
# container nodes. The live BATS suite checks the resulting Node status; this
# file pins the arithmetic and the cross-process plumbing without a cluster.
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
CONTAINER_UP="$HACK_DIR/e2e-container-up.sh"
E2E_CONTAINER_UP_LIB=true . "$CONTAINER_UP"
@test "container reservations reduce host capacity to one node limit" {
reservations=$(calculate_node_reservations 32 131897116 8 24576 3)
expected=$(printf '%s\n' 24000m 106731292Ki)
if [ "$reservations" != "$expected" ]; then
echo "unexpected kubelet reservations: $reservations" >&2
return 1
fi
}
@test "container reservations require aggregate host headroom" {
if calculate_node_reservations 24 131897116 8 24576 3 >/dev/null 2>&1; then
echo "three 8-CPU nodes unexpectedly fit on exactly 24 host CPUs" >&2
return 1
fi
if calculate_node_reservations 32 75497472 8 24576 3 >/dev/null 2>&1; then
echo "three 24-GiB nodes unexpectedly fit in exactly 72 GiB host memory" >&2
return 1
fi
}
@test "container storage pools match the QEMU lane capacity" {
if ! grep -Fq 'ZPOOL_SIZE="${ZPOOL_SIZE:-200G}"' "$CONTAINER_UP"; then
echo "container ZFS pools do not provide the QEMU lane's 200G per node" >&2
return 1
fi
}
@test "Kubernetes canonical memory units normalize to KiB" {
for fixture in \
'25165824Ki:25165824' \
'24476Mi:25063424' \
'23Gi:24117248'; do
quantity=${fixture%%:*}
expected=${fixture#*:}
actual=$(kubernetes_memory_to_kib "$quantity")
if [ "$actual" != "$expected" ]; then
echo "$quantity normalized to $actual KiB, expected $expected KiB" >&2
return 1
fi
done
for invalid in 24476 24.0Gi garbage; do
if kubernetes_memory_to_kib "$invalid" >/dev/null 2>&1; then
echo "invalid quantity was accepted: $invalid" >&2
return 1
fi
done
}
@test "container capacity values cross host compose sandbox and live assertion" {
make_command=$(make -n -C packages/core/testing SANDBOX_NAME=test prepare-cluster-container)
for expected in \
'COZY_E2E_NODE_CPUS="8" COZY_E2E_NODE_MEMORY_MIB="24576"' \
'-e COZY_E2E_NODE_CPUS="8" -e COZY_E2E_NODE_MEMORY_MIB="24576"'; do
if ! printf '%s\n' "$make_command" | grep -Fq -- "$expected"; then
echo "testing Makefile does not forward container capacity: $expected" >&2
return 1
fi
done
if ! grep -Fq 'cpus: ${COZY_E2E_NODE_CPUS:-8}' hack/e2e-compose.yaml \
|| ! grep -Fq 'mem_limit: ${COZY_E2E_NODE_MEMORY_MIB:-24576}m' hack/e2e-compose.yaml; then
echo "compose limits are not sourced from the shared capacity values" >&2
return 1
fi
if ! grep -Fq 'cpu: "${COZY_E2E_SYSTEM_RESERVED_CPU}"' "$CONTAINER_UP" \
|| ! grep -Fq 'memory: "${COZY_E2E_SYSTEM_RESERVED_MEMORY}"' "$CONTAINER_UP"; then
echo "generated Talos config does not carry the calculated reservations" >&2
return 1
fi
if ! grep -Fq '@test "Node allocatable matches the container limits"' hack/e2e-prepare-cluster-container.bats; then
echo "container bring-up does not assert scheduler-visible capacity" >&2
return 1
fi
}
@test "container compose overrides reach both startup and teardown" {
compose_file=/tmp/cozy-custom-compose.yaml
make_command=$(make -n -C packages/core/testing \
SANDBOX_NAME=test \
COMPOSE_PROJECT=cozy-custom \
COMPOSE_FILE="$compose_file" \
ZPOOL_BACKING_DIR=/tmp/cozy-custom-zpools \
prepare-cluster-container \
delete-cluster-container)
if ! printf '%s\n' "$make_command" | grep -Fq \
"COMPOSE_PROJECT=\"cozy-custom\" COMPOSE_FILE=\"$compose_file\""; then
echo "container startup does not receive compose overrides" >&2
return 1
fi
if ! printf '%s\n' "$make_command" | grep -Fq \
"docker compose -p \"cozy-custom\" -f \"$compose_file\" down -v"; then
echo "container teardown does not reuse compose overrides" >&2
return 1
fi
if ! printf '%s\n' "$make_command" | grep -Fq \
'ZPOOL_BACKING_DIR="/tmp/cozy-custom-zpools"'; then
echo "container startup does not receive the backing-directory override" >&2
return 1
fi
for expected in \
'zpool destroy "data-srv${n}"' \
'rm -f "/tmp/cozy-custom-zpools/srv${n}.img"' \
'rmdir "/tmp/cozy-custom-zpools"'; do
if ! printf '%s\n' "$make_command" | grep -Fq "$expected"; then
echo "container teardown does not remove its host storage: $expected" >&2
return 1
fi
done
cleanup_line=$(printf '%s\n' "$make_command" | grep -n 'zpool destroy' | cut -d: -f1)
startup_line=$(printf '%s\n' "$make_command" | grep -n 'e2e-container-up.sh' | cut -d: -f1)
if [ -z "$cleanup_line" ] || [ -z "$startup_line" ] || [ "$cleanup_line" -ge "$startup_line" ]; then
echo "container storage cleanup does not run before startup" >&2
return 1
fi
}
@test "container teardown reports compose failure after reclaiming host storage" {
tmp=$(mktemp -d)
mkdir -p "$tmp/bin" "$tmp/zpools"
calls="$tmp/calls"
: >"$calls"
for n in 1 2 3; do
: >"$tmp/zpools/srv${n}.img"
done
printf '%s\n' \
'#!/bin/sh' \
'printf "docker:%s\n" "$*" >>"$CALLS"' \
'exit 23' >"$tmp/bin/docker"
printf '%s\n' \
'#!/bin/sh' \
'printf "zpool:%s\n" "$*" >>"$CALLS"' \
'exit 0' >"$tmp/bin/zpool"
printf '%s\n' \
'#!/bin/sh' \
'exec "$@"' >"$tmp/bin/sudo"
chmod +x "$tmp/bin/docker" "$tmp/bin/zpool" "$tmp/bin/sudo"
if PATH="$tmp/bin:$PATH" CALLS="$calls" make -s -C packages/core/testing \
COMPOSE_PROJECT=cozy-cleanup-test \
COMPOSE_FILE="$PWD/hack/e2e-compose.yaml" \
ZPOOL_BACKING_DIR="$tmp/zpools" \
delete-cluster-container >"$tmp/output" 2>&1; then
echo "container teardown hid the compose failure" >&2
return 1
fi
grep -Fq 'container compose teardown failed with exit 23' "$tmp/output"
for n in 1 2 3; do
grep -Fq "zpool:destroy data-srv${n}" "$calls"
if [ -e "$tmp/zpools/srv${n}.img" ]; then
echo "backing file srv${n}.img survived failed compose teardown" >&2
return 1
fi
done
if [ -d "$tmp/zpools" ]; then
echo "empty backing directory survived failed compose teardown" >&2
return 1
fi
rm -r "$tmp"
}
@test "privileged Talos container image is digest pinned" {
image=$(yq '.["x-node"].image' hack/e2e-compose.yaml)
if ! printf '%s\n' "$image" | grep -Eq '^ghcr\.io/siderolabs/talos:v1\.13\.5@sha256:[0-9a-f]{64}$'; then
echo "mutable or unexpected Talos container image: $image" >&2
return 1
fi
}
@test "container startup does not mask stale compose teardown failure" {
if grep -Eq 'docker compose .* down -v.*\|\| true' "$CONTAINER_UP"; then
echo "container startup can proceed after stale compose teardown fails" >&2
return 1
fi
grep -Fq 'die "failed to remove a stale ${COMPOSE_PROJECT} compose project before startup"' "$CONTAINER_UP"
}
@test "container teardown keeps backing file when zpool destroy fails" {
tmp=$(mktemp -d)
mkdir -p "$tmp/bin" "$tmp/zpools"
calls="$tmp/calls"
: >"$calls"
for n in 1 2 3; do
: >"$tmp/zpools/srv${n}.img"
done
printf '%s\n' '#!/bin/sh' 'exit 0' >"$tmp/bin/docker"
printf '%s\n' \
'#!/bin/sh' \
'printf "zpool:%s\n" "$*" >>"$CALLS"' \
'if [ "$1" = destroy ] && [ "$2" = data-srv2 ]; then exit 29; fi' \
'exit 0' >"$tmp/bin/zpool"
printf '%s\n' '#!/bin/sh' 'exec "$@"' >"$tmp/bin/sudo"
chmod +x "$tmp/bin/docker" "$tmp/bin/zpool" "$tmp/bin/sudo"
if PATH="$tmp/bin:$PATH" CALLS="$calls" make -s -C packages/core/testing \
COMPOSE_PROJECT=cozy-cleanup-test \
COMPOSE_FILE="$PWD/hack/e2e-compose.yaml" \
ZPOOL_BACKING_DIR="$tmp/zpools" \
delete-cluster-container >"$tmp/output" 2>&1; then
echo "container teardown hid the zpool destroy failure" >&2
return 1
fi
grep -Fq 'failed to destroy zpool data-srv2 (exit 29)' "$tmp/output"
grep -Fq 'keeping '"$tmp/zpools/srv2.img"' because data-srv2 is still imported' "$tmp/output"
if [ ! -e "$tmp/zpools/srv2.img" ]; then
echo "backing file was removed from the still-imported data-srv2 pool" >&2
return 1
fi
for n in 1 3; do
if [ -e "$tmp/zpools/srv${n}.img" ]; then
echo "cleanup stopped before removing srv${n}.img" >&2
return 1
fi
done
rm -r "$tmp"
}
@test "container teardown keeps backing files when zpool inventory is unreadable" {
tmp=$(mktemp -d)
mkdir -p "$tmp/bin" "$tmp/zpools"
calls="$tmp/calls"
: >"$calls"
for n in 1 2 3; do
: >"$tmp/zpools/srv${n}.img"
done
printf '%s\n' '#!/bin/sh' 'exit 0' >"$tmp/bin/docker"
printf '%s\n' \
'#!/bin/sh' \
'printf "zpool:%s\n" "$*" >>"$CALLS"' \
'if [ "$1" = list ]; then exit 31; fi' \
'exit 0' >"$tmp/bin/zpool"
printf '%s\n' '#!/bin/sh' 'exec "$@"' >"$tmp/bin/sudo"
chmod +x "$tmp/bin/docker" "$tmp/bin/zpool" "$tmp/bin/sudo"
if PATH="$tmp/bin:$PATH" CALLS="$calls" make -s -C packages/core/testing \
COMPOSE_PROJECT=cozy-cleanup-test \
COMPOSE_FILE="$PWD/hack/e2e-compose.yaml" \
ZPOOL_BACKING_DIR="$tmp/zpools" \
delete-cluster-container >"$tmp/output" 2>&1; then
echo "container teardown treated an unreadable zpool inventory as absence" >&2
return 1
fi
grep -Fq 'failed to inventory zpools after data-srv1 lookup returned exit 31 (inventory exit 31)' "$tmp/output"
if grep -Fq 'zpool:destroy ' "$calls"; then
echo "container teardown destroyed a pool without a readable inventory" >&2
return 1
fi
for n in 1 2 3; do
if [ ! -e "$tmp/zpools/srv${n}.img" ]; then
echo "backing file srv${n}.img was removed without proving its pool absent" >&2
return 1
fi
done
rm -r "$tmp"
}
hack/e2e-chainsaw/_lib/backup-access-preflight.sh
+127
View File
@@ -0,0 +1,127 @@
#!/bin/sh
# E2E-only proof that a freshly granted COSI BucketAccess is already usable by
# the S3 server. accessGranted describes the COSI object state; it does not
# prove that the backing server has reloaded the new IAM identity.
cozy_backup_access_preflight() (
set -eu
namespace="$1"
access_name="$2"
timeout_seconds="${3:-90}"
s3_namespace="${COZY_E2E_S3_NAMESPACE:-tenant-root}"
s3_service="${COZY_E2E_S3_SERVICE:-seaweedfs-s3}"
local_port="${COZY_E2E_S3_PREFLIGHT_PORT:-18333}"
workdir=''
pf_pid=''
uploaded=false
access_key=''
secret_key=''
bucket=''
deadline=''
attempt=''
object=''
case "${timeout_seconds}:${local_port}" in
*[!0-9:]*)
echo "backup access preflight timeout and local port must be integers" >&2
return 1
;;
esac
if ! workdir=$(mktemp -d); then
echo "could not create backup access preflight work directory" >&2
return 1
fi
# Invoked indirectly by the exit trap below.
# shellcheck disable=SC2329
cleanup() {
rc=$?
trap - 0
if [ -n "${pf_pid}" ] && kill -0 "${pf_pid}" 2>/dev/null; then
if ! kill "${pf_pid}" 2>/dev/null; then
echo "WARNING: could not stop S3 preflight port-forward process ${pf_pid}" >&2
fi
# wait reports the SIGTERM used above as a non-zero child status. That is
# the expected port-forward shutdown, not a preflight failure.
if ! wait "${pf_pid}" 2>/dev/null; then
:
fi
fi
if [ "${uploaded}" = true ]; then
if ! timeout 15 mc rm --insecure "backup-preflight/${bucket}/${object}" >/dev/null 2>&1; then
echo "WARNING: could not remove failed S3 preflight object ${bucket}/${object}" >&2
fi
fi
rm -rf -- "${workdir}"
exit "${rc}"
}
trap cleanup 0
# Keep the producer and decoder as separate commands. This helper is sourced
# by cozytest.sh under /bin/sh, where pipefail is unavailable; a pipeline
# would otherwise let a failed Secret read look successful when base64 exits
# zero on empty input.
if ! kubectl -n "${namespace}" get secret "${access_name}" \
-o jsonpath='{.data.BucketInfo}' > "${workdir}/bucket-info.b64"; then
echo "failed to read BucketInfo Secret ${namespace}/${access_name}" >&2
return 1
fi
if ! base64 -d < "${workdir}/bucket-info.b64" > "${workdir}/bucket-info.json"; then
echo "BucketInfo Secret ${namespace}/${access_name} is not valid base64" >&2
return 1
fi
access_key=$(jq -r '.spec.secretS3.accessKeyID // ""' "${workdir}/bucket-info.json")
secret_key=$(jq -r '.spec.secretS3.accessSecretKey // ""' "${workdir}/bucket-info.json")
bucket=$(jq -r '.spec.bucketName // ""' "${workdir}/bucket-info.json")
if [ -z "${access_key}" ] || [ -z "${secret_key}" ] || [ -z "${bucket}" ]; then
echo "BucketInfo Secret ${namespace}/${access_name} lacks access key, secret key, or bucket name" >&2
return 1
fi
echo "Checking that BucketAccess ${namespace}/${access_name} is live in S3 before provisioning the backup workload..."
kubectl -n "${s3_namespace}" port-forward \
"service/${s3_service}" "${local_port}:8333" >"${workdir}/port-forward.log" 2>&1 &
pf_pid=$!
if ! timeout 30 sh -ec "until nc -z 127.0.0.1 ${local_port}; do sleep 1; done"; then
echo "S3 preflight port-forward did not become ready" >&2
sed 's/^/ port-forward: /' "${workdir}/port-forward.log" >&2
return 1
fi
export MC_CONFIG_DIR="${workdir}/mc"
if ! mc alias set backup-preflight "https://127.0.0.1:${local_port}" \
"${access_key}" "${secret_key}" --insecure >/dev/null; then
echo "failed to configure S3 client for BucketAccess ${namespace}/${access_name}" >&2
return 1
fi
object=".cozystack-e2e-preflight/${access_name}-$(date +%s)-$$"
printf 'cozystack backup access preflight: %s\n' "${access_name}" > "${workdir}/source"
deadline=$(( $(date +%s) + timeout_seconds ))
attempt=0
while :; do
attempt=$(( attempt + 1 ))
: > "${workdir}/mc.log"
if timeout 15 mc cp --insecure "${workdir}/source" \
"backup-preflight/${bucket}/${object}" >"${workdir}/mc.log" 2>&1; then
uploaded=true
if timeout 15 mc cp --insecure "backup-preflight/${bucket}/${object}" \
"${workdir}/download" >>"${workdir}/mc.log" 2>&1 \
&& cmp "${workdir}/source" "${workdir}/download" \
&& timeout 15 mc rm --insecure "backup-preflight/${bucket}/${object}" \
>>"${workdir}/mc.log" 2>&1; then
uploaded=false
echo "BucketAccess ${namespace}/${access_name} passed PUT, GET, compare, and DELETE on attempt ${attempt}."
return 0
fi
fi
if [ "$(date +%s)" -ge "${deadline}" ]; then
echo "BucketAccess ${namespace}/${access_name} was granted but did not pass S3 PUT/GET/DELETE within ${timeout_seconds}s" >&2
sed 's/^/ s3-preflight: /' "${workdir}/mc.log" >&2
return 1
fi
sleep 5
done
)
hack/e2e-chainsaw/_lib/ghcr-mirror.sh
-713
View File
@@ -1,713 +0,0 @@
# shellcheck shell=bash
# Shared helper: point tenant Kubernetes e2e worker nodes at the in-sandbox
# ghcr.io pull-through registry (hack/e2e-ghcr-mirror.yaml) when it is up, via the
# chart's `talos.registryMirrors` knob, otherwise emit nothing so workers fall back
# to pulling ghcr.io directly.
#
# Why: tenant worker Talos nodes pull `ghcr.io/siderolabs/kubelet` directly; that
# egress is flaky/rate-limited from the CI runner and the pull times out with a TLS
# handshake timeout, so the kubelet service never starts and no tenant node joins.
# Diagnosed in cozystack/cozystack#3548 (in-guest Talos dmesg), tracked by #3513.
# Same flaky-public-egress class the talos-image-cache (#3231) fixed for the OS image.
# One variant of that flake, not the whole of it: #3513 also records failures that
# lose the budget before Talos ever reaches the kubelet pull, and those this cannot
# touch.
#
# WHAT THE GATE BELOW ESTABLISHES, precisely, because it is easy to overstate: the
# Deployment went Available, and the API accepted the egress allow. Neither observes
# the tenant-side datapath. If the allow selects an identity the worker's pull does
# not actually carry (see assumption 2 in hack/e2e-ghcr-mirror.yaml), workers are
# pointed at a ClusterIP whose SYNs are dropped, and Talos burns a TCP connect
# timeout per pull before falling back to ghcr.io -- slower than not mirroring at
# all, on exactly the runs this exists to protect. So: the fallback paths cannot
# make CI worse, the committed path is only as safe as the assumptions listed
# there.
#
# talos-image-cache.sh gates the equivalent decision on a byte-level 206 from a Pod
# carrying the consumer's own label, which is what turns that assumption into a
# measurement. Doing the same here is the hardening follow-up, and it is also what
# would let the suite assert the mirror served the pull rather than the worker
# quietly falling back.
GHCR_MIRROR_SVC_URL="${GHCR_MIRROR_SVC_URL:-http://ghcr-mirror.kube-system.svc}"
_GHCR_MIRROR_DECISION_FILE="${_GHCR_MIRROR_DECISION_FILE:-/tmp/e2e-ghcr-mirror-endpoint}"
GHCR_MIRROR_MANIFEST="${GHCR_MIRROR_MANIFEST:-hack/e2e-ghcr-mirror.yaml}"
GHCR_WARM_VERSIONS_FILE="${GHCR_WARM_VERSIONS_FILE:-packages/apps/kubernetes/files/versions.yaml}"
GHCR_WARM_JOB_NAME="${GHCR_WARM_JOB_NAME:-ghcr-mirror-warm}"
GHCR_WARM_REPO="${GHCR_WARM_REPO:-siderolabs/kubelet}"
GHCR_WARM_READY_TIMEOUT="${GHCR_WARM_READY_TIMEOUT:-600}"
# How many minors to warm. Two, because the kubernetes-latest and
# kubernetes-previous suites resolve the top two keys and no other suite spins a
# tenant on a third. Raise it if a suite starts selecting deeper; the cost is
# ~61 MiB of upstream fetch per extra tag on the leg this feature exists for
# (one platform's config plus layers, measured for v1.35.6 against ghcr.io).
GHCR_WARM_TAG_COUNT="${GHCR_WARM_TAG_COUNT:-2}"
# ghcr_warm_tags: print the kubelet image tags worth pre-fetching, one per line,
# newest minor first.
#
# Read from the chart's own version map rather than a list kept here. The suites
# pick their Kubernetes version out of that same file at runtime, so a copy would
# drift silently on the next version bump: the tenant would boot on a tag nothing
# warmed and the first worker would pay the cold upstream fetch this exists to
# remove, with no signal that the warm-up had missed.
#
# Ordered by minor, highest first, which is the order the suites resolve: they pass
# `keys | sort_by(.) | .[-1]` and `.[-2]` to run_kubernetes_test. Sorting here rather
# than reading the file's own order is not a guard against the file being written
# some other way -- packages/apps/kubernetes/hack/update-versions.sh regenerates it
# through `sort -V -r`, so highest-first is what the generator guarantees and a new
# minor lands at the top, not the bottom. It is so that this list and the suites
# agree by construction instead of by both happening to read the same file the same
# way. The caller warms a prefix of this, so the order decides what gets warmed.
ghcr_warm_tags() {
if ! command -v yq >/dev/null 2>&1; then
echo "yq is not installed, so the kubelet version map could not be read" >&2
return 1
fi
yq 'to_entries | sort_by(.key) | reverse | .[].value' \
"$GHCR_WARM_VERSIONS_FILE"
}
# ghcr_warm_job_manifest <image> <tag>...: print the warm-up Job, or fail.
#
# Pure string builder, kept separate so it can be unit-tested, same reason
# talos-image-cache.sh keeps its probe overrides out of the caller.
#
# The Job asks the mirror for each tag's index, every manifest the index lists, and
# every blob those manifests reference, discarding the bytes. distribution's proxy
# populates its blobstore on the way past, so the tenant worker's later pull is a
# local cache hit instead of a live fetch across the leg that stalls.
#
# "On the way past" is not free, and the cost is worth stating exactly. In
# distribution 2.8 a local miss reads the blob from upstream TWICE: ServeBlob starts
# `go storeLocal(...)`, which copies remote -> local, and separately runs
# copyContent(ctx, dgst, w) to serve the client; both open remoteStore. There is no
# tee. So a warm-up blob GET crosses the throttled leg twice.
#
# What keeps that from being a net addition is warming ONE platform per tag -- the
# one the workers run. Then the Job fetches the same blobs the first worker would
# have fetched, and the 2x is paid once either way: by the Job now, or by the worker
# later inside its node-Ready budget. Walking both platforms broke exactly this: the
# arm64 half is bytes nobody in the sandbox pulls, so it was a true addition to the
# leg, and the claim that the total was unchanged was false while it was there.
#
# The byte figures below are image sizes, not measurements of the mirror's upstream
# side, so read them as a lower bound on install-time egress.
#
# Selecting linux/amd64 out of the index stays a grep because digest and platform
# share a record: normalise the whitespace, split on record boundaries, take the
# record that names the architecture. Pulling every sha256 out of a document is a grep,
# and a digest that turns out not to be a blob answers 404. That is free on the
# manifest leg, where the walk just moves on; on the blob leg a 404 counts as a miss
# and fails the Job, which is what a single-arch tag would do -- its config and layer
# digests are not manifests, so the walk finds no index to descend.
ghcr_warm_job_manifest() {
local image="$1" tag tags=""
shift
[ "$#" -gt 0 ] || return 1
# An empty endpoint or repo does not fail anything downstream, which is why it has
# to fail here. The Job would still build, run, exit 0, and print "index
# unavailable" for every tag while the cache stayed cold -- and from inside the
# Pod that is indistinguishable from a mirror that is simply unreachable.
[ -n "$GHCR_MIRROR_SVC_URL" ] || return 1
[ -n "$GHCR_WARM_REPO" ] || return 1
for tag in "$@"; do
# Refuse anything that is not a bare version. These tags land in a `for tag in
# ...` inside the container's shell, where a substitution would run as the Job.
# The version map is in-tree today, which is why the guard is cheap to add now
# and expensive to retrofit later. No real kubelet tag is rejected by this.
# Character class first, regex second. `grep -q` succeeds when ANY line of its
# input matches and `$` anchors end-of-LINE, so a value whose first line is a
# bare version passes the pattern while everything after the newline is pasted
# into the heredoc as shell. The earlier list of rejected inputs enumerated ways
# to inject -- $(), backticks, semicolon, space -- and missed this one, because
# it was reasoned about as "more than one token" rather than as what the anchor
# treats as a boundary. This case does not depend on that.
case "$tag" in
*[!v0-9.]*) return 1 ;;
esac
printf '%s' "$tag" | grep -qE '^v[0-9]+\.[0-9]+\.[0-9]+$' || return 1
tags="${tags}${tags:+ }${tag}"
done
cat <<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: ${GHCR_WARM_JOB_NAME}
namespace: kube-system
labels:
app.kubernetes.io/name: ${GHCR_WARM_JOB_NAME}
spec:
# Bounded two ways, because retries alone are not a bound here. The fetches are
# slow by nature on the leg this exists for, so a Job with only backoffLimit can
# sit pulling on a sandbox node for hours after the suites it was meant to help
# have finished, competing for the CPU the tenant guests are already short of.
backoffLimit: 3
activeDeadlineSeconds: 3600
# Outlives the suites on purpose. The node-join failure block is the only reader
# of this Job's outcome and it fires at the 29m node-join deadline, in a suite
# that starts tens of minutes after install; a shorter TTL deletes the Job and
# its log before the one moment anybody asks whether the warm-up worked.
ttlSecondsAfterFinished: 14400
template:
metadata:
labels:
app.kubernetes.io/name: ${GHCR_WARM_JOB_NAME}
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: warm
image: ${image}
command: ["/bin/sh", "-ec"]
args:
- |
root="${GHCR_MIRROR_SVC_URL}"
base="\$root/v2/${GHCR_WARM_REPO}"
acc='Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
# -U names this traffic in the registry's access log. The node-join
# diagnostic counts GETs for the kubelet image to answer "did a worker
# pull through the mirror"; without a way to tell the two apart, this
# Job's own 50-odd requests would make that answer yes on every run.
get() { wget -q -U ${GHCR_WARM_JOB_NAME} "\$@"; }
digests() { grep -o 'sha256:[0-9a-f]\{64\}' | sort -u; }
# Wait for the mirror to serve before concluding anything. This Job is
# created seconds after the mirror Deployment, the registry image is
# deliberately not in the pre-pull set, and the readiness probe adds
# its own delay -- so at submit time the Service usually has no
# endpoints at all. Treating that as "nothing to warm" is how a
# warm-up silently no-ops on the run it was needed for.
# The bound is the loop condition, not a branch inside the loop. A shape
# that loops until the fetch succeeds and exits from an "over budget"
# arm inside the body puts termination in something removable, and
# removing it does not fail anything: it hangs, which is worse than
# either outcome and takes whatever is watching down with it.
#
# No backticks anywhere in this heredoc. It is unquoted, so a backtick
# in a COMMENT is still command substitution at render time, and the
# first version of this note contained a shell snippet that expanded
# into an endless loop -- the builder hung, and every structural test
# passed because the YAML it never finished printing was still valid.
# A real clock, not a per-attempt constant. How long an attempt
# costs is not knowable here: Cilium answers a Service with no
# ready endpoints by rejecting (serviceNoBackendResponse defaults
# to reject), so the connect fails instantly and the -T budget is
# never spent, while a mirror that is up but slow spends all of
# it. A constant is wrong in both regimes, and it was wrong in
# both directions across two attempts at guessing it.
ready=0
start=\$(date +%s)
while [ \$(( \$(date +%s) - start )) -lt ${GHCR_WARM_READY_TIMEOUT} ]; do
if get -T 10 -O /dev/null "\$root/v2/"; then
ready=1
break
fi
sleep 5
done
if [ "\$ready" -ne 1 ]; then
echo "warm: mirror did not start serving within ${GHCR_WARM_READY_TIMEOUT}s"
exit 1
fi
# Pick the one platform the workers run. The index lists digest and
# platform inside the SAME record, so normalising whitespace and
# splitting on record boundaries keeps this a grep -- no JSON parser,
# which the mirror's image does not carry.
#
# Walking both platforms instead is what the earlier version did, and
# it was wrong twice over: arm64 is bytes nobody in the sandbox ever
# pulls, and the digest sort orders manifests by hex, which put arm64 first
# for both warmed tags. On the 40 KB/s floor this feature exists for,
# that spent the first half of the Job's deadline on an architecture
# the workers do not use and left the second tag cold.
select_amd64() {
tr -d ' \t\n' |
sed 's/},{/}\n{/g' |
grep -E '"architecture":"amd64"' |
grep -o 'sha256:[0-9a-f]\{64\}' |
head -1
}
failed=0
got=0
warmed=""
missed=""
for tag in ${tags}; do
idx=\$(get -T 120 -O - --header "\$acc" "\$base/manifests/\$tag") || {
echo "warm: index for \$tag unavailable"
failed=\$((failed + 1))
missed="\$missed \$tag"
continue
}
before=\$got
tag_failed=0
# A single-arch tag has no platform record: the response IS the
# manifest, so walk it directly rather than looking for an index
# entry that does not exist.
sel=\$(printf '%s' "\$idx" | select_amd64)
if [ -n "\$sel" ]; then
bodies=\$(get -T 120 -O - --header "\$acc" "\$base/manifests/\$sel") || bodies=""
else
bodies="\$idx"
fi
if [ -n "\$bodies" ]; then
for blob in \$(printf '%s' "\$bodies" | digests); do
# "fetched", not "cached": a successful GET says the mirror
# answered, which is equally true of a cache hit and of a live
# upstream fetch. The blobstore write is distribution's business
# and this Job cannot observe it.
if get -T 600 -O /dev/null "\$base/blobs/\$blob"; then
got=\$((got + 1))
echo "warm: \$tag \$blob fetched"
else
failed=\$((failed + 1))
tag_failed=\$((tag_failed + 1))
echo "warm: \$tag \$blob failed"
fi
done
fi
# "warmed" has to mean every blob of the tag landed. A live run
# showed why: three of four blobs failed on an upstream reset and the
# tag still reported warmed, because the counter asked whether ANY
# blob arrived. A partially cached tag still makes the worker fetch
# the rest across the slow leg, which is the thing being measured.
if [ "\$got" -gt "\$before" ] && [ "\$tag_failed" -eq 0 ]; then
warmed="\$warmed \$tag"
else
missed="\$missed \$tag"
fi
done
# Per-tag, not just a total: a Job that hits its deadline halfway warms
# one suite and not the other, and a bare count cannot say which.
echo "warm: warmed:\$warmed"
echo "warm: not warmed:\$missed"
echo "warm: fetched \$got, failed \$failed"
# One question, about the subject: is every selected tag fully warm.
# The counters are proxies for it and both turned out to be strictly
# weaker -- a failed blob or a failed index already puts its tag in the
# missed list, and a zero fetch count with nothing missed would need zero
# tags, which the builder refuses. Keeping them as extra conjuncts made
# their own mutations survive: anything they would have caught, this
# catches first. A proxy also misses whatever path forgets to update it,
# and that is how a cold tag rode out on a Complete Job.
[ -z "\$missed" ]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- ALL
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 500m
memory: 128Mi
ephemeral-storage: 64Mi
EOF
}
# warm_ghcr_mirror: start the warm-up Job. Best-effort and never fails the caller:
# it runs during install, and the run has to survive without it exactly as it did
# before the warm-up existed.
warm_ghcr_mirror() {
local out image tags manifest
if ! out=$(kubectl -n kube-system get deploy ghcr-mirror 2>&1); then
# Only the API's own NotFound means the mirror was never applied here. A missing
# kubectl, a bad context and an apiserver blip all fail this read too, and the
# captured message is the only place their cause exists -- the same distinction
# resolve_ghcr_mirror_endpoint documents below.
case "$out" in
*NotFound*)
echo "ghcr-mirror not deployed -- nothing to warm" >&2 ;;
*)
echo "WARNING: could not query the ghcr-mirror Deployment; skipping the warm-up. Reason: ${out:-none reported}" >&2 ;;
esac
return 0
fi
# Warm only the minors a suite can actually select. The suites resolve `.[-1]` and
# `.[-2]` of the sorted keys, ghcr_warm_tags emits that same order, so a prefix of
# it is exactly their set without restating their expression here.
#
# The whole map would be ~300 MiB at the walk this file ships: five tags at ~61 MiB
# each for the single platform the Job fetches. (634 MiB was the measured figure
# while the walk still covered both platforms of every tag; quoting it against
# one-platform code was the same accounting slip the paragraph above pins on the
# platform walk itself.) It crosses the one leg
# this feature exists because of, the one the access log measures at 40-165 KB/s and
# which answers 500 mid-stream. Warming
# tags nothing selects spends that leg on nothing while the install and the
# pre-pull step are still using it.
# No 2>/dev/null. Command substitution captures stdout only, so ghcr_warm_tags'
# complaint about a missing yq cannot reach $tags in the first place -- redirecting
# it merely deleted the one line that names the cause, leaving the caller to blame
# versions.yaml for a tool that is not installed.
tags=$(ghcr_warm_tags | head -n "$GHCR_WARM_TAG_COUNT") || tags=""
if [ -z "$tags" ]; then
echo "WARNING: could not read kubelet tags from ${GHCR_WARM_VERSIONS_FILE}; skipping the warm-up" >&2
return 0
fi
# Reuse the mirror's own image so there is one place to bump. It is NOT already
# cached on the nodes: hack/e2e-ghcr-mirror.yaml keeps it out of the pre-pull set
# on purpose, and the pre-pull step runs after this one anyway. The Job's own wait
# for the mirror to serve is what absorbs that.
if ! image=$(kubectl -n kube-system get deploy ghcr-mirror \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null); then
image=""
fi
if [ -z "$image" ]; then
echo "WARNING: could not read the ghcr-mirror image; skipping the warm-up" >&2
return 0
fi
# The image is the only builder input that arrives from the cluster, and it is
# interpolated into the Job YAML. Writing it needs Deployment write access, so this
# is not a trust boundary. The point is narrower: the endpoint and the repository
# are checked and the tags are pattern-matched, so leaving the one input that comes
# from outside unchecked reads as an oversight. Other locally-set values
# (GHCR_WARM_JOB_NAME, GHCR_WARM_READY_TIMEOUT) also reach the script unquoted and
# are equally unchecked -- they are set in this file and nowhere else.
# Reference characters only.
case "$image" in
*[!A-Za-z0-9./:@_-]*)
echo "WARNING: the ghcr-mirror image reads as [$image], which is not an image reference; skipping the warm-up" >&2
return 0 ;;
esac
# Capture the manifest with stderr NOT folded in. `2>&1` here would put anything
# written to stderr inside the YAML, and under a runner that traces the body (the
# bats runner does) that is the trace itself: the Job then fails to parse for a
# reason that has nothing to do with the Job. The builder communicates through its
# exit status, so there is nothing on stderr worth keeping.
#
# shellcheck disable=SC2086 # tags is a builder-validated whitespace-separated list
if ! manifest=$(ghcr_warm_job_manifest "$image" $tags 2>/dev/null); then
echo "WARNING: refused to build the warm-up Job (a tag is not a plain version); skipping" >&2
return 0
fi
if ! out=$(printf '%s\n' "$manifest" | kubectl apply -f - 2>&1); then
echo "WARNING: could not start the ghcr-mirror warm-up; tenant workers may pay a cold fetch. Reason: ${out:-none reported}" >&2
return 0
fi
echo "ghcr-mirror warm-up started for: $(printf '%s' "$tags" | tr '\n' ' ')" >&2
return 0
}
# ghcr_registry_mirrors_block: pure string builder. Given a mirror endpoint URL
# ($1, may be empty), print the `registryMirrors:` sub-block for a tenant Kubernetes
# CR `spec.talos` (4-space indented, i.e. nested under ` talos:`), or nothing when
# the endpoint is empty. Kept separate so it can be unit-tested: an indentation or
# quoting slip here would emit an invalid CR and every kubernetes-* test would fail.
ghcr_registry_mirrors_block() {
local endpoint="$1"
[ -n "$endpoint" ] || return 0
printf ' registryMirrors:\n ghcr.io:\n endpoints:\n - %s\n' "$endpoint"
}
# _apply_ghcr_mirror_egress_policy: install the CiliumClusterwideNetworkPolicy that
# lets tenant worker VM (virt-launcher) Pods egress to the mirror. It ships in the
# manifest but is applied here, after Cilium's CRDs exist. Idempotent, best-effort.
#
# On failure it PRINTS the reason to stdout for the caller to surface. The causes
# are not all alike -- a missing yq, an apiVersion the cluster does not serve, an
# RBAC denial are permanent, while an apiserver blip is not -- and the exit code
# cannot tell them apart. Discarding the message would leave the caller announcing
# a cause it does not know, in the subsystem this exists to make legible.
_apply_ghcr_mirror_egress_policy() {
if ! command -v yq >/dev/null 2>&1; then
echo "yq is not installed, so the policy could not be extracted from ${GHCR_MIRROR_MANIFEST}"
return 1
fi
# Braces, not a bare `2>&1 >/dev/null`: the goal is to keep stderr (the reason)
# on stdout for the caller and drop kubectl's success chatter, and the braced
# form says that unambiguously.
{ yq 'select(.kind == "CiliumClusterwideNetworkPolicy")' "$GHCR_MIRROR_MANIFEST" \
| kubectl apply -f - >/dev/null; } 2>&1
}
# resolve_ghcr_mirror_endpoint: print the mirror endpoint URL to use, or an empty
# string to signal "pull ghcr.io directly". Resolved once and cached in a /tmp file
# that persists across bats files (same sandbox container), so only the first tenant
# test pays the readiness wait.
resolve_ghcr_mirror_endpoint() {
if [ -f "$_GHCR_MIRROR_DECISION_FILE" ]; then
cat "$_GHCR_MIRROR_DECISION_FILE"
return 0
fi
local endpoint="" cache=1 out rc avail ready apply_err
# One `get deploy` serves both the existence check and the not-deployed-vs-error
# split: a genuine NotFound is a stable decision worth caching, but any other
# failure (transient apiserver blip) must NOT be cached, or one blip would
# disable the mirror for every later test in the shared sandbox.
#
# Capture inside `if` (not a bare `out=$(...); rc=$?`): the chainsaw suites run
# this script under `sh -c` with `set -eu`, and dash inherits errexit into a
# command substitution, so a non-zero kubectl in a bare assignment would kill
# the script before either fallback branch runs. bash does not inherit it,
# which is why a bare assignment survives locally but not in CI.
if out=$(kubectl -n kube-system get deploy ghcr-mirror 2>&1); then
rc=0
else
rc=$?
fi
if [ "$rc" -ne 0 ]; then
case "$out" in
# The API's own NotFound only. A bare "not found" also matches a shell
# reporting a missing kubectl and a bad --context, neither of which says
# anything about whether the mirror was applied -- and both would then be
# cached as the stable answer for every later suite in the sandbox.
*NotFound*)
echo "ghcr-mirror not deployed -- tenant workers pull ghcr.io directly" >&2 ;;
*)
echo "WARNING: could not query ghcr-mirror Deployment (transient); not caching, will retry" >&2
cache=0 ;;
esac
else
# Readiness is decided once, then acted on once. The watch is the cheap way to
# learn it, but `rollout status` also exits non-zero on a broken or aborted
# watch, and its output goes to /dev/null -- so when it fails, ask the API what
# the Deployment's own condition says instead of reading the watch's exit code
# as the answer.
#
# 2m, not longer: the wait is serial with the rest of the 67m e2e step, whose
# budget inventory has to stay above the sum of its legs, and spending more here
# eats the margin the failure-path diagnostics need in order to be collected at
# all. It is already generous -- the Deployment pulls one small digest-pinned
# image and was applied at install time, tens of minutes earlier.
if kubectl -n kube-system rollout status deploy/ghcr-mirror --timeout=2m >/dev/null 2>&1; then
ready=yes
else
if avail=$(kubectl -n kube-system get deploy ghcr-mirror \
-o jsonpath='{.status.conditions[?(@.type=="Available")].status}' 2>/dev/null); then
:
else
avail=""
fi
case "$avail" in
True) ready=yes ;;
False) ready=no ;;
*) ready=unknown ;;
esac
fi
case "$ready" in
yes)
# Only commit the endpoint once the tenant egress allow is actually in
# place. The mirror is a kube-system ClusterIP that the tenant default-deny
# egress drops without this allow; pointing workers at it when the policy
# did not apply just adds dial-timeout latency before Talos falls back.
# Leaving the endpoint empty is what keeps that cost off the run -- it
# removes the one failure mode this function can see, not every one of them
# (see the header on what the gate does and does not establish).
if apply_err=$(_apply_ghcr_mirror_egress_policy); then
endpoint="$GHCR_MIRROR_SVC_URL"
echo "ghcr-mirror ready and egress allow applied -- tenant workers mirror ghcr.io via ${endpoint}" >&2
else
# Not cached, for the same reason as the failed query above: a failed
# apply says nothing about whether the next attempt would succeed. The
# reason is printed rather than characterised -- some causes here are
# permanent and retrying will not help, and only the message says which.
echo "WARNING: ghcr-mirror ready but its egress allow failed to apply; not caching, will retry. Reason: ${apply_err:-none reported}" >&2
cache=0
fi
;;
no)
# Cached, and this is a trade rather than an invariant: a Deployment can
# still go Available later on its own -- a kubelet image-pull retry or a
# Pending Pod finally scheduling would both do it, with nothing re-applying
# the manifest. What caching buys is not re-paying the wait in every later
# suite, serial with the rest of the step; what it costs is missing a mirror
# that recovers mid-run. For a best-effort accelerator that trade is worth
# taking, and re-reading the condition cheaply on a cached miss would take
# it back if it ever bites.
echo "WARNING: ghcr-mirror not Available -- tenant workers pull ghcr.io directly" >&2
;;
*)
echo "WARNING: could not confirm ghcr-mirror readiness (transient); not caching, will retry" >&2
cache=0
;;
esac
fi
# Cache atomically (temp + mv) so a concurrent reader in the shared sandbox never
# sees a half-written decision file.
if [ "$cache" -eq 1 ]; then
printf '%s' "$endpoint" > "${_GHCR_MIRROR_DECISION_FILE}.tmp" \
&& mv "${_GHCR_MIRROR_DECISION_FILE}.tmp" "$_GHCR_MIRROR_DECISION_FILE"
fi
printf '%s' "$endpoint"
}
# _ghcr_mirror_bounded_read <label> <command...>: run one diagnostic read under a
# wall-clock bound and name the outcome when it does not finish, so a read cut off
# mid-flight is not misread as the cluster being silent. Always returns 0: the only
# caller sits before the node-join block's `exit 1`, and a non-zero here would
# replace that exit under `set -e`.
#
# Bounded by the caller's own COZY_DIAG_READ_TIMEOUT/COZY_DIAG_READ_GRACE, the same
# knobs cozy_diag_read uses, so lowering the node-join block's budget lowers these
# too -- a private knob here would sit at its own default while the caller's budget
# dropped. The block validates those values before this runs; standalone use
# (hack/ghcr-mirror_test.bats exercises this without run-kubernetes.sh) falls back
# to the block's own 20s/5s defaults. The bound lives here rather than in a call to
# cozy_diag_read because that function is defined in run-kubernetes.sh and this file
# is sourced on its own, the same reason talos-image-cache.sh carries its own copy.
_ghcr_mirror_bounded_read() {
local label="$1"
shift
local to="${COZY_DIAG_READ_TIMEOUT:-20}" grace="${COZY_DIAG_READ_GRACE:-5}" rc=0
if command -v timeout >/dev/null 2>&1; then
timeout -k "$grace" "$to" "$@" || rc=$?
else
"$@" || rc=$?
fi
case "$rc" in
0) ;;
124|137)
echo "=== ${label}: read did not finish within ${to}s and was cut off; what it had not printed is absent from this log, not from the cluster ===" >&2 ;;
*)
echo "=== ${label}: read failed (exit ${rc}); what it would have shown was not observed ===" >&2 ;;
esac
return 0
}
# ghcr_mirror_diagnose: on-demand diagnostic for the node-join failure path.
#
# Pointing workers at the mirror and their nodes joining does not establish that
# the mirror served anything -- Talos falls back to public ghcr.io on its own, so
# a pass is consistent with the mirror never being touched, and a failure is
# consistent with it being unreachable. registry:2's access log is what tells the
# two apart: a `GET /v2/siderolabs/kubelet/...` line means a worker really came
# through here. Without it #3513 gains another run that has to be reasoned about
# from the outside, which is how it survived four attempts.
#
# All output to stderr; never fails the caller. Every read is wall-clock bounded
# by the caller's COZY_DIAG_READ_TIMEOUT/COZY_DIAG_READ_GRACE.
ghcr_mirror_diagnose() {
local hits log gate_rc=0
local to="${COZY_DIAG_READ_TIMEOUT:-20}" grace="${COZY_DIAG_READ_GRACE:-5}"
# The existence gate is bounded too: a wedged apiserver hangs `get deploy` just
# as it hangs the reads below.
if command -v timeout >/dev/null 2>&1; then
timeout -k "$grace" "$to" \
kubectl -n kube-system get deploy ghcr-mirror >/dev/null 2>&1 || gate_rc=$?
else
kubectl -n kube-system get deploy ghcr-mirror >/dev/null 2>&1 || gate_rc=$?
fi
if [ "$gate_rc" -ne 0 ]; then
echo "ghcr-mirror not deployed -- tenant workers pulled ghcr.io directly; nothing to diagnose" >&2
return 0
fi
{
echo "--- ghcr-mirror deploy/pod/svc/endpointslice ---"
_ghcr_mirror_bounded_read 'ghcr-mirror deploy/pod/svc/endpointslice' \
kubectl -n kube-system get deploy,pod,svc,endpointslice \
-l app.kubernetes.io/name=ghcr-mirror -o wide
echo "--- worker egress allow policy ---"
_ghcr_mirror_bounded_read 'worker egress allow policy' \
kubectl get ciliumclusterwidenetworkpolicy e2e-ghcr-mirror-worker-egress -o wide
# The answer to "did a worker actually pull through the mirror".
#
# Filter the whole log rather than tailing it. The readinessProbe requests /v2/
# every 5s and registry:2 writes an access line per request, so ~12 lines a
# minute pile up after the worker's pull; by the time this runs, past a
# 29m node-join budget, that is 200+ probe lines on top of the one request
# worth finding, and any fixed tail window reports "no kubelet pull" for a
# mirror that served it. --tail=-1 is explicit because kubectl defaults a label-selected
# read to the last 10 lines, not to the whole log. It is the wall clock, not the
# line count, that bounds this read: --tail=-1 stays, timeout caps the time.
echo "--- did a worker pull ${GHCR_WARM_REPO} through the mirror? ---"
# Read and filter as separate steps. Piping them would make an unreadable log
# and a log with no kubelet request produce the same 0, and the second is a
# finding about the workers while the first is a statement about nothing.
local log_rc=0 durations=''
if command -v timeout >/dev/null 2>&1; then
log=$(timeout -k "$grace" "$to" \
kubectl -n kube-system logs -l app.kubernetes.io/name=ghcr-mirror \
--tail=-1 2>/dev/null) || log_rc=$?
else
log=$(kubectl -n kube-system logs -l app.kubernetes.io/name=ghcr-mirror \
--tail=-1 2>/dev/null) || log_rc=$?
fi
if [ "$log_rc" -eq 0 ]; then
# Exclude the warm-up's own requests. It asks for the same paths from
# kube-system before any tenant exists, so counting them would make "workers
# did reach it" true of every run and retire the only signal that tells a
# served pull from a worker that fell back to public ghcr.io. The warm-up
# names itself with -U; see ghcr_warm_job_manifest.
#
# Count access lines, not every line carrying the path. registry:2.8.3 writes
# three lines per GET against the pinned image -- a logrus "response completed",
# an upstream-challenge line, and one nginx-combined access line -- so matching
# the path alone counted one request as three. The combined line is the only
# shape that appears exactly once per request, which makes this a request count
# rather than a line count, and it is also the shape whose user-agent field is
# positional and always present.
#
# GET and HEAD both count. containerd resolves a manifest with a HEAD before it
# GETs anything, and the combined access line records HEAD in the same shape
# (measured on the pinned image with a containerd user-agent), so a filter
# anchored on GET alone drops the first request of every pull -- and on a log
# holding only that HEAD it declares the worker absent while the duration
# section below prints that same request's timing. The warm-up issues only GETs,
# so counting HEAD reintroduces none of its traffic.
hits=$(printf '%s\n' "$log" \
| grep -F -e "\"GET /v2/${GHCR_WARM_REPO}" -e "\"HEAD /v2/${GHCR_WARM_REPO}" \
| grep -vcF "$GHCR_WARM_JOB_NAME") || hits=0
if [ "${hits:-0}" -gt 0 ]; then
echo "a worker pulled ${GHCR_WARM_REPO} through the mirror (${hits} request(s), excluding the warm-up's own)"
else
echo "no ${GHCR_WARM_REPO} request appears in the mirror's access log; whether workers fell back to public ghcr.io or never got as far as the pull, this log cannot tell apart"
fi
# How long the mirror took to answer the worker. This is the only signal that
# separates a warm cache from a cold one after the fact: distribution serves a
# cached blob from disk in milliseconds and streams a missed one from ghcr.io for
# minutes, and nothing else in the artifacts distinguishes them. A warm-up that
# reports success is not proof the blob reached the blobstore -- the proxy stores
# in the background, so a completed GET and a stored blob are different events.
# The warm-up's own requests are excluded for the same reason as above.
#
# Filter here rather than handing the log to `sh -c` as an argument. Linux caps
# a single argv element at MAX_ARG_STRLEN, 131072 bytes, and that cap is
# independent of ARG_MAX and of how much room the rest of the command line
# leaves: measured on 6.8, an argument of 131000 bytes execve()s and one of
# 131072 fails with E2BIG. The readiness probe writes an access line every 5s,
# so this log passes that after roughly twenty minutes of mirror uptime -- which
# is always true here, since the mirror is created during install and this runs
# at the 29m node-join deadline, in a suite that starts long after it. As an
# argument the read would have failed on every real run and succeeded only
# against a fixture small enough to fit, which is the one case the tests
# exercise.
echo "--- how long did the mirror take to answer the worker? ---"
durations=$(printf '%s\n' "$log" | grep -F "/v2/${GHCR_WARM_REPO}" \
| grep -F 'http.response.duration' \
| grep -vF "$GHCR_WARM_JOB_NAME" \
| grep -oE 'http\.response\.duration=[^ ]+' \
| tail -5) || durations=''
if [ -n "$durations" ]; then
printf '%s\n' "$durations"
else
echo "no timed request for ${GHCR_WARM_REPO} outside the warm-up's own; the durations that separate a warm cache from a cold one were not observed"
fi
else
echo "could not read the mirror's log; this says nothing either way about whether workers used it"
fi
# Whether the warm-up itself got anywhere. Without this the exclusion above makes
# the run unfalsifiable: a red suite looks identical whether the cache was warm
# and the guest still starved, or the warm-up never came up at all.
echo "--- did the warm-up populate the cache? ---"
_ghcr_mirror_bounded_read 'warm-up Job status' \
kubectl -n kube-system get job "$GHCR_WARM_JOB_NAME" -o wide
_ghcr_mirror_bounded_read 'warm-up Job result' \
kubectl -n kube-system logs "job/${GHCR_WARM_JOB_NAME}" --tail=5
echo "--- registry access log (last 50 lines, for context) ---"
_ghcr_mirror_bounded_read 'registry access log (last 50 lines)' \
kubectl -n kube-system logs -l app.kubernetes.io/name=ghcr-mirror \
--tail=50 --prefix
} >&2
return 0
}
hack/e2e-chainsaw/_lib/run-kubernetes.sh
+650 -197
View File
File diff suppressed because it is too large. Load diff
hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml
+4 -4
View File
@@ -336,13 +336,13 @@ spec:
# Inner budgets, sized against what CI actually takes rather than guessed
# generously — the whole bucket suite completes in ~37s and a backup-enabled
# ClickHouse plus Keeper reaches Ready in ~98s (measured on run 30353894332):
# 03 660s = 300 bucket HR + 180 claim + 180 access
# 03 750s = 300 bucket HR + 180 claim + 180 access + 90 S3 preflight
# 04 600s = 300 HR + 300 StatefulSet
# 05 480s = BackupJob to Succeeded
# 06 480s = in-place RestoreJob
# 07 1080s = 300 HR + 300 StatefulSet + 480 to-copy RestoreJob
# sum = 3300s (55m)
# 60m therefore leaves ~5m of headroom for the kubectl apply / exec SQL
# sum = 3390s (56m30s)
# 60m therefore leaves ~3m30s of headroom for the kubectl apply / exec SQL
# calls interleaved between those waits (convention 6: justified in-line).
#
# Reaching the ceiling at all takes a pathological hang: every wait fails
@@ -355,7 +355,7 @@ spec:
set -eu
cd ../../..
[ -x examples/backups/clickhouse/run-all.sh ] || { echo "clickhouse backup example scripts not found at examples/backups/clickhouse" >&2; exit 1; }
NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
COZY_E2E_BACKUP_PREFLIGHT=1 NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
examples/backups/clickhouse/run-all.sh
check:
($error == null): true
hack/e2e-chainsaw/foundationdb/chainsaw-test.yaml
+48 -1
View File
@@ -63,7 +63,6 @@ spec:
health:
available: true
fullReplication: true
healthy: true
# volumeClaimTemplate size is pinned on the CR spec.
- assert:
resource:
@@ -129,6 +128,54 @@ spec:
check:
($error == null): true
- name: refresh-and-assert-full-health
try:
- script:
timeout: 5m
content: |
set -eu
ns="$NAMESPACE"
cl=foundationdb-test
pod=$(kubectl -n "$ns" get pods \
-l "foundationdb.org/fdb-cluster-name=$cl,foundationdb.org/fdb-process-class=storage" \
-o jsonpath='{.items[0].metadata.name}')
status_json() {
timeout -k 5 20 kubectl -n "$ns" exec "$pod" -c foundationdb -- \
fdbcli -C /var/fdb/data/fdb.cluster --exec 'status json' 2>/dev/null
}
database_healthy() {
status_json | jq -e '.client.database_status.healthy == true' >/dev/null
}
# The v2.30 operator can finish its initial reconciliation from a
# transient status_incomplete response, then never requeue solely to
# refresh health. Wait on the authoritative machine-readable status
# before asking the operator for exactly one fresh read.
deadline=$(( $(date +%s) + 270 ))
until database_healthy; do
if [ "$(date +%s)" -ge "$deadline" ]; then
echo "FoundationDB machine status did not become fully healthy" >&2
status_json | jq '{databaseStatus: .client.database_status, messages: .cluster.messages}' >&2 || true
exit 1
fi
# TODO(e2e-replace-fixed-timeouts): fdbcli health has no watchable
# Kubernetes object; replace this poll when the operator exposes one.
sleep 5
done
kubectl -n "$ns" annotate foundationdbclusters.apps.foundationdb.org "$cl" \
"e2e.cozystack.io/health-refresh=$(date +%s)" --overwrite >/dev/null
check:
($error == null): true
- assert:
timeout: 3m
resource:
apiVersion: apps.foundationdb.org/v1beta2
kind: FoundationDBCluster
metadata:
name: foundationdb-test
status:
health:
healthy: true
- name: verify-monitoring-and-storage
try:
# Dashboard resource map ConfigMap is created by the controller.
hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
+12 -7
View File
@@ -48,7 +48,6 @@ spec:
# (those alternate rather than run to completion in turn: each takes
# its first reading, one wait passes, each takes its second)
# guest Talos capture
# ghcr-mirror state, access log and warm-up Job
# talos-image-cache diagnosis
# The four collectors that bracket the node-join wait -- the sandbox
# kernel KVM counters, the runner kernel CPU time, the sandbox QEMU
@@ -84,24 +83,30 @@ spec:
set -eu
cd ../../..
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
# 4th arg "true" folds the ouroboros hairpin-NAT assertions onto this
# cluster instead of a second ~25m bringup.
run_kubernetes_test 'keys | sort_by(.) | .[-1]' 'test-latest-version' '59991' 'true'
# 4th arg folds the ouroboros hairpin-NAT assertions onto this cluster;
# the 5th folds the System -> CustomConfig OIDC lifecycle onto the same
# ready workers. Neither feature needs another tenant-cluster bringup.
run_kubernetes_test 'keys | sort_by(.) | .[-1]' 'test-latest-version' '59991' 'true' 'true'
check:
($error == null): true
finally:
# Always runs after try/catch — i.e. after the crust-gather snapshots on a
# failed run. Reclaims the tenant Kubernetes cluster, which run_kubernetes_test
# applies imperatively (kubectl), so Chainsaw's auto-cleanup does not track it.
# On success run_kubernetes_test already tore it down, so this is a no-op then.
# This is the single authoritative child -> parent teardown on both success
# and failure, followed by the compute/storage reclamation barriers.
- description: tear down the tenant Kubernetes cluster
script:
timeout: 10m
# 30s diagnostic Certificate + 5m child + 5m parent + 5m compute/PVC
# drain + 5m LINSTOR reclamation = 20m30s of inner bounds. Keep the op
# above that sum so an inner timeout reports its stage instead of a
# Chainsaw SIGKILL hiding it; normal cleanup returns immediately.
timeout: 22m
content: |
set -eu
cd ../../..
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
cozy_cleanup
cozy_cleanup test-latest-version
---
# Version-independent regression guard for PR #2872 B1 (runs once, here, not
# in kubernetes-previous): with the `replicated` StorageClass absent on the
hack/e2e-chainsaw/kubernetes-oidc-customconfig/chainsaw-test.yaml
-117
View File
@@ -1,117 +0,0 @@
# Chainsaw port of hack/e2e-apps/kubernetes-oidc-customconfig.bats.
#
# Phase-1 OIDC CustomConfig selector e2e — render-side only. Pairs with the
# kubernetes-oidc-system suite; see its header for why no Kamaji bringup and
# no HR Ready wait (minReplicas: 0, HR asserted to exist only).
#
# What is exercised on a live cluster: spec.oidc.mode: CustomConfig is
# admitted; the HelmRelease renders; the tenant-supplied
# AuthenticationConfiguration is wired onto the KamajiControlPlane; and —
# crucially — NO Keycloak objects are created in the cozy realm when the
# tenant brings their own issuer, and the placeholder OIDC kubeconfig Secret
# is NOT pre-rendered (System mode pre-renders it; CustomConfig distributes
# out-of-band).
#
# The absence checks run AFTER the positive authn-config Secret assert
# establishes the chart has finished rendering — otherwise a not-yet-
# reconciled HR would false-pass them, a hazard the sequential bats never had.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: kubernetes-oidc-customconfig
spec:
catch:
- events: {}
- describe:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
name: kubernetes-oidc-byo
- describe:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
name: kubernetes-oidc-byo
steps:
- name: create-kubernetes-oidc-byo
try:
- apply:
file: kubernetes-oidc-byo.yaml
- assert:
timeout: 1m
resource:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: kubernetes-oidc-byo
- assert:
timeout: 3m
resource:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
name: kubernetes-oidc-byo
- name: kamajicontrolplane-still-carries-authentication-config
try:
- assert:
resource:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
name: kubernetes-oidc-byo
spec:
apiServer:
(contains(extraArgs, '--authentication-config=/etc/kubernetes/authentication-config/config.yaml')): true
- name: authn-config-secret-carries-tenant-issuer
try:
- assert:
timeout: 1m
resource:
apiVersion: v1
kind: Secret
metadata:
name: kubernetes-oidc-byo-oidc-authn-config
- script:
content: |
set -eu
body=$(kubectl -n "$NAMESPACE" get secret kubernetes-oidc-byo-oidc-authn-config \
-o jsonpath='{.data.config\.yaml}' | base64 -d)
echo "$body" | grep -qF "url: https://idp.byo.example.test"
echo "$body" | grep -qF "cozystack-byo-oidc-byo"
# Fail-loud negation (a bare `! ... | grep` is vacuous under set -e):
# the cozy-realm issuer must NOT appear in a BYO config.
if echo "$body" | grep -qE 'url: https://keycloak\.[^/]+/realms/cozy'; then
echo "FAIL: BYO AuthenticationConfiguration must not carry the cozy realm issuer" >&2
exit 1
fi
check:
($error == null): true
# Absence proofs — ordered after the positive asserts above (chart has
# rendered), so `error` op polling to its timeout is a real negative, not a
# not-yet-reconciled false pass.
- name: no-keycloak-objects-and-no-prerendered-kubeconfig
try:
- error:
timeout: 30s
resource:
apiVersion: v1.edp.epam.com/v1
kind: KeycloakClient
metadata:
name: tenant-test-kubernetes-oidc-byo
- error:
timeout: 30s
resource:
apiVersion: v1.edp.epam.com/v1
kind: KeycloakClientScope
metadata:
name: tenant-test-kubernetes-oidc-byo-audience
# System mode pre-renders <release>-oidc-kubeconfig as a placeholder;
# CustomConfig must not.
- error:
timeout: 30s
resource:
apiVersion: v1
kind: Secret
metadata:
name: kubernetes-oidc-byo-oidc-kubeconfig
hack/e2e-chainsaw/kubernetes-oidc-customconfig/kubernetes-oidc-byo.yaml
-48
View File
@@ -1,48 +0,0 @@
# No KubernetesNodes pool is created — this is a render-side test that only
# asserts on the KamajiControlPlane + the tenant-supplied issuer
# AuthenticationConfiguration Secret. Since the Phase 2 split the kubernetes
# chart renders the control plane only (worker pools are separate KubernetesNodes
# resources), so there are zero MachineDeployments, KubevirtMachines or worker
# DataVolumes — no DRBD churn between CI runs.
# (see #3231).
# Unlike the bats original, no talos.imageFactoryURL block is injected here:
# the helper emits it only when the in-sandbox Talos image cache is up, and
# with zero worker replicas this render-side test instantiates no DataVolume
# imports for it to redirect — the static fixture stays equivalent.
apiVersion: apps.cozystack.io/v1alpha1
kind: Kubernetes
metadata:
name: oidc-byo
spec:
addons:
ingressNginx:
enabled: false
hosts: []
valuesOverride: {}
controlPlane:
apiServer:
resources: {}
resourcesPreset: small
replicas: 1
version: v1.35
oidc:
mode: CustomConfig
customConfig:
config: |
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://idp.byo.example.test
audiences:
- cozystack-byo-oidc-byo
claimMappings:
username:
claim: preferred_username
prefix: ""
groups:
claim: groups
prefix: ""
users:
- email: byo-admin@example.test
role: admin
hack/e2e-chainsaw/kubernetes-oidc-system/chainsaw-test.yaml
-123
View File
@@ -1,123 +0,0 @@
# Chainsaw port of hack/e2e-apps/kubernetes-oidc-system.bats.
#
# Phase-1 OIDC System-mode e2e — render-side only. Pairs with the
# kubernetes-oidc-customconfig suite (BYO issuer path).
#
# Does NOT spin up a full Kamaji control plane: that path is covered by the
# kubernetes-latest/previous suites and would add another ~25 min bringup for
# a feature whose end-to-end OIDC browser flow is explicitly out of scope
# (see docs/oidc-tenant.md -> Phase 1). minReplicas: 0 means no worker VMs;
# the suite runs in minutes. Accordingly the HR is only asserted to EXIST
# (cozystack-api materialised it), never Ready.
#
# What is exercised on a live cluster: the new spec.oidc.* fields are
# admitted by the aggregated API; the HelmRelease renders the OIDC templates;
# and the resulting KamajiControlPlane / AuthenticationConfiguration Secret /
# KeycloakClient / KeycloakClientScope carry the expected per-cluster shape.
# The Keycloak asserts are unconditional: the chart capability-gates their
# rendering on v1.edp.epam.com/v1 (oidc-keycloak.yaml), and the CI platform
# always ships the operator — the bats' api-resources guard was dead code
# there (and its `skip` did not exist under cozytest anyway).
#
# The Kubernetes CR is applied declaratively, so Chainsaw auto-cleanup
# deletes it at test end — the bats' hand-rolled cleanup_kr/cozy_cleanup
# hooks have no equivalent here by construction.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: kubernetes-oidc-system
spec:
catch:
- events: {}
- describe:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
name: kubernetes-oidc-system
- describe:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
name: kubernetes-oidc-system
steps:
- name: create-kubernetes-oidc-system
try:
- apply:
file: kubernetes-oidc-system.yaml
# cozystack-api materialises the HelmRelease…
- assert:
timeout: 1m
resource:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: kubernetes-oidc-system
# …and the chart renders the KamajiControlPlane (no Ready wait — that
# would require the Kamaji bringup this suite deliberately avoids).
- assert:
timeout: 3m
resource:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
name: kubernetes-oidc-system
- name: kamajicontrolplane-oidc-wiring
try:
- assert:
resource:
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
name: kubernetes-oidc-system
spec:
apiServer:
# --authentication-config is appended to extraArgs; the
# chart-owned mount lands on extraVolumeMounts.
(contains(extraArgs, '--authentication-config=/etc/kubernetes/authentication-config/config.yaml')): true
(extraVolumeMounts[?name == 'authentication-config']):
- name: authentication-config
deployment:
# The chart-owned volume lands on extraVolumes.
(extraVolumes[?name == 'authentication-config']):
- name: authentication-config
- name: authn-config-secret
try:
- assert:
timeout: 1m
resource:
apiVersion: v1
kind: Secret
metadata:
name: kubernetes-oidc-system-oidc-authn-config
# The issuer check is a regex (any keycloak.<host>), so it stays a script;
# the per-cluster audience literal rides along.
- script:
content: |
set -eu
body=$(kubectl -n "$NAMESPACE" get secret kubernetes-oidc-system-oidc-authn-config \
-o jsonpath='{.data.config\.yaml}' | base64 -d)
echo "$body" | grep -qE 'url: https://keycloak\.[^/]+/realms/cozy'
echo "$body" | grep -qF -- "- tenant-test-kubernetes-oidc-system"
check:
($error == null): true
- name: keycloak-objects-in-cozy-realm
try:
- assert:
timeout: 1m
resource:
apiVersion: v1.edp.epam.com/v1
kind: KeycloakClient
metadata:
name: tenant-test-kubernetes-oidc-system
spec:
public: true
- assert:
timeout: 1m
resource:
apiVersion: v1.edp.epam.com/v1
kind: KeycloakClientScope
metadata:
name: tenant-test-kubernetes-oidc-system-audience
spec:
(protocolMappers[0].protocolMapper): oidc-audience-mapper
hack/e2e-chainsaw/kubernetes-oidc-system/kubernetes-oidc-system.yaml
-33
View File
@@ -1,33 +0,0 @@
# No KubernetesNodes pool is created — this is a render-side test that only
# asserts on the KamajiControlPlane + the cozy-realm Keycloak objects. Since the
# Phase 2 split the kubernetes chart renders the control plane only (worker
# pools are separate KubernetesNodes resources), so there are zero
# MachineDeployments, KubevirtMachines or worker DataVolumes — no DRBD churn
# overlapping the kubernetes-previous suite.
# Unlike the bats original, no talos.imageFactoryURL block is injected here:
# the helper emits it only when the in-sandbox Talos image cache is up, and
# with zero worker replicas this render-side test instantiates no DataVolume
# imports for it to redirect — the static fixture stays equivalent.
apiVersion: apps.cozystack.io/v1alpha1
kind: Kubernetes
metadata:
name: oidc-system
spec:
addons:
ingressNginx:
enabled: false
hosts: []
valuesOverride: {}
controlPlane:
apiServer:
resources: {}
resourcesPreset: small
replicas: 1
version: v1.35
oidc:
mode: System
users:
- email: e2e-admin@example.test
role: admin
- email: e2e-viewer@example.test
role: view
hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
+8 -4
View File
@@ -35,7 +35,6 @@ spec:
# (those alternate rather than run to completion in turn: each takes
# its first reading, one wait passes, each takes its second)
# guest Talos capture
# ghcr-mirror state, access log and warm-up Job
# talos-image-cache diagnosis
# The four collectors that bracket the node-join wait -- the sandbox
# kernel KVM counters, the runner kernel CPU time, the sandbox QEMU
@@ -78,12 +77,17 @@ spec:
# Always runs after try/catch — i.e. after the crust-gather snapshots on a
# failed run. Reclaims the tenant Kubernetes cluster, which run_kubernetes_test
# applies imperatively (kubectl), so Chainsaw's auto-cleanup does not track it.
# On success run_kubernetes_test already tore it down, so this is a no-op then.
# This is the single authoritative child -> parent teardown on both success
# and failure, followed by the compute/storage reclamation barriers.
- description: tear down the tenant Kubernetes cluster
script:
timeout: 10m
# 30s diagnostic Certificate + 5m child + 5m parent + 5m compute/PVC
# drain + 5m LINSTOR reclamation = 20m30s of inner bounds. Keep the op
# above that sum so an inner timeout reports its stage instead of a
# Chainsaw SIGKILL hiding it; normal cleanup returns immediately.
timeout: 22m
content: |
set -eu
cd ../../..
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
cozy_cleanup
cozy_cleanup test-previous-version
hack/e2e-chainsaw/mariadb/chainsaw-test.yaml
+1 -1
View File
@@ -290,7 +290,7 @@ spec:
set -eu
cd ../../..
[ -x examples/backups/mariadb/run-all.sh ] || { echo "mariadb backup example scripts not found at examples/backups/mariadb" >&2; exit 1; }
NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
COZY_E2E_BACKUP_PREFLIGHT=1 NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
examples/backups/mariadb/run-all.sh
check:
($error == null): true
hack/e2e-chainsaw/mongodb/chainsaw-test.yaml
+1 -1
View File
@@ -437,7 +437,7 @@ spec:
set -eu
cd ../../..
[ -x examples/backups/mongodb/run-all.sh ] || { echo "mongodb backup example scripts not found at examples/backups/mongodb" >&2; exit 1; }
NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
COZY_E2E_BACKUP_PREFLIGHT=1 NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
SKIP_RESTORE=1 examples/backups/mongodb/run-all.sh
check:
($error == null): true
hack/e2e-chainsaw/postgres/chainsaw-test.yaml
+1 -1
View File
@@ -326,7 +326,7 @@ spec:
set -eu
cd ../../..
[ -x examples/backups/postgres/run-all.sh ] || { echo "postgres backup example scripts not found at examples/backups/postgres" >&2; exit 1; }
NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
COZY_E2E_BACKUP_PREFLIGHT=1 NAMESPACE="$NAMESPACE" S3_ENDPOINT="https://seaweedfs-s3.tenant-root:8333" \
examples/backups/postgres/run-all.sh
check:
($error == null): true
hack/e2e-chainsaw/vminstance/vmdisk-vmi.yaml
+1 -1
View File
@@ -8,4 +8,4 @@ spec:
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
optical: false
storage: 5Gi
storageClass: replicated
storageClass: ($values.storageClass)
hack/e2e-chainsaw/vminstance/vmdisk.yaml
+1 -1
View File
@@ -8,4 +8,4 @@ spec:
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
optical: false
storage: 5Gi
storageClass: replicated
storageClass: ($values.storageClass)
hack/e2e-compose.yaml
+143
View File
@@ -0,0 +1,143 @@
# Talos nodes as containers, the L2 half of the e2e substrate experiment.
#
# Sits beside hack/e2e-prepare-cluster.bats' QEMU guests rather than replacing
# them. Addressing, CPU and memory deliberately mirror that file so everything
# downstream -- the MetalLB 192.168.123.200-250 pool, hack/e2e-post-install-prep.sh,
# the chainsaw suites -- is unchanged between the two lanes.
#
# NOT `talosctl cluster create docker`: that wrapper forces a single control
# plane (it has --workers but no --controlplanes), has no --disks, and hides the
# container spec. Compose keeps three control planes, as today.
#
# The contract below comes from siderolabs/talos
# pkg/provision/providers/docker/node.go: PLATFORM/TALOSSKU/USERDATA, privileged,
# seccomp:unconfined, read_only, and the tmpfs + volume layout. Two mounts are
# NOT part of that contract and are mandatory anyway -- see the per-mount notes.
name: cozy-e2e
networks:
talos:
driver: bridge
ipam:
config:
- subnet: 192.168.123.0/24
gateway: 192.168.123.1
x-node: &node
image: ghcr.io/siderolabs/talos:v1.13.5@sha256:0535cf71b65c8c9fdb0b745e697be174bfe7394ecac7ccb2621caf4a606fc5a6
privileged: true
security_opt:
- seccomp:unconfined
read_only: true
stop_grace_period: 30s
# Matches the QEMU lane's `-smp 8 -m 24576` so neither lane is given more of
# the runner than the other. Talos reports host capacity in container mode;
# e2e-container-up.sh counterbalances that with dynamic kubelet reservations,
# and e2e-prepare-cluster-container.bats asserts the resulting allocatable.
cpus: ${COZY_E2E_NODE_CPUS:-8}
mem_limit: ${COZY_E2E_NODE_MEMORY_MIB:-24576}m
tmpfs:
- /run
- /system
- /tmp
environment: &node-env
PLATFORM: container
TALOSSKU: ${COZY_E2E_NODE_CPUS:-8}CPU-${COZY_E2E_NODE_MEMORY_MIB:-24576}RAM
USERDATA: ${USERDATA:?USERDATA must be set; hack/e2e-container-up.sh generates it}
# The two host binds every node repeats below, and why each is mandatory:
#
# /dev:/dev Docker gives a privileged container its own 64 MB tmpfs /dev,
# seeded ONCE at creation. /dev/kvm works because it exists at
# start, but ZFS zvols created later never appear, and
# linstor-csi then fails ControllerPublishVolume with "could
# not determine device path". Binding the host devtmpfs fixes
# it, and is why `talosctl cluster create docker` can never
# support LINSTOR.
#
# /run/udev The LINSTOR satellite hostPath-mounts it. In a container /run
# is tmpfs with no udev, so the pod hangs Init:0/1 on
# "hostPath type check failed: /run/udev is not a directory".
#
# Written out per service rather than shared through a YAML anchor: compose
# merges mappings but not sequences, so an anchored list nests instead of
# flattening and is rejected as "volumes.0 must be a string".
services:
srv1:
<<: *node
hostname: srv1
container_name: srv1
environment:
<<: *node-env
networks:
talos:
ipv4_address: 192.168.123.11
volumes:
- /dev:/dev
- /run/udev:/run/udev
- srv1-var:/var
- srv1-state:/system/state
- srv1-cni:/etc/cni
- srv1-k8s:/etc/kubernetes
- srv1-kubelet:/usr/libexec/kubernetes
- srv1-opt:/opt
srv2:
<<: *node
hostname: srv2
container_name: srv2
environment:
<<: *node-env
networks:
talos:
ipv4_address: 192.168.123.12
volumes:
- /dev:/dev
- /run/udev:/run/udev
- srv2-var:/var
- srv2-state:/system/state
- srv2-cni:/etc/cni
- srv2-k8s:/etc/kubernetes
- srv2-kubelet:/usr/libexec/kubernetes
- srv2-opt:/opt
srv3:
<<: *node
hostname: srv3
container_name: srv3
environment:
<<: *node-env
networks:
talos:
ipv4_address: 192.168.123.13
volumes:
- /dev:/dev
- /run/udev:/run/udev
- srv3-var:/var
- srv3-state:/system/state
- srv3-cni:/etc/cni
- srv3-k8s:/etc/kubernetes
- srv3-kubelet:/usr/libexec/kubernetes
- srv3-opt:/opt
volumes:
srv1-var: {}
srv1-state: {}
srv1-cni: {}
srv1-k8s: {}
srv1-kubelet: {}
srv1-opt: {}
srv2-var: {}
srv2-state: {}
srv2-cni: {}
srv2-k8s: {}
srv2-kubelet: {}
srv2-opt: {}
srv3-var: {}
srv3-state: {}
srv3-cni: {}
srv3-k8s: {}
srv3-kubelet: {}
srv3-opt: {}
hack/e2e-container-up.sh
Executable
+307
View File
@@ -0,0 +1,307 @@
#!/bin/sh
# Bring up the container-lane substrate on the RUNNER HOST.
#
# The QEMU lane boots its guests inside the e2e sandbox. Containers cannot nest
# that way for free, so the Talos nodes are started as SIBLINGS on the runner's
# docker daemon and the sandbox is joined to their network. That keeps the host
# bind mounts in hack/e2e-compose.yaml resolving against the real host devtmpfs,
# which is the whole reason the lane can carry LINSTOR at all.
#
# Everything Talos-side (bootstrap, etcd, kubeconfig) stays in
# hack/e2e-prepare-cluster-container.bats, which runs inside the sandbox exactly
# as the QEMU lane's file does. This script owns only what must happen on the
# host and before the containers exist.
#
# POSIX sh with grep/find on purpose: this runs on CI runners and contributor
# machines, where rg/fd/bash-isms are not available.
set -eu
# kubernetes_memory_to_kib <quantity>
#
# Kubernetes canonicalizes binary memory quantities to the largest exact unit.
# A Node that previously reported Ki can therefore switch to Mi when kubelet
# reservations make the result an exact MiB value. Normalize the units used by
# Node status so the live capacity assertion validates the value, not its
# presentation.
kubernetes_memory_to_kib() {
cozy_memory_quantity=$1
case "$cozy_memory_quantity" in
*Ki)
cozy_memory_value=${cozy_memory_quantity%Ki}
cozy_memory_multiplier=1
;;
*Mi)
cozy_memory_value=${cozy_memory_quantity%Mi}
cozy_memory_multiplier=1024
;;
*Gi)
cozy_memory_value=${cozy_memory_quantity%Gi}
cozy_memory_multiplier=$(( 1024 * 1024 ))
;;
*)
echo "unsupported Kubernetes memory quantity: '$cozy_memory_quantity'" >&2
return 1
;;
esac
case "$cozy_memory_value" in
'' | *[!0-9]*)
echo "invalid Kubernetes memory quantity: '$cozy_memory_quantity'" >&2
return 1
;;
esac
printf '%s\n' "$(( cozy_memory_value * cozy_memory_multiplier ))"
}
# calculate_node_reservations <host-cpus> <host-memory-kib>
# <node-cpus> <node-memory-mib> <node-count>
#
# Talos container nodes read capacity from the shared host kernel, not from
# their Docker CPU/memory cgroups. Return the kubelet systemReserved quantities
# that reduce each node's scheduler-visible capacity to its container limit.
# Also reject a host that cannot hold all node limits with strict headroom,
# matching the QEMU lane's runner contract instead of relying on cgroup OOMs.
calculate_node_reservations() {
cozy_host_cpus=$1
cozy_host_memory_kib=$2
cozy_node_cpus=$3
cozy_node_memory_mib=$4
cozy_node_count=$5
for cozy_quantity in "$cozy_host_cpus" "$cozy_host_memory_kib" "$cozy_node_cpus" "$cozy_node_memory_mib" "$cozy_node_count"; do
case "$cozy_quantity" in
'' | *[!0-9]* | 0)
echo "all container capacity inputs must be positive integers, got '$cozy_quantity'" >&2
return 1
;;
esac
done
cozy_required_cpus=$(( cozy_node_cpus * cozy_node_count ))
cozy_required_memory_kib=$(( cozy_node_memory_mib * 1024 * cozy_node_count ))
if [ "$cozy_host_cpus" -le "$cozy_required_cpus" ]; then
echo "container lane needs more than ${cozy_required_cpus} host CPUs, found ${cozy_host_cpus}" >&2
return 1
fi
if [ "$cozy_host_memory_kib" -le "$cozy_required_memory_kib" ]; then
echo "container lane needs more than ${cozy_required_memory_kib} KiB host memory, found ${cozy_host_memory_kib} KiB" >&2
return 1
fi
printf '%sm\n' "$(( (cozy_host_cpus - cozy_node_cpus) * 1000 ))"
printf '%sKi\n' "$(( cozy_host_memory_kib - cozy_node_memory_mib * 1024 ))"
}
# Unit tests and the live capacity assertion source only the helpers above.
if [ "${E2E_CONTAINER_UP_LIB:-false}" = true ]; then
return 0 2>/dev/null || exit 0
fi
SANDBOX_NAME="${SANDBOX_NAME:?SANDBOX_NAME must be set (packages/core/testing/Makefile sets it)}"
# Resolved from this script's own location, not the caller's cwd: the Makefile
# target runs it from packages/core/testing, where a relative hack/ does not
# exist, while a contributor runs it from the repo root, where it does. Deriving
# the root here makes both work and keeps the compose project name, the file and
# the teardown in packages/core/testing/Makefile referring to the same thing.
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
REPO_ROOT=$(CDPATH= cd -- "${SCRIPT_DIR}/.." && pwd)
COMPOSE_FILE="${COMPOSE_FILE:-${REPO_ROOT}/hack/e2e-compose.yaml}"
COMPOSE_PROJECT="${COMPOSE_PROJECT:-cozy-e2e}"
# A smaller pool is not safe even when its sparse backing file fits physically:
# root-tenant local PVCs can leave a satellite with less than the ~21 GiB needed
# by CDI scratch. Once a worker's 20 GiB target PV is bound there, node affinity
# forces the importer and scratch volume onto that same full satellite. Match the
# QEMU lane's per-node 200 GB /dev/vdc so placement cannot turn node join into a
# storage-capacity lottery.
ZPOOL_SIZE="${ZPOOL_SIZE:-200G}"
ZPOOL_BACKING_DIR="${ZPOOL_BACKING_DIR:-/var/lib/cozy-e2e-zpools}"
KUBERNETES_VERSION="${KUBERNETES_VERSION:-v1.33.12}"
COZY_E2E_NODE_CPUS="${COZY_E2E_NODE_CPUS:-8}"
COZY_E2E_NODE_MEMORY_MIB="${COZY_E2E_NODE_MEMORY_MIB:-24576}"
export COZY_E2E_NODE_CPUS COZY_E2E_NODE_MEMORY_MIB
log() { echo "[e2e-container-up] $*"; }
die() { echo "[e2e-container-up] FATAL: $*" >&2; exit 1; }
# modprobe and zpool need root; docker does not, and must NOT be run through
# sudo, or the compose project ends up owned by a different user than the
# teardown in packages/core/testing/Makefile runs as. So escalate per command
# rather than re-exec'ing the whole script. Empty when already root, which is
# how this runs on a dev box; `sudo` on a CI runner, which runs as an
# unprivileged user with passwordless sudo.
SUDO=""
if [ "$(id -u)" -ne 0 ]; then
command -v sudo >/dev/null 2>&1 \
|| die "not root and no sudo available; this script needs to load kernel
modules and create zpools on the host."
SUDO="sudo"
fi
HOST_CPUS=$(getconf _NPROCESSORS_ONLN)
HOST_MEMORY_KIB=$(awk '/^MemTotal:/ {print $2; exit}' /proc/meminfo)
NODE_RESERVATIONS=$(calculate_node_reservations \
"$HOST_CPUS" "$HOST_MEMORY_KIB" \
"$COZY_E2E_NODE_CPUS" "$COZY_E2E_NODE_MEMORY_MIB" 3) \
|| die "the runner cannot safely host three configured Talos containers"
SYSTEM_RESERVED_CPU=$(printf '%s\n' "$NODE_RESERVATIONS" | sed -n '1p')
SYSTEM_RESERVED_MEMORY=$(printf '%s\n' "$NODE_RESERVATIONS" | sed -n '2p')
log "node limit ${COZY_E2E_NODE_CPUS} CPU/${COZY_E2E_NODE_MEMORY_MIB} MiB; kubelet systemReserved ${SYSTEM_RESERVED_CPU}/${SYSTEM_RESERVED_MEMORY} against host ${HOST_CPUS} CPU/${HOST_MEMORY_KIB} KiB"
# ---------------------------------------------------------------------------
# 1. Host kernel modules.
#
# machine.kernel.modules is a SILENT no-op in container mode --
# internal/app/machined/pkg/controllers/runtime/kernel_module_spec.go returns
# early on ModeContainer with no error and no event -- so a module the QEMU lane
# declares in its machine config simply never loads here. Asserting it now makes
# absence fail at provisioning instead of surfacing an hour later as a storage
# or CNI error with no obvious cause.
# ---------------------------------------------------------------------------
for mod in openvswitch zfs; do
if ! $SUDO modprobe "$mod" 2>/dev/null; then
log "modprobe ${mod} returned non-zero; verifying the loaded-module state"
fi
if ! grep -q "^${mod} " /proc/modules; then
die "kernel module '${mod}' is not loaded on the host.
The container lane takes its kernel from the host and Talos cannot load
modules in container mode, so this must be loaded before the nodes start.
openvswitch ships with the distro kernel, so 'modprobe openvswitch' is
usually enough; zfs needs the OpenZFS packages first (on Ubuntu,
'apt-get install zfsutils-linux', then 'modprobe zfs')."
fi
log "host module ${mod}: loaded"
done
# ---------------------------------------------------------------------------
# 2. Per-node ZFS pools.
#
# zpools are GLOBAL PER KERNEL, so every node sees every pool. They must be
# node-distinct and each satellite pinned to its own; nothing enforces the
# pinning. Acceptable for CI, not a production pattern.
# ---------------------------------------------------------------------------
$SUDO mkdir -p "$ZPOOL_BACKING_DIR"
for n in 1 2 3; do
if $SUDO zpool list "data-srv${n}" >/dev/null 2>&1; then
log "zpool data-srv${n}: already present"
continue
fi
$SUDO truncate -s "$ZPOOL_SIZE" "${ZPOOL_BACKING_DIR}/srv${n}.img"
$SUDO zpool create -f "data-srv${n}" "${ZPOOL_BACKING_DIR}/srv${n}.img"
log "zpool data-srv${n}: created (${ZPOOL_SIZE})"
done
# ---------------------------------------------------------------------------
# 3. Machine config, generated inside the sandbox so it uses the SAME pinned
# talosctl the QEMU lane does.
#
# Differences from the QEMU lane's patches, each forced by container mode:
# * no machine.kernel.modules -- silent no-op, see above
# * no /etc/lvm/lvm.conf filter -- no LVM in play
# * no ARP VIP -- unavailable in container mode, so the
# apiServerEndpoint is a node IP
# Everything else is deliberately identical.
# ---------------------------------------------------------------------------
docker exec \
-e COZY_E2E_SYSTEM_RESERVED_CPU="$SYSTEM_RESERVED_CPU" \
-e COZY_E2E_SYSTEM_RESERVED_MEMORY="$SYSTEM_RESERVED_MEMORY" \
"$SANDBOX_NAME" sh -ec '
cd /workspace
cat > container-patch.yaml <<PATCH
machine:
kubelet:
nodeIP:
validSubnets:
- 192.168.123.0/24
extraConfig:
maxPods: 512
systemReserved:
cpu: "${COZY_E2E_SYSTEM_RESERVED_CPU}"
memory: "${COZY_E2E_SYSTEM_RESERVED_MEMORY}"
registries:
mirrors:
docker.io:
endpoints:
- https://mirror.gcr.io
files:
- content: |
[plugins]
[plugins."io.containerd.cri.v1.runtime"]
device_ownership_from_security_context = true
path: /etc/cri/conf.d/20-customization.part
op: create
cluster:
apiServer:
extraArgs:
oidc-issuer-url: "https://keycloak.example.org/realms/cozy"
oidc-client-id: "kubernetes"
oidc-username-claim: "preferred_username"
oidc-groups-claim: "groups"
network:
cni:
name: none
dnsDomain: cozy.local
podSubnets:
- 100.65.0.0/16
serviceSubnets:
- 10.96.0.0/16
PATCH
cat > container-patch-controlplane.yaml <<PATCH
machine:
nodeLabels:
node.kubernetes.io/exclude-from-external-load-balancers:
\$patch: delete
cluster:
allowSchedulingOnControlPlanes: true
controllerManager:
extraArgs:
bind-address: 0.0.0.0
scheduler:
extraArgs:
bind-address: 0.0.0.0
apiServer:
certSANs:
- 127.0.0.1
- 192.168.123.11
- 192.168.123.12
- 192.168.123.13
proxy:
disabled: true
discovery:
enabled: false
etcd:
advertisedSubnets:
- 192.168.123.0/24
PATCH
[ -f secrets.yaml ] || talosctl gen secrets
rm -f controlplane.yaml worker.yaml talosconfig kubeconfig
talosctl gen config --with-secrets secrets.yaml cozystack https://192.168.123.11:6443 \
--kubernetes-version '"$KUBERNETES_VERSION"' \
--output-types controlplane,talosconfig -o . --force \
--config-patch @container-patch.yaml \
--config-patch-control-plane @container-patch-controlplane.yaml
'
log "machine config generated (kubernetes ${KUBERNETES_VERSION})"
USERDATA=$(docker exec "$SANDBOX_NAME" base64 -w0 /workspace/controlplane.yaml)
export USERDATA
[ -n "$USERDATA" ] || die "USERDATA is empty; talosctl gen config produced nothing"
# ---------------------------------------------------------------------------
# 4. Start the nodes and put the sandbox on their network.
# ---------------------------------------------------------------------------
if ! docker compose -p "$COMPOSE_PROJECT" -f "$COMPOSE_FILE" down -v; then
die "failed to remove a stale ${COMPOSE_PROJECT} compose project before startup"
fi
docker compose -p "$COMPOSE_PROJECT" -f "$COMPOSE_FILE" up -d
log "three Talos containers started"
NETWORK="${COMPOSE_PROJECT}_talos"
if docker network inspect "$NETWORK" -f '{{range $k,$v := .Containers}}{{$v.Name}} {{end}}' | grep -q "\b${SANDBOX_NAME}\b"; then
log "sandbox already attached to ${NETWORK}"
else
docker network connect "$NETWORK" "$SANDBOX_NAME"
log "sandbox attached to ${NETWORK}"
fi
log "substrate up; hack/e2e-prepare-cluster-container.bats takes it from here"
hack/e2e-ghcr-mirror.yaml
-195
View File
@@ -1,195 +0,0 @@
# In-sandbox pull-through registry mirror for ghcr.io (tenant worker image pulls).
#
# Why this exists: tenant Kubernetes worker Talos nodes pull their kubelet image
# `ghcr.io/siderolabs/kubelet:<ver>` directly from public ghcr.io. That egress is
# intermittently flaky/rate-limited from the CI runner -- the pull hits
# `net/http: TLS handshake timeout host=ghcr.io`, the Talos kubelet service never
# starts, the worker never registers, and the `kubernetes-*` e2e tests fail with
# zero tenant nodes inside the 18-minute node-join budget. Diagnosed in
# cozystack/cozystack#3548 (in-guest Talos dmesg) and tracked by #3513. This is the
# same flaky-public-egress class as the Talos OS image (#3231 / hack/e2e-talos-image-cache.yaml).
#
# This addresses ONE variant of #3513, and the issue stays open past it. The same
# thread records failures that issue no apid CSR at all inside the whole budget --
# time lost before Talos gets anywhere near the kubelet image -- which no registry
# mirror can touch. A green suite after this lands is not evidence the flake is
# closed.
#
# This Deployment runs a standard registry:2 configured as a PULL-THROUGH cache for
# ghcr.io: worker pulls are routed here by the chart's `talos.registryMirrors` knob
# (machine.registries.mirrors), the cache fetches from ghcr.io ONCE per blob and
# serves every subsequent worker locally over plain HTTP, so the many per-worker
# pulls no longer each depend on live public egress. The retry around a failed
# fetch is containerd's, in the guest -- distribution 2.8's proxy blobstore has
# none of its own, so a first fetch that fails is simply a failed pull.
#
# CI band-aid, not a product artifact. The durable knob is the chart's
# `talos.registryMirrors` this manifest exercises; a production operator points it at
# their own mirror/pull-through registry.
#
# ASSUMPTIONS THAT NEED A CI RUN TO CONFIRM:
# 1. Talos honours a plain-http mirror endpoint for ghcr.io via
# machine.registries.mirrors[ghcr.io].endpoints=[http://...]. If a scheme/TLS
# knob is additionally required, add machine.registries.config here too.
# 2. The containerd pull from inside the worker VM leaves with the virt-launcher
# Pod's own Cilium identity, which is what the policy below selects. The label
# half is settled -- run-kubernetes.sh already reads tenant worker Pods by
# `kubevirt.io=virt-launcher` -- but that the pull is subject to that Pod's
# egress policy at all is what a run has to show.
#
# Not an assumption, though it looks like one: the guest resolving the short name
# `ghcr-mirror.kube-system.svc`. The worker machineconfig this repo renders sets
# machine.network.searchDomains to svc.<mgmtClusterDomain> and <mgmtClusterDomain>
# (see the nameservers block in the reconcile Job), and the same guest already
# depends on that list to reach `linstor-csi-nfs.cozy-linstor.svc` for NFS mounts.
# The mechanism is in-tree and load-bearing elsewhere. If a run ever shows
# otherwise, point GHCR_MIRROR_SVC_URL at the FQDN.
#
# What the registry:2 image itself no longer assumes: it is digest-pinned on
# mirror.gcr.io below, and that exact digest was checked to run under the security
# context here (non-root 65532, read-only rootfs, dropped caps) and to serve
# ghcr.io/siderolabs/kubelet manifests and blobs as an anonymous pull-through. It is
# deliberately NOT in the pre-pull set: hack/e2e-install-cozystack.bats applies this
# manifest before the pre-pull DaemonSet runs, so adding it there would cache it
# after the pulls that matter. Two Pods now pull it in that window -- this
# Deployment and the warm-up Job the same install step starts -- and the Job carries
# no affinity, so it may land on a second node and pull it again.
apiVersion: v1
kind: Service
metadata:
name: ghcr-mirror
namespace: kube-system
labels:
app.kubernetes.io/name: ghcr-mirror
spec:
selector:
app.kubernetes.io/name: ghcr-mirror
ports:
- name: http
port: 80
targetPort: registry
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ghcr-mirror
namespace: kube-system
labels:
app.kubernetes.io/name: ghcr-mirror
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ghcr-mirror
strategy:
type: Recreate
template:
metadata:
labels:
app.kubernetes.io/name: ghcr-mirror
spec:
# No API access is needed; do not mount a ServiceAccount token.
automountServiceAccountToken: false
# Run unprivileged and PSS-restricted. fsGroup makes the emptyDir cache and
# tmp volumes group-writable so the non-root user can write them under a
# read-only root filesystem.
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: registry
# registry:2 as a pull-through cache. Pulled from mirror.gcr.io and
# digest-pinned (as every other hack/ helper manifest does) so the
# mirror's own image is not a fresh anonymous Docker Hub pull -- the very
# flaky-egress dependency this PR exists to remove.
image: mirror.gcr.io/library/registry:2.8.3@sha256:a3d8aaa63ed8681a604f1dea0aa03f100d5895b6a58ace528858a7b332415373
env:
# Pull-through cache: proxy every request to ghcr.io and cache locally.
- name: REGISTRY_PROXY_REMOTEURL
value: https://ghcr.io
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
ports:
- name: registry
containerPort: 5000
readinessProbe:
# /v2/ returns 200 once the registry is serving.
httpGet:
path: /v2/
port: registry
initialDelaySeconds: 2
periodSeconds: 5
resources:
requests:
cpu: 50m
memory: 64Mi
ephemeral-storage: 100Mi
limits:
memory: 512Mi
# Bounds the container writable layer plus the cache/tmp emptyDirs
# below, so a runaway cache cannot fill node-local storage and evict
# unrelated Pods. Sized for the CI image set (kubelet + a few).
ephemeral-storage: 3Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
volumeMounts:
- name: cache
mountPath: /var/lib/registry
# readOnlyRootFilesystem: give the registry a writable scratch dir for
# any temp it touches outside its storage rootdirectory.
- name: tmp
mountPath: /tmp
volumes:
- name: cache
emptyDir:
sizeLimit: 2Gi
- name: tmp
emptyDir:
sizeLimit: 256Mi
---
# CI-only: allow tenant worker VM Pods to reach the mirror.
#
# Tenant namespaces run under a default-deny Cilium egress installed by the tenant
# chart whose union allows same-namespace endpoints, the tenant tree and the `world`
# entity (so public ghcr.io works, just flakily) but NOT an arbitrary kube-system
# Service -- so a worker's containerd resolves ghcr-mirror.kube-system.svc yet its
# TCP connect to the ClusterIP is silently dropped. This adds one tightly-scoped
# allow: worker VM (virt-launcher) Pods in tenant-test may egress to the mirror Pod.
# Mirrors the e2e-talos-image-cache-importer-egress policy pattern, but selects the
# worker VM identity (kubevirt.io: virt-launcher) that actually performs the
# kubelet-image pull, not the CDI importer.
#
# The helper commits the mirror endpoint once the API accepts this object, which is
# not the same as the flow being allowed: if the selector misses the identity the
# in-guest pull really carries, the SYNs are dropped and every worker pull pays a
# connect timeout before Talos falls back. Assumption 2 above is exactly that gap.
#
# Applied at point-of-use by hack/e2e-chainsaw/_lib/ghcr-mirror.sh (after Cilium's
# CRDs exist), not with the rest of this manifest at install time.
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: e2e-ghcr-mirror-worker-egress
labels:
app.kubernetes.io/name: ghcr-mirror
spec:
endpointSelector:
matchLabels:
"k8s:kubevirt.io": virt-launcher
"k8s:io.kubernetes.pod.namespace": tenant-test
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
"k8s:app.kubernetes.io/name": ghcr-mirror
hack/e2e-install-cozystack.bats
+65 -109
View File
@@ -69,56 +69,6 @@
fi
}
@test "Deploy ghcr.io pull-through mirror (best-effort e2e mirror)" {
# Tenant Kubernetes worker Talos nodes pull ghcr.io/siderolabs/kubelet straight
# from public ghcr.io. That egress is intermittently rate-limited from the CI
# runner, the pull dies on a TLS handshake timeout, the kubelet service never
# starts and no tenant node joins (cozystack/cozystack#3513). This Deployment
# caches ghcr.io locally; tenant CRs point spec.talos.registryMirrors at its
# Service. Readiness is gated at point-of-use in
# hack/e2e-chainsaw/_lib/ghcr-mirror.sh, which falls back to direct pulls.
#
# Its own @test, not folded into the Talos image factory cache above: that step
# returns early when it cannot read talos.schematicID/version, and this manifest
# substitutes no placeholders and needs neither value. Sharing a block would let
# a values.yaml restructure silently stop deploying this mirror.
#
# The CiliumClusterwideNetworkPolicy in the manifest is deferred to point-of-use
# for the same reason as above: Cilium's CRDs do not exist until Cozystack is
# installed.
yq 'select(.kind != "CiliumClusterwideNetworkPolicy")' hack/e2e-ghcr-mirror.yaml \
| kubectl apply -f - || echo "WARNING: failed to apply ghcr-mirror (fallback to direct ghcr.io pulls)"
if kubectl -n kube-system get deploy ghcr-mirror >/dev/null 2>&1; then
echo "ghcr-mirror Deployment created (pull-through cache for ghcr.io)"
else
echo "WARNING: ghcr-mirror Deployment NOT created — tenant workers will pull ghcr.io directly"
fi
# Start filling the cache now, not when the first tenant worker asks for it.
# An empty pull-through cache makes the first pull a live fetch from ghcr.io, and
# that leg is what the tenant worker cannot afford: measured at 40-165 KB/s for a
# 61 MiB image, with some blob streams dying mid-transfer. At the low end that
# single pull runs about twenty-six minutes, and it is one leg of a join that
# also has to boot Talos and get a CNI onto the node inside the same node-Ready
# budget. Deliberately not stated as a fraction of that budget: the figure is a
# measured transfer rate and stays true whatever the deadline is set to, while a
# ratio would have to be re-derived every time the deadline moves and would read
# as comfortable the moment it did. The kubernetes suites start tens of minutes
# after this step, so a background Job has time to finish.
#
# What is guaranteed: a warm-up that fails, times out or never starts does not
# fail this step, and the suites behave as they did before it existed.
#
# What is NOT guaranteed, because it would be untrue: that this is free. The Job
# pulls the two selected kubelet tags across the same throttled egress the rest of
# the install and the pre-pull step below are using, and if a worker cannot reach
# the mirror (assumption 2 in hack/e2e-ghcr-mirror.yaml) Talos falls back to public
# ghcr.io and the two then compete for it. That is why the warm set is the suites'
# two minors (~61 MiB each, one platform) rather than the whole version map, which
# would be ~300 MiB at the same one-platform walk.
. hack/e2e-chainsaw/_lib/ghcr-mirror.sh
warm_ghcr_mirror || true
}
@test "Required installer chart exists" {
if [ ! -f packages/core/installer/Chart.yaml ]; then
echo "Missing: packages/core/installer/Chart.yaml" >&2
@@ -137,6 +87,13 @@
# actually pulls (skips configmap fields and CRD examples that happen to
# contain an `image:` key). Add a chart here when a new peer-sensitive
# workload is found.
#
# `select(. != null)` drops container entries that carry no image. The pod
# templates embedded in a LinstorCluster CR are strategic-merge fragments,
# so one that overrides only `args` has no image of its own to pull.
# Without the filter yq emits a bare `null`, which reaches the prepull
# DaemonSet as a container with an empty image and fails apply with a
# message naming a container index rather than the chart behind it.
# Stage each render AND the yq filter through tmp files instead of
# piping. Two constraints stack here: `set -x` would expand any
# `var=$(helm ...)` capture into the trace and balloon CI logs, and
@@ -154,8 +111,8 @@
helm template packages/system/linstor > "$linstor_yaml"
helm template packages/system/cert-manager > "$certmanager_yaml"
yq -N '
(..|select(has("containers"))|.containers[]|.image),
(..|select(has("initContainers"))|.initContainers[]|.image)
(..|select(has("containers"))|.containers[]|.image|select(. != null)),
(..|select(has("initContainers"))|.initContainers[]|.image|select(. != null))
' "$kubeovn_yaml" "$linstor_yaml" "$certmanager_yaml" > "$images_list"
# The failure-only Talos diagnostics Pod can land on any node. The sandbox's
# upstream talosctl binary is static, so cache its pinned Alpine base on every
@@ -224,72 +181,38 @@
# Wait for operator to create the platform PackageSource
timeout 120 sh -ec 'until kubectl get packagesource cozystack.cozystack-platform >/dev/null 2>&1; do sleep 2; done'
# Create platform Package with isp-full variant
kubectl apply -f - <<EOF
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
name: cozystack.cozystack-platform
spec:
variant: isp-full
components:
platform:
values:
networking:
podCIDR: "10.244.0.0/16"
podGateway: "10.244.0.1"
serviceCIDR: "10.96.0.0/16"
joinCIDR: "100.64.0.0/16"
publishing:
host: "example.org"
apiServerEndpoint: "https://192.168.123.10:6443"
bundles:
enabledPackages:
- cozystack.external-dns-application
EOF
# Stage the generated manifest instead of piping it into kubectl: cozytest
# runs under /bin/sh without pipefail, so a generator failure in a pipeline
# could be hidden by kubectl successfully applying empty input.
local platform_packages
platform_packages=$(mktemp)
hack/e2e-platform-packages.sh > "$platform_packages"
kubectl apply -f "$platform_packages"
rm -f "$platform_packages"
# Launch storage + LB configuration in the background. It waits for its
# own prerequisites (linstor-controller deploy, MetalLB CRDs) and finishes
# while the parallel HR wait below is still running, so the cost overlaps
# with the platform reconcile instead of compounding it.
# Launch storage + LB configuration in the background. It waits for its own
# prerequisites while the platform controllers reconcile HelmReleases, so
# the costs overlap even though the authoritative readiness gate starts only
# after prep has finished.
hack/e2e-post-install-prep.sh > /tmp/post-install-prep.log 2>&1 &
POST_PREP_PID=$!
# Wait until HelmReleases appear & reconcile them
timeout 180 sh -ec 'until [ $(kubectl get hr -A --no-headers 2>/dev/null | wc -l) -gt 10 ]; do sleep 1; done'
# TODO(e2e-replace-fixed-timeouts): genuine sleep. The threshold of 10 is a
# heuristic for "enough HRs visible to start waiting"; the snapshot below
# uses whatever HRs have appeared by then. There is no objective k8s API
# signal for "all platform HRs have been emitted" without hard-coding the
# expected list, so the 5s pad lets a few late-arrivals join the snapshot.
sleep 5
# Pacing only: names every HR that timed out in the trace; the authoritative
# gate re-lists below, covering HRs created after this snapshot (#2822).
kubectl wait hr --all -A --timeout=15m --for=condition=ready || true
echo "Waiting for post-install-prep to complete"
if ! wait $POST_PREP_PID; then
if ! wait "$POST_PREP_PID"; then
cat /tmp/post-install-prep.log >&2
echo "post-install-prep failed" >&2
exit 1
fi
cat /tmp/post-install-prep.log
# Fail the test if any HelmRelease is not Ready. Wait again on a fresh
# listing so HelmReleases created after the snapshot above are gated too;
# the window absorbs momentary Unknown flaps from drift reconciles.
if ! kubectl wait hr --all -A --timeout=15m --for=condition=ready; then
kubectl get hr -A || true
# kubectl's STATUS column truncates long messages; dump the full Ready
# condition per non-ready HR so the real error (e.g. a rejected CRD) is
# visible in the test output instead of only inside the cozyreport.
kubectl get hr -A --no-headers | awk '$4 != "True"' | while read -r ns name _; do
echo "--- Non-ready HelmRelease: $ns/$name" >&2
kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2 || true
done
echo "Some HelmReleases failed to reconcile" >&2
exit 1
fi
# One authoritative gate, on fresh lists. A bare `kubectl wait --all` takes
# one snapshot and can miss releases created a moment later; it also waits
# the full timeout after Flux has already declared Stalled=True. The helper
# re-lists, requires the all-Ready name set to stay stable for 5s, and fails
# immediately on Stalled. The 11-release floor preserves the old >10
# existence backstop without the masked pacing wait.
. hack/e2e-wait-helmreleases.sh
cozy_wait_all_helmreleases_ready 900 11 5 2
}
@test "Wait for Cluster‑API provider deployments" {
@@ -312,6 +235,39 @@ EOF
# wait in the previous test does not cover that. Prove an AUTHENTICATED
# request against the actual resource succeeds before the patch's own GET.
timeout 120 sh -ec 'until kubectl get tenants.apps.cozystack.io root -n tenant-root >/dev/null 2>&1; do sleep 2; done'
local storage_class
storage_class="${COZY_E2E_STORAGE_CLASS:-replicated}"
case "$storage_class" in
replicated) ;;
local)
# VictoriaLogs is the only root-tenant install workload that explicitly
# defaults to replicated instead of inheriting the cluster's default
# StorageClass. The container lane has no DRBD and intentionally exposes
# only local. Put the override in the values Secret BEFORE monitoring is
# enabled, so the monitoring HelmRelease's first render creates local
# claims. Patching that HelmRelease after creation races the operator: a
# replicated PVC can be created first, and storageClassName is immutable.
# Flux drift detection is disabled, so the owning cozystack-basics release
# does not rewrite this E2E-only live override on its ordinary interval.
local root_values_b64_file root_values_file root_values_patch_file
root_values_b64_file=$(mktemp)
root_values_file=$(mktemp)
root_values_patch_file=$(mktemp)
kubectl get secret cozystack-values -n tenant-root -o jsonpath='{.data.values\.yaml}' > "$root_values_b64_file"
base64 -d "$root_values_b64_file" > "$root_values_file"
COZY_E2E_STORAGE_CLASS="$storage_class" yq -i \
'.logsStorages = [{"name":"generic","retentionPeriod":"1","storage":"10Gi","storageClassName":strenv(COZY_E2E_STORAGE_CLASS)}]' \
"$root_values_file"
jq -Rs '{"stringData":{"values.yaml":.}}' < "$root_values_file" > "$root_values_patch_file"
kubectl patch secret cozystack-values -n tenant-root --type merge --patch-file "$root_values_patch_file"
rm -f "$root_values_b64_file" "$root_values_file" "$root_values_patch_file"
;;
*)
echo "COZY_E2E_STORAGE_CLASS must be local or replicated, got '$storage_class'" >&2
return 1
;;
esac
kubectl patch tenants/root -n tenant-root --type merge -p '{"spec":{"host":"example.org","ingress":true,"monitoring":true,"etcd":true,"isolated":true, "seaweedfs": true}}'
timeout 60 sh -ec 'until kubectl get hr -n tenant-root etcd ingress monitoring seaweedfs tenant-root >/dev/null 2>&1; do sleep 1; done'
@@ -430,8 +386,8 @@ EOF
# Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy,
# so the dashboard is the consumer that proves the internal-URL default works.
# The install-time `kubectl wait hr --all -A` ran before this test flipped the
# flag and only ever saw the token-proxy shape, so nothing has re-checked the
# The install-time all-HelmRelease gate ran before this test flipped the flag
# and only ever saw the token-proxy shape, so nothing has re-checked the
# dashboard on the OIDC path.
#
# Waiting on hr/dashboard directly would be vacuous: it is still Ready=True
hack/e2e-node-join-soft-red.sh
-146
View File
@@ -1,146 +0,0 @@
#!/bin/sh
# Decide whether a failed Chainsaw run may leave the job green.
#
# Exactly one e2e failure is tolerated: the tenant node-join deadline in
# hack/e2e-chainsaw/_lib/run-kubernetes.sh. That wait stopped discriminating --
# nested virtualisation on the shared runners degrades far enough that a worker
# which registers in minutes elsewhere can miss any deadline the suite can
# afford -- so a red there says as much about the machine as about the product,
# and what it produces is re-runs.
#
# The softening lives HERE, one layer above the suite, and not in the suite,
# which is the whole design. The test still fails: Chainsaw records a failure,
# its catch runs in full (previous-instance container logs, the host snapshot,
# the data-plane capture, the event dump) and the JUnit report says so. A
# softening inside the script would have bought a green check by throwing all of
# that away, because every one of those collectors is keyed on the test failing.
#
# What changes is the lane's verdict -- and, with it, anything the lane gates on
# a failed job. Such a step stops firing on a softened run unless the lane hands
# it the decision separately, which is a coupling to look for rather than to
# assume; the guard file below holds the one that exists today.
#
# The discriminator is a marker file the failing test writes for exactly this
# purpose, never an inference about which suite failed or what its log said. A
# marker is written only on the deadline expiring -- not on the wait failing to
# run, and not on any assertion after the join.
#
# Every branch that cannot answer the question answers "block". A report that
# cannot be read, a failed test the report does not name, a failed test whose
# name no marker matches, a run with no failures recorded at all: each of those
# is a red this must not tolerate, because the one thing worse than a flaky gate
# is a gate that opens when it does not understand what it is looking at. That
# holds branch by branch and is not a claim that every shape has been thought of;
# what makes it hold is that reaching the tolerating exit takes a name, a lookup
# and an answer, so a name that goes missing anywhere between the report and the
# lookup ends up in the unexplained list rather than out of the accounting.
#
# Not every lane that runs the suite asks this. Which ones do, and why the rest
# stay hard, is held in hack/node-join-soft-red_test.bats.
#
# Usage: hack/e2e-node-join-soft-red.sh <sandbox-container>
# exit 0 every recorded failure is a node-join deadline; the lane may pass
# exit 1 something else failed, or the question could not be answered
set -eu
SANDBOX=${1:?usage: e2e-node-join-soft-red.sh <sandbox-container>}
# Both paths are the ones the sandbox itself uses: the report name comes from
# `report.name` in hack/e2e-chainsaw/.chainsaw.yaml and lands beside the suites,
# and the snapshot root is spelled the way _lib/run-kubernetes.sh spells it.
#
# The override in that expression is read in the SANDBOX's environment, not the
# runner's: `docker exec` starts a process with the container's environment and
# passes none of its own, so a COZY_REPORT_DIR exported beside this script has
# no effect here at all. Nothing in this repository exports it, so both sides
# take the default and agree by default rather than by coupling.
REPORT=/workspace/hack/e2e-chainsaw/chainsaw-report.xml
SNAPSHOT_ROOT_EXPR='${COZY_REPORT_DIR:-/workspace/_out/cozyreport}/snapshots'
xml=$(docker exec "$SANDBOX" sh -c "cat '$REPORT'" 2>/dev/null) || xml=
if [ -z "$xml" ]; then
echo "the chainsaw report is missing or empty, so which tests failed is unknown; treating this run as a failure" >&2
exit 1
fi
# One record per testcase, so a failure element can only be attributed to the
# testcase it is nested in. Splitting on the element rather than matching a line
# is what keeps a `failures="N"` attribute on the enclosing testsuite -- which
# carries no `<` -- from reading as a failed case.
#
# The name is how the marker below is looked up, so a failed case this cannot
# name is a failure it cannot attribute. Those are counted rather than skipped,
# and a tally that does not match the names collected ends the run here: without
# it an unnamed or empty-named failure drops silently out of the list and the
# gate opens on a failure it never read.
#
# The name is read out of the element's opening tag and NOT out of the record,
# which is the difference between attributing a failure and guessing at one. A
# record runs to the NEXT `<testcase`, so it carries the `</testsuite>` that
# closes this suite and the `<testsuite name="...">` that opens the following
# one; searching all of it lets an unnamed case inherit its neighbour's suite
# name, and the tally above then sees a case that is "named" and says nothing. A
# record with no `>` at all yields an empty head, no match, and a block.
#
# Inside that head the leading space is load-bearing for the same class of
# reason: `classname="..."` ends in `name="..."`, so without it a case carrying a
# classname and no name reports the classname as the test to look a marker up by.
failed=$(printf '%s\n' "$xml" | awk '
BEGIN { RS = "<testcase" }
NR == 1 { next }
$0 ~ /<(failure|error)[ >\/]/ {
seen++
head = substr($0, 1, index($0, ">"))
if (match(head, / name="[^"]+"/)) {
print substr(head, RSTART + 7, RLENGTH - 8)
named++
}
}
END { if (seen != named) exit 3 }
') || {
echo "the chainsaw report records a failed test it does not name, so the marker that would explain it cannot be looked up; that is a failure this cannot attribute, so it blocks" >&2
exit 1
}
if [ -z "$failed" ]; then
echo "the chainsaw run did not pass, yet its report records no failed test; that is a failure this cannot attribute, so it blocks" >&2
exit 1
fi
# One name per line, read as a line rather than split into words. The report is
# input to this decision, not something it may assume the shape of, and word
# splitting silently DROPS a name made only of blanks: the loop body would not
# run for it, no marker would be looked up, and a failure nobody examined would
# leave the run green. Read as a line it survives, finds no marker, and blocks.
# Splitting also expands a name carrying `*` or `?` against the runner's working
# directory, which turns one unexplained failure into a handful of filenames.
#
# The lookup reads from /dev/null as a guard on what may be put in this loop
# later, not against anything measured: `docker exec` without `-i` leaves the
# list alone, and a command that did read stdin would eat it silently.
unexplained=
while IFS= read -r test_name; do
# The name goes in as an argument rather than being pasted into the script
# text: interpolating it would put report content inside a command the sandbox
# then parses, and the quoting that holds for a Chainsaw Test name holds only
# because of what such a name is allowed to contain.
if docker exec "$SANDBOX" sh -c \
"[ -f \"$SNAPSHOT_ROOT_EXPR/\$1/SOFT-RED-node-join.txt\" ]" \
soft-red "$test_name" </dev/null 2>/dev/null; then
echo "» ${test_name}: failed on the node-join deadline, which the lane does not block on"
continue
fi
unexplained="${unexplained} ${test_name}"
done <<EOF
${failed}
EOF
if [ -n "$unexplained" ]; then
echo "these tests failed for reasons the lane does not tolerate:${unexplained}" >&2
exit 1
fi
# Deliberately not an annotation. The failing test already wrote one naming the
# deadline; a second one here would report the same run twice, and this line is
# about a different fact -- what the lane decided -- which belongs in the log
# beside the decision.
echo "» every failed test was the node-join deadline; the job is not blocked on it, and the marker beside each test's diagnostics in cozyreport.tgz records that this run did not prove what it set out to"
hack/e2e-platform-packages.sh
Executable
+83
View File
@@ -0,0 +1,83 @@
#!/bin/sh
set -eu
api_server_endpoint=${COZY_APISERVER_ENDPOINT:-https://192.168.123.10:6443}
linstor_drbd_enabled=${COZY_LINSTOR_DRBD_ENABLED:-true}
# Container-mode Talos shares the runner kernel and cannot load DRBD. Keep the
# normal package graph for QEMU, but let that lane replace only LINSTOR with an
# otherwise-identical Package whose logger sidecar is disabled.
case "$linstor_drbd_enabled" in
true|false) ;;
*)
echo "COZY_LINSTOR_DRBD_ENABLED must be true or false, got: $linstor_drbd_enabled" >&2
exit 2
;;
esac
cat <<EOF
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
name: cozystack.cozystack-platform
spec:
variant: isp-full
components:
platform:
values:
networking:
podCIDR: "10.244.0.0/16"
podGateway: "10.244.0.1"
serviceCIDR: "10.96.0.0/16"
joinCIDR: "100.64.0.0/16"
publishing:
host: "example.org"
apiServerEndpoint: "$api_server_endpoint"
bundles:
enabledPackages:
- cozystack.external-dns-application
EOF
if [ "$linstor_drbd_enabled" = false ]; then
cat <<'EOF'
disabledPackages:
- cozystack.linstor
- cozystack.kubevirt-cdi
---
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
name: cozystack.linstor
spec:
variant: default
components:
linstor:
values:
drbd:
enabled: false
---
# CDI's own 600M ceiling was seen to OOM the decompress+convert of a tenant
# worker disk near 100% on this lane, after which CDI retries from scratch and
# the DataVolume cycles forever instead of failing. It is an override here and
# not a chart default because the same import completes at 600M on a QEMU-node
# cluster, so what has been shown to need the headroom is this substrate, not
# the product. Move it into packages/system/kubevirt-cdi/values.yaml the day it
# reproduces somewhere that is not a container lane.
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
name: cozystack.kubevirt-cdi
spec:
variant: default
components:
kubevirt-cdi:
values:
importerResources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 750m
memory: 4Gi
EOF
fi
hack/e2e-post-install-prep.sh
+101 -34
View File
@@ -41,6 +41,98 @@
# MetalLB wait below, and still well inside the timeout the job carries.
set -eu
# The QEMU lane gives every satellite a private /dev/vdc and lets LINSTOR
# create the zpool. Container nodes share the runner kernel, where zpools are
# global, so hack/e2e-container-up.sh creates one node-distinct pool ahead of
# time and the satellite must register that existing pool instead.
validate_linstor_storage_mode() {
case "${COZY_LINSTOR_DRBD_ENABLED:-true}" in
true | false) return 0 ;;
*)
echo "[post-install-prep] COZY_LINSTOR_DRBD_ENABLED must be true or false, got '${COZY_LINSTOR_DRBD_ENABLED}'" >&2
return 1
;;
esac
}
create_linstor_storage_pool() {
e2e_linstor_node=$1
case "${COZY_LINSTOR_DRBD_ENABLED:-true}" in
true)
kubectl exec -n cozy-linstor deploy/linstor-controller -- \
linstor physical-storage create-device-pool zfs "$e2e_linstor_node" /dev/vdc \
--pool-name data --storage-pool data
;;
false)
kubectl exec -n cozy-linstor deploy/linstor-controller -- \
linstor storage-pool create zfs "$e2e_linstor_node" data "data-$e2e_linstor_node"
;;
*) return 2 ;;
esac
}
render_linstor_storageclasses() {
cat <<'EOF'
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: linstor.csi.linbit.com
parameters:
linstor.csi.linbit.com/storagePool: "data"
linstor.csi.linbit.com/layerList: "storage"
linstor.csi.linbit.com/allowRemoteVolumeAccess: "false"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF
if [ "${COZY_LINSTOR_DRBD_ENABLED:-true}" = false ]; then
return 0
fi
cat <<'EOF'
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: replicated
provisioner: linstor.csi.linbit.com
parameters:
linstor.csi.linbit.com/storagePool: "data"
linstor.csi.linbit.com/autoPlace: "3"
linstor.csi.linbit.com/layerList: "drbd storage"
linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: suspend-io
property.linstor.csi.linbit.com/DrbdOptions/Resource/on-no-data-accessible: suspend-io
property.linstor.csi.linbit.com/DrbdOptions/Resource/on-suspended-primary-outdated: force-secondary
property.linstor.csi.linbit.com/DrbdOptions/Net/rr-conflict: retry-connect
volumeBindingMode: Immediate
allowVolumeExpansion: true
EOF
}
# CDI infers ReadWriteMany/Block first for the LINSTOR provisioner because it
# assumes a DRBD-capable StorageClass. The container lane deliberately exposes
# only LINSTOR's local storage layer, and linstor-csi rejects RWX without DRBD.
# Pin the one class used by that lane to the RWO/Block combination phase-0
# proved before any DataVolume can consume the inferred default.
patch_local_cdi_storage_profile() {
kubectl patch storageprofile local --type merge \
-p '{"spec":{"claimPropertySets":[{"accessModes":["ReadWriteOnce"],"volumeMode":"Block"}]}}'
}
# Unit tests source the pure helpers above without reaching a cluster.
if [ "${E2E_POST_INSTALL_PREP_LIB:-false}" = true ]; then
return 0 2>/dev/null || exit 0
fi
if ! validate_linstor_storage_mode; then
exit 2
fi
# Per-link budget in seconds, applied by wait_for_linstor to each link separately.
LINK_BUDGET=900
@@ -148,7 +240,7 @@ for node in srv1 srv2 srv3; do
case $created_pools in
*" $node "*) echo " pool 'data' already exists on $node"; continue;;
esac
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor ps cdp zfs ${node} /dev/vdc --pool-name data --storage-pool data &
create_linstor_storage_pool "$node" &
pids="$pids $!"
done
for pid in $pids; do
@@ -156,39 +248,14 @@ for pid in $pids; do
done
echo "[post-install-prep] applying StorageClasses"
kubectl apply -f - <<'EOF'
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: linstor.csi.linbit.com
parameters:
linstor.csi.linbit.com/storagePool: "data"
linstor.csi.linbit.com/layerList: "storage"
linstor.csi.linbit.com/allowRemoteVolumeAccess: "false"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: replicated
provisioner: linstor.csi.linbit.com
parameters:
linstor.csi.linbit.com/storagePool: "data"
linstor.csi.linbit.com/autoPlace: "3"
linstor.csi.linbit.com/layerList: "drbd storage"
linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: suspend-io
property.linstor.csi.linbit.com/DrbdOptions/Resource/on-no-data-accessible: suspend-io
property.linstor.csi.linbit.com/DrbdOptions/Resource/on-suspended-primary-outdated: force-secondary
property.linstor.csi.linbit.com/DrbdOptions/Net/rr-conflict: retry-connect
volumeBindingMode: Immediate
allowVolumeExpansion: true
EOF
render_linstor_storageclasses | kubectl apply -f -
if [ "${COZY_LINSTOR_DRBD_ENABLED:-true}" = false ]; then
echo "[post-install-prep] waiting for CDI StorageProfile/local"
timeout 600 sh -ec 'until kubectl get storageprofile local >/dev/null 2>&1; do sleep 2; done'
echo "[post-install-prep] forcing CDI StorageProfile/local to RWO/Block"
patch_local_cdi_storage_profile
fi
echo "[post-install-prep] waiting for MetalLB CRDs"
timeout 300 sh -ec 'until kubectl get crd ipaddresspools.metallb.io l2advertisements.metallb.io >/dev/null 2>&1; do sleep 2; done'
hack/e2e-prepare-cluster-container.bats
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env bats
# -----------------------------------------------------------------------------
# Cozystack E2E provisioning — CONTAINER lane
#
# The counterpart to hack/e2e-prepare-cluster.bats. That file boots three QEMU
# guests; this one takes over after hack/e2e-container-up.sh has started three
# Talos containers on the runner's docker daemon and attached this sandbox to
# their network. Everything from bootstrap onward is deliberately the same shape
# as the QEMU lane so the two stay comparable, and both hand off an identical
# cluster to hack/e2e-install-cozystack.bats.
#
# Why the split: containers cannot nest inside the sandbox the way QEMU guests
# do, so the nodes must be siblings on the host daemon for hack/e2e-compose.yaml's
# /dev and /run/udev binds to resolve against the real host devtmpfs. Only the
# host-side half needs a docker socket; this half needs only the network.
#
# Addressing, CPU and memory match the QEMU lane exactly, so the MetalLB
# 192.168.123.200-250 pool and hack/e2e-post-install-prep.sh are unchanged.
# The one deliberate difference is the API endpoint: container mode has no ARP
# VIP, so .11 stands in for the QEMU lane's 192.168.123.10.
# -----------------------------------------------------------------------------
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
E2E_CONTAINER_UP_LIB=true . "$HACK_DIR/e2e-container-up.sh"
@test "Talos API is reachable on all three container nodes" {
# e2e-container-up.sh has already started them; this is the handoff check, and
# it is a wait rather than a probe because container start is asynchronous.
if ! timeout 180 sh -ec 'until nc -nz 192.168.123.11 50000 && nc -nz 192.168.123.12 50000 && nc -nz 192.168.123.13 50000; do sleep 1; done'; then
echo "Talos API never came up on 192.168.123.11-13:50000." >&2
echo "The sandbox reaches the nodes over the compose network it was attached to;" >&2
echo "if this times out, check that hack/e2e-container-up.sh ran and that the" >&2
echo "sandbox is on the cozy-e2e_talos network." >&2
exit 1
fi
}
@test "Host kernel modules are usable from inside the nodes" {
# THE trap of this lane. machine.kernel.modules is a silent no-op in container
# mode -- kernel_module_spec.go returns early on ModeContainer with no error
# and no event -- so a lane that carried the QEMU patch over would look like it
# worked and fail much later as a CNI or storage error. e2e-container-up.sh
# loads these on the host; this asserts the nodes actually see them, which is
# the property that matters and is not implied by the host check alone.
for mod in openvswitch zfs; do
if ! talosctl read /proc/modules -n 192.168.123.11 -e 192.168.123.11 | grep -q "^${mod} "; then
echo "Kernel module '${mod}' is not visible from srv1." >&2
echo "Container nodes share the host kernel and Talos cannot load modules in" >&2
echo "container mode, so this must be loaded on the host before they start." >&2
exit 1
fi
done
}
@test "/dev/kvm is present inside the nodes" {
# KubeVirt's device plugin needs it, and its presence is what makes the tenant
# workers run under real KVM rather than TCG emulation.
if ! talosctl ls /dev/kvm -n 192.168.123.11 -e 192.168.123.11 >/dev/null 2>&1; then
echo "/dev/kvm is missing inside srv1 — KubeVirt cannot advertise devices.kubevirt.io/kvm." >&2
echo "hack/e2e-compose.yaml binds the host /dev; check the host exposes /dev/kvm." >&2
exit 1
fi
}
@test "Bootstrap Talos cluster" {
# Retry for up to 120s: talosctl bootstrap refuses with "time is not in sync
# yet" until NTP converges on a freshly started node. This is the same
# allowance the QEMU lane makes, and it is a genuine infrastructure wait, not
# a retry around product logic.
#
# If it never converges, the usual cause is that the nodes cannot reach the
# internet at all rather than slow NTP -- most often because docker sets
# `-P FORWARD DROP` and the compose bridge's traffic is being dropped.
if ! timeout 120 sh -ec 'until talosctl bootstrap -n 192.168.123.11 -e 192.168.123.11; do sleep 2; done'; then
echo "talosctl bootstrap never succeeded." >&2
echo "If the error was 'time is not in sync yet', the nodes likely have no egress:" >&2
echo "check 'iptables -S FORWARD' on the host for a DROP policy over the compose bridge." >&2
talosctl dmesg -n 192.168.123.11 -e 192.168.123.11 || true
exit 1
fi
}
@test "Wait until etcd is healthy" {
# Same 5m budget and the same reason as the QEMU lane: a 3-node etcd converges
# through Talos's serialized learner promotion, so the third member is only
# promoted once the second has caught up.
if ! timeout 300 sh -ec 'until talosctl etcd members -n 192.168.123.11,192.168.123.12,192.168.123.13 -e 192.168.123.11 >/dev/null 2>&1; do sleep 1; done'; then
talosctl dmesg -n 192.168.123.11,192.168.123.12,192.168.123.13 -e 192.168.123.11 || true
exit 1
fi
timeout 60 sh -ec 'while talosctl etcd members -n 192.168.123.11,192.168.123.12,192.168.123.13 -e 192.168.123.11 2>&1 | grep -q "rpc error"; do sleep 1; done'
}
@test "Retrieve kubeconfig and register all three nodes" {
rm -f kubeconfig
# No ARP VIP in container mode, so the endpoint is a node IP rather than the
# QEMU lane's 192.168.123.10.
talosctl kubeconfig kubeconfig -e 192.168.123.11 -n 192.168.123.11
if ! timeout 120 sh -ec 'until [ $(kubectl get node --no-headers | wc -l) -eq 3 ]; do sleep 1; done'; then
echo "Fewer than three nodes registered:" >&2
kubectl get nodes -o wide >&2 || true
exit 1
fi
}
@test "Node allocatable matches the container limits" {
# Container mode exposes host-wide capacity through /proc, so without the
# systemReserved values generated by e2e-container-up.sh every one of these
# nodes advertises the runner's full 32 CPU / ~125 GiB. That lets the
# scheduler pack far beyond each 8 CPU / 24 GiB cgroup and turns later OOMs
# into apparent workload flakes. Check the scheduler-visible result, not just
# the machine-config input.
local node cpu cpu_m memory memory_kib
local node_cpu_limit="${COZY_E2E_NODE_CPUS:-8}"
local node_memory_limit_mib="${COZY_E2E_NODE_MEMORY_MIB:-24576}"
local max_cpu_m=$(( node_cpu_limit * 1000 ))
local min_cpu_m=$(( max_cpu_m - 1000 ))
local max_memory_kib=$(( node_memory_limit_mib * 1024 ))
local min_memory_kib=$(( max_memory_kib - 1024 * 1024 ))
for node in srv1 srv2 srv3; do
cpu=$(kubectl get node "$node" -o jsonpath='{.status.allocatable.cpu}')
case "$cpu" in
*m) cpu_m=${cpu%m} ;;
*[!0-9]* | '')
echo "node $node reported an unreadable allocatable CPU quantity: '$cpu'" >&2
return 1
;;
*) cpu_m=$(( cpu * 1000 )) ;;
esac
memory=$(kubectl get node "$node" -o jsonpath='{.status.allocatable.memory}')
if ! memory_kib=$(kubernetes_memory_to_kib "$memory"); then
echo "node $node reported an unreadable allocatable memory quantity: '$memory'" >&2
return 1
fi
if [ "$cpu_m" -lt "$min_cpu_m" ] || [ "$cpu_m" -gt "$max_cpu_m" ]; then
echo "node $node allocatable CPU $cpu is outside ${min_cpu_m}-${max_cpu_m}m" >&2
return 1
fi
if [ "$memory_kib" -lt "$min_memory_kib" ] || [ "$memory_kib" -gt "$max_memory_kib" ]; then
echo "node $node allocatable memory $memory is outside ${min_memory_kib}-${max_memory_kib}Ki" >&2
return 1
fi
echo "$node allocatable: ${cpu}/${memory}"
done
}
hack/e2e-prepull-images.sh
+15
View File
@@ -31,6 +31,21 @@ fi
echo "Pre-pulling ${#images[@]} image(s):"
printf ' %s\n' "${images[@]}"
# A literal `null` means the caller's yq filter matched a container that has
# no `image:` field -- typically a strategic-merge pod-template fragment
# inside a CR, which overrides a few fields and pulls nothing. Catch it here:
# passed through, it becomes a container with an empty image and `kubectl
# apply` rejects the DaemonSet by container index, naming neither the filter
# nor the chart that produced it.
for image in "${images[@]}"; do
if [[ "$image" == "null" ]]; then
echo "e2e-prepull-images: refusing a literal 'null' image reference." >&2
echo " Some container in the caller's yq filter has no 'image:' field." >&2
echo " Drop nulls at the source (see hack/e2e-install-cozystack.bats)." >&2
exit 1
fi
done
containers=""
i=0
for image in "${images[@]}"; do
hack/e2e-talos-image-cache.yaml
+26 -6
View File
@@ -159,6 +159,7 @@ data:
import http.server
import os
import sys
import time
ROOT = sys.argv[1] if len(sys.argv) > 1 else "/data"
PORT = int(sys.argv[2]) if len(sys.argv) > 2 else 8080
@@ -184,6 +185,8 @@ data:
ctype = self.guess_type(path)
rng = self.headers.get("Range", "")
if not rng.startswith("bytes="):
self._remaining = size
self._transfer_range = "full"
self.send_response(200)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(size))
@@ -203,6 +206,7 @@ data:
end = min(end, size - 1)
f.seek(start)
self._remaining = end - start + 1
self._transfer_range = "%d-%d/%d" % (start, end, size)
self.send_response(206)
self.send_header("Content-Type", ctype)
self.send_header("Content-Range", "bytes %d-%d/%d" % (start, end, size))
@@ -217,12 +221,28 @@ data:
super().copyfile(source, outputfile)
return
self._remaining = None
while remaining > 0:
chunk = source.read(min(65536, remaining))
if not chunk:
break
outputfile.write(chunk)
remaining -= len(chunk)
requested = remaining
sent = 0
started = time.monotonic()
try:
while remaining > 0:
chunk = source.read(min(65536, remaining))
if not chunk:
break
outputfile.write(chunk)
sent += len(chunk)
remaining -= len(chunk)
finally:
status = "complete" if sent == requested else "interrupted"
self.log_message(
"transfer-finished status=%s path=%s range=%s requested=%d sent=%d duration=%.3fs",
status,
self.path,
getattr(self, "_transfer_range", "unknown"),
requested,
sent,
time.monotonic() - started,
)
with http.server.ThreadingHTTPServer(("0.0.0.0", PORT), RangeHandler) as httpd:
hack/e2e-wait-helmreleases.sh
+92
View File
@@ -0,0 +1,92 @@
#!/bin/bash
# Readiness gate for the platform HelmReleases created during E2E install.
cozy_wait_all_helmreleases_ready() {
local timeout_seconds="${1:-900}"
local minimum_count="${2:-11}"
local quiet_seconds="${3:-5}"
local poll_seconds="${4:-2}"
local deadline
local snapshot= count= stalled= not_ready= fingerprint=
local stable_fingerprint= stable_since=0 now
case "${timeout_seconds}:${minimum_count}:${quiet_seconds}:${poll_seconds}" in
*[!0-9:]*)
echo "HelmRelease readiness timing arguments must be integers" >&2
return 1
;;
esac
if [ "${minimum_count}" -eq 0 ] || [ "${poll_seconds}" -eq 0 ]; then
echo "HelmRelease readiness minimum count and poll interval must be positive" >&2
return 1
fi
deadline=$(( $(date +%s) + timeout_seconds ))
echo "Waiting for at least ${minimum_count} HelmReleases to become Ready and remain a stable set for ${quiet_seconds}s..."
while :; do
now=$(date +%s)
if ! snapshot=$(kubectl get helmreleases.helm.toolkit.fluxcd.io -A -o json); then
echo "HelmRelease readiness list failed; keeping the gate closed" >&2
stable_fingerprint=
stable_since=0
else
if ! count=$(printf '%s\n' "${snapshot}" | jq -r '.items | length'); then
echo "Could not parse the HelmRelease readiness list" >&2
return 1
fi
stalled=$(printf '%s\n' "${snapshot}" | jq -r '
.items[] |
select((.status.observedGeneration // 0) == .metadata.generation) |
select([.status.conditions[]? | select(.type == "Stalled" and .status == "True")] | length > 0) |
"\(.metadata.namespace)/\(.metadata.name): " +
(([.status.conditions[]? | select(.type == "Stalled") | .message][0]) // "Stalled=True")
')
if [ -n "${stalled}" ]; then
echo "HelmRelease reconciliation reached a terminal Stalled condition:" >&2
printf '%s\n' "${stalled}" | sed 's/^/ /' >&2
return 1
fi
not_ready=$(printf '%s\n' "${snapshot}" | jq -r '
.items[] |
(.status.observedGeneration // 0) as $observed |
.metadata.generation as $generation |
([.status.conditions[]? | select(.type == "Ready" and .status == "True")] | length) as $ready |
select($observed != $generation or $ready == 0) |
"\(.metadata.namespace)/\(.metadata.name): " +
(if $observed != $generation then
"generation \($generation) not observed (status.observedGeneration=\($observed))"
else
(([.status.conditions[]? | select(.type == "Ready") | "\(.reason): \(.message)"][0]) // "Ready condition absent")
end)
')
if [ "${count}" -ge "${minimum_count}" ] && [ -z "${not_ready}" ]; then
fingerprint=$(printf '%s\n' "${snapshot}" | jq -r '
.items | map("\(.metadata.namespace)/\(.metadata.name)") | sort | join("\n")
')
if [ "${fingerprint}" != "${stable_fingerprint}" ]; then
stable_fingerprint="${fingerprint}"
stable_since="${now}"
elif [ $(( now - stable_since )) -ge "${quiet_seconds}" ]; then
echo "All ${count} HelmReleases are Ready; the release set stayed stable for ${quiet_seconds}s."
return 0
fi
else
stable_fingerprint=
stable_since=0
fi
fi
if [ "${now}" -ge "${deadline}" ]; then
echo "HelmReleases did not all become Ready within ${timeout_seconds}s" >&2
if [ -n "${not_ready}" ]; then
printf '%s\n' "${not_ready}" | sed 's/^/ /' >&2
fi
if ! kubectl get helmreleases.helm.toolkit.fluxcd.io -A >&2; then
echo "The final HelmRelease diagnostic list also failed" >&2
fi
return 1
fi
sleep "${poll_seconds}"
done
}
hack/ghcr-mirror_test.bats
-1509
View File
File diff suppressed because it is too large. Load diff
hack/helmrelease-readiness.bats
+127
View File
@@ -0,0 +1,127 @@
#!/usr/bin/env bats
# Unit coverage for the single post-install HelmRelease readiness gate.
helmrelease_snapshot() {
count="$1"
not_ready="${2:-0}"
stalled="${3:-0}"
stale="${4:-0}"
jq -nc --argjson count "$count" --argjson notReady "$not_ready" --argjson stalled "$stalled" --argjson stale "$stale" '
{items: [range(1; $count + 1) as $i |
{metadata: {namespace: "ns", name: ("hr-" + ($i | tostring)), generation: 2},
status: {observedGeneration: (if $i == $stale then 1 else 2 end), conditions:
([{type: "Ready", status: (if $i == $notReady then "False" else "True" end), reason: "Progressing", message: "working"}]
+ (if $i == $stalled then [{type: "Stalled", status: "True", message: "terminal failure"}] else [] end))}}
]}
'
}
@test "stale Ready stays closed until the current generation is observed" {
. hack/e2e-wait-helmreleases.sh
clock=$(mktemp)
printf '0\n' > "$clock"
date() { sed -n '1p' "$clock"; }
sleep() {
now=$(sed -n '1p' "$clock")
printf '%s\n' "$(( now + $1 ))" > "$clock"
}
kubectl() {
now=$(sed -n '1p' "$clock")
if [ "$now" -lt 4 ]; then
helmrelease_snapshot 11 0 0 3
else
helmrelease_snapshot 11
fi
}
cozy_wait_all_helmreleases_ready 30 11 4 2 >/dev/null
[ "$(sed -n '1p' "$clock")" -ge 8 ] || {
echo "the gate accepted Ready=True from an unobserved generation" >&2
exit 1
}
}
@test "a late HelmRelease resets stability and is included before success" {
. hack/e2e-wait-helmreleases.sh
clock=$(mktemp)
calls=$(mktemp)
printf '0\n' > "$clock"
date() { sed -n '1p' "$clock"; }
sleep() {
now=$(sed -n '1p' "$clock")
printf '%s\n' "$(( now + $1 ))" > "$clock"
}
kubectl() {
printf '%s\n' "$*" >> "$calls"
now=$(sed -n '1p' "$clock")
if [ "$now" -lt 2 ]; then
helmrelease_snapshot 11
elif [ "$now" -lt 4 ]; then
helmrelease_snapshot 12 12
else
helmrelease_snapshot 12
fi
}
cozy_wait_all_helmreleases_ready 30 11 5 2 >/dev/null
[ "$(sed -n '1p' "$clock")" -ge 10 ] || {
echo "the gate returned before the late release became Ready and stabilized" >&2
exit 1
}
[ "$(grep -c -- '-o json' "$calls")" -ge 6 ]
}
@test "Stalled HelmRelease fails immediately without sleeping to timeout" {
. hack/e2e-wait-helmreleases.sh
calls=$(mktemp)
date() { printf '100\n'; }
sleep() { echo "sleep must not run after Stalled=True" >&2; return 1; }
kubectl() {
printf '%s\n' "$*" >> "$calls"
helmrelease_snapshot 11 3 3
}
if cozy_wait_all_helmreleases_ready 900 11 5 2 >/dev/null 2>&1; then
echo "a terminal Stalled HelmRelease passed readiness" >&2
exit 1
fi
[ "$(wc -l < "$calls")" -eq 1 ] || {
echo "the Stalled path kept polling instead of failing immediately" >&2
exit 1
}
}
@test "a transient list error keeps the gate closed" {
. hack/e2e-wait-helmreleases.sh
clock=$(mktemp)
calls=$(mktemp)
printf '0\n' > "$clock"
date() { sed -n '1p' "$clock"; }
sleep() {
now=$(sed -n '1p' "$clock")
printf '%s\n' "$(( now + $1 ))" > "$clock"
}
kubectl() {
count=$(wc -l < "$calls")
printf '%s\n' "$*" >> "$calls"
if [ "$count" -eq 0 ]; then
echo 'temporary apiserver error' >&2
return 1
fi
helmrelease_snapshot 11
}
cozy_wait_all_helmreleases_ready 30 11 4 2 >/dev/null 2>&1
[ "$(sed -n '1p' "$clock")" -ge 6 ] || {
echo "the list error was mistaken for an all-Ready snapshot" >&2
exit 1
}
}
@test "install suite has one unmasked HelmRelease gate" {
[ "$(grep -c 'cozy_wait_all_helmreleases_ready 900 11 5 2' hack/e2e-install-cozystack.bats)" -eq 1 ]
if grep -q 'kubectl wait hr --all -A.*|| true' hack/e2e-install-cozystack.bats; then
echo "the old masked HelmRelease wait survived" >&2
exit 1
fi
}
hack/linstor-storage-mode_test.bats
+313
View File
@@ -0,0 +1,313 @@
#!/usr/bin/env bats
# Unit coverage for the lane-specific E2E storage contract: LINSTOR pool
# registration, management StorageClasses, and downstream Chainsaw fixtures.
# The library guard keeps this suite cluster-free under hack/cozytest.sh.
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
POST_PREP="$HACK_DIR/e2e-post-install-prep.sh"
# shellcheck source=/dev/null
E2E_POST_INSTALL_PREP_LIB=true . "$POST_PREP"
kubectl() { printf '%s\n' "$*"; }
@test "QEMU mode creates data from the satellite block device" {
unset COZY_LINSTOR_DRBD_ENABLED
command=$(create_linstor_storage_pool srv2)
expected='exec -n cozy-linstor deploy/linstor-controller -- linstor physical-storage create-device-pool zfs srv2 /dev/vdc --pool-name data --storage-pool data'
if [ "$command" != "$expected" ]; then
echo "unexpected QEMU pool command: $command" >&2
return 1
fi
}
@test "container mode registers the node-specific pre-created zpool" {
COZY_LINSTOR_DRBD_ENABLED=false
command=$(create_linstor_storage_pool srv2)
expected='exec -n cozy-linstor deploy/linstor-controller -- linstor storage-pool create zfs srv2 data data-srv2'
if [ "$command" != "$expected" ]; then
echo "unexpected container pool command: $command" >&2
return 1
fi
}
@test "QEMU mode renders local and replicated StorageClasses" {
unset COZY_LINSTOR_DRBD_ENABLED
manifest=$(render_linstor_storageclasses)
names=$(printf '%s\n' "$manifest" | yq -N '.metadata.name')
expected_names=$(printf '%s\n' local replicated)
replicated_layers=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "replicated") | .parameters."linstor.csi.linbit.com/layerList"')
if [ "$names" != "$expected_names" ]; then
echo "unexpected QEMU StorageClasses: $names" >&2
return 1
fi
if [ "$replicated_layers" != "drbd storage" ]; then
echo "replicated StorageClass lost its DRBD layer: $replicated_layers" >&2
return 1
fi
}
@test "container mode renders only the production local StorageClass" {
COZY_LINSTOR_DRBD_ENABLED=false
manifest=$(render_linstor_storageclasses)
names=$(printf '%s\n' "$manifest" | yq -N '.metadata.name')
local_layers=$(printf '%s\n' "$manifest" | yq '.parameters."linstor.csi.linbit.com/layerList"')
remote=$(printf '%s\n' "$manifest" | yq '.parameters."linstor.csi.linbit.com/allowRemoteVolumeAccess"')
if [ "$names" != "local" ]; then
echo "container mode emitted unexpected StorageClasses: $names" >&2
return 1
fi
if [ "$local_layers" != "storage" ] || [ "$remote" != "false" ]; then
echo "container local StorageClass has unexpected parameters: layerList=$local_layers allowRemoteVolumeAccess=$remote" >&2
return 1
fi
}
@test "container mode pins CDI local StorageProfile to RWO Block" {
command=$(patch_local_cdi_storage_profile)
expected='patch storageprofile local --type merge -p {"spec":{"claimPropertySets":[{"accessModes":["ReadWriteOnce"],"volumeMode":"Block"}]}}'
if [ "$command" != "$expected" ]; then
echo "unexpected CDI StorageProfile patch: $command" >&2
return 1
fi
if ! grep -Fq 'timeout 600 sh -ec '\''until kubectl get storageprofile local' "$POST_PREP"; then
echo "container prep does not wait for CDI to create StorageProfile/local" >&2
return 1
fi
}
@test "invalid DRBD mode is rejected before cluster access" {
COZY_LINSTOR_DRBD_ENABLED=unsupported
if validate_linstor_storage_mode; then
echo "invalid DRBD mode unexpectedly passed validation" >&2
return 1
fi
}
@test "tenant Kubernetes storage defaults to replicated and accepts local only explicitly" {
# Sourcing inside the test keeps run-kubernetes.sh's live-cluster
# cozy_cleanup helper out of cozytest.sh's file-level cleanup discovery.
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
unset COZY_E2E_STORAGE_CLASS
default_class=$(cozy_e2e_storage_class)
COZY_E2E_STORAGE_CLASS=local
container_class=$(cozy_e2e_storage_class)
if [ "$default_class" != replicated ] || [ "$container_class" != local ]; then
echo "unexpected tenant storage classes: default=$default_class container=$container_class" >&2
return 1
fi
COZY_E2E_STORAGE_CLASS=unsupported
if cozy_e2e_storage_class; then
echo "invalid tenant storage class unexpectedly passed validation" >&2
return 1
fi
}
@test "local storage uses its saved LINSTOR pool baseline" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_E2E_STORAGE_CLASS=local
cozy_wait_linstor_pool_free() { printf 'unexpected pool wait\n'; return 99; }
cozy_wait_linstor_pool_baseline() { printf 'baseline-wait:%s:%s\n' "$1" "$2"; }
output=$(cozy_wait_linstor_pool_reclaimed 90 300)
if printf '%s\n' "$output" | grep -Fq 'unexpected pool wait'; then
echo "local storage entered the replicated-capacity wait" >&2
return 1
fi
[ "$output" = 'baseline-wait:300:524288' ]
}
@test "replicated storage retains the LINSTOR free-capacity barrier" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_E2E_STORAGE_CLASS=replicated
cozy_wait_linstor_pool_free() { printf 'pool-wait:%s:%s\n' "$1" "$2"; }
output=$(cozy_wait_linstor_pool_reclaimed 91 17)
[ "$output" = 'pool-wait:91:17' ]
}
@test "local pool baseline comparison is per-node and tolerates only metadata drift" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
baseline=$(printf 'srv1:67108864\nsrv2:20787200\nsrv3:60817408')
within_tolerance=$(printf 'srv1:67108864\nsrv2:20262912\nsrv3:60817408')
leaked_worker=$(printf 'srv1:67108864\nsrv2:20787200\nsrv3:39845888')
missing_node=$(printf 'srv1:67108864\nsrv3:60817408')
cozy_linstor_pools_at_baseline "$baseline" "$within_tolerance" 524288
if cozy_linstor_pools_at_baseline "$baseline" "$leaked_worker" 524288; then
echo "a 20 GiB worker leak passed the local-pool baseline" >&2
return 1
fi
if cozy_linstor_pools_at_baseline "$baseline" "$missing_node" 524288; then
echo "a missing LINSTOR node passed the local-pool baseline" >&2
return 1
fi
}
@test "local pool baseline capture persists all three numeric satellite rows" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_LINSTOR_POOL_BASELINE_FILE="_out/tmp/linstor-baseline-$$"
kubectl() { printf 'srv3:60817408\nsrv1:67108864\nsrv2:20787200\n'; }
output=$(cozy_capture_linstor_pool_baseline)
captured=$(cat "$COZY_LINSTOR_POOL_BASELINE_FILE")
rm -f "$COZY_LINSTOR_POOL_BASELINE_FILE"
[ "$captured" = "$(printf 'srv3:60817408\nsrv1:67108864\nsrv2:20787200')" ]
printf '%s\n' "$output" | grep -Fq 'local LINSTOR pool baseline recorded'
}
@test "Kubernetes cleanup fails when local LINSTOR capacity is not reclaimed" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
kubectl() { return 0; }
cozy_wait_tenant_drained() { return 0; }
cozy_wait_linstor_pool_reclaimed() { return 42; }
if cozy_cleanup test-latest-version; then
echo "cleanup hid the failed LINSTOR reclamation barrier" >&2
return 1
fi
}
@test "local Kubernetes suite cannot start without a LINSTOR baseline" {
# shellcheck source=/dev/null
. "$HACK_DIR/e2e-chainsaw/_lib/run-kubernetes.sh"
COZY_E2E_STORAGE_CLASS=local
yq() { printf 'v1.33.12\n'; }
kubectl() { return 0; }
cozy_wait_tenant_drained() { return 0; }
cozy_capture_linstor_pool_baseline() { return 42; }
helm() { printf 'UNEXPECTED_HELM_CALL\n'; return 99; }
rc=0
output=$(run_kubernetes_test '.' test-latest-version 59991) || rc=$?
if [ "$rc" -eq 0 ]; then
echo "suite accepted a failed local LINSTOR baseline capture" >&2
return 1
fi
if printf '%s\n' "$output" | grep -Fq UNEXPECTED_HELM_CALL; then
echo "suite continued into Helm after the baseline capture failed" >&2
return 1
fi
}
@test "Kubernetes cleanup operations outlive their bounded reclamation stages" {
for suite in latest previous; do
timeout=$(yq "select(.metadata.name == \"kubernetes-${suite}\") | .spec.steps[0].finally[0].script.timeout" "hack/e2e-chainsaw/kubernetes-${suite}/chainsaw-test.yaml")
if [ "$timeout" != 22m ]; then
echo "kubernetes-${suite} cleanup timeout is $timeout, expected 22m" >&2
return 1
fi
done
}
@test "both merge-gating lanes forward local storage through install and Chainsaw" {
# Checked in both workflows rather than one: pull-requests.yaml gates same-repo
# PRs and e2e-fork.yaml gates fork PRs, they carry the override separately, and
# a lane that installs on `local` while templating Chainsaw with `replicated`
# fails deep inside a suite rather than at the mismatch.
install_command=$(make -n -C packages/core/testing SANDBOX_NAME=test COZY_E2E_STORAGE_CLASS=local install-cozystack)
make_command=$(make -n -C packages/core/testing SANDBOX_NAME=test COZY_E2E_STORAGE_CLASS=local CHAINSAW_SUITES=vminstance test-chainsaw)
for wf in .github/workflows/pull-requests.yaml .github/workflows/e2e-fork.yaml; do
install_value=$(yq '.jobs.e2e.steps[] | select(.name == "Install Cozystack into sandbox") | .env.COZY_E2E_STORAGE_CLASS' "$wf")
run_value=$(yq '.jobs.e2e.steps[] | select(.name == "Run E2E tests") | .env.COZY_E2E_STORAGE_CLASS' "$wf")
if [ "$install_value" != local ] || [ "$run_value" != local ]; then
echo "unexpected storage modes in $wf: install=$install_value chainsaw=$run_value" >&2
return 1
fi
done
if ! printf '%s\n' "$install_command" | grep -Fq -- '-e COZY_E2E_STORAGE_CLASS="local"'; then
echo "testing Makefile does not forward COZY_E2E_STORAGE_CLASS into installation" >&2
return 1
fi
if ! printf '%s\n' "$make_command" | grep -Fq -- '-e COZY_E2E_STORAGE_CLASS="local"'; then
echo "testing Makefile does not forward COZY_E2E_STORAGE_CLASS into the sandbox" >&2
return 1
fi
if ! printf '%s\n' "$make_command" | grep -Fq -- '--set-string storageClass="${COZY_E2E_STORAGE_CLASS:-replicated}" vminstance'; then
echo "testing Makefile does not pass the lane storage class to Chainsaw values" >&2
return 1
fi
}
@test "container install supplies VictoriaLogs local storage before monitoring exists" {
install_suite=hack/e2e-install-cozystack.bats
if ! grep -Fq 'storage_class="${COZY_E2E_STORAGE_CLASS:-replicated}"' "$install_suite"; then
echo "install suite does not retain the replicated QEMU default" >&2
return 1
fi
if ! grep -Fq 'kubectl patch secret cozystack-values -n tenant-root --type merge --patch-file "$root_values_patch_file"' "$install_suite"; then
echo "install suite does not put the monitoring override in root values" >&2
return 1
fi
if grep -Fq 'kubectl patch hr/monitoring' "$install_suite"; then
echo "install suite still races monitoring reconciliation with a HelmRelease patch" >&2
return 1
fi
if ! grep -Fq '.logsStorages = [{"name":"generic","retentionPeriod":"1","storage":"10Gi","storageClassName":strenv(COZY_E2E_STORAGE_CLASS)}]' "$install_suite"; then
echo "monitoring override does not preserve the required VictoriaLogs values" >&2
return 1
fi
secret_patch_line=$(grep -nF 'kubectl patch secret cozystack-values' "$install_suite" | cut -d: -f1)
tenant_patch_line=$(grep -nF 'kubectl patch tenants/root' "$install_suite" | cut -d: -f1)
if [ -z "$secret_patch_line" ] || [ -z "$tenant_patch_line" ] || [ "$secret_patch_line" -ge "$tenant_patch_line" ]; then
echo "root values must be patched before monitoring is enabled: secret=$secret_patch_line tenant=$tenant_patch_line" >&2
return 1
fi
rendered=$(mktemp)
helm template monitoring packages/extra/monitoring --namespace tenant-root \
--set-string 'logsStorages[0].name=generic' \
--set-string 'logsStorages[0].retentionPeriod=1' \
--set-string 'logsStorages[0].storage=10Gi' \
--set-string 'logsStorages[0].storageClassName=local' > "$rendered"
rendered_class=$(yq 'select(.kind == "HelmRelease" and .metadata.name == "monitoring-system") | .spec.values.logsStorages[0].storageClassName' "$rendered")
rm -f "$rendered"
if [ "$rendered_class" != local ]; then
echo "monitoring chart did not carry the root values override into its child release: $rendered_class" >&2
return 1
fi
}
@test "DRBD-dependent fixtures and Kubernetes checks consume the lane storage mode" {
for manifest in hack/e2e-chainsaw/vminstance/vmdisk.yaml hack/e2e-chainsaw/vminstance/vmdisk-vmi.yaml; do
storage_class=$(yq '.spec.storageClass' "$manifest")
if [ "$storage_class" != '($values.storageClass)' ]; then
echo "$manifest does not consume the Chainsaw storageClass value: $storage_class" >&2
return 1
fi
done
kubernetes_script=hack/e2e-chainsaw/_lib/run-kubernetes.sh
rendered_uses=$(grep -Fc 'storageClass: "${storage_class}"' "$kubernetes_script")
if [ "$rendered_uses" -ne 2 ]; then
echo "expected both tenant Kubernetes resources to consume storage_class, found $rendered_uses" >&2
return 1
fi
if ! grep -Fq 'if [ "$storage_class" = local ]; then' "$kubernetes_script"; then
echo "local-only Kubernetes suites do not omit DRBD/RWX assertions" >&2
return 1
fi
if ! grep -Fq 'if [ "$storage_class" = replicated ]; then' "$kubernetes_script"; then
echo "fallback regression does not limit replicated StorageClass restoration to QEMU" >&2
return 1
fi
}
hack/node-join-soft-red_test.bats
-675
View File
@@ -1,675 +0,0 @@
#!/usr/bin/env bats
# -----------------------------------------------------------------------------
# Unit tests for hack/e2e-node-join-soft-red.sh -- the decision one layer above
# the suite about whether a failed Chainsaw run may leave the job green.
#
# Named without the `e2e-` prefix the script itself carries, and deliberately:
# the root Makefile builds the unit lane as
# `$(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))`, so a file named after
# the script would be excluded from `make unit-tests` and run nowhere.
#
# What these pin is the direction the gate fails in. It opens on exactly one
# shape -- every failed test carrying the marker the node-join deadline writes
# -- and blocks on everything else, including every way it can fail to answer
# the question. A gate that opens because it could not read its own input is
# worse than no gate, because the run looks examined.
#
# `docker` is stubbed rather than run: the real script reaches into the sandbox
# container for the report and for each marker, and the stub is what lets a test
# say "this report, these markers" without one.
#
# cozytest.sh's awk parser ends an @test block at the first bare closing brace,
# so the mocks stay at top level. EXIT-trap cleanup is banned in hack/*.bats, so
# each test removes its own temp dir on its last line.
#
# Run with: hack/cozytest.sh hack/node-join-soft-red_test.bats
# -----------------------------------------------------------------------------
GATE=hack/e2e-node-join-soft-red.sh
# The sandbox stands in for a container, and it has to be a real executable on
# PATH rather than a shell function: the script under test is invoked through
# `sh`, and a function does not survive a new shell. It answers the two
# questions the script asks -- cat the report, test for a marker -- off files a
# test writes, so a test states its world instead of staging a container.
#
# `rm -f` before the redirect is the house idiom for staging a fake binary: a
# name that already exists as a symlink to a real one is written THROUGH.
stage_sandbox_stub() {
mkdir -p "$1/bin"
rm -f "$1/bin/docker"
cat >"$1/bin/docker" <<'STUB'
#!/bin/sh
case "$*" in
*chainsaw-report.xml*)
[ -f "${COZY_STUB_DIR}/report-fails" ] && exit 1
cat "${COZY_STUB_DIR}/report"
exit 0
;;
*SOFT-RED-node-join.txt*)
# The test name is the LAST argument, not part of the script text: the gate
# hands it to the sandbox shell as an argument so that report content never
# reaches a command the sandbox parses. Reading it from anywhere else here
# would leave that property untested.
for _a in "$@"; do _name=$_a; done
# Logged verbatim: a test that cares whether a name reached the lookup at all
# cannot tell that from the exit code, which is the same for a name that
# missed and for a lookup that never happened.
printf '%s\n' "$_name" >>"${COZY_STUB_DIR}/lookups"
grep -qxF "$_name" "${COZY_STUB_DIR}/markers" 2>/dev/null && exit 0
exit 1
;;
esac
echo "unexpected docker call: $*" >&2
exit 99
STUB
chmod +x "$1/bin/docker"
export COZY_STUB_DIR="$1"
: >"$1/report"
: >"$1/markers"
: >"$1/lookups"
PATH="$1/bin:$PATH"
}
# A report in the shape chainsaw 0.2.15 writes: one testsuite element per suite
# directory, one testcase per Test, and a nested failure element on the ones
# that failed. Taking the real shape matters here -- the enclosing testsuite
# carries a `failures="N"` attribute, and a matcher that read that as a failed
# case would report every suite as failing.
report_with() {
printf '%s\n' '<testsuites name="chainsaw-report" time="1" tests="2" failures="1">'
for _spec in "$@"; do
_name=${_spec%:*}
_verdict=${_spec##*:}
printf ' <testsuite name="%s" tests="1" failures="1" errors="0" id="0" time="">\n' "${_name}"
printf ' <testcase name="%s" classname="" time="1">\n' "${_name}"
[ "${_verdict}" = failed ] && printf ' <failure message="exit status 1"></failure>\n'
printf ' </testcase>\n </testsuite>\n'
done
printf '%s\n' '</testsuites>'
}
@test "a run whose only failure carries the node-join marker does not block the job" {
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
report_with kubernetes-latest:failed bucket:passed >"$tmp/report"
echo kubernetes-latest >"$tmp/markers"
rc=0
( set +x; sh "$GATE" sandbox ) >"$tmp/out" 2>&1 || rc=$?
[ "$rc" -eq 0 ]
grep -q 'kubernetes-latest' "$tmp/out"
rm -rf "$tmp"
}
@test "a failed test without the marker blocks the job even beside one that has it" {
# The pair matters: a gate that answered "some failure was a deadline" instead
# of "every failure was" would open here, and this is the shape that happens
# on a real run -- the node-join deadline expires on one suite while something
# unrelated breaks in another.
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
report_with kubernetes-latest:failed bucket:failed >"$tmp/report"
echo kubernetes-latest >"$tmp/markers"
rc=0
( set +x; sh "$GATE" sandbox ) >"$tmp/out" 2>&1 || rc=$?
[ "$rc" -ne 0 ]
grep -q 'bucket' "$tmp/out"
rm -rf "$tmp"
}
@test "a failed kubernetes suite with no marker blocks, because only the deadline writes one" {
# Every other way the kubernetes suites can go red -- an assertion after the
# join, the wait failing to run at all -- leaves no marker, and must be
# indistinguishable to this gate from any other suite failing.
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
report_with kubernetes-previous:failed >"$tmp/report"
rc=0
( set +x; sh "$GATE" sandbox ) >/dev/null 2>&1 || rc=$?
[ "$rc" -ne 0 ]
rm -rf "$tmp"
}
@test "a report that cannot be read blocks rather than opening on an unknown" {
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
: >"$tmp/report-fails"
echo kubernetes-latest >"$tmp/markers"
rc=0
( set +x; sh "$GATE" sandbox ) >/dev/null 2>&1 || rc=$?
[ "$rc" -ne 0 ]
rm -rf "$tmp"
}
@test "a report recording no failure at all blocks, because the run failed anyway" {
# This is reached when chainsaw died before it could record why, and the
# temptation is to read "no failures" as "nothing wrong". The step only calls
# this after a non-zero make, so an empty failure list means the report and
# the run disagree, and the run is the one that happened.
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
report_with bucket:passed >"$tmp/report"
rc=0
( set +x; sh "$GATE" sandbox ) >/dev/null 2>&1 || rc=$?
[ "$rc" -ne 0 ]
rm -rf "$tmp"
}
# A report whose failed case cannot be named, with a NAMED suite after it. The
# order is the whole point rather than a detail of the fixture: an awk record
# split on `<testcase` runs to the next one, so it carries the following
# `<testsuite name="...">` with it, and a case that cannot be named from its own
# opening tag can be named from that. Put the unnameable case last and every
# assertion below passes against a gate that has the bug -- which is what these
# three fixtures did before they were written this way.
#
# The suite that follows is the one carrying a marker, so a gate that leaks the
# name across the boundary attributes BOTH failures to it, finds a marker for
# both, and exits zero on a failure it never read.
report_with_unnameable() {
printf '%s\n' '<testsuites name="chainsaw-report" tests="2" failures="2">'
printf ' <testsuite name="kafka" tests="1" failures="1" errors="0" id="0" time="">\n'
printf ' <testcase %s time="1">\n' "$1"
printf ' <failure message="exit status 1"></failure>\n'
printf ' </testcase>\n </testsuite>\n'
printf ' <testsuite name="kubernetes-latest" tests="1" failures="1" errors="0" id="0" time="">\n'
printf ' <testcase name="kubernetes-latest" classname="" time="1">\n'
printf ' <failure message="exit status 1"></failure>\n'
printf ' </testcase>\n </testsuite>\n'
printf '%s\n' '</testsuites>'
}
@test "a failed testcase the report leaves unnamed blocks instead of dropping out of the list" {
# The name is the only key a marker can be looked up by, so a failed case with
# no name is a failure the gate cannot attribute. Enumerating names alone would
# skip it silently and open on a report whose failures it never finished
# reading -- the failure mode this counts against the names collected to catch.
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
report_with_unnameable 'classname=""' >"$tmp/report"
echo kubernetes-latest >"$tmp/markers"
rc=0
( set +x; sh "$GATE" sandbox ) >"$tmp/out" 2>&1 || rc=$?
[ "$rc" -ne 0 ]
rm -rf "$tmp"
}
@test "a failed testcase with a classname and no name blocks, rather than being read as the classname" {
# `classname="..."` ends in `name="..."`. A match that does not require the
# attribute boundary reads the classname as the test name -- and the classname
# here is the name of the suite that DOES carry a marker, so that misreading
# opens the gate rather than merely mislabelling a line.
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
report_with_unnameable 'classname="kubernetes-latest"' >"$tmp/report"
echo kubernetes-latest >"$tmp/markers"
rc=0
( set +x; sh "$GATE" sandbox ) >"$tmp/out" 2>&1 || rc=$?
[ "$rc" -ne 0 ]
rm -rf "$tmp"
}
@test "a failed testcase named with the empty string blocks, and blocks as one it cannot name" {
# `name=""` is the shape a pattern written for "a name attribute is present"
# accepts and a lookup cannot use: it matches and yields nothing.
#
# The REASON is asserted, not only the exit. Reading the failed list a line at
# a time means an empty name survives to the lookup, misses, and blocks as an
# unexplained test -- so a pattern that accepts it still ends in a red, and an
# assertion on the exit alone could not tell the two apart. What the exit code
# cannot say, the message can: a case the report did not name is an accounting
# failure, and saying so is what sends a reader at the report rather than at a
# test with a blank name.
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
report_with_unnameable 'name="" classname=""' >"$tmp/report"
echo kubernetes-latest >"$tmp/markers"
rc=0
( set +x; sh "$GATE" sandbox ) >"$tmp/out" 2>&1 || rc=$?
[ "$rc" -ne 0 ]
grep -q 'does not name' "$tmp/out"
rm -rf "$tmp"
}
@test "a failed test whose name is only blanks blocks, instead of dropping out of the loop" {
# The report is input, not something this gate gets to assume the shape of, so
# a name of blanks is a name it has to survive. Split into words it disappears:
# the loop body never runs for it, no marker is looked up, and the run ends
# green having examined nothing.
#
# The REASON is asserted, not only the exit, because two different repairs end
# in a red here and only one of them is this one. A name that reaches the
# lookup and misses is reported as a failure the lane does not tolerate; a name
# mangled on the way there -- by dropping `IFS=`, say, which hands the lookup
# an empty string -- blocks as well, and an assertion on the exit alone would
# call that fixed.
tmp=$(mktemp -d)
stage_sandbox_stub "$tmp"
{
printf '%s\n' '<testsuites name="chainsaw-report" tests="1" failures="1">'
printf ' <testsuite name="weird" tests="1" failures="1" errors="0" id="0" time="">\n'
printf ' <testcase name=" " classname="" time="1">\n'
printf ' <failure message="exit status 1"></failure>\n'
printf ' </testcase>\n </testsuite>\n'
printf '%s\n' '</testsuites>'
} >"$tmp/report"
rc=0
( set +x; sh "$GATE" sandbox ) >"$tmp/out" 2>&1 || rc=$?
[ "$rc" -ne 0 ]
grep -qxF ' ' "$tmp/lookups"
rm -rf "$tmp"
}
# The lanes that run the Chainsaw suite, split by whether a failed run there
# blocks a merge. Named rather than derived because "does this lane block a
# merge" is not a property any grep over the file can read, and checked against
# the derived set below so that a lane added to the tree has to be placed here
# before either guard passes.
SOFT_LANES=".github/workflows/pull-requests.yaml .github/workflows/e2e-fork.yaml"
HARD_LANES=".github/workflows/nightly.yaml .github/workflows/e2e-tag.yaml"
chainsaw_lanes() {
grep -lE 'make -C packages/core/testing .*test-chainsaw' .github/workflows/*.yaml | sort
}
@test "the merge-blocking lanes ask this gate and the other two never do" {
# The tolerance buys one thing -- that a runner degraded enough to miss the
# node-join deadline does not stand between a correct change and its merge --
# and it costs the ability to see that failure. So it belongs where a merge is
# at stake and nowhere else, and the two lanes left hard are hard for reasons
# of their own.
#
# e2e-tag is the release candidate's own e2e check. A soft pass there ships a
# release whose tenant clusters brought up no worker nodes, which is the one
# question this suite exists to answer.
#
# nightly is the only place an e2e failure reaches anyone who did not open a
# pull request: no lane in this tree notifies anywhere, so the red job IS the
# notification, and a soft pass turns it into silence.
lanes=$(chainsaw_lanes)
if [ -z "$lanes" ]; then
echo "found no workflow running the chainsaw suite at all, so this guard checked nothing" >&2
return 1
fi
expected=$(printf '%s\n' $SOFT_LANES $HARD_LANES | sort)
if [ "$lanes" != "$expected" ]; then
echo "the workflows running the chainsaw suite are not the ones this guard knows about; a new lane has to be declared as blocking or not before it can run the suite" >&2
printf 'found:\n%s\nknown:\n%s\n' "$lanes" "$expected" >&2
return 1
fi
for lane in $SOFT_LANES; do
if ! grep -q 'e2e-node-join-soft-red.sh' "$lane"; then
echo "$lane blocks merges and runs the chainsaw suite, but never asks hack/e2e-node-join-soft-red.sh, so the same failed run blocks there and not in the other blocking lane" >&2
return 1
fi
done
for lane in $HARD_LANES; do
if grep -q 'e2e-node-join-soft-red.sh' "$lane"; then
echo "$lane tolerates the node-join deadline; it must not -- e2e-tag would pass a release candidate whose workers never joined, and nightly is the only surface that reports an e2e failure to anyone" >&2
return 1
fi
done
}
@test "a soft lane hands the softening decision to whatever it gates on a failed job" {
# A softened red finishes the job GREEN, so the job's result stops answering
# "did the suite pass". Two kinds of consumer read it and each breaks its own
# way: a step gated on `failure()` silently stops running, and a job that
# REPORTS the result silently starts reporting a pass. Both have to be handed
# the decision instead of inferring it.
#
# The reporting handover is four links, and each is read inside the block it
# has to live in -- the step with the gate's id, the job that runs the suite,
# the job that reports its result -- because a link matched anywhere in the
# file passes while it sits on a neighbour, and the value is then empty for
# good with nothing to say so. Link 1 is read tighter still: the decision has
# to be written in the same branch as the exit that makes the job green, and
# no other exit may leave that block, or a run the gate REFUSED to soften
# leaves green with nothing recorded.
#
# What this does NOT do, and the list is the part to keep honest when this
# comment is next rewritten, because dropping it is how the previous two
# versions came to promise more than they held. It cannot tell one reporter
# from two: a second one added later rides on the strings the first put in its
# job. It reads the branch as text, so it holds that the softened case is
# written before the general one and reports success, not that it is the branch
# GitHub evaluates -- which also means a legitimate refactor into a ternary or
# a negated guard reds it. It knows nothing of a reader that consumes the
# result through a composite action or a reusable workflow. Every block here is
# delimited by indentation, so a shape that indents differently is outside what
# it can see rather than something it will report. And it reads the step as
# lines, so an `exit 0` inside a quoted script the step hands to something else
# counts the same as one of its own.
for lane in $SOFT_LANES; do
id=$(awk '/^ id: / { last = $2 } /e2e-node-join-soft-red\.sh/ { print last; exit }' "$lane")
if [ -z "$id" ]; then
echo "$lane calls the gate from a step with no id, so nothing downstream can read what the gate decided" >&2
return 1
fi
job=$(awk '/^ [A-Za-z0-9_-]+:$/ { j = substr($1, 1, length($1) - 1) }
/make -C packages\/core\/testing .*test-chainsaw/ { print j; exit }' "$lane")
if [ -z "$job" ]; then
echo "could not find which job in $lane runs the chainsaw suite; without it this guard reports success for having lost its input" >&2
return 1
fi
# Link 1, and the exit it has to travel with. Taken from the step's own first
# line rather than from its id, so a key written above the id is inside the
# block too, and ended at the next step whatever key THAT one leads with: a
# step led by `id:` or `uses:` is a step, and a terminator that only knows
# `- name:` reads it as more of this one and accepts a write that belongs to
# somebody else.
id_line=$(grep -n "^ id: ${id}\$" "$lane" | head -n 1 | cut -d: -f1)
step_start=$(awk -v i="$id_line" 'NR <= i && /^ - / { l = NR } END { print l }' "$lane")
if [ -z "$step_start" ]; then
echo "could not find where step ${id} of $lane begins; without that boundary the checks below cannot tell what is inside it" >&2
return 1
fi
step=$(awk -v s="$step_start" 'NR < s { next } NR > s && /^ - / { exit } { print }' "$lane")
# A step that cannot fail the job cannot block on a red the gate refused to
# soften either: it would go green with nothing recorded, and the status
# would call that run a pass. Read as a key with a value rather than as the
# word `true`, because the value may be an expression, and one that reads
# `${{ github.event_name == 'pull_request' }}` is `true` on this lane.
coe=$(printf '%s\n' "$step" | grep -E '^ *continue-on-error:' | head -n 1)
case "${coe##*: }" in
""|false) ;;
*)
echo "step ${id} in $lane carries continue-on-error (${coe# }), so a failure the gate REFUSED to soften leaves the job green with nothing recorded and the status calls it a pass" >&2
return 1
;;
esac
gate_i=$(printf '%s\n' "$step" | grep -n 'e2e-node-join-soft-red\.sh' | head -n 1 | cut -d: -f1)
if [ -z "$gate_i" ]; then
echo "step ${id} in $lane does not call the gate, so the id this guard followed is not the step that decides" >&2
return 1
fi
gate_indent=$(printf '%s\n' "$step" | sed -n "${gate_i}p" | sed 's/[^ ].*//')
end_i=$(printf '%s\n' "$step" | awk -v s="$gate_i" -v ind="$gate_indent" 'NR > s && $0 == ind "fi" { print NR; exit }')
if [ -z "$end_i" ]; then
echo "the gate's branch in step ${id} of $lane does not close where this can see it; without that boundary the checks below cannot tell what is inside it" >&2
return 1
fi
branch=$(printf '%s\n' "$step" | awk -v a="$gate_i" -v b="$end_i" 'NR > a && NR < b')
# Both spellings of the sink: `${GITHUB_OUTPUT}` is the same write. What is
# NOT the same write is $GITHUB_ENV, which reaches the same job and publishes
# nothing.
write=$(printf '%s\n' "$branch" | grep -F 'soft_red=true' | head -n 1)
if ! printf '%s\n' "$write" | grep -q 'GITHUB_OUTPUT'; then
echo "the branch the gate opens in step ${id} of $lane does not write soft_red=true to \$GITHUB_OUTPUT; whatever it writes instead is invisible to the job output below it, so everything downstream reads this run as one where the suite passed" >&2
return 1
fi
# And writes it whenever that branch is taken. A write put under a condition
# of its own -- scoped to full runs, say -- records the decision on some
# softened runs and not others, and the ones it skips go green with nothing
# recorded, which reads downstream as a suite that passed. Held as a shape
# rather than by parsing: a bare statement at the branch's own indentation.
# A write spelled some third way is outside this and reds rather than being
# judged.
case "$write" in
"${gate_indent} echo "*|"${gate_indent} printf "*) ;;
*)
echo "the write in step ${id} of $lane is not a bare statement in the branch the gate opens (${write}); a conditional or nested write records the decision on some runs and not others, and a softened run it skipped is reported as a pass" >&2
return 1
;;
esac
if ! printf '%s\n' "$branch" | grep -qE '^ *exit 0$'; then
echo "the branch the gate opens in step ${id} of $lane records the decision and does not exit zero, so the record is written for a run that goes on to fail anyway" >&2
return 1
fi
# In that order. A write standing after the exit is a write that never runs,
# and every check above it passes on a branch that records nothing.
write_i=$(printf '%s\n' "$branch" | grep -nF 'soft_red=true' | head -n 1 | cut -d: -f1)
exit_i=$(printf '%s\n' "$branch" | grep -nE '^ *exit 0$' | head -n 1 | cut -d: -f1)
if [ "$write_i" -gt "$exit_i" ]; then
echo "the branch the gate opens in step ${id} of $lane exits before it records the decision, so the record is never written and the run goes green carrying nothing" >&2
return 1
fi
# Outside that branch the step may do neither of the two things the handover
# rests on. Both are asked as "not at all" rather than "not this spelling",
# because a spelling is what an edit changes: the sink is named, not the
# value written to it, and an exit is judged by its operand rather than by
# matching the digit zero -- `exit $rc` is a green exit on every run where
# that variable is zero, and the decision can be assembled from a variable
# just as easily.
outside=$(printf '%s\n' "$step" | awk -v a="$gate_i" -v b="$end_i" 'NR <= a || NR >= b')
if printf '%s\n' "$outside" | grep -q 'GITHUB_OUTPUT'; then
echo "step ${id} in $lane writes to the output sink outside the branch the gate opens; a run the gate never softened would carry a decision anyway, and the status would report a passing suite as one that missed the deadline" >&2
return 1
fi
green=$(printf '%s\n' "$outside" | grep -oE '(^|[^[:alnum:]_])exit[[:space:]]+[^;&|)]*' \
| grep -vE 'exit[[:space:]]+[1-9][0-9]*$' | head -n 1)
if [ -n "$green" ]; then
echo "step ${id} in $lane can leave without a red exit outside the branch the gate opens (${green# }); a failure the gate REFUSED to soften would then go green with nothing recorded, and the status would call it a pass" >&2
return 1
fi
# And what the branch falls through to has to be a non-zero exit written
# out. Falling out of the block instead reaches whatever the step ends with,
# which on a step that ends by announcing success is a green exit; and an
# exit through a variable is a green exit whenever that variable is zero.
# The first EXIT after the branch, not the first line: a comment or a line
# of logging between the two leaves the property untouched, and a check that
# reds on one is a check somebody switches off.
after=$(printf '%s\n' "$step" | awk -v b="$end_i" -v ind="$gate_indent" 'NR > b && $0 ~ "^" ind "exit " { print; exit }')
case "$after" in
"${gate_indent}exit "[1-9]*) ;;
*)
echo "the branch the gate opens in step ${id} of $lane does not fall through to a written non-zero exit (${after:-nothing follows it}); a failure the gate refused to soften would then leave by whatever ends the step, and a step that ends by announcing success ends green" >&2
return 1
;;
esac
# Link 2: the job that ran the suite publishes that step's decision, read
# inside that job's block.
jobblock=$(awk -v want="$job" '
$0 ~ "^ " want ":$" { inj = 1; next }
inj && /^ [A-Za-z0-9_-]+:$/ { exit }
inj { print }' "$lane")
# The same question one level up, and it is not the same answer: a job that
# cannot fail reports `success` to everything that reads its result, so the
# step's own exit stops meaning anything at all.
jcoe=$(printf '%s\n' "$jobblock" | grep -E '^ continue-on-error:' | head -n 1)
case "${jcoe##*: }" in
""|false) ;;
*)
echo "job ${job} in $lane carries continue-on-error (${jcoe# }), so its result is success even when it failed, and a red the gate refused to soften reaches the status as a pass" >&2
return 1
;;
esac
if ! printf '%s\n' "$jobblock" | grep -q '^ outputs:$'; then
echo "job ${job} in $lane declares no outputs at all, so step ${id}'s decision stops at the job boundary" >&2
return 1
fi
if ! printf '%s\n' "$jobblock" | grep -vE '^ *#' | grep -qF " soft_red: \${{ steps.${id}.outputs.soft_red }}"; then
echo "job ${job} in $lane does not publish step ${id}'s decision as its own output; an output on another job, or one naming a step that does not exist, resolves to the empty string in silence" >&2
return 1
fi
# Links 3 and 4 live in the job that reports the result, so they are read
# there. Found by which job reads that result rather than by name.
# The reporter is the job that reads the suite's result AND publishes the
# check a merge waits on, not merely the first job that mentions the result:
# a notify or cleanup job keyed on the same result is not what a reader sees
# in front of the merge button. `E2E Tests` is that check's name, and the
# only literal here about another file.
reporter=""
for candidate in $(awk '/^ [A-Za-z0-9_-]+:$/ { print substr($1, 1, length($1) - 1) }' "$lane"); do
cblock=$(awk -v want="$candidate" '
$0 ~ "^ " want ":$" { inj = 1; next }
inj && /^ [A-Za-z0-9_-]+:$/ { exit }
inj { print }' "$lane")
if printf '%s\n' "$cblock" | grep -qF "needs.${job}.result" \
&& printf '%s\n' "$cblock" | grep -qE "context: ['\"]E2E Tests['\"]"; then
reporter="$candidate"
break
fi
done
if [ -z "$reporter" ]; then
echo "no job in $lane both reads the ${job} job's result and publishes the E2E Tests check, so nothing in it can carry the softening decision to what a merge waits on" >&2
return 1
fi
repblock=$(awk -v want="$reporter" '
$0 ~ "^ " want ":$" { inj = 1; next }
inj && /^ [A-Za-z0-9_-]+:$/ { exit }
inj { print }' "$lane")
seen_as=$(printf '%s\n' "$repblock" | grep -F "needs.${job}.outputs.soft_red" | grep -E '^ *[A-Z0-9_]+:' | head -n 1)
seen_as=$(printf '%s' "${seen_as%%:*}" | tr -d ' \t')
if [ -z "$seen_as" ]; then
echo "job ${reporter} in $lane reports the ${job} job's result without reading what that job decided; a red the lane tolerated finishes green, so that report would tell a reader the suite passed on a run where it failed" >&2
return 1
fi
result_as=$(printf '%s\n' "$repblock" | grep -F "needs.${job}.result" | grep -E '^ *[A-Z0-9_]+:' | head -n 1)
result_as=$(printf '%s' "${result_as%%:*}" | tr -d ' \t')
if [ -z "$result_as" ]; then
echo "could not read the name job ${reporter} in $lane knows the ${job} result by; without it the ordering check below has nothing to order against and would report success for having lost its input" >&2
return 1
fi
# Link 4: and branches on it, ahead of the branch that maps a successful
# result to "the suite passed", and to a success of its own. Comment lines
# are excluded on both sides: a branch named in a comment is not a branch.
soft_line=$(printf '%s\n' "$repblock" | grep -nF "${seen_as} === 'true'" | grep -vE ':[[:space:]]*//' | head -n 1 | cut -d: -f1)
plain_line=$(printf '%s\n' "$repblock" | grep -nF "${result_as} === 'success'" | grep -vF "${seen_as}" | grep -vE ':[[:space:]]*//' | head -n 1 | cut -d: -f1)
if [ -z "$soft_line" ] || [ -z "$plain_line" ]; then
echo "job ${reporter} in $lane passes ${seen_as} to its reporting script and never branches on it, so the status it publishes is the one it published before the softening existed" >&2
return 1
fi
if [ "$soft_line" -gt "$plain_line" ]; then
echo "job ${reporter} in $lane branches on ${seen_as} behind the branch that already answers a successful result; the softened case is written and never reached" >&2
return 1
fi
verdict=$(printf '%s\n' "$repblock" | awk -v s="$soft_line" 'NR > s && /^ *}/ { exit } NR > s && /state = / { print; exit }')
case "$verdict" in
*"'success'"*) ;;
*)
echo "job ${reporter} in $lane branches on ${seen_as} and that branch does not report success (${verdict:-no verdict in its body}); the lane decided not to block, and the check a merge waits on has to say so" >&2
return 1
;;
esac
done
# The step side, and the one step that stands on `failure()` today. The SSH
# breakpoint is the only interactive way into a sandbox whose node-join never
# finished -- the failure this lane stopped blocking on -- so skipping it there
# would take the one debugging path away on exactly the run it was built for.
checked=0
for lane in $SOFT_LANES; do
if ! grep -q 'name: Breakpoint on E2E failure' "$lane"; then
continue
fi
checked=$((checked + 1))
id=$(awk '/^ id: / { last = $2 } /e2e-node-join-soft-red\.sh/ { print last; exit }' "$lane")
block=$(awk '/name: Breakpoint on E2E failure/ { inb = 1; next }
inb && /^ uses: / { exit }
inb { print }' "$lane")
case "$block" in
*"steps.${id}.outputs.soft_red"*) ;;
*)
echo "the breakpoint in $lane opens only on a failed job, and a softened node-join red is a job that succeeded; it would be skipped on the one failure worth attaching to" >&2
return 1
;;
esac
# And it must not be able to fail the job it now runs inside. On the
# softened path everything else has already succeeded, so this step is the
# only thing left that could turn the lane's decision not to block into a
# block, decided by a debugging tool rather than by the tests.
case "$block" in
*"continue-on-error: true"*) ;;
*)
echo "the breakpoint in $lane can fail the job, and on a softened red that is a job which had decided not to block; a debugging session that cannot open would then be what blocks the merge" >&2
return 1
;;
esac
done
if [ "$checked" -eq 0 ]; then
echo "no lane that tolerates the node-join deadline carries a breakpoint step any more; this guard now checks nothing and should go with the step it was written for" >&2
return 1
fi
}
@test "the e2e job cap covers the kubernetes pair at its worst, and every lane carries the same one" {
# What this holds is the part that can be read out of source: the two bringup
# Tests run sequentially (parallel: 1), each can spend its whole operation
# ceiling and then the failure catch behind it, and the job cap has to sit
# above that or the run ends `cancelled` with cozyreport.tgz and the job log
# both lost.
#
# What it does NOT hold, stated because the cap in the workflows is larger
# than this check needs and a reader is owed the reason. The full worst case
# of those suites counts their `finally` legs and the storageclass-fallback
# Test beside them too: 67m operation, 19m catch and a 10m `finally` for each
# of the two bringup Tests, plus 15m and a 19m catch for the fallback Test,
# which is 226 minutes. No cap in this file has ever covered that sum -- the
# same arithmetic came to 192 against the previous 180-minute cap -- and one
# that did would be a cap that never fires.
#
# The margin above this bound is empirical, and this is the measurement: three
# nightlies that failed BOTH bringup Tests ran their full-suite E2E job in 149,
# 158 and 160 minutes under the previous ceilings (runs 32324686063,
# 32208653925, 32348997754). A run of that shape is the one that gains here,
# and it gains at most 12 minutes per suite -- 11 from the node-join deadline,
# 1 from the diagnostics phase budget -- so the top of that range projects to
# about 184 minutes, which the cap clears by about 30 where the old one cleared
# 160 by 20. The scoped lanes run a subset of that suite, so the full run is
# the one worth sizing against.
lib=hack/e2e-chainsaw/_lib/run-kubernetes.sh
op=$(grep -oE '^COZY_OP_CEILING=[0-9]+' "$lib" | head -n 1 | sed -E 's/.*=//')
# Both catch legs, read from the ops that carry them rather than restated: the
# previous-instance logs and the host snapshot each declare their own ceiling
# in the global config, and they run one after the other on a failed test.
#
# The sum reads whole minutes only, so a leg written in any other unit would
# be dropped from it silently and priced at zero. Counting the legs against
# the ones this could add up is what turns that into a red.
config=hack/e2e-chainsaw/.chainsaw.yaml
legs=$(grep -cE '^ timeout: ' "$config" || true)
priced=$(grep -cE '^ timeout: [0-9]+m$' "$config" || true)
if [ "$legs" != "$priced" ]; then
echo "a catch leg in $config declares its ceiling in something other than whole minutes ($legs legs, $priced this can add up); the sum below would price the rest at zero" >&2
return 1
fi
catch=$(grep -oE '^ timeout: [0-9]+m$' "$config" \
| grep -oE '[0-9]+' | awk '{ total += $1 } END { print total }')
for v in op catch; do
eval "n=\$$v"
if [ -z "$n" ] || [ "$n" -lt 1 ]; then
echo "expected to read $v from source; without it this guard reports success for having lost its input" >&2
return 1
fi
done
worst=$(( 2 * ((op / 60) + catch) ))
# The cap of the job that runs chainsaw, not every cap in the file: take the
# last one declared before the chainsaw invocation, which is the enclosing
# job's. Picking by value or by position in the file would be right only until
# a job is added above. Derived over every lane that runs the suite, so a new
# one is held to the same figure without being listed here.
caps=$(for lane in $(chainsaw_lanes); do
awk '/^ timeout-minutes: [0-9]+$/ { cap = $2 }
/make -C packages\/core\/testing .*test-chainsaw/ { if (cap) print cap; exit }' "$lane"
done | sort -u)
# One figure across all of them, so a lane cannot be raised alone and leave the
# others killing the same run they were meant to survive. The split in the
# guard above does not reach this: a hard lane runs the same suites for the
# same time and only decides differently at the end.
if [ "$(printf '%s\n' "$caps" | grep -c .)" -ne 1 ]; then
echo "the e2e lanes carry different job caps ($(printf '%s ' $caps)); the same failed run would then be cancelled in one lane and finish in another" >&2
return 1
fi
if [ "$caps" -lt "$worst" ]; then
echo "the e2e job cap is ${caps}m against a kubernetes pair that can spend ${worst}m on its own (2 x (${op}s ceiling + ${catch}m catch)); the cap would kill the run and take cozyreport.tgz and the job log with it" >&2
return 1
fi
}
hack/platform-packages_test.bats
+111
View File
@@ -0,0 +1,111 @@
#!/usr/bin/env bats
# Unit coverage for the lane-specific Package manifest emitted by
# hack/e2e-platform-packages.sh. This suite runs through hack/cozytest.sh, so it
# uses direct shell assertions rather than bats' run/status helpers.
@test "QEMU defaults keep the platform-managed LINSTOR package" {
manifest=$(hack/e2e-platform-packages.sh)
names=$(printf '%s\n' "$manifest" | yq -N '.metadata.name')
endpoint=$(printf '%s\n' "$manifest" | yq '.spec.components.platform.values.publishing.apiServerEndpoint')
if [ "$names" != "cozystack.cozystack-platform" ]; then
echo "default manifest unexpectedly replaces a platform package: $names" >&2
return 1
fi
if [ "$endpoint" != "https://192.168.123.10:6443" ]; then
echo "unexpected default API endpoint: $endpoint" >&2
return 1
fi
}
@test "container mode replaces LINSTOR with DRBD disabled" {
manifest=$(COZY_APISERVER_ENDPOINT=https://192.168.123.11:6443 COZY_LINSTOR_DRBD_ENABLED=false hack/e2e-platform-packages.sh)
names=$(printf '%s\n' "$manifest" | yq -N '.metadata.name')
expected_names=$(printf '%s\n' cozystack.cozystack-platform cozystack.linstor cozystack.kubevirt-cdi)
endpoint=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "cozystack.cozystack-platform") | .spec.components.platform.values.publishing.apiServerEndpoint')
disabled=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "cozystack.cozystack-platform") | .spec.components.platform.values.bundles.disabledPackages | join(",")')
drbd=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "cozystack.linstor") | .spec.components.linstor.values.drbd.enabled')
if [ "$names" != "$expected_names" ]; then
echo "container manifest has unexpected packages: $names" >&2
return 1
fi
if [ "$endpoint" != "https://192.168.123.11:6443" ]; then
echo "container API endpoint was not forwarded: $endpoint" >&2
return 1
fi
if [ "$disabled" != "cozystack.linstor,cozystack.kubevirt-cdi" ]; then
echo "platform-managed LINSTOR was not disabled: $disabled" >&2
return 1
fi
if [ "$drbd" != "false" ]; then
echo "replacement LINSTOR package did not disable DRBD: $drbd" >&2
return 1
fi
}
@test "container mode raises the CDI importer memory ceiling" {
manifest=$(COZY_APISERVER_ENDPOINT=https://192.168.123.11:6443 COZY_LINSTOR_DRBD_ENABLED=false hack/e2e-platform-packages.sh)
disabled=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "cozystack.cozystack-platform") | .spec.components.platform.values.bundles.disabledPackages | join(",")')
mem=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "cozystack.kubevirt-cdi") | .spec.components.kubevirt-cdi.values.importerResources.limits.memory')
cpu=$(printf '%s\n' "$manifest" | yq 'select(.metadata.name == "cozystack.kubevirt-cdi") | .spec.components.kubevirt-cdi.values.importerResources.limits.cpu')
if [ "$disabled" != "cozystack.linstor,cozystack.kubevirt-cdi" ]; then
echo "container mode did not replace both packages: $disabled" >&2
return 1
fi
if [ "$mem" != "4Gi" ]; then
echo "replacement CDI package did not raise the memory ceiling: $mem" >&2
return 1
fi
# The CPU ceiling deliberately stays at CDI's own default: the failure this
# override answers was memory, and widening CPU too would be an unmeasured
# second change riding along.
if [ "$cpu" != "750m" ]; then
echo "replacement CDI package moved the CPU ceiling off CDI's default: $cpu" >&2
return 1
fi
}
@test "the CDI chart default does not move production behaviour" {
# The override above is a lane concern. The chart itself must restate CDI's
# built-in requirements, so an install that sets nothing behaves as it did
# before importerResources existed.
defaults=$(helm template packages/system/kubevirt-cdi --set _cluster.root-host=example.org \
| yq -N 'select(.kind == "CDI") | .spec.config.podResourceRequirements | [.limits.cpu, .limits.memory, .requests.cpu, .requests.memory] | join(" ")' | grep .)
if [ "$defaults" != "750m 600M 100m 60M" ]; then
echo "kubevirt-cdi default podResourceRequirements drifted from CDI's own: $defaults" >&2
return 1
fi
}
@test "invalid DRBD mode fails before emitting YAML" {
if manifest=$(COZY_LINSTOR_DRBD_ENABLED=unsupported hack/e2e-platform-packages.sh); then
echo "invalid DRBD mode unexpectedly succeeded" >&2
return 1
fi
if [ -n "$manifest" ]; then
echo "invalid DRBD mode emitted a partial manifest" >&2
return 1
fi
}
@test "both gating workflows forward the E2E-only override into the sandbox" {
workflow_value=$(yq '.jobs.e2e.steps[] | select(.name == "Install Cozystack into sandbox") | .env.COZY_LINSTOR_DRBD_ENABLED' .github/workflows/pull-requests.yaml)
fork_value=$(yq '.jobs.e2e.steps[] | select(.name == "Install Cozystack into sandbox") | .env.COZY_LINSTOR_DRBD_ENABLED' .github/workflows/e2e-fork.yaml)
make_command=$(make -n -C packages/core/testing SANDBOX_NAME=test COZY_LINSTOR_DRBD_ENABLED=false install-cozystack)
if [ "$workflow_value" != "false" ]; then
echo "container workflow does not set COZY_LINSTOR_DRBD_ENABLED=false" >&2
return 1
fi
if [ "$fork_value" != "false" ]; then
echo "fork workflow does not set COZY_LINSTOR_DRBD_ENABLED=false" >&2
return 1
fi
if ! printf '%s\n' "$make_command" | grep -Fq -- '-e COZY_LINSTOR_DRBD_ENABLED="false"'; then
echo "testing Makefile does not forward COZY_LINSTOR_DRBD_ENABLED" >&2
return 1
fi
}
hack/run-kubernetes-cpu-throttle_test.bats
+5 -7
View File
@@ -172,11 +172,10 @@ run_capture() {
*'exit 127 and collect nothing'*) ;;
*) echo "expected the warning to still name the 127 class it is contrasting against" >&2; return 1 ;;
esac
# The sentence went through two false shapes before this one: "except the
# worker CPU throttling capture" (reads as sole exception, and ghcr-mirror has
# the same shape) and then "the two that carry a fallback" (there are three --
# talos-image-cache guards its call too). Both were enumerations, and an
# enumeration of a growing set is wrong the next time someone adds to it.
# The sentence went through false shapes before this one, including "except
# the worker CPU throttling capture", which read as a sole exception even
# though talos-image-cache has the same shape. An enumeration of a growing set
# is wrong the next time someone adds to it.
# What is pinned now is the DISCRIMINATOR: guarding the call with command -v
# is what separates the two outcomes, and every collector that guards is
# named. A fourth one added without a mention fails this.
@@ -184,7 +183,7 @@ run_capture() {
*'guard the call with command -v'*) ;;
*) echo "expected the warning to state what separates the two outcomes" >&2; return 1 ;;
esac
for c in 'CPU throttling' 'ghcr-mirror' 'talos-image-cache'; do
for c in 'CPU throttling' 'talos-image-cache'; do
case "$line" in
*"$c"*) ;;
*) echo "expected the warning to name the guarded collector: $c" >&2; return 1 ;;
@@ -714,7 +713,6 @@ run_capture() {
# count on purpose: the loop below is the enumeration, and a sentence that
# also counted would go stale the first time a leg is added to it.
for leg in 'cozy_capture_tenant_talos "${test_name}" || true' \
'ghcr_mirror_diagnose || true' \
'talos_image_cache_diagnose || true'; do
line=$(grep -n -F -x " ${leg}" "$lib" | head -n 1 | cut -d: -f1)
if [ -z "$line" ]; then
hack/run-kubernetes-drain_test.bats
+34 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bats
# -----------------------------------------------------------------------------
# Unit tests for cozy_tenant_drained in hack/e2e-chainsaw/_lib/run-kubernetes.sh
# Unit tests for the scoped tenant-drain helpers in run-kubernetes.sh.
#
# cozy_tenant_drained is the pure exit-condition of the inter-test drain loop
# (cozy_wait_tenant_drained). Each argument is one resource-probe capture: the
@@ -79,3 +79,36 @@
exit 1
fi
}
@test "PVC filtering ignores leftovers from databases and other Kubernetes clusters" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
pvcs=$(printf '%s\n' \
'persistentvolumeclaim/clickhouse-data-test' \
'persistentvolumeclaim/disk-system-kubernetes-test-previous-version-md0-worker-a' \
'persistentvolumeclaim/disk-system-kubernetes-test-latest-version-md0-worker-b')
got=$(cozy_filter_cluster_pvcs test-latest-version "$pvcs")
expected='persistentvolumeclaim/disk-system-kubernetes-test-latest-version-md0-worker-b'
[ "$got" = "$expected" ] || {
echo "PVC filter crossed cluster ownership: [$got]" >&2
exit 1
}
}
@test "scoped drain queries the CAPI label and ignores unrelated PVCs" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
calls=$(mktemp)
kubectl() {
printf '%s\n' "$*" >> "$calls"
case "$*" in
*" get pvc "*) printf 'persistentvolumeclaim/clickhouse-data-test\n' ;;
*) printf '' ;;
esac
}
cozy_wait_tenant_drained test-latest-version 0 >/dev/null
[ "$(grep -c 'cluster.x-k8s.io/cluster-name=kubernetes-test-latest-version' "$calls")" -eq 2 ] || {
echo "VM and VMI probes were not scoped to the current CAPI cluster" >&2
cat "$calls" >&2
exit 1
}
}
hack/run-kubernetes-join-timing_test.bats
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env bats
# Successful-node-join timing stays in the normal job log; these mocks keep its
# resource/event rendering cluster-free under hack/cozytest.sh.
kubectl() {
case "$*" in
*"get events.events.k8s.io"*)
printf '%b\n' \
'\t\t2026-08-27T13:50:07Z\t2026-08-27T13:49:00Z\tDataVolume\tkubernetes-test-latest-version-md0-a-worker-disk-system\tImportInProgress\tImport in progress' \
'2026-08-27T13:51:02Z\t\t\t2026-08-27T13:49:00Z\tDataVolume\tkubernetes-test-latest-version-md0-a-worker-disk-system\tCompleted\tImport Complete' \
'2026-08-27T13:51:06Z\t\t\t2026-08-27T13:49:00Z\tPod\tvirt-launcher-kubernetes-test-latest-version-md0-a-worker-abc\tPulled\tSuccessfully pulled image in 3.355s' \
'2026-08-27T13:51:54Z\t\t\t2026-08-27T13:49:00Z\tMachine\tkubernetes-test-latest-version-md0-a-worker\tSuccessfulSetNodeRef\tkubernetes-test-latest-version-md0-a-worker'
;;
*"get nodes"*)
printf '%b\n' 'worker-a\tcreated=2026-08-27T13:51:45Z\tReady=2026-08-27T13:52:03Z'
;;
*"logs deploy/talos-image-cache"*)
printf '%s\n' '2026-08-27T13:51:01Z transfer-finished status=complete path=/image/openstack-amd64.raw.xz range=0-123/124 requested=124 sent=124 duration=0.432s'
;;
*)
return 1
;;
esac
}
timeout() {
shift 3
"$@"
}
assert_contains() {
case "$1" in
*"$2"*) return 0 ;;
esac
echo "output did not contain: $2" >&2
return 1
}
@test "successful join report separates wait, lifecycle and image transfer timing" {
# shellcheck source=/dev/null
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
output=$(cozy_report_node_join_timing \
test-latest-version tenant-kubeconfig 100 140 345 2026-08-27T13:48:00Z)
assert_contains "$output" 'worker-pool request -> Ready wait: 40s'
assert_contains "$output" 'two-node Ready wait: 205s'
assert_contains "$output" 'worker-pool request -> two nodes Ready: 245s'
expected=$(printf 'DataVolume\tkubernetes-test-latest-version-md0-a-worker-disk-system\tImportInProgress')
assert_contains "$output" "$expected"
expected=$(printf '2026-08-27T13:50:07Z\tDataVolume')
assert_contains "$output" "$expected"
assert_contains "$output" 'Successfully pulled image in 3.355s'
expected=$(printf 'worker-a\tcreated=2026-08-27T13:51:45Z\tReady=2026-08-27T13:52:03Z')
assert_contains "$output" "$expected"
assert_contains "$output" 'duration=0.432s'
}
@test "worker timing starts immediately before the KubernetesNodes request" {
lib=hack/e2e-chainsaw/_lib/run-kubernetes.sh
stamp_line=$(grep -n '_worker_pool_requested_at=$(date +%s)' "$lib" | cut -d: -f1)
nodes_line=$(grep -n '^kind: KubernetesNodes$' "$lib" | cut -d: -f1)
[ -n "$stamp_line" ]
[ -n "$nodes_line" ]
[ "$stamp_line" -lt "$nodes_line" ]
[ "$(( nodes_line - stamp_line ))" -le 5 ]
}
@test "guest console summary prices direct GHCR kubelet image work" {
# shellcheck source=/dev/null
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
fixture="_out/tmp/guest-console-kubelet-$$.log"
mkdir -p "${fixture%/*}"
printf '%s\n' \
'[ 12.037680] [talos] pulling image ghcr.io/siderolabs/kubelet:v1.35.6: starting...' \
'[ 15.793093] [talos] service[kubelet](Preparing): Creating service runner' \
'[ 17.837505] [talos] service[kubelet](Running): Health check successful' \
>"$fixture"
output=$(_cozy_guest_kubelet_image_timing worker-a "$fixture")
rm -f "$fixture"
assert_contains "$output" 'kubelet image ghcr.io/siderolabs/kubelet:v1.35.6'
assert_contains "$output" 'fetch+unpack 3.755s (guest 12.038s -> 15.793s)'
assert_contains "$output" 'kubelet healthy at guest 17.838s'
assert_contains "$output" 'source=ghcr.io (no e2e registry mirror)'
}
hack/run-kubernetes-node-join_test.bats
+54 -88
View File
@@ -250,11 +250,10 @@ stub_collectors() {
# reached it outside a `timeout` wrapper. Stubbing a collector there does not
# make that test pass more easily; it removes the collector from the audit
# entirely, and the test stays green because there is nothing left to audit.
# Each of them carries reads the audit is the only instrument that sees:
# ghcr_mirror_diagnose issues five, and every cadvisor capture issues a Pod
# listing and a node read apiece. Their own suites substitute a grep over the
# source, which by construction cannot see a read that was added without a
# bound.
# Each of them carries reads the audit is the only instrument that sees: every
# cAdvisor capture issues a Pod listing and a node read apiece. Their own suites
# substitute a grep over the source, which by construction cannot see a read
# that was added without a bound.
#
# They are stubbed only where the test is about the phase budget rather than
# about the reads, which is the one place their real bodies would drown the
@@ -265,7 +264,6 @@ stub_gated_collectors() {
cozy_capture_tenant_worker_block_io() { printf 'block-io-stub\n'; }
cozy_capture_sandbox_node_cpu_time() { printf 'sandbox-cpu-time-stub\n'; }
cozy_capture_tenant_worker_thread_cpu() { printf 'thread-cpu-stub\n'; }
ghcr_mirror_diagnose() { printf 'ghcr-mirror-stub\n'; }
}
assert_file_contains() {
@@ -411,10 +409,6 @@ read_cost_inputs() {
# listing-keyed pin is satisfied by whichever of them ran, and stays green
# with the other absent from the audit entirely. The directory is the one
# thing each produces alone.
if ! grep -q 'get deploy ghcr-mirror' "$kubectl_calls"; then
echo "FAIL: ghcr_mirror_diagnose did not run, so its reads were not audited" >&2
false
fi
# Keyed on the directory having CONTENT, not on the directory existing. Each
# capture mkdir -p's its report directory before it lists anything, so an
# empty one is exactly what a capture that issued no read leaves behind -- and
@@ -797,8 +791,7 @@ read_cost_inputs() {
# A rule written for additions does not catch a removal, so both directions
# are named here.
for marker in serial-console-stub talos-stub image-cache-stub cpu-throttle-stub \
network-counters-stub block-io-stub sandbox-cpu-time-stub thread-cpu-stub \
ghcr-mirror-stub; do
network-counters-stub block-io-stub sandbox-cpu-time-stub thread-cpu-stub; do
if grep -q "$marker" "$tmp/out"; then
echo "FAIL: $marker ran after the phase ran out of budget" >&2
false
@@ -810,7 +803,6 @@ read_cost_inputs() {
assert_file_contains '(d) tenant worker CPU counters, sandbox node CPU time and worker per-thread CPU time: not collected' "$tmp/out"
assert_file_contains '(d2) tenant worker network counters: not collected' "$tmp/out"
assert_file_contains '(d3) tenant worker block IO counters: not collected' "$tmp/out"
assert_file_contains 'ghcr-mirror state, access log and warm-up Job: not collected' "$tmp/out"
rm -rf "$tmp"
}
@@ -966,7 +958,6 @@ read_cost_inputs() {
cozy_capture_runner_kernel_cpu_time() { :; }
cozy_capture_sandbox_qemu_thread_cpu() { :; }
cozy_capture_runner_canary() { :; }
ghcr_mirror_diagnose() { :; }
kubectl() { :; }
cozy_report_node_join_failure test-latest-version
' 2>&1) || true
@@ -1037,7 +1028,6 @@ read_cost_inputs() {
cozy_capture_sandbox_qemu_thread_cpu() { :; }
cozy_capture_runner_canary() { :; }
cozy_capture_tenant_worker_block_io() { :; }
ghcr_mirror_diagnose() { :; }
cozy_capture_sandbox_node_cpu_time() { :; }
cozy_capture_tenant_worker_thread_cpu() { :; }
kubectl() { printf "KUBECTL_RAN\n" >&2; }
@@ -1116,7 +1106,6 @@ read_cost_inputs() {
cozy_capture_sandbox_qemu_thread_cpu() { :; }
cozy_capture_runner_canary() { :; }
cozy_capture_tenant_worker_block_io() { :; }
ghcr_mirror_diagnose() { :; }
cozy_report_node_join_failure test-latest-version
' 2>&1) || true
printf '%s\n' "$out" >"$tmp/postsource"
@@ -1153,7 +1142,6 @@ read_cost_inputs() {
cozy_capture_sandbox_qemu_thread_cpu() { :; }
cozy_capture_runner_canary() { :; }
cozy_capture_tenant_worker_block_io() { :; }
ghcr_mirror_diagnose() { :; }
cozy_report_node_join_failure test-latest-version
' 2>&1) || true
printf '%s\n' "$out" >"$tmp/empty"
@@ -1252,16 +1240,15 @@ read_cost_inputs() {
console=$(grep -n "cozy_capture_tenant_serial_console 'node-join failed" "$lib" | head -n 1 | cut -d: -f1)
cpu=$(grep -n 'cozy_capture_tenant_worker_cpu_throttle "${_sample}" || true' "$lib" | head -n 1 | cut -d: -f1)
talos=$(grep -n 'cozy_capture_tenant_talos "${test_name}" || true' "$lib" | head -n 1 | cut -d: -f1)
mirror=$(grep -n 'ghcr_mirror_diagnose || true' "$lib" | head -n 1 | cut -d: -f1)
cache=$(grep -n 'talos_image_cache_diagnose || true' "$lib" | head -n 1 | cut -d: -f1)
for v in volumes console cpu talos mirror cache; do
for v in volumes console cpu talos cache; do
eval "n=\$$v"
if [ -z "$n" ]; then
echo "expected to locate $v in $lib" >&2
exit 1
fi
done
for later in console cpu talos mirror cache; do
for later in console cpu talos cache; do
eval "n=\$$later"
if [ "$volumes" -ge "$n" ]; then
echo "the volume state read (line $volumes) must precede $later (line $n), or a tight run declines it" >&2
@@ -1284,16 +1271,15 @@ read_cost_inputs() {
console=$(grep -n "cozy_capture_tenant_serial_console 'node-join failed" "$lib" | head -n 1 | cut -d: -f1)
cpu=$(grep -n 'cozy_capture_tenant_worker_cpu_throttle "${_sample}" || true' "$lib" | head -n 1 | cut -d: -f1)
talos=$(grep -n 'cozy_capture_tenant_talos "${test_name}" || true' "$lib" | head -n 1 | cut -d: -f1)
mirror=$(grep -n 'ghcr_mirror_diagnose || true' "$lib" | head -n 1 | cut -d: -f1)
cache=$(grep -n 'talos_image_cache_diagnose || true' "$lib" | head -n 1 | cut -d: -f1)
for v in block console cpu talos mirror cache; do
for v in block console cpu talos cache; do
eval "n=\$$v"
if [ -z "$n" ]; then
echo "expected to locate $v in $lib" >&2
exit 1
fi
done
for later in console cpu talos mirror cache; do
for later in console cpu talos cache; do
eval "n=\$$later"
if [ "$block" -ge "$n" ]; then
echo "the block IO counters (line $block) must precede $later (line $n), or a tight run declines them" >&2
@@ -1329,16 +1315,15 @@ read_cost_inputs() {
console=$(grep -n "cozy_capture_tenant_serial_console 'node-join failed" "$lib" | head -n 1 | cut -d: -f1)
cpu=$(grep -n 'cozy_capture_tenant_worker_cpu_throttle "${_sample}" || true' "$lib" | head -n 1 | cut -d: -f1)
talos=$(grep -n 'cozy_capture_tenant_talos "${test_name}" || true' "$lib" | head -n 1 | cut -d: -f1)
mirror=$(grep -n 'ghcr_mirror_diagnose || true' "$lib" | head -n 1 | cut -d: -f1)
cache=$(grep -n 'talos_image_cache_diagnose || true' "$lib" | head -n 1 | cut -d: -f1)
for v in cpu net console talos mirror cache; do
for v in cpu net console talos cache; do
eval "n=\$$v"
if [ -z "$n" ]; then
echo "expected to locate $v in $lib" >&2
exit 1
fi
done
for later in console cpu talos mirror cache; do
for later in console cpu talos cache; do
eval "n=\$$later"
if [ "$net" -ge "$n" ]; then
echo "the network counters (line $net) must precede $later (line $n), or a tight run declines them" >&2
@@ -1545,13 +1530,11 @@ read_cost_inputs() {
# enclosing definition so the name reported is the function's, not the line's.
#
# Over the sourced libraries as well as this file, and the list of them is
# read from the source rather than written out here. Two of the four captures
# the sentence names -- the ghcr-mirror and talos-image-cache diagnoses --
# live in libraries this file sources, so a scan of this file alone finds
# neither, and their arms in the table below sat unreachable while reading as
# coverage. A guarded read added to any sourced library is what this has to
# see, and hardcoding today's two would put the next library outside the scan
# in exactly the same silent way.
# read from the source rather than written out here. The talos-image-cache
# diagnosis named by the sentence lives in a library this file sources, so a
# scan of this file alone would not find it. A guarded read added to any
# sourced library is what this has to see, and hardcoding today's library
# would put the next one outside the scan in exactly the same silent way.
libs="$lib $(awk '/^\. hack\/e2e-chainsaw\/_lib\/[a-z-]+\.sh$/ { print $2 }' "$lib")"
for f in $libs; do
if [ ! -f "$f" ]; then
@@ -1599,7 +1582,6 @@ read_cost_inputs() {
'cozy_capture_sandbox_qemu_thread_cpu:sandbox QEMU per-thread CPU time:cozy_capture_sandbox_qemu_thread_cpu' \
'cozy_capture_runner_canary:runner fixed-work canary:_cozy_canary_run_arm' \
'cozy_capture_tenant_worker_thread_cpu:worker per-thread CPU time:cozy_capture_tenant_worker_thread_cpu _cozy_virt_launcher_listing' \
'ghcr_mirror_diagnose:ghcr-mirror:ghcr_mirror_diagnose _ghcr_mirror_bounded_read' \
'talos_image_cache_diagnose:talos-image-cache:talos_image_cache_diagnose _talos_image_cache_bounded_read'; do
fn=${entry%%:*}
rest=${entry#*:}
@@ -1644,7 +1626,6 @@ ${carrier}
# and the arm body is exempted below rather than given a phrase of its own.
cozy_capture_runner_canary) phrase='runner fixed-work canary' ;;
cozy_capture_tenant_worker_thread_cpu) phrase='worker per-thread CPU time' ;;
ghcr_mirror_diagnose) phrase='ghcr-mirror' ;;
talos_image_cache_diagnose) phrase='talos-image-cache' ;;
# The sentence enumerates the CAPTURES. Two other kinds of function guard
# the same way and are deliberately not in it, each exempt for a stated
@@ -1654,12 +1635,16 @@ ${carrier}
# them in the sentence too would be a second copy of that claim to keep in
# step. A function that is neither still fails below, which is what makes
# this a list of exemptions rather than a list of everything.
cozy_diag_read | _ghcr_mirror_bounded_read | _talos_image_cache_bounded_read) continue ;;
cozy_diag_read | _talos_image_cache_bounded_read) continue ;;
_cozy_cadvisor_node_stream | _cozy_virt_launcher_listing) continue ;;
# The canary carries its guard in the body that runs one arm, the way the
# cAdvisor captures carry theirs in the stream they share, so the sentence
# names the capture and the table above names what makes it true of it.
_cozy_canary_run_arm) continue ;;
# This is the successful-path timing report, called only after the join
# completed. Its bounds protect optional reads without belonging to the
# failure diagnostics phase whose missing-timeout warning is under test.
cozy_report_node_join_timing) continue ;;
cozy_report_node_join_failure | _talos_image_cache_deploy_state) continue ;;
*)
echo "$fn guards its call with command -v but this test has no phrase for it; add one here and to the warning" >&2
@@ -1716,7 +1701,6 @@ ${carrier}
cozy_capture_tenant_worker_block_io) phrase='worker block IO counters' ;;
cozy_capture_tenant_serial_console) phrase='serial-console family' ;;
cozy_capture_tenant_talos) phrase='guest Talos capture' ;;
ghcr_mirror_diagnose) phrase='ghcr-mirror state' ;;
talos_image_cache_diagnose) phrase='talos-image-cache diagnosis' ;;
# Called with the same suffix but not behind the phase gate: it runs ahead
# of the headline so the console experiment's own failure is named before
@@ -1736,6 +1720,9 @@ ${carrier}
cozy_capture_sandbox_qemu_thread_cpu) continue ;;
# The fourth bracket half, and also not behind the gate.
cozy_capture_runner_canary) continue ;;
# This is outside the failure diagnostics phase and deliberately remains
# best-effort: a missing timing report must not turn a completed join red.
cozy_report_node_join_timing) continue ;;
*)
echo "$fn runs in the diagnostics block but this test has no phrase for it; add one here and to both chainsaw comments" >&2
exit 1
@@ -1978,7 +1965,6 @@ EOF
cozy_capture_sandbox_qemu_thread_cpu() { :; }
cozy_capture_runner_canary() { :; }
cozy_capture_tenant_worker_block_io() { :; }
ghcr_mirror_diagnose() { :; }
kubectl() { :; }
COZY_DIAG_PHASE_BUDGET=0
cozy_report_node_join_failure test-latest-version
@@ -2128,53 +2114,46 @@ EOF
rm -rf "$tmp"
}
@test "the node-join deadline leaves an annotation and a marker, and does not touch the exit" {
@test "the node-join deadline leaves an annotation and does not touch the exit" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
tmp=$(mktemp -d)
use_temp_report_dir "$tmp"
export COZY_SNAPSHOT_NAME=kubernetes-latest
_COZY_NODE_JOIN_SOFT_RED=0
# `set +x` like every other capture in this file, and here it is the assertion
# that depends on it rather than the readability: cozytest.sh runs each test
# under `set -x`, the trace of the `echo` carries the annotation text verbatim,
# and it lands in the same file this test then greps. Left traced, all three
# and it lands in the same file this test then greps. Left traced, the
# assertions below pass against a collector that emits nothing -- and only
# under the runner CI actually uses.
rc=0
( set +x; cozy_soft_red_node_join test-latest-version ) >"$tmp/out" 2>&1 || rc=$?
( set +x; cozy_note_node_join_deadline test-latest-version ) >"$tmp/out" 2>&1 || rc=$?
# This function records; the caller fails. A non-zero return here would reach
# the caller under `set -e` and replace the exit it chose, which is the one
# thing a recorder must not do.
[ "$rc" -eq 0 ]
# The annotation is the only surface that reaches a reader who sees a green
# job and never opens the log, so it is asserted as an annotation and not
# merely as a line of text.
# Asserted as an annotation and not merely as a line of text: the `::warning`
# has to start the line or GitHub renders it as ordinary log output.
assert_file_contains '::warning title=node-join::' "$tmp/out"
# And it carries the deadline it is about. A warning that says only "the join
# failed" leaves a reader unable to tell it from the wait having been raised.
assert_file_contains 'Ready within 29m' "$tmp/out"
# It names the suite, because the report holds one directory per suite and a
# run carries more than one tenant cluster.
assert_file_contains 'test-latest-version' "$tmp/out"
# `::error` over a step that exits zero states a contradiction, and GitHub
# renders it beside genuine failures.
assert_file_lacks_pattern '::error' "$tmp/out"
marker="$tmp/report/snapshots/kubernetes-latest/SOFT-RED-node-join.txt"
[ -f "$marker" ]
# Named for the run it belongs to: the report holds one directory per suite,
# and a marker that does not say which cluster it is about cannot be counted
# across runs, which is the only reason it is written to a file at all.
assert_file_contains 'test-latest-version' "$marker"
unset COZY_SNAPSHOT_NAME
rm -rf "$tmp"
}
@test "no exit in the suite body is a zero, and the marker is written under the deadline test" {
@test "no exit in the suite body is a zero, and the deadline annotation is gated" {
lib=hack/e2e-chainsaw/_lib/run-kubernetes.sh
# The suite fails on every failure it has, the node-join deadline included.
# That is what keeps Chainsaw's catch running and the report honest, and it is
# what the lane's tolerance is built on top of rather than instead of: a zero
# That is what keeps Chainsaw's catch running and the report honest: a zero
# exit here would buy a green check by throwing away the four collectors keyed
# on the test failing, which is the shape this design exists to avoid.
#
@@ -2212,15 +2191,15 @@ EOF
echo "run_kubernetes_test has $zeros zero exits of its own; a failure that exits zero takes Chainsaw's catch, the previous-instance logs, the host snapshot and the event dump with it, and buys a green check with the evidence" >&2
return 1
fi
# And the marker call sits inside the branch the node-join wait opens, under
# And the annotation sits inside the branch the node-join wait opens, under
# the test that says which non-zero the wrapper gave. Without that gate the
# branch marks a wait that could not run at all as a deadline the lane
# tolerates, which is a different claim about a different failure.
# branch announces an expired deadline for a wait that could not run at all,
# which is a different claim about a different failure.
# Anchored on the wait's own body rather than on its shape: `timeout Nm bash -c`
# matches more than one line in the library, and picking the first would
# re-anchor this check the day another one is added above it.
wait_line=$(awk '/^ timeout [0-9]+m bash -c/ { cand = NR; next } cand && NR == cand + 1 && /get nodes --no-headers/ { print cand; exit } { cand = 0 }' "$lib")
call_line=$(grep -n 'cozy_soft_red_node_join "' "$lib" | head -n 1 | cut -d: -f1)
call_line=$(grep -n 'cozy_note_node_join_deadline "' "$lib" | head -n 1 | cut -d: -f1)
gate_line=$(grep -n 'if cozy_node_join_deadline_expired ' "$lib" | head -n 1 | cut -d: -f1)
for v in wait_line call_line gate_line; do
eval "n=\$$v"
@@ -2231,11 +2210,11 @@ EOF
done
# The discriminator is 124, which `timeout` gives when it fired. A `-k` on this
# call -- the house idiom on every other bounded read in the file -- makes a
# child that ignores SIGTERM come back as 137 instead, and the softening then
# silently stops applying to the case it was written for. It fails toward hard
# red, so it breaks nothing; it just stops working with no sign.
# child that ignores SIGTERM come back as 137 instead, and the branch then
# silently stops recognising the case it was written for: the run still fails,
# but it stops saying the deadline is why.
if sed -n "${wait_line}p" "$lib" | grep -q ' -k '; then
echo "the node-join wait carries -k; a child that ignores SIGTERM then returns 137 rather than 124, and the softened deadline stops recognising its own case" >&2
echo "the node-join wait carries -k; a child that ignores SIGTERM then returns 137 rather than 124, and the deadline branch stops recognising its own case" >&2
return 1
fi
end_line=$(awk -v s="$wait_line" 'NR > s && /^ fi$/ { print NR; exit }' "$lib")
@@ -2244,44 +2223,31 @@ EOF
return 1
fi
if [ "$call_line" -lt "$wait_line" ] || [ "$call_line" -gt "$end_line" ]; then
echo "cozy_soft_red_node_join is called at line $call_line, outside the node-join wait's failure branch (lines $wait_line to $end_line); the softening covers that deadline and nothing else" >&2
echo "cozy_note_node_join_deadline is called at line $call_line, outside the node-join wait's failure branch (lines $wait_line to $end_line); the annotation covers that deadline and nothing else" >&2
return 1
fi
if [ "$gate_line" -lt "$wait_line" ] || [ "$gate_line" -gt "$call_line" ]; then
echo "the softening at line $call_line is not gated by cozy_node_join_deadline_expired between the wait (line $wait_line) and itself; a branch that softens whatever the wrapper returned reports slow workers for a wait that never ran" >&2
return 1
fi
# And it is written before the diagnostics rather than after them. That phase
# spends minutes under a job cap that can end the run inside it, and a marker
# written on the far side of it is missing on exactly the runs that spent
# longest -- which the lane reads as a failure it cannot attribute.
diag_line=$(awk -v s="$wait_line" 'NR > s && /^ cozy_report_node_join_failure "/ { print NR; exit }' "$lib")
if [ -z "$diag_line" ]; then
echo "expected the node-join failure diagnostics to be called from the wait's branch in $lib; without it this guard reports success for having lost its input" >&2
return 1
fi
if [ "$call_line" -gt "$diag_line" ]; then
echo "the marker is written at line $call_line, after the diagnostics phase at line $diag_line; a run the job cap ends inside that phase then carries no marker and blocks as an unattributable red" >&2
echo "the annotation at line $call_line is not gated by cozy_node_join_deadline_expired between the wait (line $wait_line) and itself; a branch that annotates whatever the wrapper returned reports slow workers for a wait that never ran" >&2
return 1
fi
}
@test "only the status that means the deadline expired is softened" {
@test "only the status that means the deadline expired is reported as one" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
# 124 is what `timeout` returns when it fired, and it is the whole of what the
# softening is about. The wrapper's other failures arrive in the same shape
# and mean the wait did not run: 125 is `timeout` itself failing, 126 and 127
# are the shell being unusable or absent, and 128+n is that shell killed by a
# signal -- 137 above all, which on this path is the OOM killer on the loaded
# runner the whole change is about. Reporting any of those as "the workers
# were slow" is the one claim this must not start making.
# deadline branch is about. The wrapper's other failures arrive in the same
# shape and mean the wait did not run: 125 is `timeout` itself failing, 126 and
# 127 are the shell being unusable or absent, and 128+n is that shell killed by
# a signal -- 137 above all, which on this path is the OOM killer on a loaded
# runner. Reporting any of those as "the workers were slow" is the one claim
# this must not start making.
if ! cozy_node_join_deadline_expired 124; then
echo "exit 124 is timeout firing, and it is the status the softening exists for" >&2
echo "exit 124 is timeout firing, and it is the status the deadline branch exists for" >&2
return 1
fi
for rc in 1 2 125 126 127 137 143; do
if cozy_node_join_deadline_expired "$rc"; then
echo "exit $rc was read as the node-join deadline expiring; that status means the wait did not run, and softening it reports a slow cluster for a broken harness" >&2
echo "exit $rc was read as the node-join deadline expiring; that status means the wait did not run, and announcing it reports a slow cluster for a broken harness" >&2
return 1
fi
done
@@ -2318,7 +2284,7 @@ EOF
# being written anywhere, and every branch below would then agree vacuously.
# Set just under what the tree carries today, so a copy going out of reach is
# noticed while a copy legitimately deleted is not a failure.
if [ "$count" -lt 18 ]; then
if [ "$count" -lt 15 ]; then
echo "this pattern found the node-join deadline quoted in $count places, and the tree quotes it in more than that; it has stopped matching what it was written to sweep, so the check below would pass over whatever it no longer sees" >&2
return 1
fi
hack/run-kubernetes-oidc_test.bats
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env bats
# Unit coverage for the OIDC lifecycle folded into kubernetes-latest.
# Sourcing run-kubernetes.sh only defines functions; every kubectl interaction
# below is replaced with a shell stub, so this file needs no cluster.
@test "HelmRelease upgrade wait rejects stale Ready and stale observedGeneration" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
state_file=$(mktemp)
printf '0\n' > "$state_file"
kubectl() {
count=$(sed -n '1p' "$state_file")
count=$(( count + 1 ))
printf '%s\n' "$count" > "$state_file"
case "$count" in
1) printf '7|7|True|' ;;
2) printf '8|7|True|' ;;
*) printf '8|8|True|' ;;
esac
}
sleep() { :; }
cozy_wait_helmrelease_upgrade tenant-test kubernetes-demo 7 5 >/dev/null
[ "$(sed -n '1p' "$state_file")" -eq 3 ] || {
echo "the wait accepted Ready before both generation gates advanced" >&2
exit 1
}
}
@test "HelmRelease upgrade wait fails immediately on Stalled" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
calls=$(mktemp)
kubectl() {
printf '%s\n' "$*" >> "$calls"
case "$*" in
*" get helmrelease "*) printf '8|8|False|True' ;;
*" describe helmrelease "*) return 0 ;;
*) return 1 ;;
esac
}
sleep() { echo "sleep must not run after Stalled=True" >&2; return 1; }
if cozy_wait_helmrelease_upgrade tenant-test kubernetes-demo 7 600 >/dev/null 2>&1; then
echo "a Stalled HelmRelease was accepted" >&2
exit 1
fi
[ "$(wc -l < "$calls")" -eq 2 ] || {
echo "the Stalled path did not stop after status plus diagnostics" >&2
cat "$calls" >&2
exit 1
}
}
@test "System OIDC assertion proves rendered objects bootstrap kubeconfig and tenant RBAC" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
kubectl() {
case "$*" in
*" wait job kubernetes-demo-oidc-bootstrap "*) return 0 ;;
*" get kamajicontrolplane kubernetes-demo "*)
printf '[--authentication-config=/etc/kubernetes/authentication-config/config.yaml]'
;;
*" get secret kubernetes-demo-oidc-authn-config "*)
printf 'url: https://keycloak.example.test/realms/cozy\naudiences:\n- tenant-test-kubernetes-demo\n' | base64 | tr -d '\n'
;;
*" get keycloakclient.v1.edp.epam.com tenant-test-kubernetes-demo "*) printf 'true' ;;
*" get keycloakclientscope.v1.edp.epam.com tenant-test-kubernetes-demo-audience "*) printf 'oidc-audience-mapper' ;;
*" get secret kubernetes-demo-oidc-kubeconfig "*)
printf 'args:\n- oidc-login\n- --oidc-client-id=tenant-test-kubernetes-demo\n' | base64 | tr -d '\n'
;;
*) echo "unexpected kubectl call: $*" >&2; return 1 ;;
esac
}
cozy_oidc_bindings() {
printf 'e2e-admin@example.test\tcluster-admin\ne2e-viewer@example.test\tview\n'
}
cozy_assert_oidc_system demo
}
@test "CustomConfig transition patches the live cluster waits for the new generation and rejects System leftovers" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
calls=$(mktemp)
wait_args=$(mktemp)
patch_body=$(mktemp)
kubectl() {
printf '%s\n' "$*" >> "$calls"
case "$*" in
*" get helmrelease kubernetes-demo "*) printf '7' ;;
*" patch kuberneteses.apps.cozystack.io demo "*)
previous=
for argument in "$@"; do
if [ "$previous" = --patch ]; then
printf '%s\n' "$argument" > "$patch_body"
break
fi
previous="$argument"
done
;;
*" wait job kubernetes-demo-oidc-bootstrap "*) return 0 ;;
*" get kamajicontrolplane kubernetes-demo "*)
printf '[--authentication-config=/etc/kubernetes/authentication-config/config.yaml]'
;;
*" get secret kubernetes-demo-oidc-authn-config "*)
printf 'url: https://idp.byo.example.test\naudiences:\n- cozystack-byo-demo\n' | base64 | tr -d '\n'
;;
*" get keycloakclient.v1.edp.epam.com tenant-test-kubernetes-demo") return 1 ;;
*" get keycloakclientscope.v1.edp.epam.com tenant-test-kubernetes-demo-audience") return 1 ;;
*" get secret kubernetes-demo-oidc-kubeconfig") return 1 ;;
*) echo "unexpected kubectl call: $*" >&2; return 1 ;;
esac
}
cozy_wait_helmrelease_upgrade() { printf '%s\n' "$*" > "$wait_args"; }
cozy_oidc_bindings() { printf 'byo-admin@example.test\tcluster-admin\n'; }
cozy_switch_and_assert_oidc_custom_config demo
[ "$(sed -n '1p' "$wait_args")" = 'tenant-test kubernetes-demo 7 600' ] || {
echo "the transition did not wait from the pre-patch generation" >&2
cat "$wait_args" >&2
exit 1
}
jq -e '.spec.oidc.mode == "CustomConfig"' "$patch_body" >/dev/null
jq -e '.spec.oidc.users == [{"email":"byo-admin@example.test","role":"admin"}]' "$patch_body" >/dev/null
jq -er '.spec.oidc.customConfig.config' "$patch_body" | grep -q 'https://idp.byo.example.test'
jq -er '.spec.oidc.customConfig.config' "$patch_body" | grep -q 'cozystack-byo-demo'
}
hack/run-kubernetes-serial-console_test.bats
+17 -7
View File
@@ -1044,7 +1044,7 @@ run_capture() {
fi
}
@test "the passing path captures the console too, before the tenant is torn down" {
@test "the passing path captures the console and leaves teardown to Chainsaw finally" {
lib=hack/e2e-chainsaw/_lib/run-kubernetes.sh
# An instrument that only runs on the failure path answers in one direction.
# Every console this tree has ever captured came from a run that failed, so
@@ -1054,8 +1054,8 @@ run_capture() {
# that value comes from, and it exists only until the tenant is deleted.
gate=$(grep -n '^ *cozy_assert_guest_console_attached || attach_rc=$?$' "$lib" | head -n 1 | cut -d: -f1)
green=$(grep -n "^ *if ! cozy_capture_tenant_serial_console 'the suite passed" "$lib" | head -n 1 | cut -d: -f1)
teardown=$(grep -n '^ *kubectl -n tenant-test delete kuberneteses.apps.cozystack.io "${test_name}" --ignore-not-found --wait=false' "$lib" | tail -n 1 | cut -d: -f1)
for v in gate green teardown; do
function_end=$(awk -v start="$green" 'NR > start && /^}$/ { print NR; exit }' "$lib")
for v in gate green function_end; do
eval "n=\$$v"
if [ -z "$n" ]; then
echo "expected to locate $v in $lib" >&2
@@ -1063,12 +1063,22 @@ run_capture() {
fi
done
# After the attach check, because that check is what says there is a container
# to read; before the delete, because the Pod holding the stream goes with the
# tenant and nothing recovers it afterwards.
if [ "$green" -le "$gate" ] || [ "$green" -ge "$teardown" ]; then
echo "the passing-path capture (line $green) must sit between the attach check ($gate) and the teardown ($teardown)" >&2
# to read. The function must not start an asynchronous tenant teardown after
# the capture: each suite's finally block owns the single authoritative delete.
if [ "$green" -le "$gate" ]; then
echo "the passing-path capture (line $green) must sit after the attach check ($gate)" >&2
return 1
fi
if sed -n "${green},${function_end}p" "$lib" | grep -Eq 'delete kubernetes(nodeses|es)[.]apps[.]cozystack[.]io'; then
echo "run_kubernetes_test still starts tenant teardown on its success path" >&2
return 1
fi
for suite in latest previous; do
grep -Fq "cozy_cleanup test-${suite}-version" "hack/e2e-chainsaw/kubernetes-${suite}/chainsaw-test.yaml" || {
echo "kubernetes-${suite} finally does not own authoritative tenant teardown" >&2
return 1
}
done
}
@test "the passing path asks for the pool minimum, not the pool" {
hack/run-kubernetes-talos-diagnostics_test.bats
+1 -1
View File
@@ -471,7 +471,7 @@ assert_file_lacks_pattern() {
tmp=$(mktemp -d)
kubectl_calls="$tmp/kubectl.calls"
cozy_cleanup
cozy_cleanup test-latest-version
[ "$(sed -n '2p' "$kubectl_calls")" = '-n tenant-test delete certificates.cert-manager.io -l cozystack-e2e.io/tenant-talos-diagnostics --ignore-not-found --wait=true --timeout=30s' ]
[ "$(sed -n '3p' "$kubectl_calls")" = '-n tenant-test delete pod,secret -l cozystack-e2e.io/tenant-talos-diagnostics --ignore-not-found --wait=false' ]
hack/run-kubernetes-talos-spec_test.bats
+22 -60
View File
@@ -2,23 +2,21 @@
# -----------------------------------------------------------------------------
# Unit tests for talos_spec_block in hack/e2e-chainsaw/_lib/run-kubernetes.sh
#
# It is the composition point for the tenant Kubernetes CR's `spec.talos`: two
# independent in-sandbox mirrors (the Talos OS image cache's imageFactoryURL and
# the ghcr.io pull-through's registryMirrors) each contribute a fragment, and
# either, both or neither may be present on a given run. The result is spliced
# into an unquoted heredoc directly under `spec:`, so an indentation slip or a
# stray `talos:` key does not fail this function -- it produces a CR the API
# rejects, and every kubernetes-* suite dies at tenant creation with an error
# that points at the CR rather than at this line.
# It is the composition point for the tenant Kubernetes CR's `spec.talos`: the
# Talos OS image cache contributes imageFactoryURL when available. The result is
# spliced into an unquoted heredoc directly under `spec:`, so an indentation slip
# does not fail this function -- it produces a CR the API rejects, and every
# kubernetes-* suite dies at tenant creation with an error that points at the CR
# rather than at this line.
#
# Both resolvers are replaced with stubs after sourcing, which reduces this to
# the pure composition it is: the resolvers have their own coverage
# (hack/ghcr-mirror_test.bats) and need a cluster to mean anything.
# The resolver is replaced with a stub after sourcing, which reduces this to the
# pure composition it is: the resolver has its own coverage in
# hack/talos-image-cache_test.bats and needs a cluster to mean anything.
#
# The four-combination assertions go through yq on a spliced document rather
# than through grep on the string, so what they pin is "the CR parses and the
# keys land under spec.talos", which is the actual contract. A string match
# would keep passing on a block indented into the wrong parent.
# The assertion goes through yq on a spliced document rather than through grep
# on the string, so what it pins is "the CR parses and the key lands under
# spec.talos", which is the actual contract. A string match would keep passing
# on a block indented into the wrong parent.
#
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
# own line; there is no bats `run`/`$status`. Sourcing run-kubernetes.sh only
@@ -33,66 +31,30 @@ spec_doc() {
printf 'apiVersion: apps.cozystack.io/v1alpha1\nkind: Kubernetes\nmetadata:\n name: t\nspec:\n%s\n host: ""\n' "$1"
}
@test "no mirror up renders nothing, so the chart defaults apply" {
@test "no Talos image cache renders nothing, so the chart default applies" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
resolve_talos_image_factory_url() { printf ''; }
resolve_ghcr_mirror_endpoint() { printf ''; }
out=$(talos_spec_block)
[ -z "$out" ] || { echo "expected no spec.talos block when neither mirror is up, got [$out]" >&2; exit 1; }
[ -z "$out" ] || { echo "expected no spec.talos block when the image cache is down, got [$out]" >&2; exit 1; }
}
@test "only the Talos image cache up renders imageFactoryURL and no registryMirrors" {
@test "the Talos image cache renders imageFactoryURL and no registryMirrors" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
resolve_talos_image_factory_url() { printf 'http://talos-image-cache.kube-system.svc'; }
resolve_ghcr_mirror_endpoint() { printf ''; }
work=$(mktemp -d)
spec_doc "$(talos_spec_block)" > "$work/cr.yaml"
got=$(yq '.spec.talos.imageFactoryURL' "$work/cr.yaml")
[ "$got" = "http://talos-image-cache.kube-system.svc" ] || { echo "imageFactoryURL did not land under spec.talos, got [$got]" >&2; cat "$work/cr.yaml" >&2; rm -rf "$work"; exit 1; }
mirrors=$(yq '.spec.talos.registryMirrors' "$work/cr.yaml")
[ "$mirrors" = "null" ] || { echo "expected no registryMirrors when the ghcr mirror is down, got [$mirrors]" >&2; rm -rf "$work"; exit 1; }
rm -rf "$work"
}
@test "only the ghcr.io mirror up renders registryMirrors and no imageFactoryURL" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
resolve_talos_image_factory_url() { printf ''; }
resolve_ghcr_mirror_endpoint() { printf 'http://ghcr-mirror.kube-system.svc'; }
work=$(mktemp -d)
spec_doc "$(talos_spec_block)" > "$work/cr.yaml"
got=$(yq '.spec.talos.registryMirrors["ghcr.io"].endpoints[0]' "$work/cr.yaml")
[ "$got" = "http://ghcr-mirror.kube-system.svc" ] || { echo "mirror endpoint did not land under spec.talos.registryMirrors, got [$got]" >&2; cat "$work/cr.yaml" >&2; rm -rf "$work"; exit 1; }
url=$(yq '.spec.talos.imageFactoryURL' "$work/cr.yaml")
[ "$url" = "null" ] || { echo "expected no imageFactoryURL when the Talos cache is down, got [$url]" >&2; rm -rf "$work"; exit 1; }
rm -rf "$work"
}
@test "both mirrors up render under one talos key" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
resolve_talos_image_factory_url() { printf 'http://talos-image-cache.kube-system.svc'; }
resolve_ghcr_mirror_endpoint() { printf 'http://ghcr-mirror.kube-system.svc'; }
work=$(mktemp -d)
block=$(talos_spec_block)
spec_doc "$block" > "$work/cr.yaml"
url=$(yq '.spec.talos.imageFactoryURL' "$work/cr.yaml")
ep=$(yq '.spec.talos.registryMirrors["ghcr.io"].endpoints[0]' "$work/cr.yaml")
[ "$url" = "http://talos-image-cache.kube-system.svc" ] || { echo "imageFactoryURL missing when both mirrors are up, got [$url]" >&2; cat "$work/cr.yaml" >&2; rm -rf "$work"; exit 1; }
[ "$ep" = "http://ghcr-mirror.kube-system.svc" ] || { echo "mirror endpoint missing when both mirrors are up, got [$ep]" >&2; cat "$work/cr.yaml" >&2; rm -rf "$work"; exit 1; }
# A second `talos:` would still parse -- yq keeps the last duplicate -- and
# would silently drop whichever key came first, so count the key rather than
# trusting the reads above.
keys=$(printf '%s\n' "$block" | grep -c '^ talos:$')
[ "$keys" -eq 1 ] || { echo "expected exactly one talos key, found $keys" >&2; rm -rf "$work"; exit 1; }
[ "$mirrors" = "null" ] || { echo "e2e must not inject registryMirrors, got [$mirrors]" >&2; rm -rf "$work"; exit 1; }
rm -rf "$work"
}
@test "the key the sandbox keys off is spelled the way the chart declares it" {
. hack/e2e-chainsaw/_lib/run-kubernetes.sh
# talos_spec_block writes registryMirrors and imageFactoryURL by hand. If the
# chart ever renames either, the CR would be rejected only at e2e time, and
# this file is the cheapest place to notice.
for key in imageFactoryURL registryMirrors; do
yq -e ".talos | has(\"${key}\")" packages/apps/kubernetes/values.yaml >/dev/null \
|| { echo "packages/apps/kubernetes/values.yaml has no talos.${key}; talos_spec_block would emit a key the chart rejects" >&2; exit 1; }
done
# talos_spec_block writes imageFactoryURL by hand. If the chart ever renames
# it, the CR would be rejected only at e2e time, and this file is the cheapest
# place to notice.
yq -e '.talos | has("imageFactoryURL")' packages/apps/kubernetes/values.yaml >/dev/null \
|| { echo "packages/apps/kubernetes/values.yaml has no talos.imageFactoryURL; talos_spec_block would emit a key the chart rejects" >&2; exit 1; }
}
hack/sandbox-cidr-disjoint.bats
+1 -1
View File
@@ -51,7 +51,7 @@
@test "sandbox cluster CIDR overlaps no platform range" {
talos_file=hack/e2e-prepare-cluster.bats
install_file=hack/e2e-install-cozystack.bats
install_file=hack/e2e-platform-packages.sh
# Talos cluster.network.podSubnets: every list item under the key, including
# ones separated from the first by a comment or a blank line, so a second
hack/select-e2e.sh
+1 -1
View File
@@ -240,7 +240,7 @@ src_to_suites() {
# mappings their derived names point at suites that do not exist, so an
# ingress-only package change can omit the suite that exercises it.
ingress-application|ingress-nginx) echo gateway ;;
kubernetes-application) echo "kubernetes-latest kubernetes-previous kubernetes-oidc-system kubernetes-oidc-customconfig" ;;
kubernetes-application) echo "kubernetes-latest kubernetes-previous" ;;
securitygroup-controller) echo securitygroup ;;
*-application) echo "${1%-application}" ;;
*) echo "$1" ;;
hack/select-e2e_test.bats
+7 -8
View File
@@ -194,7 +194,7 @@ assert_full_suite() {
[ -z "$output" ]
}
@test "kubernetes-application maps to the four kubernetes suites" {
@test "kubernetes-application maps to the two kubernetes suites" {
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp -r packages/core/platform/sources "$tmp/sources"
@@ -202,10 +202,8 @@ assert_full_suite() {
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
echo "$output" | grep -q "kubernetes-latest"
echo "$output" | grep -q "kubernetes-previous"
# The OIDC render-side suites exercise the same kubernetes app chart, so a
# chart-only change must select them too.
echo "$output" | grep -q "kubernetes-oidc-system"
echo "$output" | grep -q "kubernetes-oidc-customconfig"
# OIDC System -> CustomConfig lifecycle coverage is folded into latest.
! echo "$output" | grep -q "kubernetes-oidc-"
}
@test "dashboards-only diff selects nothing (path is plural)" {
@@ -870,14 +868,15 @@ assert_full_suite() {
@test "an ingress-nginx change selects the gateway admission regression" {
# system/ingress-nginx is shared by the root ingress package, the tenant
# ingress application, and tenant Kubernetes clusters. The gateway suite
# owns the host-cluster Ingress admission regression, while the four
# Kubernetes suites cover the copies installed inside tenant clusters.
# owns the host-cluster Ingress admission regression, while the two
# Kubernetes suites cover the copies installed inside tenant clusters --
# two, not four, since the OIDC pair was folded into kubernetes-latest.
tmp=$(mktemp -d)
cp -r packages/core/platform/sources "$tmp/sources"
echo "packages/system/ingress-nginx/templates/admission-webhook-egress-policy.yaml" > "$tmp/diff"
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
assert_selection "an ingress-nginx change must exercise every installed copy" \
"$output" "gateway kubernetes-latest kubernetes-oidc-customconfig kubernetes-oidc-system kubernetes-previous"
"$output" "gateway kubernetes-latest kubernetes-previous"
rm -rf "$tmp"
}
hack/talos-image-cache_test.bats
+8
View File
@@ -79,6 +79,14 @@
[ "$kinds" = "CiliumClusterwideNetworkPolicy" ]
}
@test "range server records transfer outcome, bytes and duration" {
script=$(yq 'select(.kind == "ConfigMap") | .data."serve.py"' hack/e2e-talos-image-cache.yaml)
printf '%s\n' "$script" | python3 -c 'import sys; compile(sys.stdin.read(), "serve.py", "exec")'
printf '%s\n' "$script" | grep -Fq 'transfer-finished status=%s path=%s range=%s requested=%d sent=%d duration=%.3fs'
printf '%s\n' "$script" | grep -Fq 'time.monotonic() - started'
}
@test "egress policy selects the importer label and namespace the probe pod uses" {
manifest=hack/e2e-talos-image-cache.yaml
helper=hack/e2e-chainsaw/_lib/talos-image-cache.sh
packages/core/testing/Makefile
+106 -2
View File
@@ -3,8 +3,13 @@ NAME := sandbox
CLEAN := 1
TESTING_APPS := $(shell find ../../apps -maxdepth 1 -mindepth 1 -type d | awk -F/ '{print $$NF}')
SANDBOX_NAME := cozy-e2e-sandbox-$(shell echo "$$(hostname):$$(pwd)" | sha256sum | cut -c -6)
COZY_E2E_NODE_CPUS ?= 8
COZY_E2E_NODE_MEMORY_MIB ?= 24576
ROOT_DIR = $(dir $(abspath $(firstword $(MAKEFILE_LIST))/../../..))
COMPOSE_PROJECT ?= cozy-e2e
COMPOSE_FILE ?= $(ROOT_DIR)hack/e2e-compose.yaml
ZPOOL_BACKING_DIR ?= /var/lib/cozy-e2e-zpools
include ../../../hack/common-envs.mk
@@ -36,8 +41,105 @@ copy-nocloud-image:
prepare-cluster: copy-nocloud-image
docker exec "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-prepare-cluster.bats'
# Container lane: Talos nodes as containers instead of QEMU guests, so tenant
# workloads sit one nesting level shallower. Sits BESIDE prepare-cluster rather
# than replacing it; nothing downstream changes, because the addressing, CPU and
# memory in hack/e2e-compose.yaml mirror the QEMU lane exactly.
#
# Two steps, not one, because the halves must run in different places. The nodes
# have to be siblings on the runner's docker daemon for the /dev and /run/udev
# binds in hack/e2e-compose.yaml to resolve against the real host devtmpfs --
# that bind is what lets this lane carry LINSTOR at all -- so the host-side half
# needs a docker socket the sandbox does not have. Everything Talos-side then
# runs inside the sandbox exactly as the QEMU lane does.
#
# No copy-nocloud-image dependency: containers boot from the Talos image
# directly and need no nocloud asset, which also means this lane does not wait
# on `make -C packages/core/talos assets`.
prepare-cluster-container: delete-cluster-container
SANDBOX_NAME="${SANDBOX_NAME}" COMPOSE_PROJECT="${COMPOSE_PROJECT}" COMPOSE_FILE="${COMPOSE_FILE}" ZPOOL_BACKING_DIR="${ZPOOL_BACKING_DIR}" COZY_E2E_NODE_CPUS="${COZY_E2E_NODE_CPUS}" COZY_E2E_NODE_MEMORY_MIB="${COZY_E2E_NODE_MEMORY_MIB}" $(ROOT_DIR)hack/e2e-container-up.sh
docker exec -e COZY_E2E_NODE_CPUS="${COZY_E2E_NODE_CPUS}" -e COZY_E2E_NODE_MEMORY_MIB="${COZY_E2E_NODE_MEMORY_MIB}" "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-prepare-cluster-container.bats'
# Tear the container lane down. Docker volumes hold etcd and kubelet state, and
# the host zpools hold LINSTOR data, so both halves must disappear before a
# retry or the next run can adopt a half-dead cluster.
# USERDATA is set to a placeholder because compose interpolates the whole file
# before it decides what the subcommand needs, so the `${USERDATA:?}` guard that
# makes a missing machine config fail loudly on `up` also fails `down`, where no
# machine config is required. The value is never read: `down` matches containers
# by project and service name. Without it teardown always errored, and ignoring
# that error left the nodes and their /var volumes alive for the next run -- the
# exact half-dead cluster this target exists to prevent.
delete-cluster-container:
@set -u; \
cleanup_rc=0; \
if USERDATA=teardown docker compose -p "${COMPOSE_PROJECT}" -f "${COMPOSE_FILE}" down -v; then \
:; \
else \
step_rc=$$?; \
echo "container compose teardown failed with exit $$step_rc; continuing host-storage cleanup" >&2; \
cleanup_rc=$$step_rc; \
fi; \
if [ -z "${ZPOOL_BACKING_DIR}" ] || [ "${ZPOOL_BACKING_DIR}" = / ]; then \
echo "refusing unsafe ZPOOL_BACKING_DIR '${ZPOOL_BACKING_DIR}'" >&2; \
exit 1; \
fi; \
if [ "$$(id -u)" -eq 0 ]; then sudo_cmd=; else sudo_cmd=sudo; fi; \
for n in 1 2 3; do \
pool_detached=1; \
if $$sudo_cmd zpool list "data-srv$${n}" >/dev/null 2>&1; then \
if $$sudo_cmd zpool destroy "data-srv$${n}"; then \
:; \
else \
step_rc=$$?; \
echo "failed to destroy zpool data-srv$${n} (exit $$step_rc)" >&2; \
if [ "$$cleanup_rc" -eq 0 ]; then cleanup_rc=$$step_rc; fi; \
pool_detached=0; \
fi; \
else \
direct_list_rc=$$?; \
inventory_rc=0; \
pool_names=$$($$sudo_cmd zpool list -H -o name 2>/dev/null) || inventory_rc=$$?; \
if [ "$$inventory_rc" -ne 0 ]; then \
echo "failed to inventory zpools after data-srv$${n} lookup returned exit $$direct_list_rc (inventory exit $$inventory_rc); keeping its backing file" >&2; \
if [ "$$cleanup_rc" -eq 0 ]; then cleanup_rc=$$inventory_rc; fi; \
pool_detached=0; \
elif printf '%s\n' "$$pool_names" | grep -Fxq "data-srv$${n}"; then \
echo "zpool inventory still lists data-srv$${n} after its direct lookup returned exit $$direct_list_rc; keeping its backing file" >&2; \
if [ "$$cleanup_rc" -eq 0 ]; then cleanup_rc=$$direct_list_rc; fi; \
pool_detached=0; \
fi; \
fi; \
if [ "$$pool_detached" -eq 0 ]; then \
echo "keeping ${ZPOOL_BACKING_DIR}/srv$${n}.img because data-srv$${n} is still imported" >&2; \
else \
if $$sudo_cmd rm -f "${ZPOOL_BACKING_DIR}/srv$${n}.img"; then \
:; \
else \
step_rc=$$?; \
echo "failed to remove ${ZPOOL_BACKING_DIR}/srv$${n}.img (exit $$step_rc)" >&2; \
if [ "$$cleanup_rc" -eq 0 ]; then cleanup_rc=$$step_rc; fi; \
fi; \
fi; \
done; \
if [ -d "${ZPOOL_BACKING_DIR}" ]; then \
if $$sudo_cmd rmdir "${ZPOOL_BACKING_DIR}"; then \
:; \
else \
step_rc=$$?; \
echo "failed to remove backing directory ${ZPOOL_BACKING_DIR} (exit $$step_rc)" >&2; \
if [ "$$cleanup_rc" -eq 0 ]; then cleanup_rc=$$step_rc; fi; \
fi; \
fi; \
exit "$$cleanup_rc"
# The API endpoint and E2E-only storage mode cross into the sandbox explicitly:
# docker exec starts a fresh environment, so values exported by the caller do
# not reach the manifest generator or root-tenant setup. Empty values retain the
# QEMU defaults; the container lane supplies its node endpoint, disables DRBD,
# and selects local storage because its Talos nodes share one kernel.
install-cozystack:
docker exec "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-install-cozystack.bats'
docker exec -e COZY_APISERVER_ENDPOINT="${COZY_APISERVER_ENDPOINT}" -e COZY_LINSTOR_DRBD_ENABLED="${COZY_LINSTOR_DRBD_ENABLED}" -e COZY_E2E_STORAGE_CLASS="${COZY_E2E_STORAGE_CLASS}" "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-install-cozystack.bats'
test-openapi:
docker exec "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozytest.sh hack/e2e-test-openapi.bats'
@@ -48,8 +150,10 @@ test-openapi:
# job uploads. CHAINSAW_SUITES restricts the run to the named suite dirs
# (space-separated, e.g. "postgres bucket"); empty runs the whole suite. The
# CI Test-Impact-Analysis step (hack/select-e2e.sh) sets it from the PR diff.
# COZY_E2E_STORAGE_CLASS is passed both to script operations and as a Chainsaw
# value for templated resource files; empty keeps the QEMU default, replicated.
test-chainsaw:
docker exec "${SANDBOX_NAME}" sh -c 'cd /workspace/hack/e2e-chainsaw && chainsaw test ${CHAINSAW_SUITES}'
docker exec -e COZY_E2E_STORAGE_CLASS="${COZY_E2E_STORAGE_CLASS}" "${SANDBOX_NAME}" sh -c 'cd /workspace/hack/e2e-chainsaw && chainsaw test --set-string storageClass="$${COZY_E2E_STORAGE_CLASS:-replicated}" ${CHAINSAW_SUITES}'
collect-report: ## Collect the test report from the sandbox.
docker exec "${SANDBOX_NAME}" sh -c 'cd /workspace && hack/cozyreport.sh cozyreport'