Commit Graph
26 Commits
Author SHA1 Message Date
Aleksei Sviridkin c1fe1e9cb2 ci(e2e): say what the 215-minute e2e budget actually covers
The comment on every lane claimed the budget fits two tenant
Kubernetes suites failing at their node-join deadline. A green full
suite on the container lane already spends most of it, and a suite that
runs into the deadline costs about 25 minutes more than one that
passes, so the claim does not hold. State the real bound so a cancelled
run is read as a red suite, not as a budget to raise.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 16:57:44 +03:00
Aleksei Sviridkin da28ea7ee8 ci(e2e): run nightly and release-tag e2e on the container lane
Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.

Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.

DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 16:57:43 +03:00
Aleksei Sviridkin 6df5d076b1 ci(workflows): pin flux, yq and helm-unittest installs
Every flux install fetched fluxcd.io/install.sh and ran it as root, so
whatever that host served at the moment executed on runners that later
hold registry credentials. Use the upstream setup action instead, pinned
by commit and given an explicit version: it checks the release tarball
against that release's checksums file and runs no script fetched at run
time.

nightly and the fork e2e lane still took yq from the floating latest
release. They now use the pinned, checksum-verified install the
promotion steps already have.

The helm-unittest plugin embeds its own Helm and was installed at
whatever version was newest, so a plugin release could change chart
test results on a PR that touched no chart. Pin it, and replace a
different version a persistent runner may carry.

A contract test scans every workflow for these three tools, so an
unpinned flux, yq or helm-unittest install copied from an older step
fails the unit lane.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 11:02:18 +03:00
Aleksei Sviridkin 312409ecd2 test(e2e): fail the suite whose pods outlive its cleanup (#4492)
## What this PR does

A Chainsaw suite could pass while the workloads it created kept running
after its own cleanup, and nothing noticed until a later suite failed.
On one E2E run the brokers of four Kafka releases from suites that had
passed and deleted their applications (#4488), plus a TenantControlPlane
sitting on its finalizer with two apiserver pods (#4489), pushed pod
memory requests to the node limit. Harbor then failed on `Insufficient
memory` two suites later, under the wrong name.

This makes the suite that leaked fail, and name what it left behind.

- Chainsaw v0.2.15 has no hook that runs after a test's cleanup: its
`Configuration` carries `error.catch`, which runs only on failure. So
`make test-chainsaw` now runs `hack/e2e-chainsaw-run.sh`, which invokes
`chainsaw test` once per suite directory and checks the cluster between
invocations with `hack/e2e-chainsaw/_lib/leak-guard.sh`.
- Attribution: a pod is charged to a suite when all three hold. It is
not terminal. Its UID was not in the pod list taken before the suite.
The root of its ownerReferences chain was created at or after the suite
started. A platform pod replaced mid-suite by a rollout, an eviction or
a crash traces back to an object the install created, so it is not
charged. A pre-existing pod adopted by a controller recreated mid-suite
is excluded by the UID baseline. A leak is charged once, to its own
suite.
- The suite start is read off an apiserver, from the `creationTimestamp`
of a server-side dry-run create, rather than off the sandbox. With
several control-plane nodes the reading and a root's timestamp can come
from different apiservers, so the rule assumes their clocks agree more
closely than the gap between the reading and the suite's first object;
no skew margin is added. Source for that behaviour: the generic registry
`Store.create` stamps it through `rest.FillObjectMetaSystemFields`, then
`DryRunnableStorage.Create` copies it back without persisting
(`staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go`
and `dryrun.go` at Kubernetes v1.33.0).
- Budget: charged pods get 300s to disappear, the default
`timeouts.cleanup` in `.chainsaw.yaml`; a unit test keeps the two equal.
A suite that overrides its own cleanup timeout (kafka sets 8m) is
covered the same way, because the check runs only after `chainsaw test`
has returned, which is after every cleanup in the suite has finished
under its own timeout; the 300s start from there. The leaks this targets
sat about 110 minutes (Kafka brokers) and 30+ minutes (the
TenantControlPlane) past their suites. Every green suite logs
`leak-guard: suite=<dir> clear after Ns`, so the number can be checked
against later runs.
- Past the budget the run prints `LEAKED-PODS suite=<dir> budget=300s
count=N`, then one line per pod with its phase, deletion state, root and
`app.kubernetes.io/instance`. It raises a `leaked-pods` error annotation
and adds a failed `leak-guard` test case to `chainsaw-report.xml`. The
run continues through the remaining suites and exits non-zero at the
end, as the single invocation did on a red test.
- A suite that already failed on its own is checked once without the
wait. Its leftovers are reported as a `::warning` with `LEAKED-PODS
suite=<dir> checked-once=chainsaw-failed`, since some may still be
tearing down.
- Each pod gets one `kubectl wait` inside the remaining budget, and
nothing is retried. A timeout or a spent budget gives the verdict from
the last good pod list, also when the re-list after it fails, so a leak
fails the suite. A wait that fails only after running for its whole
`--timeout` counts as a timeout whatever it printed: kubectl maps an
expired context to "timed out waiting for the condition" only inside the
watch, so a deadline that passes during its initial lookup surfaces as
"context deadline exceeded" (`wait.go` and `delete.go` in kubectl
v1.33.0). A spent budget is caught before calling kubectl and never
passed as `--timeout=0`, which means "check once".
- Any other error on a wait or a re-list, before the budget is spent and
before any timeout, stops the waiting and is reported as `LEAKED-PODS
suite=<dir> count=N inconclusive="<error>"`. That is a warning, not a
leak failure, so an apiserver blip cannot become a leak red.
- A failed first reading after a suite, including a pod list that cannot
be parsed, leaves the suite unchecked. It fails with a `leak-guard`
annotation and a report case saying the cluster could not be read, not
that it leaked. Errors that are not leaks carry the annotation title
`leak-guard`, not `leaked-pods`.
- Known leaks are listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt`
as `<suite-dir> <issue-ref>`: `kubernetes-latest` and
`kubernetes-previous` against #4489 (both suites create a
TenantControlPlane through `_lib/run-kubernetes.sh`). `kafka` and
`kafka-metadata` were listed against #4488 until #4490 fixed it; those
entries are gone. An allowlisted suite prints its leak as a `::warning`
and does not fail. An entry that names a directory that is not a suite
stops the run before the first suite, so a rename cannot keep an
exemption alive. The PR fixing each issue deletes its lines.

Suite selection is unchanged. The runner runs the named directories when
there are any, and otherwise every directory directly under
`hack/e2e-chainsaw/`, each handed to Chainsaw's own discovery; it
refuses a tree with a test at its top level, where that split would stop
being exact. I verified this with Chainsaw v0.2.15's own
`discovery.DiscoverTests`, built from source at that tag. For the full
tree, for `kafka redis harbor` and for `kubernetes-latest`, the union of
the per-suite discoveries equals what a single `chainsaw test [suites]`
discovers: 57, 7 and 2 tests, identical lists. 57 matches the test count
in an earlier full run's report. A unit test pins that the runner passes
chainsaw the suites and flags it was given, in order. The order changes
too: suites used to run in Chainsaw's own order and now run one
directory at a time in alphabetical order, which moves the kubernetes
suites after the kafka ones. Before #4490 fixed #4488 this starved them
(see the E2E criterion below). With the fix merged the order has no
effect, and a future leak that lands before a memory-heavy suite is
named at the suite that leaked. One behaviour does change: a selected
name that is not a directory used to fail the single invocation before
any test ran; now only that suite fails and the others run.

`chainsaw-report.xml` keeps its path and shape: one `<testsuites>` root
with the summed counts, one `<testsuite>` per suite directory. Its
consumers are the `Collect chainsaw report` / `Upload chainsaw report`
steps in `pull-requests.yaml`, `e2e-fork.yaml`, `nightly.yaml` and
`e2e-tag.yaml`, which `docker cp` the file and upload it as an artifact.
Nothing in the repository parses it. The merge is unit-tested on two
per-suite reports in Chainsaw's real shape, one of them failed, and the
result is checked to parse as XML.

Tests: `hack/chainsaw-leak-guard_test.bats` (31 tests, in `make
unit-tests`; named without the `e2e-` prefix, which the unit lane
excludes) covers the attribution rule, the budget, the allowlist, the
merge and the runner, all against stub `kubectl` and `chainsaw`. Every
rule was mutation-checked: removing the guard call, having the Makefile
bypass the runner, inverting the allowlist match, flipping or tightening
the root-timestamp comparison, removing the UID baseline, removing the
terminal-phase filter, skipping the budget wait, changing the budget,
lowering or raising the depth cap, accepting a stale allowlist entry,
reading an unreadable pod list as empty, dropping the leak case from the
report, skipping directories in the selection, treating any failed owner
read as a missing owner, dropping the separate report for a suite that
already failed, dropping the not-run reason from the report, reading any
wait error as a timeout, removing the inconclusive branch, letting a
failed re-list after a timeout turn a leak inconclusive, ignoring a
failed re-list, reporting an unreadable cluster as a leak, reading an
unparsable pod list as clean, reporting a kubectl warning instead of the
error, passing the report name without its extension, wording a failed
re-list as a failed wait, dropping an allowlist's last line when it has
no newline, dropping the check that `--set-string` carries a value, and
reading a wait that ran its whole timeout but printed a context error as
inconclusive. Each one turns its named test red, and restoring turns it
green.

E2E criterion, registered before the run: in the `Run E2E tests` log of
this PR, every suite that ran prints `leak-guard: suite=<dir> clear
after`, a `LEAKED-PODS suite=<dir>` line carrying `allowlisted=`, or one
carrying `checked-once=` (a suite that failed on its own) or
`inconclusive=` (waiting stopped on a failed read), and no line matches
`LEAKED-PODS suite=.* budget=`. With #4490 merged, `kafka` and
`kafka-metadata` are no longer allowlisted and must print `leak-guard:
suite=kafka clear after` and `leak-guard: suite=kafka-metadata clear
after`, which is the live check that #4490 fixed #4488. The first run of
this PR (before #4490, run 36223896793) is the positive control: it
printed `LEAKED-PODS suite=kafka count=15` and `suite=kafka-metadata
count=8`, allowlisted, and every other suite cleared in 0-37s. That run
also went red on both kubernetes suites, because those 23 leaked brokers
left no memory for the second tenant worker VM (`0/3 nodes are
available: 3 Insufficient memory`), which the order change described
above made possible. The guard has not yet run against a real cluster,
so the first run of this PR should carry the `full-e2e` label; its
`clear after` lines are the first measured teardown times, and any
unexpected `LEAKED-PODS` line is either a real leak or a false positive
to understand before merge.

Follow-ups, known and out of scope here:

- The allowlist exempts a whole suite, so a new leak in an allowlisted
suite stays hidden until the issue's fix removes the line.
- Attribution is per suite directory, not per Chainsaw `Test`; the leak
lines carry the release instance, which usually names the test.
- Only pods are checked. A leaked PVC, LoadBalancer IP or bucket holds
no node memory and is not seen.
- The 300s budget rests on Chainsaw's cleanup timeout, not on a measured
pod-gone latency; the `clear after` lines are where that measurement
will come from.
- A platform object recreated mid-suite (for example a HelmRelease
uninstall/reinstall remediation, or a force-replace on an immutable
field) has a root newer than the suite start, and its pods would be
charged to whichever suite was running as a leak; the first CI runs are
where to look for this.
- The per-pod wait recomputes the remaining budget before each pod, and
no unit test pins that.
- `kubernetes-previous` is allowlisted for #4489 although the stuck
finalizer was seen only in `kubernetes-latest`; both suites create a
TenantControlPlane the same way, and the exemption hides any new leak
there until #4489 is fixed.
- Each clean suite now waits for its own pods to finish terminating
before the next starts, where the tail used to overlap the next suite.
The first full run's `clear after` lines say what that costs against the
E2E job's 215-minute cap.
- After a timeout followed by a failed re-list, the leak lines come from
the list taken before the wait, so a pod whose own wait had already
succeeded is still named and the count can be too high; the verdict is
unaffected.
- Likewise, when every wait succeeded and the re-list then fails, the
inconclusive report still lists the pods whose deletion was just
confirmed. In the boundary case where the budget runs out during the
last successful wait and the re-list then fails, the result is
inconclusive rather than a verdict from the last good list, because the
guard only treats the budget as spent when it finds it exhausted at a
check.
- A mistyped `COZY_LEAK_ALLOWLIST` path passes allowlist validation; it
fails closed, since no suite is then exempt.
- An allowlisted suite is not waited on, so `kubernetes-latest` and
`kubernetes-previous` will print `LEAKED-PODS ... allowlisted=` on most
runs even for pods still in normal teardown; waiting out the budget
first and applying the allowlist only to what remains would stop that
noise.
- A slow teardown in each of several suites can add up to 300s apiece,
against the E2E job's 215-minute cap. On the first full run of this PR
(run 36223896793) the `clear after` values of the suites that were
waited on summed to 210s; the last full run before this change took 158
of the 215 minutes.
- A pod whose owner's resource type is no longer served gets "the server
doesn't have a resource type" rather than NotFound, so its suite is
reported unchecked instead of leaked; it still fails.
- `kubectl wait --for=delete` waits for deletion, while the guard skips
pods in `Succeeded` or `Failed`; a pod that finishes but is never
deleted (a Job under a leaked root) spends the rest of the budget before
the suite shows `clear`. The verdict is right; only job time is lost.
- The unit tests do not pin the one-line empty report Chainsaw writes
for a directory with no tests (`_lib`), nor the branch that catches a
spent budget before calling `kubectl wait`; both are verified against
the Chainsaw and kubectl sources rather than by a test.
- The runner's per-suite report directory from `mktemp -d` is not
removed; the sandbox is discarded after the job.
- Each non-suite directory under `hack/e2e-chainsaw/` (today `_lib`)
costs one empty Chainsaw invocation, because the runner leaves the
decision of what is a test to Chainsaw.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds files under `hack/`, changes the `test-chainsaw` target,
and rewords one comment and the failure line in the four E2E workflows.
The trigger map's `hack/` entry is for cozystack/ccp, whose skills
anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`,
none of which change here. A GitHub code search for `chainsaw` across
the cozystack organisation finds hits only in this repository.

### Release note

```release-note
test(e2e): the E2E app suite now fails the suite whose pods outlive its own cleanup by more than 300s, naming each leftover pod and its owner; suites with a filed leak are listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt and only warn.
```
2026-10-01 14:03:13 +03:00
Aleksei Sviridkin 2f08832392 Merge remote-tracking branch 'origin/feat/talos-arm64' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 19:49:46 +03:00
Aleksei Sviridkin 2df6015414 ci(talos): keep the Talos builds in CI amd64
The talos package now builds for every arch in PLATFORM, and the jobs
that build it without setting PLATFORM would pick up arm64 as well:
the PR Talos build, the nightly disk and the release assets. None of
them has a consumer for arm64 output, since e2e boots amd64 and the
release publishes the amd64 assets only, so each one pins amd64 the
way the image builds already do.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 17:44:19 +03:00
Aleksei Sviridkin b6a30bd504 docs(e2e): describe the per-suite runner and the pod-leak guard
The agent guide, the suite README and a workflow comment said the E2E
step is a single chainsaw invocation; it is now one per suite, and a
suite whose pods outlive its cleanup fails. Say how a pod is charged
to a suite, what the budget is, and how the allowlist is kept honest.
The workflows' failure line now points at LEAKED-PODS as well, since a
run that failed only on a leak has no assertion diff to point at.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:19:49 +03:00
IvanHunters 95e68bad86 ci(workflows): use a literal match in the flux version guards
The guard ran `grep -qx "flux: v${FLUX_VERSION}"`, where the dots in the
version are regex metacharacters, so a client printing `flux: v2X8X6`
would satisfy it. Use `grep -Fqx` for a fixed-string match in the build
toolchain guards this change added.

Also correct the step comments that still read "install if absent
(idempotent)": the step now installs whenever the pinned version is not
the one present, not only when flux is missing.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
IvanHunters 8c0f91ff00 ci(workflows): pin flux version in remaining install-latest steps
The nightly toolchain steps still fetched flux through the installer's
floating "latest". Pin them to FLUX_VERSION 2.8.6 with the same
version-gated guard so nightly's flux push artifact step uses a fixed,
reproducible CLI rather than whatever latest resolves to.

Also switch e2e-fork's already-pinned step from a presence check to the
same version gate, so a runner with a pre-cached older flux is corrected
rather than left as is, and fix that step's comment to point at the flux
distro cozystack actually ships
(internal/fluxinstall/manifests/fluxcd.yaml, v2.8.0).

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:47 +03:00
renovate[bot] a7dfbe6f8c chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:04 +03:00
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-05 13:19:17 +05:00
Aleksei Sviridkin ad62a8d62b test(e2e): stop blocking merges on the tenant node-join deadline
The tenant node-join wait no longer discriminates. Nested
virtualisation on the shared runners degrades far enough that a guest
which registers in minutes on real hardware can miss any deadline this
test can afford, so a red on that one wait says as much about the
machine the run landed on as about the product, and it teaches a
reader to re-run the job.

The suite still fails on it. That is what keeps Chainsaw's catch
running: the previous-instance container logs, the host snapshot, the
data-plane capture and the event dump are all keyed on the test
failing, and the report stays honest. What changes is one layer up.
The wait writes a marker beside its diagnostics, and a new lane-side
gate reads that marker to decide whether the job blocks. It opens only
when every failed test in the run carries one, and blocks on anything
it cannot attribute: a report it could not read, a failed test the
report does not name, a failure with no marker.

Only the lanes that block a merge ask it. The two pull-request lanes
do; nightly and the tag lane run the same suites and stay hard. A soft
pass on the tag lane would ship a release candidate whose tenant
clusters brought up no worker nodes, and nightly is the only place an
e2e failure reaches anyone who did not open a pull request, since no
lane in this tree notifies anywhere.

A softened red finishes the job green, so the job's result stops
answering whether the suite passed, and both kinds of consumer are
handed the decision instead of inferring it. A step gated on the job
failing would silently stop running: the SSH breakpoint is that case,
and it still opens for a pull request marked for debugging. A job that
reports the result would silently start reporting a pass: the required
commit status now says the deadline was missed and that the merge is not
blocked on it, rather than that the suite passed.

The deadline goes to 29m. That is the test's own budget and nothing
more: its clock starts after the machinedeployment wait, the
LoadBalancer assignment and the healthz probe, while the chart's
autoscaler and health-check timers have been running since the Machine
was created, so no ordering against them is claimed. The chart's 20m
and 30m give a scale to pick against, and a budget over the first
leaves room for a remediation cycle without promising that one fits.

The operation ceiling both suites give the script rises with the
deadline, and the diagnostics phase budget rises inside it, because
the collectors gated ahead of the serial console cost more between
them than the old budget covered. Both budget guards read their terms
from the sources that set them. The e2e job cap rises with them, and
its guard holds what a guard can hold: the pair's operation and catch
ceilings, and one figure across every lane that runs the suite. It
also records that the suites' full summed ceilings have never fit any
cap in that file, so the margin above the bound is taken from a
measured run instead. The prose that restates the cap elsewhere moves
with it, and the sweep that holds the deadline's restatements now
reads the figure spelled in words as well as in minutes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-21 12:52:42 +03:00
Aleksei Sviridkin db2ddfab2b fix(ci): give the nightly e2e runner headroom over its guest vCPUs
The e2e sandbox boots three QEMU management nodes at `-smp 8`, so 24
guest vCPUs are committed before any tenant VM starts. The nightly lane
ran that sandbox on a 24-vCPU runner, a 1:1 commit that leaves the host
nothing for QEMU's emulation and IO threads, containerd, the kernel and
the runner agent. The tag, fork and pull-request lanes already run the
same sandbox on the 32-vCPU class, so move nightly there too.

This aligns headroom and claims no performance effect. Measured boot
data shows the load a workflow puts on the host, not the runner class,
dominates guest starvation, and the two are confounded in that data; no
red run is attributed to the class here. The case is the arithmetic: a
1:1 commit leaves no headroom, and two lanes running the identical
sandbox on different classes is drift nothing was watching.

Add hack/sandbox-runner-headroom.bats to keep the two sides paired. It
derives the guest vCPU and memory demand from the sandbox bats file, and
the runner class of every job that drives the sandbox make targets, then
requires the runner to be strictly larger on both axes. Neither side is
a literal list, so raising `-smp`, adding a management node or adding a
lane on a smaller runner turns it red instead of drifting apart
silently.

Input the derivation cannot read in full is refused rather than guessed:
a guess produces a demand smaller than the real one, which passes a
runner that cannot hold the sandbox, while a refusal names the line.
That covers a launch written outside any `@test` block, a block left
open at end of file, a `for` opener the strict pattern misses, a
non-literal loop list, and a `while` loop sharing the block.

The derivation runs against the live tree, which passes whether or not
those refusals still work, so each one is pinned by a fixture feeding a
synthetic file to the same program. Loosening the parser turns a fixture
red rather than quietly widening what the check accepts.

Record the rule alongside the other environment invariants in
docs/agents/e2e-testing.md, so a new lane picks its runner class from
the documentation rather than from a failing unit suite.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-15 02:41:19 +03:00
Aleksei Sviridkin 7e2db4da7e fix(e2e): bound every remaining read in the report collector
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and
the rest -- nodes, namespaces, events, HelmReleases, packages,
packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads
and every module gate -- read with none. The tarball is written on the
collector's last line, so a single read that never returns does not
truncate the report, it loses it, on the run where it is the only
evidence left.

Each read now goes through the shape that fits it: the existing bounded
reader for a read whose output is a file, the existing selector for one
that feeds a loop, and a new probe for the `if kubectl get deploy ...`
gates. A gate is a read like any other, and an unbounded one is worse
than the reads it guards, since the module below it never starts. Its
cut-off is recorded, because skipping the module leaves the same empty
tree as a cluster that never had the component; a gate that answers "not
installed" is recorded nowhere, since that is the cluster answering
correctly and a marker firing on every install without KubeVirt is one a
reader learns to skip. The reads that stay on the bare wrapper say so
where they sit; what they share is that the output is not the reader's
to write, going into a variable or through a pipeline that transforms
it, so there is no moment at which a marker could attach.

The header claimed a per-section list of what was bounded, and the tests
froze that list plus a per-section count of the unbounded reads. Both are
gone. The list had already gone stale, naming four sections while the
file had ten, and the counts were an undercount by construction -- they
matched a redirect to `$VAR/`, which no gate, selector or command
substitution has. One test replaces both: no read below the library guard
runs without a ceiling, checked by looking for a counterexample.

A ceiling per read is not a ceiling on the report, and the header now
says so where it used to imply the opposite. The pod, VM, VMI and
services walks carry a count cap and a time budget; the walks over nodes,
namespaces, HelmReleases, packages, PVCs and certificates carry neither,
so their cost is the cluster's to set. What this removes is the read that
never returns.

The bounded reader now names the tool it ran and describes what that tool
returned. It served kubectl alone until this change gave it talosctl and
the four host commands, so a `df` that hit a stale mount left a marker
reading "kubectl exited 1 ... not because the object ended" -- two false
statements in one line, in an artifact whose premise is that its notes
can be believed.

The exec reader keeps the remote command's whole message in the listing
it belongs to. `kubectl exec` appends its own `command terminated with
exit code N` after whatever the command said, so quoting the last line
there drops the cause and keeps the trailer, and sending the message
away from the file leaves a zero-byte listing -- the "returned nothing,
or never ran" question this collector exists to answer. Its cut-off note
also stops asserting that the archive is beside it: a stream that got
members across is kept and truncated, one killed before its first byte
leaves zero bytes and the call site's emptiness check removes it, and
the reader runs before that check, so it names both.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 14:38:29 +03:00
Aleksei Sviridkin 96e33fe656 ci(e2e): give the report collection step a ceiling of its own
The step that runs hack/cozyreport.sh carried no timeout-minutes in any
of the three workflows that run it, so it inherited only the job's 180.
The collector writes its tarball on its last line and the upload is the
next step, so a read that never returns spends what is left of the job
budget, the job is cancelled, and the upload never runs -- the artifact
is lost entirely rather than truncated, on exactly the run where it is
the only evidence left. `|| true` guards against a non-zero exit, not
against a read that does not return.

30 minutes is measured against the step rather than divided out of the
job: it takes 3-42s across sampled runs, on failed e2e jobs as well as
green ones, and 30 leaves five sixths of the job budget for the steps
that upload what this one produced.

A test pins the ceiling in all three workflows and requires it to stay
under the job's own, since a step ceiling at or above the job's cap is
the same as no ceiling at all.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 14:38:29 +03:00
Andrei Kvapil 78fc3f9178 refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows)
The E2E job once ran as several state-sharing self-hosted-runner jobs
that rendezvoused through a sha256(REPO:WORKFLOW:REF) key and a per-run
/tmp/$SANDBOX_NAME copy of the checkout. Since the move to the ephemeral
pool it is a single job: there is no cross-job rendezvous and the
checkout is never reused, so copying it into /tmp and removing it
afterwards is dead weight.

Run every step in $GITHUB_WORKSPACE instead: drop the Prepare/Remove
workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes, and point the
upload paths at _out/. This deletes the unguarded `rm -rf
/tmp/$SANDBOX_NAME` rather than guarding it: when an upstream step fails
or the job is cancelled, the unconditional `Set sandbox ID` is skipped,
SANDBOX_NAME stays empty, and the `if: always()` steps interpolate it —
the removal expanding to `rm -rf /tmp/` and failing the cleanup step.
`Set sandbox ID` stays, since the container still needs a name.

Apply the same treatment to nightly.yaml, and drop the Makefile dead
code the single-runner retirement left behind: the world-writable
mkdir/chmod of /tmp/$SANDBOX_NAME (backed a bind mount removed long ago)
and the `-e SANDBOX_NAME` container env (nothing in the e2e-sandbox
image reads it).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 17:49:02 +02:00
Myasnikov Daniil 90254efa6d fix(ci): resolve nightly packages digest without image-specific skopeo ops
The Nightly workflow has never completed a run. Every scheduled run since
NIGHTLY_ENABLED was set has failed in `mirror` at "Resolve source commit
and version", ~80s in:

  level=fatal msg="unsupported image-specific operation on artifact with
  type \"application/vnd.cncf.flux.config.v1+json\""

cozystack-packages is a Flux OCI artifact, not a container image: its
config mediaType is application/vnd.cncf.flux.config.v1+json. Plain
`skopeo inspect` parses the config blob as an image config and fatals on
any non-image artifact, so resolving the digest with
`inspect --format '{{.Digest}}'` could never work.

Fetch the :main manifest once with `inspect --raw` and derive both the
digest (`skopeo manifest-digest`, which just hashes the manifest bytes —
no image-specific parsing) and the revision annotation from those exact
bytes. This also tightens the race the original comment describes: the
digest and the annotation now come from a single fetch, so there is no
window at all between pinning the tag and reading its revision, and one
registry round-trip is saved.

Because the workflow never ran past this step, the rest was unproven.
Also fix both `flux pull artifact` calls, which passed an --output
directory that does not exist and would have failed with "invalid output
path": create it first, matching the established idiom in
pull-requests.yaml ("Pull current-main packages tree").

Verified against the live OCIR artifact: the fixed step resolves
packages_ref to the same digest the registry reports and yields main's
current HEAD sha; the fixed pull extracts the tree. skopeo copy on Flux
artifacts, the flux push digest awk parse, the cozy-installer chart name
and nightly-mirror.sh's per-image digest verification were each checked
and are correct as-is.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-16 22:40:05 +05:00
Myasnikov DaniilandClaude Opus 4.8 302a72f14a Merge origin/main into feat/e2e-chainsaw-example
Reconciles the talos-image-cache conflicts created when #3254 merged to
main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh,
a superset of #3254's logic + the node-join diagnose helper), drop the
BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path
versions of the mirror manifest, install bats, and unit test. Also picks
up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and
Talos v1.13.6 (#3240).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-10 12:00:01 +05:00
Myasnikov DaniilandClaude Opus 4.8 bece285678 ci: pin action refs flagged by the zizmor gate (#3223)
The zizmor "Audit workflows" gate (added in #3223) is red on main: five
actions/* references in the release workflows from #3017 are pinned to a
floating tag instead of a commit SHA (blanket unpinned-uses policy, High
severity):

- nightly.yaml (x3)  actions/checkout@v4
- promote-rc.yaml     actions/checkout@v4
- retention.yaml      actions/create-github-app-token@v1

Pin each to the SHA the rest of the repo already standardizes on:
- actions/checkout                -> 34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- actions/create-github-app-token -> d72941d797fd3113feb6b93fd0dec494b13a2547 # v1

`zizmor --offline .github/workflows/` now reports no findings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-09 13:15:39 +05:00
Myasnikov DaniilandClaude Fable 5 efbf372d25 test(e2e): fix post-merge seams the conflict markers missed
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed
  test-apps-% target; switch it to the single chainsaw invocation
  (empty CHAINSAW_SUITES = full suite) and upload the JUnit report
- pin the chainsaw-report upload-artifact step to the same SHA the
  rest of the workflows use (scorecard/zizmor gate)
- repoint hack/run-kubernetes-drain_test.bats at the relocated
  hack/e2e-chainsaw/_lib/run-kubernetes.sh
- replicate cozytest.sh's NotReady-pod data-plane capture
  (hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so
  chainsaw-suite failures keep diagnostic parity with the BATS lanes
- refresh stale bats path comments

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-08 18:06:22 +05:00
Myasnikov DaniilandClaude Fable 5 70adc01e79 ci(release): nits — skopeo login via stdin; tighten dashboard version regex
- nightly.yaml: pass the registry tokens to `skopeo login` over stdin
  (--password-stdin) instead of `-p` on argv, matching promote-rc/finalize and
  keeping the token out of the process list.
- dashboard configmap: require digits-then-dot (^v?[0-9]+\.) when deriving the
  displayed platform version from the console image tag. The old ^v?[0-9] let a
  tagless, port-carrying ref (ghcr.io:443/.../console, whose last ":"-segment is
  "443/.../console") pass as a version. Verified: v1.5.0 / 1.5.0 still surface;
  the port ref and :latest no longer do.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-06 17:49:40 +05:00
Myasnikov DaniilandClaude Opus 4.8 c69f0f61c7 ci(release): annotate the nightly nocloud push with image.source
The first-ever `oras push` of cozystack-nocloud creates the GHCR package,
but without org.opencontainers.image.source it is created unlinked to the
repo, and the follow-up `oras tag … nightly` can 403 permission_denied:
write_package until someone links it by hand. Set the annotation on push so
the package is linked from the start (matches the --label the image builds
already carry).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-06 16:49:24 +05:00
Myasnikov DaniilandClaude Opus 4.8 0e4d550780 ci(release): don't persist checkout credentials in nightly jobs
Address review feedback from coderabbitai on .github/workflows/nightly.yaml:
actions/checkout writes the token into .git/config by default, and these jobs
copy the checkout into /tmp/$SANDBOX_NAME before running repo targets. Set
persist-credentials: false on all three source checkouts so the token is not
carried into the sandbox copies.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-06 10:03:54 +05:00
Myasnikov DaniilandClaude Opus 4.8 14c684ad5b ci(release): make each nightly tag immutable
Address review feedback from coderabbitai on .github/workflows/nightly.yaml:
the version was 0.0.0-nightly.<date>, so a same-day re-dispatch or re-run
republished every image, chart and artifact under the same tag — the dated
nightly was not write-once. Append the run id and attempt so each build gets a
distinct, immutable tag, and widen retention.yaml's selector regex to match the
extra numeric components (still excluding the floating :nightly and any release
tag).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-06 10:03:53 +05:00
Myasnikov DaniilandClaude Opus 4.8 d233288d37 ci(release): pin nightly source artifact to a digest
Address review feedback from coderabbitai on .github/workflows/nightly.yaml:
the resolve step read cozystack-packages:main and the pull step re-fetched
the mutable :main tag, so a concurrent build-main push could check out commit
A while mirroring tree B. Resolve the :main digest once, derive the revision
sha from it, and pull the baked tree by that same digest.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-06 10:03:52 +05:00
Myasnikov DaniilandClaude Opus 4.8 c1634be861 feat(release): add nightly workflow (mirror main to GHCR + full e2e)
Scheduled nightly that publishes an installable nightly of main without
rebuilding it, then runs the full e2e suite against the published bits:

- mirror: resolves the exact commit cozystack-packages:main was built
  from, runs nightly-mirror.sh to copy every cozystack-owned component
  image OCIR->GHCR by digest, re-publishes the rewritten packages
  artifact and the cozy-installer chart to GHCR (0.0.0-nightly.<date>
  + floating :nightly).
- build-disk: assembles the Talos nocloud disk from the upstream imager
  (no cozystack-image dependency, no rebuild) and publishes it as a GHCR
  OCI artifact (cozystack-nocloud) via oras, so a nightly is installable
  on real hardware, not just e2e.
- e2e: stages the GHCR install closure (rewritten tree + chart pinned to
  the GHCR packages artifact + the disk) and runs the FULL app suite (no
  Test Impact Analysis - a nightly has no diff to scope against).

No GitHub release and no api/apps/v1alpha1 Go-module tag are created.
Gated by the NIGHTLY_ENABLED repo variable. actionlint clean.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-06 10:03:50 +05:00