Commit Graph
847 Commits
Author SHA1 Message Date
Myasnikov Daniil ffae003d6b fix(release): stop a failed org package listing from exiting green
The nightly sweep's own listing shared one `|| true` with the grep
that filters it, and grep legitimately exits 1 when no cozystack/*
package matches. A failed /orgs/.../packages read therefore arrived
as an empty list, took the "No cozystack/* container packages found"
branch and exited 0 -- below the nightly sweep but ABOVE the deferred
guard, so a protection set that could not be resolved reported
success having swept nothing. That bypasses the guard rather than
varying it: replaying the step against a stub gh, a failed /pulls
combined with a legitimately empty org listing exited 0 after
printing its own ::error::.

List and filter as two statements so one `|| true` stops meaning two
things, handle the listing failure like the other two (::error::,
blocked flag, emptied list), and consult the flag before the early
exit. That exit is non-zero, matching the other two listing
failures: no sweep is left to carry the run down to the guard, so
this is the last place the flag can be honoured, and an unresolved
protection set is exactly what fail-closed means. The legitimate
empty-org case still prints the same line and exits 0.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil a37a3178e5 docs(release): name all four sourcers of the image-ref enumeration
The library header still claimed three call sites. There are four:
this PR made hack/verify-promoted-packages.sh source it to compare
the candidate's refs against the rc artifact's. The historical
drift story stays with the original three, since the verifier has
only ever read the shared enumeration.

The consumer table gained hack/promote-packages-artifact.sh under a
bullet requiring every consumer of references to read all three
shapes through the library, while the exception below it still named
only hack/overlay-main-images.sh. The publisher sources nothing and
needs nothing: it hands the whole tree to `flux push artifact` and
touches one reference, the installer's own self-pin, by key with yq.
Say that, rather than implying a gap it does not have -- the real
exception remains overlay-main-images.sh, which does read references
and does walk the tree itself.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil dee398f3b3 fix(release): defer a retention listing failure to the promotion guard
Both candidate-protection sources open with a `gh api --paginate`
listing and neither handled a failure. Under `set -euo pipefail` a
transient /pulls or /branches blip therefore exited the step before
the unrelated nightly GHCR sweep had run -- the exact coupling the
deferred guard at the bottom exists to remove, and which
hack/promotion-retention-contract.bats already asserts positionally.

Route both listings into that guard instead: raise the blocked flag
and empty the list, so the promotion sweep is skipped and the job
still fails while the nightly sweep finishes. Emptying is not
cosmetic -- a half-paginated list left in place is a smaller
protected set, and a smaller protected set is the one shape that can
authorise a delete.

Replaying the step's real shell against a stub gh: with /pulls
failing it exited 5 and never reached the nightly line; it now exits
1 with the nightly sweep done and the promotion sweep skipped.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil b65b6aa3ae test(release): pin every requiredBaseFiles entry, not just the list
The promote gate refuses a target base that lacks any of five
[path, marker] pairs, and the contract test asserted the list as a
block: deleting the whole thing turned it red, deleting the
hack/lib/image-refs.sh entry alone did not. Four of five pairs were
therefore unpinned individually.

Pin the missing one. The verifier sources that library through the
same `dirname $0` as lib/promoted-packages.sh, so a base carrying
the script without it fails at exactly the same point -- after the
candidate, the staging branch and the draft release exist.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil acd0a9e0a2 test(release): cover the verifier's rc-reference scan guard
The scan treats any grep diagnostic as a failed scan, because
`find ... -exec ... +` reports the legitimate "nothing matched" as a
failure too and the `|| true` absorbing that also absorbs a file
that went unscanned. Deleting the whole guard left the suite green:
its two siblings -- a failing collector and an empty collection --
each have a dedicated test, and this one had none.

Reach the state the guard exists for by putting a grep on PATH that
answers the scan's invocation the way the real one answers a file it
cannot open. A chmod 000 fixture would stop proving anything
wherever the suite runs as root.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil 04fa8471fe docs(release): say what the gates actually guarantee
Three claims that were stronger than the code, plus one legibility fix.

The PR gate's comment said a `packages/` edit "on the PR or its base"
fails the status before merge. The base half is not true: GitHub fires
no `pull_request` event when the base branch advances, it only
recomputes refs/pull/N/merge, so a change landing on the base after this
went green is never re-verified and the PR merges on the stale result.
Checking the merge tree does make base drift visible WHEN the job runs,
which is worth keeping — but finalize is what actually catches the drift
case, after the merge and before any stable name. Say both, in the
workflow and in the reviewer checklist, so nobody reads the green as
covering more than it does.

"Transactional at every consumer-visible name" is a judgement call
dressed as an absolute, inherited from the "this dispatch performs NO
registry mutation" it replaced. The dispatch now does write to the
registry: the `promotion-*` candidate is a real, publicly listable tag.
What it creates is no *stable* name — no vX.Y.Z, no :latest, no stable
installer. Stated that way the sentence stops needing the reader to
already know which writes were meant.

The retention paragraph listed two of the conditions the workflow
enforces. It now lists all of them, including the ones the workflow
gained here: same-repository, bot-authored, `release-X.Y.Z` branch, the
`release` label, exactly one valid digest pin, and protection from a
base-branch tip as well as an open PR.

Also: `flux pull artifact` reads its credentials from
~/.docker/config.json, which is why finalize now runs `docker login` —
but its runner class is not one of the docker-using build ones and the
setup step installs everything except docker. Check for it there, where
the message can name what is missing, instead of failing with "command
not found" halfway down the release path. And the verifier's exclusion
list is a restatement of flux's `excludeOCI`, not a derivation — the CLI
does not expose it — so the coupling and the pinned version are now
written down next to the list.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 820e72d572 fix(release): protect a candidate until its retag, not until PR merge
Retention protected a packages candidate only while an OPEN bot release
PR pinned its digest. The digest stops being eligible for deletion only
when finalize's retag adds a stable tag to the manifest, because that is
what makes the `all(tags match promotion-*)` predicate stop matching —
and finalize runs after the merge that closes the PR.

The window between them is where a half-finished release sits. Finalize
creates the write-once stable git tag, then dies before the retag: a
lost runner, a 5xx publishing the draft, a non-fast-forward on the
maintenance branch. What is left pins the candidate from a merge commit
and a tag nobody can move, with no open PR and only a temporary tag on
the manifest. Thirty days later retention deletes it, after which
re-running finalize 404s at the candidate check and re-dispatching
promote is refused because the stable tag exists. The release can then
only be finished by hand — the exact class of manual cleanup this design
exists to remove.

Protect the tips of the branches a promote PR can merge into as well.
The two sources are read differently on purpose. An open promote PR's
head is supposed to carry exactly one candidate pin, so anything else is
an anomaly and blocks. A branch tip is an additional source: every
digest pin found is protected and a branch without one contributes
nothing, because a maintenance line cut before this pipeline existed
cannot pin a candidate and must not wedge retention forever.

Blocking now stops the promotion sweep alone. It used to `exit 1` inside
the resolution loop, which also killed the unrelated nightly prune
further down the same step — so a WIP commit adding a trailing comment
to a pin on a release branch would silently stop GHCR nightly pruning.
The job still fails; it fails after the nightly sweep has run.

Verified by running the step's script against a stubbed `gh`: a digest
pinned by main's tip and referenced by no open PR is now excluded while
a genuinely abandoned one is still listed; a manifest that also carries
a stable tag stays excluded by the existing predicate; and a malformed
pin on a PR head leaves the nightly sweep running, skips the promotion
sweep and exits 1.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 05b2ce90cf fix(release): close the verifier's fail-open paths and cover them
Three gaps in the guards that decide whether the published candidate is
still the tree being tagged. All three passed silently rather than
failing, which is the worst shape for a check that runs immediately
before a write-once name is created.

`normalized_refs` piped `collect_image_refs` straight into its parsing
loop, so the pipeline reported `sort -u`'s status and the collector's
was discarded. POSIX sh has no `pipefail` and this script has to stay
POSIX, so `set -e` never saw it: a failed or truncated collection became
a smaller set and the rc-vs-candidate comparison matched it happily. The
verifier then reported that no container bytes changed across promotion
having proven nothing — on the guard that carries that entire claim.
Collect into a file first, so a nonzero collector aborts before anything
is compared. An empty collection is the same failure wearing a zero exit
status and needs its own check: two empty sets compare equal, so the
proof passes having examined nothing, and every packages tree carries
image references — none means the collector understood nothing it was
handed.

The rc-leftovers scan swallowed grep's stderr, turning "could not read
this file" into "this file is clean" — the silent-skip shape
hack/promote-rewrite-tags.sh refuses by name. The `|| true` has to stay,
because `-exec … +` reports the legitimate "nothing matched" as a
failure too, so the diagnostics are the only remaining signal that a
file went unscanned. Fail on any of them. Defence in depth rather than a
live hole: a surviving rc string would also fail the per-file compare,
since the release tree has none.

The per-file content comparison had no test at all. It is the only leg
that catches a file present on BOTH sides with different bytes — exactly
the threat the job exists for, a maintainer editing packages/ on the
release PR after promotion published the candidate. The file-set compare
sees the same names, the installer compare reads one file, and the
rc-refs compare never looks at the release tree. Replacing the
comparison with a no-op left all nine other tests green.

Red phases, all observed before the fixes. The content-compare case
changes one non-image line on one side, so no other leg can fire and
claim the credit; it fails with the comparison no-op'd and passes with
it restored. The two collector cases run the verifier from a copy whose
sibling lib/ holds a stub — which is how the workflows already invoke
it, `.release-tooling/hack/…` with libraries resolved next to the
script, so the failure is reached the way production would reach it.
Against the piped version both fail on `[ "$rc" -ne 0 ]`: the verifier
exits 0 and prints its success line whether the collector returns 3 or
returns nothing.

Also pins `e2e`'s dependency on the candidate gate. The contract pinned
`e2e-report`'s, so removing `verify-release-candidate` from `e2e`'s
`needs` together with the `success` clause on its release arm left all
six tests green — and running a full-e2e promotion suite against a
candidate that never verified is the worse of the two failures.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 9fa8eeeda2 ci(release): pin the promotion toolchain and cap the candidate gate
Three review findings on the new promotion steps.

`yq` was fetched from `releases/latest/download` in all three steps this
pipeline installs it in — the promote dispatch, the promote PR gate and
finalize. That makes an unpinned moving dependency part of what decides
whether a stable release may be created, and part of what rewrites the
installer pin: two promotions of the same rc can be served different
binaries. Pin it the way FLUX_VERSION already is, version-gated so a
matching preinstalled yq is reused, on the version the e2e sandbox image
already tracks.

A version authenticates nothing about the bytes it names, though, and
these are the workflows that mint write-once stable names. yq is
therefore also pinned BY CONTENT: its SHA-256 is committed beside its
version and verified before install.

Scoped to yq deliberately. It is the one downloaded binary with write
authority over what a release contains — it stamps the candidate digest
into installer values at promote time and platformVersion into the
stable chart at finalize. skopeo, helm and oras only move bytes that are
already digest-addressed, so a substituted one is caught downstream; a
substituted yq writes the wrong pin and every later check verifies that
pin faithfully.

The Flux installer is left as it is, on purpose. It fetches
flux_${FLUX_VERSION}_checksums.txt from the pinned release and verifies
the binary against it, so the binary is not the exposure — the ~6 KB
bootstrap script from fluxcd.io is, and it is mutable by design and
fetched identically in ten other workflows here. Pinning its bytes in
these three files would break on the next regeneration of a script the
project does not control while leaving the same fetch unpinned
everywhere else. The comment names the residual risk instead.

No precedent existed either way: nothing under .github/workflows/ or
hack/ verified a downloaded tool before this, and every version-pinned
download in the tree (oras 1.2.0, cozyvalues-gen 1.6.0) is unverified.
This is a new convention rather than an application of one, confined to
the release path and justified by write authority, and the comment says
so where the next reader will be standing. The three unpinned fetches
outside this pipeline (nightly.yaml, e2e-fork.yaml) are left alone.

The checksum is trust-on-first-use, which is the honest description: it
fixes the bytes to what was checked when the version was chosen and
makes any later substitution loud. Provenance is the release's own
published checksums list cross-checked against the downloaded asset;
both agree on
fa52a4e758c63d38299163fbdd1edfb4c4963247918bf9c1c5d31d84789eded4, and
that binary self-reports v4.53.3.

Verified locally against the real asset: the version gate skips a
matching preinstalled yq and re-installs a mismatched one, the checksum
line reports OK for the genuine binary and exits 1 for a tampered copy,
which aborts the step under the runner's `bash -e`.

`verify-release-candidate` had no `timeout-minutes`, so it inherited the
6-hour default while `e2e-report` waits on it through `needs` — a
stalled `flux pull artifact` would hold the required `E2E Tests` status
pending for the whole of it. 30 minutes, matching `checks` and
`finalize`.

The contract pins that all three install steps carry both versions and
the checksum, run the checksum check, name no `latest/download`, and
agree on one yq version and one checksum. So the unpinned one-line idiom
cannot come back into a release step by copy-paste from the workflows
that still use it, and bumping the version in one file without its
checksum cannot pass. Both mutations are caught.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 2d734fb229 fix(release): preflight the rc tree's own pin before publishing
hack/promote-packages-artifact.sh pushes the candidate BEFORE rewriting
installer values, so the artifact carries the rc tree's
platformSourceUrl/platformSourceRef verbatim — and
hack/verify-promoted-packages.sh reads exactly those two fields back out
of it as the candidate's rc baseline, requiring a trusted repository and
an immutable `digest=sha256:<64 hex>` pin.

The publisher checked neither. A tag-form ref or an untrusted URL in the
rc tree published a candidate and pushed two commits regardless; the
refusal arrived later, in the promote PR gate or in finalize, once the
staging branch and the draft release already existed. Check both at
dispatch time instead, while nothing has been written.

A copied regex would drift from the guard it is bringing forward and
quietly restore the late failure, so the trusted repository and the
digest pattern move into hack/lib/promoted-packages.sh and both halves
source it. That makes the library part of what the verifier needs at run
time, so promote's base-branch precondition gains it: the list exists
because the base's own workflows run after merge, and a base carrying
the verifier without its library fails at the same two late points.

Tests: both preflights are covered, each asserting the mock registry
client was never invoked — "exited non-zero" alone does not say nothing
was written. Both fail with the preflight removed. The publisher
fixtures now pin the trusted repository the way a real rc tree does, and
still send the candidate elsewhere via REGISTRY so the rewrite
assertions keep their teeth.

Also covers a malformed PROMOTION_ID. It is not attacker-influenced, but
it is the run-uniqueness suffix of the candidate tag and retention.yaml
deletes a candidate only when every tag on the manifest matches
`promotion-v…-run-[0-9]+-[0-9]+`; a value outside that shape publishes
an artifact no cleanup pass can ever collect.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 54e8533a9d fix(release): compare executable bits unambiguously in the verifier
The mode check was written as a single chain:

  if { [ -x "$a" ] && [ ! -x "$r" ] || [ ! -x "$a" ] && [ -x "$r" ]; }

which reads like a symmetric difference and is not one. POSIX gives
`&&` and `||` equal precedence and left associativity, so it groups as
`((A && B) || C) && D` with `D = [ -x "$r" ]`. A file executable in the
candidate and not in the release tree makes `A && B` true but `D`
false, so the whole condition is false and the mismatch is accepted.
Only the mirror case was rejected — and it is the benign one. The
direction that got through is content the merge reviewer never saw made
runnable inside the very artifact the operator installs, which cmp(1)
cannot see because it compares bytes.

Resolve each side to a value and compare those, so no precedence rule
is load-bearing.

Both directions now have coverage. Before the fix the candidate-only
test failed against the shipped condition and the release-only test
passed, which is what makes the pair meaningful rather than a check
that only goes red when the whole block is deleted.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil 5d56681f8c fix(ci): put every label guard in lockstep, forks included
Promotion needs the `release` label to start work, so this PR added it
to the set of labels whose `labeled` event is not discarded. That set is
stated in four places and was updated in three.

`gate-concurrency-contract.bats` caught the shape of the problem and
then became the first casualty of it. It asserted that the concurrency
key excludes exactly ONE label from the `-label` group, by counting
occurrences of `github.event.label.name != '`. A second publishing label
made the count 2, turning the suite red — and with it `checks`, then
`finalize`, then `e2e`, then the required `E2E Tests` status. Raising
the constant to 2 would re-hardcode the same brittleness one label later
and throw away the invariant the test exists for: the set of labels
excluded from the `-label` suffix is exactly the set `plan` runs for.
Both sets are now extracted from the workflow and compared, so the
assertion holds at any number of labels.

`resolve_assets` was the guard that got missed. It sits on `e2e`'s
release arm, so a label event `plan` runs for and it skips for leaves
`e2e` with both arms false — `finalize` is force-skipped for release
PRs — while `e2e-report` still runs. On a promote PR already carrying
`full-e2e`, re-applying `release` therefore superseded the green run and
posted `E2E Tests` = failure, because `full-e2e` bypasses the "e2e not
repeated" branch and `E2E_RESULT` is `skipped`. `unlabeled` is not a
trigger, so removing the label could not clear it; only a push could.

`release` must also not publish on the fork lane. e2e-fork.yaml's
`resolve` decides "this run built nothing, so it is not a verdict" by
asking whether `Plan build` skipped. Admitting `release` at `plan` made
that false: a fork run reaches `plan`, has `build`/`finalize` gated off
by its own label, succeeds having produced no pr-patch, and the fork
lane fails closed on the missing artifact — red on a suite that was
never going to run, on a PR whose only fault is a mislabel. `plan` and
`resolve_assets` now admit `release` for same-repo PRs only, and the
concurrency key carries the complementary term, so
NOT(A OR B OR (C AND NOT F)) stays exactly (NOT A) AND (NOT B) AND
(NOT C OR F). For a same-repo PR F is false and every one of them
reduces to what it was, so none of this changes non-fork behaviour.

The same shape still reaches the fork lane from a PUSH to a fork PR
somebody mislabelled `release` — `build` is gated on the label, not on
the event, so this predates promotion using the label. Left alone; the
comment in e2e-fork.yaml now records it instead of asserting an
invariant this PR would have made false.

The contract enumerated `plan` and `e2e-report` by name, which is why it
did not catch `resolve_assets`: a named-job loop cannot notice a job
nobody thought to name. It now derives the job list from the file and
requires every job that decides from a label name to decide from the
same set, and pins the fork term on both halves of the complement. The
file's existing properties are kept — extraction runs over
executable-only lines, so a commented-out key cannot satisfy the
contract, and every comparison asserts a non-empty input first, so an
empty extraction fails closed under both bats and cozytest.sh.

Verified by mutation, each caught: excluding a label `plan` skips for,
admitting one the key does not exclude, reverting `resolve_assets`,
deleting `e2e-report`'s guard, narrowing `verify-release-candidate`'s,
renaming the label in one guard only, commenting out the group key, and
dropping the fork term from either side alone. Deleting `plan`'s
step-level fork guard is caught too, by an assertion that had to stop
matching on the bare identifier once `plan`'s own `if:` named it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil f21d277eba fix(release): retain live promotion candidates
Protect package candidates referenced by open release PRs while pruning abandoned run-specific promotion artifacts. Document the stable-candidate lifecycle and maintenance-branch rollout requirement.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil c2e145709b fix(release): publish stable packages candidate
Serialize the tag-rewritten packages tree during RC promotion and pin its immutable digest in the installer. Verify the candidate against the prospective merge and the original RC artifact before creating stable names.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Aleksei Sviridkin 829edfadb3 test(e2e): warm the ghcr.io mirror before the tenant suites need it
The in-sandbox pull-through cache fetches a blob only when a client
asks for it, so the first tenant worker's kubelet pull is a live fetch
from ghcr.io. That leg is what misses the node-Ready budget: measured
at 40-165 KB/s for a 61 MiB image, with two blob streams dying
mid-transfer after 7-8 MiB on an upstream HTTP/2 reset. Talos holds
the kubelet service in PreFunc until the pull completes, so the node
never registers, and the tenant cilium HelmRelease then fails for want
of a schedulable node rather than for any fault of its own.

Fill the cache from a Job started in the same install step that
deploys the mirror, tens of minutes before the kubernetes suites run.
The Job waits for the mirror to serve, then asks it for each kubelet
tag's index, walks the manifests it gets back, and discards the blob
bodies; distribution populates its blobstore on the way past.

It warms one platform per tag -- the one the workers run -- picked out of
the index by record, which keeps the walk a grep. Walking both platforms
was a true addition to the same throttled leg: arm64 is bytes nothing in
the sandbox pulls, and the digest ordering put it first for both tags, so
on the slow runs this exists for the deadline expired with the second tag
still cold.

It warms the two minors the suites select, not the whole version map.
The map would be ~300 MiB at the one-platform walk this ships (~61 MiB
per tag), three fifths of it for minors no suite can resolve, spent on
the same throttled egress the rest of the install is using. The tag
list still comes from the chart's map rather than a copy, ordered the
way the suites resolve it, so the set is theirs without restating
their expression.

The readiness wait is a wall clock, not a per-attempt count. Cilium
answers a Service with no ready endpoints by rejecting, so a probe
costs nothing when the mirror is absent and the full timeout when it
is merely slow; a constant was wrong in both directions across two
attempts at picking one.

Tags are refused by character class before the pattern is applied, so a
value cannot pass by having a valid first line: grep matches per line
and the pattern's anchor is end-of-line, not end-of-input.

The Job exits non-zero on any miss, so backoffLimit is a real bound
and a warm-up that reached nothing cannot report success. It names its
own requests, because the node-join diagnostic counts kubelet GETs in
the access log to tell a worker that pulled through the mirror from
one that fell back to public ghcr.io.

Naming them is only half of that. The same diagnostic now reports the
Job's status and result and the Job outlives the suites, because
excluding the warm-up without reporting it would leave a red run
unable to distinguish a warm cache from a warm-up that never started.
That counter also moved onto the access-line shape: registry:2.8.3
writes three lines carrying the path per request, so matching the path
counted one pull as three. It reads the repository from the same
variable the warm-up does, so a rename cannot leave the diagnostic
counting zero while the workers pull under the new name. GET and
HEAD both count: containerd resolves a manifest with a HEAD first,
so a GET-anchored count reports a pull that died before its first
GET as no request at all.

The failure path also prints how long the mirror took to answer the
worker. A warm cache and a cold one are otherwise indistinguishable in
the artifacts -- the Job reports success either way, because the proxy
stores the blob in the background and a completed GET is a different
event from a stored blob -- and server-side duration separates them by
orders of magnitude. It filters the log in the shell that read it: a
single argv element is capped at 131072 bytes on Linux regardless of
ARG_MAX, and probe traffic carries this log past that within about
twenty minutes of mirror uptime, so handing it to a helper would empty
the section on every run worth reading. Nothing is printed when the log
could not be read at all, because a heading with no lines under it
reads as a finding about the workers.

Measured against a local pull-through registry using the script the
builder emits: 32m47s cold, 1.1s warm for the same tags, and the
blobs it fetches are the same digests the failing runs stalled on.

A warm-up that fails, times out or never starts does not fail the
step, and the suites behave as they did before it existed. It is not
free, though: it pulls across the same throttled egress the install
is using, which is why the warm set is two tags and not five.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-14 11:08:49 +03:00
Aleksei Sviridkin 9dc49de1c9 fix(apps): make the post-delete cleanup hooks report failure (#3593)
## What this PR does

The qdrant and harbor post-delete cleanup hooks could not fail. Every
delete was guarded by `|| echo "WARNING: ..." >&2`, neither script set
`-e`, and both ended on an unconditional completion message, so the
container exited 0 on every path. Helm read that as a successful hook,
`helm.sh/hook-delete-policy: hook-succeeded` removed the Job together
with the pod that held the warning, and an uninstall that reclaimed
nothing looked exactly like one that had nothing to reclaim. Written up
in #3566.

Both hooks now count failures and set the exit code at the end. `set -e`
would abort at the first failed delete and skip the rest of the sweep,
which is the opposite of what is wanted: every remaining object should
still be attempted and the outcome reported once. Already-gone objects
stay a success, since every delete keeps `--ignore-not-found`. Each
failure message names the object it left behind. `packages/apps/mariadb`
already uses this shape.

Both hooks also consumed `kubectl get` output directly: harbor by
iterating `$(kubectl get ...)`, qdrant by piping it into `grep`, whose
status the pipeline then reports. A failed listing arrived as an empty
list, and the sweep reported success without having looked. Each listing
is now captured before it is used.

#3566 attributes that shape to harbor's two loops alone. qdrant has it
too: `pvcs=$(kubectl get pvc ... | grep -E ... || true)` reports grep's
status, so a `kubectl get` that failed left `$pvcs` empty and the hook
announced "no data PVCs found; nothing to delete". Both instances are
fixed here.

### The cost

A hook that exits non-zero keeps its Job and its pods, because
`hook-succeeded` does not fire for it. That is the point, the pods hold
the log, but it means a failed uninstall now leaves objects behind where
it previously left none. Nothing collects them on its own:
`before-hook-creation` only fires when the hook runs again, and a
`post-delete` hook runs only on an uninstall, so they stay until a
release of that name is installed and uninstalled again, or until the
namespace goes.

The release itself is not stranded. Helm appends the post-delete hook
error to its error list instead of returning on it, marks the release
uninstalled, purges it from storage anyway, and only then reports
`uninstallation completed with N error(s)`. The failure surfaces once
and the app still goes away.

Both Jobs get one retry instead of the Kubernetes default of six, and
instead of none. Every delete is idempotent, so a second attempt
re-issues an API call rather than re-running any logic, and one short
backoff separates a transient apiserver error from a real one. These
failures become visible for the first time with this change, and a
signal that fires on noise stops being read, including on the day it is
right. Six is the far end of the same mistake: ten minutes of backoff
across seven pods outlasts the hook timeout, so Helm reports that
timeout instead of the hook's own error.

`packages/apps/clickhouse` and `packages/apps/mariadb` already exit
non-zero from their cleanup hooks and set no retry limit, so a real
failure in either spends the full backoff and reaches the operator as a
timeout rather than as the error the hook wrote. Adding the exit code
without the cap reproduces that.

One limitation is deliberate. harbor's ACME sweep cannot tell "this kind
is not served" from "I could not look". If cert-manager is torn down
before the release, `kubectl get challenge` fails and the uninstall is
reported failed with nothing to sweep. cert-manager is a hard dependency
of the bundle harbor ships in, so the window is narrow, and a
CRD-existence probe would add a second silent-skip path of its own: it
would swallow a discovery failure the way this change stops swallowing a
listing failure.

### Tests

`hack/app-cleanup-hooks_test.bats` renders each hook's script out of its
chart and runs it against a fake `kubectl`, because the property under
test is the container's exit code and matching against manifest text
cannot establish it. There are eight failure paths across the two hooks,
six in harbor and two in qdrant, and each has a case that goes red when
that one path alone is reverted to a swallowed warning. Also covered: a
clean delete, and an object that is already gone in both its shapes, an
empty listing and a `NotFound` on the delete itself.

The fake `kubectl` models `--ignore-not-found` instead of exiting 0
unconditionally, so dropping that flag from a hook turns the
already-gone cases red. Message assertions are anchored on the hook's
own `ERROR: ` prefix, because the fake names the object in its own
diagnostic too, and an unanchored match is satisfied by the fixture.

Two manifest fields are pinned in each chart's existing helm-unittest
suite, where manifest fields belong. The retry cap is one. The other is
`helm.sh/hook-delete-policy`: adding `hook-failed` to that annotation
hands the failed Job straight back to Helm to delete, which undoes the
whole change. I checked that before adding the assert, and the reversal
left both suites fully green.

The qdrant Chainsaw suite explained its post-delete pod collector by
quoting the guard string this change removes, and concluded that Helm
always collects the pod away. That conclusion inverts here, so the note
is rewritten, along with the sentence in the harbor suite that points at
it.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. It touches two app chart templates and adds one file
under `hack/`. No package added, renamed or removed, no
`values.schema.json`, no `values.yaml` default, no version enum, no
`ApplicationDefinition`, no namespace, no node prerequisite, and neither
`hack/package.mk` nor `hack/update-crd.sh`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(apps): the qdrant and harbor post-delete cleanup hooks now exit non-zero when a delete or a listing fails, instead of reporting success on every path. `helm uninstall` reports the failure rather than returning clean, and the hook Job and its pods stay in the namespace so the log survives to be read. Nothing collects them on its own: they remain until a release of that name is installed and uninstalled again, or until the namespace is removed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved Harbor and Qdrant cleanup reliability by detecting listing
and deletion failures.
- Cleanup jobs now retry once and report failures with a non-zero
status.
- Cleanup continues processing remaining resources after individual
failures.
- Successful cleanup remains safe and repeatable, including when
resources are already missing.
- Harbor cleanup remains limited to resources associated with the
configured Harbor host.

- **Tests**
- Added comprehensive coverage for successful, repeated, partial, and
failed cleanup scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 10:55:01 +03:00
Aleksei Sviridkin 340483fdf0 fix(hack): declare the shared package targets and the root test target phony (#3353)
## What this PR does

`hack/package.mk` set `.PHONY` with `=` instead of `:`:

```make
.PHONY=help show diff apply delete update image
```

That assigns a variable which happens to be named `.PHONY`; it declares
nothing. `make -p` prints it back among the variables (`.PHONY = help
show diff apply delete update image`) rather than as a special target,
so none of the targets `package.mk` defines was phony in any package
that includes it.

Nothing in the tree collides with those names today, so this is
preventive rather than a fix for present breakage. The failure it
forecloses is quiet: with a directory named `image` beside a package
Makefile, `make image` prints `make: 'image' is up to date.` and exits 0
without building. A file of that name is enough on its own where the
rule has no prerequisites; where it has one, the recipe keeps running
until that prerequisite is shadowed too.

The list changed as well. `suspend`, `resume`, `check` and `clean` are
defined in this file and were missing, and `check` is the one that
matters most, since it is the prerequisite that keeps `show`, `diff`,
`apply` and `delete` rebuilding. `update` and `image` are gone, because
`package.mk` does not define them; the including package Makefile does.
Naming an undefined target in `.PHONY` creates it as an empty target,
turning `No rule to make target` (exit 2) into `Nothing to be done`
(exit 0) wherever the rule is missing. That reaches CI too:
`hack/build-matrix.sh` parses the root Makefile `build:` target and runs
`make -C <pkg> image` per unit, so a package arriving in that list
without an `image:` rule would go from a red job to a green one that
builds nothing.

`.DEFAULT_GOAL` on the line above keeps its `=`. That one is a real
variable, so the assignment form is right there. A comment now says
which of the two lines is which, since they read alike and one of them
just changed shape.

`hack/package-mk-phony.bats` covers the declaration and is picked up
automatically by the `hack/*.bats` glob in `bats-unit-tests`. Five of
its six tests cover `package.mk`: that the declared names are really
phony with a file of each name sitting beside the Makefile; a control
that mutates a copy back to the assignment form and requires the probe
to report up-to-date, so the first cannot pass vacuously; that the
declared list and the defined targets are the same set, checked both
directions; and two pinning the `update`/`image` exclusion, one
requiring the loud failure to survive and one extending the list in a
copy and requiring it to disappear. The loud-failure test pins both the
exit status 2 and the diagnostic. A wrapper that kept make's wording
while returning something else would pass a check on the message alone.

One point decides where the guarantee actually lives. The first test
reads its names from the `.PHONY:` line, so it cannot see a name dropped
from that line at all. The set-equality test is what guards every name's
presence.

One known gap is left in place. The first test decides by matching
make's `is up to date`, which make prints only for a target that has a
recipe, and a recipe-less alias never draws it: with no prerequisite
both forms print `Nothing to be done`, and with a prerequisite that is
itself out of date, which every target in this file is for being phony,
the working form prints that prerequisite's recipe while the broken form
prints `Nothing to be done`. The probe separates neither pair, so such a
name would pass the first test either way. Measured on both shapes, not
assumed. Every target `package.mk` defines has a recipe, so nothing is
uncovered today; the control test does not close the gap either, since
an alias is a defined target and the control requires `is up to date`
from it, which turns it red against the probe rather than against the
alias. Worth closing if an alias is ever added.

One more for whoever edits the suite next: `hack/cozytest.sh` rewrites
every bare `}` at column 0 into `return 0` followed by `}`. That is
aimed at `@test` blocks, but it does not distinguish them from a plain
shell function, so a helper factored out of these tests would silently
become unable to return a non-zero status. Tests 1 and 2 repeat a loop
rather than share one for reasons of their own, but this is the trap
waiting for anyone who merges them.

Two properties of the suite are worth stating, because both are easy to
get wrong in the other direction. The set-equality test extracts the
defined targets with a pattern that accepts a rule naming several
targets at once, so `suspend resume: check` keeps both names in the set;
anchoring the colon to a single name drops every name on such a line,
and for a target nothing else depends on that goes unnoticed. And the
control test probes the targets `package.mk` defines rather than the
names on its `.PHONY:` line, because `is up to date` is only a
meaningful answer for a name that has a rule: a declared-but-undefined
name prints `Nothing to be done` in both the working and the broken
form, so probing it in the control fails against the probe, while the
set-equality test holds the message that names the defect.

Last one, about how the tests find the repo. All six read it as
`repo=$(pwd)`, which assumes the runner starts at the repo root. That
holds for `make bats-unit-tests` and for the invocation the header
documents, so nothing is broken, and the assumption is the ordinary
shape in `hack/` rather than an outlier: about half the bats files there
resolve repo-relative paths against the cwd the same way, several
stating it in their own headers, and this is not even the only one
spelling it with `$(pwd)`. What is worth naming is the failure mode:
started elsewhere, these tests go red with `no '.PHONY:' declaration
found in hack/package.mk`, which reads like a real finding rather than a
wrong directory.

The root `Makefile` had the same defect, and there it had already fired.
`test` was not on its `.PHONY:` line and a `test/` directory sits beside
it, so `make test` printed `'test' is up to date` and exited 0 without
running anything, while `docs/agents/overview.md` advertises that
command as the way to run the full suite. CI never used it: the workflow
calls `test-controllers` and drives e2e out of `packages/core/testing`,
so the cost fell on anyone following the docs by hand. One name on the
declaration closes it, and the suite gained a sixth test that goes red
without it. Every other undeclared target in that file is unshadowed
today, so none of them is reachable the way `test` was, and they are
left alone here. #3709 covers the same class in the four
`packages/core/` Makefiles, which include only `hack/common-envs.mk` and
so are out of reach of this change entirely.

### Downstream repositories

`scripts/package.mk` in cozystack/external-apps-example was a
byte-for-byte copy of `hack/package.mk` (identical sha256 before this
change), carrying the same inert `.PHONY=` line. This PR splits the two
files, and the copy was fixed separately. It landed the same nine names,
and `update` and `image` are absent there for the reason they are absent
here: `scripts/package.mk` does not define them either, and neither does
any Makefile that includes it. What separates the two files now is the
two-line comment added here, and nothing else.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [x]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up: https://github.com/cozystack/external-apps-example/pull/4
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Fixed task declaration handling so supported commands are correctly
recognized and executed.
* Updated available commands by replacing `update` and `image` with
`suspend`, `resume`, `check`, and `clean`.
* **Tests**
* Added coverage to verify command declarations, prevent regressions,
and ensure invalid or missing commands report errors correctly.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 10:54:10 +03:00
Aleksei Sviridkin d978a371a2 test(hack): cover the .PHONY declarations this tree relies on
`.PHONY=a b c` and `.PHONY: a b c` differ in the `=` against the
`:`, and only the second declares anything. The first defines an
ordinary variable named `.PHONY` that nothing reads, leaving every
listed target file-backed. A file or directory of that name beside a
package Makefile then stands in as the product of the rule: for a
target with no prerequisites make reports it up to date, skips the
recipe and exits 0, and a target whose prerequisite is itself
file-less keeps rebuilding until that prerequisite is shadowed too.
The neighbouring `.DEFAULT_GOAL=help` is a real variable and is
correct with `=`, so the two lines read alike while only one of them
works, which is what let the inert form sit there unnoticed.

Add hack/package-mk-phony.bats, six tests, picked up automatically
by the hack/*.bats glob in `bats-unit-tests`.

Three cover the declaration itself. The first reads the names off
the `.PHONY:` line rather than repeating them, plants a file named
after each beside a stub Makefile, and requires `make --dry-run` to
print the recipe instead of "is up to date". A negative control
mutates a copy back to the assignment form and requires the probe to
report up-to-date, over the targets package.mk defines rather than
the names it declares, because "is up to date" is only a meaningful
answer for a name that has a rule. The third keeps those two sets
equal: it requires the declared list and the defined targets to be
the same set. Defined but undeclared
is the direction the list rots in: check, clean, suspend and resume
sat undeclared for as long as the declaration itself was inert.
Declared but undefined is the empty-target trap in its general
form, so a future name cannot slip in the way update and image
once did.

Two more pin what the list deliberately leaves out. package.mk
defines neither `update` nor `image`; the including package Makefile
does, and most packages define neither. Naming them here would
create them as empty targets everywhere, turning "No rule to make
target" into "Nothing to be done". Since the CI build matrix is
parsed from the root Makefile `build:` target and runs
`make -C <pkg> image` per unit, a package reaching that list without
an `image:` rule would go from a red job to a green one that builds
no image. One test requires that loud failure to survive, pinning
both of the observables it is made of: make's exit status 2 and its
diagnostic. They are independent, and a wrapper that kept the
wording while returning something else would satisfy a check on the
message alone. Its control extends the declaration with the two
names in a copy and requires the failure to disappear, so the pin
tracks the list rather than some property of the harness.

Every decoy carries one fixed timestamp. Six of the nine targets take
check as a prerequisite, and apply and delete reach it through suspend
as well, so touching the decoys in list order can leave check newer
than its own dependents. Make then rebuilds them for that reason and
prints a recipe whatever .PHONY says, and the probe reports success
against a declaration that does nothing. Equal mtimes remove the
question in any list order. Measured on GNU Make 4.3: list order left
the per-target assertion inert for up to six of the nine, moving with
where the clock tick fell, while 3.81 fits the loop inside a single
timestamp and shows none of it. The control test plants its decoys
the same way for the same reason, and goes red if that is undone in
its own loop. The two loops are independent, so neither guards the
other, and the header is what keeps the arrangement from being
unpicked.

A sixth test leaves package.mk and covers the root Makefile, where this
class had already bitten: `test` is the one target in the tree with a
path of its own name beside it. It probes with --dry-run because the
recipe wants a live cluster, and fails loudly if that path ever goes
away rather than passing on a question with no subject.

`%-update` is not coverable: .PHONY matches target names literally
and does not expand patterns.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-14 01:25:34 +03:00
IvanHunters 09dc7f47fb feat(backups): add a native MongoDB (psmdb) backup-strategy driver (#3562)
## What this PR does

Adds a native MongoDB (Percona Server for MongoDB / psmdb)
backup-strategy driver so `apps.cozystack.io/MongoDB` reaches feature
parity with the CNPG, MariaDB and ClickHouse native drivers — logical
backups, restore into a differently-named instance, and point-in-time
recovery — all through the unified `BackupJob` / `RestoreJob` /
`BackupClass` / `Plan` interface (closes #3545).

The driver orchestrates the operator's `PerconaServerMongoDBBackup` /
`PerconaServerMongoDBRestore` CRDs:

- **BackupJob**: creates a one-shot `PerconaServerMongoDBBackup`, polls
`status.state`, and persists the S3 destination + storage descriptor (by
reference — Secret name only, never a raw credential) onto the Cozystack
`Backup` artifact.
- **RestoreJob**: drives `PerconaServerMongoDBRestore` via
`backupSource` (read back from the source backup status, with a snapshot
fallback for a reaped operator CR) so the same artifact restores
in-place **or** into a differently-named instance. PITR is exposed as
`spec.options.recoveryTime` (RFC3339), matching the CNPG driver's option
exactly; unrecognised `spec.options` keys are surfaced as a
`UnknownRestoreOption` Warning event rather than silently dropped.

Because psmdb resolves the backup storage by name against the cluster's
`spec.backup.storages`, the driver names the chart's storage
(`s3-storage`) via the `cozy-default-mongodb` Strategy rather than
injecting it, and surfaces a clear `Ready=False` precondition when a
target has backups disabled instead of hanging until the deadline.

**Wiring**

- `api/backups/strategy/v1alpha1`: `MongoDB` strategy type + generated
CRD/deepcopy.
- `internal/backupcontroller`: `psmdbstrategy_controller` + `psmdbtypes`
/ `mongodbapp` partial-schema mirrors; BackupJob/RestoreJob dispatch,
supported-kinds, cleanup-on-delete branch, and scheme registration.
- `backupstrategy-controller`: `cozy-default` MongoDB route,
`cozy-default-mongodb` Strategy CR, and `psmdb.percona.com` RBAC
(get/list/watch, +create on backups/restores, no delete — the driver
does not own the S3 archive).
- `apps/mongodb`: a `backup.insecureSkipTLSVerify` value (psmdb's s3
storage has no CA-bundle field) so backups can target Cozystack's
self-signed in-cluster seaweedfs. Default `false`, fully
backward-compatible; regenerated schema/README/RD.

**Tests + e2e**

- Unit tests for the driver: applicationRef validation,
ensure-idempotency (backup + restore), the backups-enabled/storage
precondition gate, `backupSource` resolution (live-preferred / snapshot
fallback / no-destination), snapshot round-trip, the producer-seam
(`createMongoDBBackupArtifact`) driver-metadata, the cleanup-on-delete
no-Velero-fall-through guard, and `recoveryTime` → psmdb-UTC mapping +
unknown-option reporting.
- `examples/backups/mongodb/` + a `mongodb-2-backup-roundtrip` Chainsaw
suite that drives `run-all.sh` end-to-end (a sentinel document
round-trips through S3 into a restored copy while the source stays
untouched), per `docs/agents/e2e-testing.md`.
- `docs/operations/backup-classes.md` updated (new route, the
app-owns-the-storage model, and the `recoveryTime` PITR usage).

Verified end-to-end on a live dev cluster against the platform-default
`cozy-backups` bucket: backup via `cozy-default` reached the bucket and
the Cozystack `Backup` went `Ready`, and a to-copy `RestoreJob` restored
the sentinel into a differently-named instance with the source left
intact.

### Screenshots

N/A — no UI changes (controller, chart values, CRD, docs and tests
only).

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file by file
against the diff:

- **cozystack/website** — the only app-facing change is
`packages/apps/mongodb` (a new `backup.insecureSkipTLSVerify` value +
regenerated `README.md`). The managed-app reference page is regenerated
from the package `README.md` by the release bot for apps already in the
website `Makefile` (mongodb is one), so it refreshes automatically on
the next tag — no manual follow-up. `docs/operations/backup-classes.md`
is not a website-mirrored doc in the trigger map.
- **cozystack/terraform-provider-cozystack** — the one candidate trigger
("a field added to an app's `values.schema.json`") lands inside the
mongodb `backup` block, which the provider's
`internal/provider/mongodb_schema.go` explicitly states it does **not**
manage ("The deprecated backup block, the bootstrap restore block, and
detailed shardingConfig are not managed"). No enum, default,
`release.prefix`, Secret/Service name, or hand-typed CRD (`Plan`,
`RestoreJob`) changed; the new `strategy.backups.cozystack.io/MongoDB`
CRD is opaque to the provider. Not affected.
- All other repositories — no matching trigger (no `hack/` layout or
make-target change, no namespace rename, no `ApplicationDefinition`
semantics change, no `packages/core/platform` values, no telemetry
metric or `cozy-proxy` label rename).

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(backups): add a native MongoDB (psmdb) backup-strategy driver — logical backup to the cozy-default BackupClass, restore into a differently-named instance, and point-in-time recovery via spec.options.recoveryTime
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added MongoDB backup and restore support, including scheduled and
on-demand backups, restores to new or existing clusters, and
point-in-time recovery.
* Added configurable S3 TLS certificate verification for MongoDB
backups, defaulting to secure verification.
  * Added MongoDB support to the default backup configuration.

* **Documentation**
* Added MongoDB backup and restore examples, workflows, and operational
guidance.

* **Tests**
* Added automated coverage for MongoDB backup, restore, cleanup, and
end-to-end scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-13 11:18:46 +03:00
Aleksei Sviridkin 61a236cd96 feat(ci): auto-apply triage labels to issues (#3574)
## What this PR does

Issues that arrive through the API or `gh issue create` bypass the issue
templates, so they carry no `triage/*` label and nothing in the
repository adds one afterwards. Listing every open issue through the
same endpoint the workflow uses, `gh api --paginate
"/repos/cozystack/cozystack/issues?state=open&per_page=100"`, returns
567 entries, 171 of them pull requests, at the time of writing. That
leaves 396 open issues, and 252 of them carry no triage label at all.
Every count in this description is a point-in-time reading of a backlog
that keeps moving, so a later run of the same query will differ by a
few. The triage queue is impossible to filter on in that state, and
there is no way to see how much of the backlog nobody has looked at.

This adds a workflow that labels an issue when it is opened or reopened,
plus a daily sweep that backstops labels removed by hand and works
through the existing backlog. An issue somebody has assigned, or marked
with one of `priority/critical-urgent`, `priority/important-soon`,
`priority/important-longterm` or `epic`, gets `triage/accepted`, since a
maintainer has already looked at it. Everything else gets
`triage/needs-triage`. An issue already carrying any `triage/*` label is
left alone, so the sweep never overwrites a decision someone made by
hand, and pull requests returned by the same listing endpoint are
skipped. Against the listing above, the sweep applies `triage/accepted`
to 39 issues and `triage/needs-triage` to 213, and touches none of the
144 that are already triaged.

Deleting a triage label by hand does not undo any of this. An issue left
carrying no `triage/*` label is relabeled by the next sweep, so a triage
decision is changed by replacing the label. `AGENTS.md`'s labeling
section now says so, because that is the file somebody reads before
labeling by hand.

Two conditions decide what counts as a signal. Membership of
`stale.yaml`'s `exempt-issue-labels` is necessary: `triage/accepted` is
itself on that list, so applying it does more than reset a clock, the
issue stops being auto-closable, and a signal that is not already exempt
would hand out that permanent reprieve as a side effect of being
labeled. `priority/backlog`, the one tier `stale.yaml` deliberately
leaves reapable, is excluded for exactly that reason.

Membership is not sufficient, which is why the signal list is not simply
the exempt list. `lifecycle/frozen` is exempt and still not a signal: a
`lifecycle/*` label states what the stale bot may do with an issue and
says nothing about whether anybody reviewed it. Freezing an untriaged
issue to keep the bot off it is the case where reading it as accepted is
wrong. Of the five open issues whose only signal would have been
`lifecycle/frozen`, one is blocked on an upstream project and carries
`help wanted`, and one is an older feature request that an open epic now
covers. Neither is ready to be worked on, and stamping either accepted
would have kept it out of triage for good, since the sweep never
revisits an issue that already carries a triage label. `epic` passes the
same test from the other side: it is a maintainer-authored planning
artifact rather than an inbound report, and all eight open issues whose
only signal is `epic` are roadmap trackers that `needs-triage` would
describe falsely. A contract test pins the rule and not the instance, so
no label from the `lifecycle/*` namespace can quietly become a signal
later. `security/*` stays out on the same grounds, though the case is
closer: those labels record what is known about a vulnerability rather
than whether anyone triaged the report, and the cost of being wrong runs
the cheap way, since a `needs-triage` on a confirmed report is one click
to remove while a wrong `accepted` is permanent because the sweep never
revisits.

One caveat stated plainly: the signals stand for a maintainer decision
already recorded on the issue, which is a weaker claim than the one
`triage/accepted` makes in `.github/labels.yml`, and
`priority/important-longterm` says outright that the work may not be
staffed. Of the two labels this workflow can write it is still the
closer one, since `needs-triage` on an issue somebody already
prioritised or took asks for a triage that has happened, and a
maintainer who disagrees replaces the label in one click. An assignee is
the single signal that is neither a label nor already exempt, so an
issue whose only signal is an assignee does gain a permanent reprieve.
That is the intended reading rather than a side effect: somebody took
ownership, which makes it a known long-tail task rather than an
abandoned one, and it is the same argument `stale.yaml`'s own comment
makes for exempting `triage/accepted` in the first place.

Three operational details. First, the sweep paces its writes a second
apart and caps a run at 400, because GitHub allows 80 content-generating
requests a minute and 500 an hour, this client carries no throttling
plugin and no retries, and a throttled write answers `403`, which is
exempt from retry anyway. Unpaced, the first pass over the backlog would
not slow down when throttled, it would fail the job partway and leave
the backlog half labeled. Whatever a run leaves behind stays unlabeled,
so the next one picks it up. Second, adding a label bumps an issue's
`updated_at`, which `stale.yaml` reads as activity, so the first sweep
restarts the 60-day staleness clock on every issue it writes to, all 252
of them, and un-marks the 32 that carry `lifecycle/stale` today. Both
are deliberate and one-time: the backlog gets triaged now and ages again
from there, rather than being closed unread while it is still unlabeled.
Third, a run writes a job summary with the counts, the issue numbers
under each label, and both remainders it can leave behind: the write
cap's deferral, and how much of the listing it never reached when a
refusal stopped it early. A sweep over the backlog logs a line per
issue, and the summary is what makes the outcome readable without paging
through the 252 the current backlog produces.

A write that fails does not take the rest of the run with it, but the
two failure shapes are told apart rather than counted together. Any
status other than a `403` or `429` lets the run carry on, most often a
`404` or `410` from an issue transferred or deleted mid-sweep, since a
failure about one issue says nothing about the next. A `403` or `429`
means the token lost `issues: write` or the pacing stopped being enough,
which applies to every write left in the run, so the sweep stops at the
first one and the job fails loudly. Repeating a rejected request up to
the cap is what gets an integration throttled harder.

`workflow_dispatch` takes a `dry_run` input that logs what would be
labeled without writing anything. It is on by default, so a manual run
writes nothing until somebody unchecks the box, and `schedule` is the
one event that writes without being asked. The cap applies in dry-run
too, so a dry run predicts what a real one would do instead of printing
a longer list nobody gets. The job asks for `issues: write` and
everything else is read-only, the action it uses is pinned by digest,
and a 20-minute timeout keeps a hung call from holding the queue for the
default six hours.

No pull request lane can exercise this workflow, so its first real
execution is a sweep over every open issue with nothing having run it
before. `hack/issue-triage-contract.bats` pins the executable lines that
bound that blast radius: the pull-request skip, the already-triaged
skip, the pacing rate and the write cap, the job timeout, the failure
handling, the digest-pinned action, the token scopes, the serialised
queue, the dry-run default and the derivation that does not depend on
it, the cron ordering against `stale.yaml`, that both labels it writes
exist in `.github/labels.yml`, that no accepted-signal label sits
outside `stale.yaml`'s exempt list or inside the `lifecycle/*`
namespace, and that the summary is written with the `await` that makes
it flush and accounts for every outcome a run can end on. Each assertion
was checked against a mutated copy of the workflow to confirm it goes
red rather than merely existing, with the mutations run through
`hack/cozytest.sh` rather than `bats`. The two runners disagree about
negated assertions, so a pin that `bats` reports as live can be dead
under the runner CI actually uses, and every negative assertion in the
file captures grep's status instead of leading with `!`.

Suggested first step after merge: run the workflow by hand and read the
summary before the 05:53 cron arms itself. The box arrives checked, so
that run costs nothing.

### Screenshots

Not a UI change, so there is nothing to show.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff. The diff adds two files, `.github/workflows/issue-triage.yaml` and
`hack/issue-triage-contract.bats`, and appends one line to `AGENTS.md`.
No row matches: it adds and renames no package, changes no chart values,
no schema, no version enum, no `ApplicationDefinition`, no CRD, no
namespace, no variant or bundle, no label or annotation any downstream
repository matches on, no metric and no release asset. Its `hack/` rows
name `hack/upload-assets.sh`, `hack/e2e-prepare-cluster.bats`,
`hack/package.mk`, `hack/update-crd.sh` and `hack/common-envs.mk`, which
downstream repositories copy, anchor on or mirror by hand, plus one row
for moving or renaming anything under `hack/`; a new bats file added
next to them is none of those. The map has no row for the agent
documentation, and no downstream repository copies `AGENTS.md`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(ci): new and reopened issues now get a `triage/*` label automatically, and a daily sweep labels any open issue that has none: `triage/accepted` when it is assigned or carries an `epic` or a priority above `priority/backlog`, `triage/needs-triage` otherwise.
```




<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added automated issue triage triggered by new or reopened issues,
scheduled reviews, or manual runs.
* Issues can be labeled as needing triage or accepted, with pull
requests and already-triaged items excluded.
  * Added dry-run support and reporting for triage results.

* **Documentation**
* Added guidance for replacing existing triage labels when making
decisions.

* **Tests**
* Added coverage for workflow behavior, permissions, scheduling, failure
handling, and safety limits.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-13 02:41:54 +03:00
Aleksei Sviridkin 59878850f9 test(e2e): capture worker CPU throttling counters on node-join failure (#3723)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

When a tenant worker misses the node-Ready deadline, the failure bundle
currently cannot tell two different causes apart: a worker held at its
own CFS ceiling, and a worker losing host CPU it was entitled to. Both
look identical from outside the Pod, and node-level utilisation answers
neither, because it shows what the node used rather than what the
container was allowed.

This adds a collector that reads the CFS counters and the ceiling itself
from the kubelet's cAdvisor endpoint, for every tenant worker on every
node in the sandbox. The counters are the only place that difference is
recorded, and nothing else in the tree collects them for these workers.

The motivating measurement is on nightly run 31290999592: the `compute`
container reports 0.968 and 0.920 core against a `limits.cpu` of 1,
which is 92-97% of a one-core ceiling. Whether the container ever
actually met that ceiling is exactly what the artifact cannot say, and
what these counters answer.

Most of the change is the failure taxonomy rather than the read. An
empty capture is the dangerous outcome here, because zero bytes reads as
a container that never hit its ceiling -- the conclusion the collector
exists to stop a reader reaching by default. So every way the capture
can come up short gets its own sentence: the kubelet was not read, the
read was cut short part way, the metric stream could not be read back on
this runner, the kubelet answered and carried no series for this
namespace, the namespace was there and the worker Pods were not. The one
non-failure outcome is written down too: a container with no CPU limit
puts a single line on the wire, which is also what a truncated read
leaves behind, so it says which one it is.

### Ordering

The collector runs first among the gated collectors, ahead of the guest
serial console. Its ceiling is four bounded reads, about 100s of a 480s
phase budget, so on a slow run the console capture starts that much
later. That is a deliberate trade and not a side effect of putting the
cheapest first: cost is not what settles the order, survivability of the
answer is. The console evidence cannot be re-taken, and this collector's
question has no other answer in the tree, so both go ahead of the
captures whose state is partly recoverable from reads above them.

### Known limits, named rather than left to be found

- The outcome ladder enumerates combinations of read status, filter
status and result emptiness. Every new source of partial output needs a
new rung, and that has already happened twice while this branch was in
review. Deriving the label from the three axes instead would make a new
source land in an existing axis.
- The uncapped note is decided over the whole per-node file, so one
quota line anywhere in it suppresses the note. That is the conservative
direction -- it never claims uncapped over a file that shows a ceiling
-- but it makes the note rare.
- The messages say `see read-error.log` while the file on disk is
`<node>.read-error.log`. That is the convention the neighbouring
serial-console collector already uses, and changing one side alone would
make the two inconsistent.
- The `mkdir -p` for the report directory is unchecked, as it is in four
neighbouring collectors in the same file. Worth fixing across the file
rather than in one collector.
- The section letters in the failure block no longer track emission
order. They already did not before this change; this adds one more entry
to the sequence.
- The count of bats files sourcing the library, in
`docs/agents/e2e-testing.md`, is corrected here from three to seven.
Nothing pins it, so it will drift again.
- The node walk iterates `for node in ${nodes}` unquoted, so pathname
expansion applies alongside word splitting. Node names are RFC1123 so it
cannot bite today; `while IFS= read -r node` would make that structural
instead of incidental.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(e2e): capture tenant worker CPU throttling counters and their CFS ceiling when a node-join fails, so a worker held at its own limit can be told apart from one losing host CPU
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added worker CPU-throttling diagnostics to Kubernetes troubleshooting
reports.
* Reports now identify CPU quota gaps, partial reads, filtering issues,
and incomplete results.
* Added safeguards for bounded node traversal and configurable
collection budgets.

* **Bug Fixes**
* Improved failure handling when diagnostic commands, timeouts, or
metric reads are unavailable.

* **Tests**
* Expanded coverage for CPU-throttling capture, node-join diagnostics,
timeout behavior, configuration validation, and partial failures.

* **Documentation**
* Updated diagnostic collector counts and collection-order
documentation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-13 02:24:50 +03:00
Aleksei Sviridkin 3ad4d452fc fix(tests): fail when a chart runs no Helm unit test suites (#3632)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

`helm unittest` is fail-closed on a suite it cannot parse and on one
declaring `tests: []`, and fail-open on suites that are not there at
all: it prints `Test Suites: 0 passed, 0 total` and exits 0.
`hack/helm-unit-tests.sh` judges each package by that exit code alone,
so a chart whose suites were deleted, moved, or renamed past the
`tests/*_test.yaml` glob reports success having asserted nothing.

That is measurable in both directions rather than argued. Move
`packages/system/velero/tests/velero_test.yaml` aside and run the script
as it exists on main today:

```text
Running tests in packages/system/velero
helm unittest .

### Chart [ cozy-velero ] .


Charts:      1 passed, 1 total
Test Suites: 0 passed, 0 total
Tests:       0 passed, 0 total
```

The run ends with `All Helm unit tests passed.` and exit 0. The chart
passed, and the count of things it checked was zero. With this PR the
same tree exits 1 and names the package and the reason. Neither
`--strict` nor an explicit non-matching `-f` glob changes this on its
own; all three forms exit 0, checked against plugin 1.1.1.

So the script now reads the captured output and fails the package when
the zero-suite line appears.

Matching that line names a cause and a remedy where it applies, but the
set of ways to run nothing is not closed. A recipe that never invokes
`helm unittest`, and a `test:` rule that make finds nothing to do for,
print no marker of any kind and exit 0, so no additional match reaches
them; enumerating absences leaves a fresh hole every time a new shape
turns up. So the script also requires the line a real run always emits,
a `Test Suites:` count of at least one. That turns the question round:
every way of asserting nothing exits 0, and "what did this run assert"
is the question that has an answer. All 76 packages the script visits
emit that line today. The tradeoff is that a `test` target which
legitimately runs no `helm unittest` would need renaming or an opt-out,
which is the same bargain the zero-suite check already strikes.

The discovery gate has the same shape from the other side. `make -C dir
-n test` succeeds against a file-backed target, so a path named `test`
beside a package Makefile whose rule is not phony would make both the
gate and the run exit 0 without the recipe ever firing. The check keys
on what make reports rather than on the path existing, because a package
that declares the target phony runs its recipe whatever sits next to it,
and refusing that would be a false failure. The quoting around the
target name differs between make 3.x and 4.x, so it accepts either
opening quote; anchoring on the trailing quote alone would also match a
sub-make reporting some other target whose name ends in `test`.

That second check is guarding a live gap rather than a hypothetical one.
`hack/package.mk` line 2 reads `.PHONY=help show diff apply delete
update image`, which is a variable assignment and not a target
declaration, so it makes nothing phony. Of the packages the script
visits that define a `test` rule, 18 declare it phony in their own
Makefile; the rest are unprotected if a path of that name ever appears.
No such path exists today. Fixing the shared include is #3353 and is not
this PR.

One deliberate non-change, stated so it is not rediscovered as an
oversight. Running the suite now captures output instead of streaming
it, because a pipeline's exit status in POSIX `sh` reports the last
command rather than `make`. Streaming reads better and loses the status,
so it stays buffered.

The run is pinned to `LC_ALL=C`, since both checks match English wording
and a localized `make` would disarm one of them silently, which is the
failure mode the whole change exists to remove.

The guard is all-or-nothing, which is worth stating plainly. A chart
that loses four of its five suites still reports `Test Suites: 1 passed`
and passes; only total disappearance is caught. Tying the expected count
to what each chart actually has would be a per-chart number to maintain,
and a number maintained in one place while the suites move in another is
the failure this repo has been paying for elsewhere, so the cheap check
that catches the total loss is the one worth having.

One behaviour worth stating because the message does not: a suite in
which every test carries `skip:` reports `Test Suites: 0 passed, 1
skipped, 1 total` and exits 0, and the positive-evidence check refuses
it. That refusal is intended, since a wholly skipped suite asserted
nothing, but the message talks about suites expected under `tests/`,
which in that case are present and skipped on purpose. No package is in
that state today. Partial skips are unaffected: `2 passed, 1 skipped, 3
total` satisfies the check.

Three known gaps in what the change ships, none of them a wrong
statement. The positive-evidence message explains its cause but stops
short of naming a remedy, where the other two messages both end in an
action; the remedy for the legitimate case, renaming the target or
giving it an opt-out, currently lives only in the code comment. That
same message enumerates two ways a run reports nothing, and the
skipped-suite case above is a third, so it reads as a diagnosis where it
is really a list of the common causes. And no document in the tree
states the convention this tightens: `docs/agents/overview.md` still
describes the script accurately as running over every package that
defines a `test` target, but the contract is now that such a package
must also report at least one passing suite. All three are improvements
to make on the next touch of these files rather than reasons to hold the
change.

One adjacent gap stays open, and it is worth naming rather than leaving
someone to assume otherwise. This catches a chart whose suite *files*
vanished; it does not catch a package that loses its `test:` rule along
with them. Nothing ties the presence of `tests/*_test.yaml` to the
presence of a `test` target, so a change that removes both is skipped in
silence, and the script only complains when no package in the tree has a
`test` target at all. The test named `a package with no test target is
skipped, not failed` pins that permissiveness deliberately, because the
many packages that legitimately define no `test` rule would otherwise
turn every run red. Closing it needs an instrument keyed on the suite
files rather than on the Makefile target, which is a separate change.

`hack/helm-unit-tests.bats` adds eight tests, picked up automatically by
the `hack/*.bats` glob in `make unit-tests`. Three pin the new refusals,
and each matches its own message so none can stay green on a refusal
that came from another. Two pin what must not be refused: a phony target
with a colliding path, and a sub-make reporting an unrelated target up
to date. One pins that a chart which does run suites still passes, which
is what stops the refusals being satisfied by rejecting everything. Two
pin behaviour that was already there, namely that a failing package is
still reported by name and that a package with no `test` rule is skipped
rather than failed. That last one builds a tree carrying a second
package that does run a suite: with only the target-less package
present, the script takes an early exit above the failure summary and
the assertion would hold whatever the script had done. The fixtures stub
`make` output instead of invoking helm, so they need no plugin
installed.

Relates to #3453.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(tests): `make helm-unit-tests` now fails a chart that runs no Helm unit test suites, instead of reporting success for a chart whose suite files were deleted, moved, or renamed out of the `tests/*_test.yaml` glob
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved Helm test validation to detect skipped test recipes, zero
discovered suites, and runs without evidence of a passing suite.
* Helm test failures now provide clearer diagnostics, including affected
directories and failed commands.
* Test output is consistently captured and replayed for easier
troubleshooting.

* **Tests**
* Added comprehensive coverage for successful, failed, skipped, empty,
and invalid Helm test scenarios.
* Expanded coverage across package test targets, including phony and
unrelated sub-make behavior.

* **Documentation**
  * Clarified requirements for package test targets and suite reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-12 23:37:43 +03:00
Aleksei Sviridkin 94c41d100f fix(hack): fail when a chart runs no Helm unit test suites
helm-unittest is fail-closed on a suite it cannot parse and on one
declaring `tests: []`, and fail-open on suites that are not there
at all: it prints "Test Suites: 0 passed, 0 total" and exits 0.
Measured on plugin 1.1.1 against a throwaway chart, all three
forms.

hack/helm-unit-tests.sh judges each package by that exit code
alone, so a chart whose suites were deleted, moved, or renamed past
the tests/*_test.yaml glob reports success having asserted nothing.
Moving packages/system/velero/tests aside reproduces it: the script
prints "All Helm unit tests passed" and exits 0.

Check the captured output for the zero-suite line and fail the
package when it appears. Neither `--strict` nor an explicit
non-matching `-f` glob does this on its own; both exit 0.

Matching that line names the cause where it applies, but the set of
ways to run nothing is not closed: a recipe that never invokes
helm-unittest, and a `test:` rule make finds nothing to do for,
both print no marker at all, so no further match reaches them. So
also require the line a real run always emits, a `Test Suites:` count
of at least one. Every way of asserting nothing exits 0, and asking
what the run asserted is the question that has an answer. All 77
packages the script visits emit that line today. The tradeoff is
that a `test` target which legitimately runs no helm-unittest would
need renaming or an opt-out, which is the bargain the zero-suite
check already struck. That is a contract contributors have to know
about, so docs/agents/overview.md states it where it describes the
sweep.

Because the check matches helm-unittest's own summary wording, a
plugin upgrade that reworded it would fail every package at once
while blaming each package's Makefile. The error text says so, so
the reader of a tree-wide red has the real cause in front of them
rather than 77 copies of the wrong one.

The discovery gate has the same shape from the other side.
`make -C dir -n test` succeeds against a file-backed target, so a
path named `test` beside a package Makefile whose rule is not phony
would make both the gate and the run exit 0 without the recipe
firing. Of the packages the script visits that define a `test` rule,
18 declare it phony, and no such path exists today. Match what make
reports rather than looking for the path, because a package that
declares the target phony runs its recipe whatever sits next to it,
and refusing that would be a false failure. The
quoting around the target name differs between make 3.x and 4.x, so
the match accepts either opening quote; the trailing quote alone
would also match a sub-make reporting another target whose name
ends in "test".

Running the suite now captures output rather than streaming it,
because a pipeline's exit status in POSIX sh reports the last
command rather than make. The capture file is allocated once and
removed by the EXIT trap that already owns the failure list, and the
run is pinned to LC_ALL=C: all three checks match English wording,
one on make's and two on helm-unittest's, so a localized make would
disarm the first of them silently, which is the failure mode this
guard exists to remove. The locale reaches every package's recipe,
not only make.

Extend hack/helm-unit-tests.bats with eight tests. Three pin the new
refusals; two pin what they must not refuse, a phony target with a
colliding path and a sub-make reporting an unrelated target up to
date; one pins that a chart which does run suites still passes; two
pin behaviour that was already there, namely that a failing package
is still reported by name and a package with no `test` rule is
skipped rather than failed. That last one carries a second package
that does run a suite, because with only the target-less package
present the script takes its early exit above the failure summary
and the assertion holds whatever the script did. The fixtures stub
make output instead of invoking helm, so they need no plugin
installed.

Both tests that assert a package is named in the failure summary
match the summary's "  - " prefix rather than the bare path. The
script announces "Running tests in $dir" before it runs anything, so
the bare path is present in the output whatever the summary does;
matching it asserted nothing, and the summary could stop naming
packages entirely with the suite still green.

The failing-package fixture reports a passing suite count alongside
its non-zero exit, so that the exit status is the only thing that
can decide the outcome. A fixture that fails without printing the
marker is refused by the positive-evidence check first, which leaves
the exit-status branch unpinned. That shape occurs in the tree: a
`test` target that runs helm-unittest and then a second step, a
render-parity script or a vendored-version guard or go tests, emits
the marker and still fails.

The two tests already in that file cover a package reached under
packages/tests, and their fixture printed a marker and nothing else.
That is the shape the positive-evidence check refuses, so the
fixture now also emits the line a real run emits; without it both
would fail for a reason unrelated to what they assert.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:45:22 +03:00
Aleksei Sviridkin b8a5b9ba1d test(e2e): capture worker CPU throttling counters on node-join failure
When a tenant worker misses the node-Ready deadline, the sandbox node
it runs on is routinely at half its capacity with a clean kernel log.
In that state "the host starved the VM" and "the VM was held at its
own CFS ceiling" produce identical evidence: node-level utilisation
reports what the node used, never what the container was allowed. The
CFS counters are the only place the difference is recorded, and
nothing in the tree collected them for the tenant workers, so every
diagnosis of this failure so far had to be inferred from outside.

Read them from the kubelet's cAdvisor endpoint on the node hosting
each worker. Five series carry the whole answer and cAdvisor publishes
all of them per container, so nothing is computed here: the scheduled
and throttled period counts, the time spent throttled, and the quota
and period that form the ceiling. The throttled counters alone say a
container hit some ceiling, not which one, and a VM capped at one core
and a VM capped at eight are the same number without the quota.

Absence of the quota series is itself a reading rather than a gap.
cAdvisor emits it only when the quota is non-zero, so a container
reporting the period and no quota is one running uncapped - which is
one of the two answers this collector exists to distinguish, arriving
without being derived.

Reading the kubelet rather than the container matters beyond
convenience. A reader running inside the container shares the cgroup
it measures, so it slows down in proportion to how much the answer
matters and would need a budget sized for the pathological case; the
kubelet is outside that cgroup. It also drops any dependency on what
the container image ships and on how the host lays cgroups out.

The namespace is not a worker filter and is not used as one: it also
holds the tenant control plane, whose apiserver and etcd carry CPU
limits of their own and can be genuinely throttled. The Pod name
carries the second half of the filter, so a throttled apiserver cannot
answer under a heading that says worker.

The walk is over nodes rather than Pods, because the endpoint is per
node and one read covers every worker it hosts. It is gated on the
diagnostics phase budget, and placed first among the five gated
collectors: the budget declines whatever has not started when it runs
out, its own ceiling is four bounded reads rather than the minutes the
four after it can spend, and it is the only one whose question has no
other answer in the tree. That order is a trade, stated here because
it is a decision rather than a detail - it puts the guest serial
console one place further from the budget, and that capture is the
only one describing a worker which never reached apid.

Both reads take the block's own read-budget knob rather than a literal
equal to it, so lowering that knob lowers these too. Each keeps its own
exit status rather than routing through the shared helper, which always
returns zero by design: this collector has to tell "the kubelet was not
read" from "the kubelet answered and carried no worker", and only the
status separates them. The stream is captured whole and filtered after,
since a pipeline reports its last command's status and would otherwise
hand that decision to grep.

Every way a read can come up short is named rather than left as zero
bytes: a kubelet that was not read, one that answered with nothing for
the namespace, a stream cut short, and exit 124 - which cannot
distinguish this collector timing out from the read exiting 124 on its
own - each get their own line, and each says whether an error log
exists to explain it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:43:11 +03:00
Aleksei Sviridkin e9411d6b8c fix(apps): make the post-delete cleanup hooks report failure
The qdrant and harbor post-delete hooks guarded every delete with
`|| echo "WARNING: ..." >&2`, set no `-e`, and ended on an
unconditional completion message, so the container exited 0 on
every path. Helm read that as a successful hook and the
hook-succeeded delete policy removed the Job together with the pod
that carried the warning, leaving an uninstall that reclaimed
nothing indistinguishable from one that had nothing to reclaim.

Count the failures instead and set the exit code at the end, so a
failed delete does not skip the deletes that follow it. An object
that is already gone stays a success: every delete keeps
--ignore-not-found, which exits 0 in that case. This is the shape
packages/apps/mariadb already uses for the same problem.

Both hooks also consumed the output of `kubectl get` directly --
harbor by iterating it, qdrant by piping it into grep, whose status
the pipeline then reports. A listing that failed therefore arrived
as an empty list and the sweep was skipped rather than reported.
Each listing is now captured before it is used, so a listing that
failed and one that returned nothing are different events.

A hook that exits non-zero keeps its Job and pods, which is the
point: they hold the log. Give the Job one retry rather than the
Kubernetes default of six or none at all. Every delete is
idempotent, so a second attempt re-issues an API call rather than
re-running any logic, and one short backoff is what separates a
transient apiserver error from a real one -- these failures become
visible for the first time here, and a signal that fires on noise
stops being read, including on the day it is right. Six is the far
end of the same mistake: ten minutes of backoff across seven pods
outlasts the hook timeout, and Helm reports that timeout instead of
the hook's own error.

The qdrant Chainsaw suite explained its post-delete pod collector
by quoting the guard string removed here and concluding the pod is
always collected away. That is now backwards: a failed cleanup is
exactly when the pod survives and the collector is the evidence.
Rewrite the note, and the sentence in the harbor suite pointing at
it.

hack/app-cleanup-hooks_test.bats renders both scripts and runs them
against a fake kubectl. There are eight failure paths across the two
hooks -- six in harbor, two in qdrant -- and each has a case that
goes red when that one path alone is reverted to a swallowed
warning. Message assertions are anchored on the hook's own `ERROR: `
prefix, because the fake kubectl names the object in its own
diagnostic too and an unanchored match is satisfied by the fixture.
The helm-unittest suites pin the retry limit and the hook delete
policy: adding hook-failed to that annotation hands the Job back to
Helm to delete on failure and destroys the evidence this change
exists to keep, and until these asserts that reversal left both
suites green.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:43:02 +03:00
Aleksei Sviridkin 91b9f58057 fix(hack): declare the shared package targets phony
package.mk set .PHONY with `=` rather than `:`, which assigns a
variable named .PHONY instead of declaring anything phony, so none
of the targets package.mk defines was phony in any package that
includes it.

Declare exactly the concrete targets package.mk defines: help, show,
diff, apply, delete, suspend, resume, check and clean. The earlier
list named update and image, which package.mk does not define —
naming an undefined target in .PHONY creates it as an empty target,
so `make image` and `make update` would report success without
running anything in the many packages that lack those recipes. It
also omitted check and clean, the targets a same-named file or
directory could actually shadow.

The neighbouring .DEFAULT_GOAL is a variable and correctly keeps its
equals sign. A comment now records which of the two adjacent lines is
which, so the corrected one does not pull the other into its shape by
analogy.

The trigger map in docs/agents/contributing.md called the two files a
byte-for-byte copy, which was exact rather than loose: they were the
same object. They stop being one here, so the entry now records that
they were identical and no longer are, and asks for a diff-and-port,
the wording the map already uses for the other copy that has
diverged. It deliberately does not enumerate which lines differ:
that set moves as the copy catches up, and an entry that names it
would be wrong again by the next port.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:42:58 +03:00
Aleksei Sviridkin 62204d4903 test(cozystack-controller): guard the Secret grant by effective permission
The suite's "does not grant a blanket wildcard verb on secrets" assert
was a notContains over a rule object. helm-unittest compares every field
it is handed by value equality, and `any: true` changes which fields are
compared rather than how, so no spelling of that assert bounds what a
verbs list may hold: `verbs: ["get", "*"]` satisfies all of them.

Measured against the rendered ClusterRole, seven realistic spellings of
a widened Secret grant left that assert green: a wildcard in resources,
a wildcard in apiGroups, a "*" appended to the declared verb list, an
extra field beside the verbs, deletecollection spelled out, and two
where the grant is added as a separate rule. Those last two leave the
whole suite green - appending apiGroups:['*'] resources:['*']
verbs:['*'] keeps the positive pin satisfied while the controller gains
every verb on every Secret cluster-wide.

Replace it with a bats guard over the rendered manifests. It unions the
verbs that every ClusterRole and Role rule confers on core Secrets,
expanding "*" on each of the three axes, and compares that union with
the declared set. Equality rather than an upper bound: the controller is
cluster-scoped, so a widening reaches every tenant namespace, and a
narrowing breaks the two reconcilers that share the grant.

The positive pin stays in tests/rbac_test.yaml. Value equality is the
right primitive for pinning a rule as written and the wrong one for
bounding what the rule may become.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:37:09 +03:00
Aleksei Sviridkin ea78d09dd0 feat(ci): auto-apply triage labels to issues
Issues created via the API or gh bypass the issue templates and arrive
without any triage/* label; 252 of the 396 open issues carry none.
Label issues on arrival (opened/reopened) and run a daily sweep as a
backstop: issues already prioritised, assigned, or marked as epics get
triage/accepted, the rest triage/needs-triage.

A signal has to be on stale.yaml's exempt-issue-labels, because
triage/accepted is itself exempt and a signal that is not would grant a
permanent stale reprieve as a side effect of being labeled. That is
necessary and not sufficient: lifecycle/frozen is exempt and still not
a signal, because a lifecycle/* label says what the stale bot may do
with an issue and nothing about whether anybody reviewed it.

The sweep paces its writes and caps how many it makes per run, so a
pass over a large backlog stays inside GitHub's secondary rate limits
instead of failing partway through. Each run writes a job summary with
the counts, the issue numbers per label, and both remainders it can
leave behind, because a backlog pass logs a line per issue and nobody
reads 252 of those to find out what happened.

The manual dispatch defaults to dry-run, so the one entry point a human
can reach writes nothing until somebody unchecks the box. The schedule
is the one event that writes without being asked.

Deleting a triage label does not undo a decision: an issue left with
none is relabeled by the next sweep, so AGENTS.md's labeling section
says to change a decision by replacing the label instead.

No PR lane can exercise this workflow, so the first real run is a sweep
over every open issue. A contract test pins the executable lines that
bound that: the pull-request skip, the already-triaged skip, the pacing
and the cap, the dry-run default, the pinned action, the token scopes,
that no signal comes from the lifecycle/* namespace, and that the
summary is written with the await that makes it flush.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 16:35:36 +03:00
myasnikovdaniil 516be4abfe fix(cozystack-basics): normalize tenant hostname VAP comparisons with lowerAscii (#3208)
## What this PR does

The Ingress, Gateway, and Route hostname `ValidatingAdmissionPolicy`
objects compare tenant hostnames against the tenant apex with
case-sensitive CEL. The resource-side host is always lowercase — the
apiserver's hostname validation permits nothing else — but both apex
operands can carry uppercase: the `namespace.cozystack.io/host` label
value, which Kubernetes permits to carry uppercase and which the tenant
chart writes verbatim from `tenant.spec.host`, and the templated
`_cluster.root-host`, a free-form platform value that nothing validates
for case.

A mixed-case apex label wrongly denies a tenant its own legitimate
lowercase in-apex hostname. An uppercase `_cluster.root-host` is the
dangerous direction: the Ingress outside-root exclusion compares a
lowercase host against the uppercase literal, never matches, so the
exclusion stops excluding and the policy admits any non-wildcard host —
every non-wildcard platform-apex name a tenant asks for, including a
hostname belonging to another tenant and the root apex itself. Only the
anti-wildcard gate still holds.

This normalizes both operands of every hostname comparison with
`.lowerAscii()` (host, apex label, and the templated root-host literal)
across all three policies, so the match follows DNS case semantics.
There is no behavior change in the default all-lowercase configuration.

The Gateway policy's fail-open on an absent or empty host label is
deliberately left as it was: that is the platform's one documented
fail-open exception in `docs/security/threat-model.md`, so changing the
posture is a security-model decision rather than a side effect of
normalizing case. Its current shape is pinned by a test.

Test coverage:

- helm-unittest pins each comparison as a whole operand pair — equality
and `endsWith`, across the rule-host and tls-host branches — so dropping
the normalization from any single operand is caught. A pattern matching
a lone `.lowerAscii()` would not: the apex appears four times in the
Ingress expression, so stripping one operand still leaves three matches.
- A rendering case with `_cluster.root-host: Example.ORG` pins the
outside-root escape shut; it is red pre-fix.
- e2e probes set a mixed-case apex on an isolated throwaway namespace
and assert both halves of the match: a lowercase in-apex HTTPRoute and
Ingress are still admitted (red pre-fix, green post-fix), while a
foreign apex and a hostname ending with the apex without a dot boundary
are still denied. Vacuous-pass guards re-read the persisted label and
assert the probe apex really is mixed-case.

Normalizing the apex at the write site — the tenant namespace template
and `_cluster.root-host` — is tracked separately in #3328. It would not
remove the need for this change: the host label is settable out-of-band
by a trusted caller, so these policies must not assume the apex arrives
normalized.

Closes #3205.

### Screenshots

N/A (no UI changes).

### Release note

```release-note
fix(cozystack-basics): tenant hostname admission policies now compare against the tenant apex and the platform root apex case-insensitively. This closes a bypass where an uppercase `_cluster.root-host` made the Ingress policy's outside-root exclusion stop matching, so the policy admitted any non-wildcard host — including a hostname under the platform apex belonging to another tenant, and the root apex itself.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Hostname validation now consistently matches case-insensitively across
Gateway, Ingress, HTTPRoute, and TLSRoute.
* Mixed-case tenant and apex hostnames are accepted when they meet the
intended domain boundaries.
* Foreign domains and prefix-adjacent hostnames remain correctly
rejected; validation behaves as documented when the tenant host is
empty.

* **Tests**
* Expanded unit and e2e coverage for case-insensitive hostname rendering
and boundary rejections across the supported resources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-12 15:35:41 +05:00
Andrey KolkovandClaude Opus 4.8 dff56aa998 feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545).

The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:

- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
  status.state, and persists the S3 destination + storage descriptor
  (by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
  from the source backup status, snapshot fallback) so the same artifact
  restores in-place or into a differently-named instance; PITR via
  spec.options.pitr (date/latest).

Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.

Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
  partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
  Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
  CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.

Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
  backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
  drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
  restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
myasnikovdaniil d04231829a docs(hack): fix a wrong claim about grep-error propagation in two contract tests (#3620)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

`hack/promote-gate-contract.bats` and
`hack/release-freeze-contract.bats` both carried the same wrong comment
above their `code_lines` helper: it said a real grep error (exit 2)
fails a pin through the helper's own exit status, distinct from an empty
match (exit 1).

That was never actually the mechanism. `code_lines` is always piped into
a further `grep` in both files, never the last stage of a pipeline, so
ordinary (no `pipefail`) pipe semantics discard whatever it returns on
its own, regardless of runner. `hack/cozytest.sh`, the runner these
tests actually execute under in CI, also appends a `return 0` to every
line that is exactly `}`, `code_lines`' closing brace included, making
that exit status even less trustworthy as a signal.

What catches a real grep error is the output, and only for one class of
pin. The error empties `code_lines`' stream, so a pin that requires
something to be present in that stream fails on it. A pin demanding an
absence stays blind: an exact count of zero is what an emptied stream
produces anyway, and what the pin was written to assert. Both files hold
pins of that shape. Nor is the `[ -n "$block" ]` guard at the top of
each test the backstop: a block here is a bare awk extraction, upstream
of `code_lines`. The new comment says both things and stops there.

Comment-only: `code_lines`'s body is untouched in both files, and all
tests pass under both `bats` and `hack/cozytest.sh` before and after.

Known limitation of the wording: the comment says the top-of-test `[ -n
"$block" ]` guard cannot see a filter failure, which is true, but both
files also carry non-empty guards on values that did pass through
`code_lines` (`promote-gate-contract.bats:72,73,165` and
`release-freeze-contract.bats:166,236,327` via `step_line`), and those
do fail on an emptied stream. The discriminator is whether the guarded
value came through the filter, not where the guard sits in the test.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified comments describing how code-line filtering errors are
detected.
* Documented pipeline status behavior and why output-based assertions
are used instead of relying solely on command status codes.
  * No user-facing behavior or executable logic changed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-11 19:40:01 +05:00
Aleksei Sviridkin cf6bf2e379 fix(ci): stop labeled events from cancelling and re-running backports (#3569)
## What this PR does

A run joins its concurrency group before any job-level `if` is
evaluated, so `prepare`'s guard in `backport.yaml` is too late to help.
By the time it works out that a label event is irrelevant, that run has
already cancelled whatever was in flight for the same PR. Labels set
with the default `GITHUB_TOKEN` start no run, which is why the in-repo
labelers never showed this. Third-party GitHub Apps do start runs, and
they arrive in bursts.

The delivery survived, which is why this went unnoticed for so long. The
old guard admitted a run on the PR's cumulative label set, so the last
label of a burst, the one nothing cancels, requalified and redid the
work. In the bursts on record the kill lands within seconds of the merge
run starting, well before the cherry-pick, so the cost was a restart
rather than a lost backport, recorded as nothing more than a cancelled
run sitting next to a green one.

Two changes to the concurrency key, and both are needed. A label event
that requests no backport now goes into a group of its own, so it can
neither cancel nor displace the run doing the work. A label event that
does request one queues behind that run instead of killing it, because
`backport` and `backport-previous` are separate requests and both have
to finish. Splitting alone would let `backport` cancel the
merge-triggered run. Queuing alone would let an unrelated label evict a
pending request, since the group holds a single pending run and a newer
one replaces it.

`prepare`'s condition is scoped in the same commit. It read the PR's
cumulative label set, so once a merged PR carried `backport`, any later
unrelated label re-entered the job to redo a backport already delivered.
That run used to get cancelled by the next label in the burst. After the
change above it runs to completion, so the scoping has to land here
rather than as a follow-up.

The coupling runs the other way too, and it is the sharper half. That
redundant run is exactly what redelivered the backport the burst had
just killed, so narrowing the guard on its own would convert a noisy
delivery into a missing one. Neither change is safe to land without the
other.

Both jobs also gain a `timeout-minutes` ceiling, which is a consequence
of the queuing rather than housekeeping. A cancelling key disposed of a
stuck run by killing it; queuing makes the next genuine request wait
behind it instead, on the six-hour job default, which turns one wedged
job into a six-hour hole in the release line. That ceiling reaches a
stuck job and nothing else: a run can also wedge at the run level with
every job already finished, and `timeout-minutes` has no run-level
equivalent, so nothing in the workflow reaches that state.
`docs/release.md` now tells an operator to look for a run still in
flight and clear it before re-applying the label, because the retry
queues behind it rather than replacing it. The ceiling bounds execution,
not the wait for a runner, and thirty minutes is generous rather than
calculated. Execution on run 29844315422 was six seconds for `prepare`
and two and sixteen for the matrix legs, against 9m24s, 5m28s and 5m45s
of runner queue for the same three jobs. That queue is a third unbounded
case: GitHub's limits page says "job execution time" for the six-hour
job cap while spelling out that the 35-day run cap "includes execution
duration, and time spent on waiting and approval", and
[orgs/community#50926](https://github.com/orgs/community/discussions/50926)
shows a job sitting six hours on "Waiting for a runner to pick up this
job" under `timeout-minutes: 5`. So the value is picked to leave room
for a cherry-pick far larger than any on record while replacing the
six-hour default, and nothing here is sized against the queue, because
nothing here can be.

This removes a mitigation, and the trade is worth naming on both sides.
Under an unconditional cancel a run wedged at the run level cleared
itself, because the next event killed whatever was sitting in the group.
That is the same behaviour that killed live backports, so the mitigation
is inseparable from the bug being fixed: keeping it means keeping label
events that cancel a backport in flight. What it buys is a change of
failure mode rather than one fewer failure. A backport lost silently,
discovered weeks later as a fix missing from a release line, becomes a
delay the operator can see in the run list, with the recovery written
into `docs/release.md`. There is no lossless alternative here: `queue:
max` removes the eviction but only by dropping the conditional cancel
entirely, and the group key cannot tell a manual retry from a second
genuine request.

The run cited above, 31232645133, is still live and retained on purpose:
it is the only observed instance of the run-level wedge, so clearing it
would remove the evidence for the paragraph that describes it. Its
exposure is one pull request, #3262, which carries no backport label, so
nothing is queued behind it.

The old condition also carried a disjunct that could never decide
anything. A `labeled` payload already lists the new label in
`pull_request.labels`, so `contains` was true whenever the explicit
`github.event.label.name` check was, and the name check only ever agreed
with a `contains` that had already matched. The restructure drops it by
construction rather than by deletion: the name check is now the whole of
what a `labeled` event may match on.

### Why this no longer touches `pull-requests.yaml`

`main` fixed it there by splitting the group key, then refined that so a
`full-e2e` label stays in the main group instead of publishing a second
`E2E Tests` status on one SHA. That is the better mechanism, it neither
queues nor delays, and `gate-concurrency-contract.bats` now pins
`cancel-in-progress: true` in that file, so what this PR originally did
would turn it red. Nothing was left to add there, so the hunk is dropped
and its pin with it. The split-key idea is applied in `backport.yaml`
instead, where it did not exist.

One claim from the earlier description is withdrawn. It said the
leftover risk was a duplicate backport PR, opened when the original had
merged and its branch had been deleted before the queued run pushed.
`main` has since gained a guard that lists closed PRs on the
`backport-<n>-to-<target>` branch and skips the target when one of them
merged. Branch deletion does not defeat it: `GET
/branches/backport-3510-to-release-1.6` is 404 while the same head
filter on `/pulls?state=closed` still returns the merged PR. So a
redundant run only cost runner minutes, and the scoping above removes
the run.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only workflow it cites is `tags.yaml`, for the release-prep
behaviour `cozystack/ccp` documents. Every `hack/` trigger it lists
either names a specific file another repository mirrors or gates on, or
covers moving and renaming under `hack/`. This PR appends tests to a
contract file the map does not name, and moves nothing. No package,
chart, values file, CRD, image reference, namespace, variant or node
prerequisite in the diff.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(ci): stop `labeled` pull request events from cancelling the automatic backport run already in flight. A burst of labels from a third-party app could previously kill the backport the merge had just started, leaving the work to be redone by whichever label arrived last.
```
2026-08-11 16:50:50 +03:00
Aleksei Sviridkin 4ac02f59d5 fix(e2e): stop the sandbox cluster CIDR overlapping the pod range (#3750)
## What this PR does

The sandbox Talos config and the platform install step both used
`10.244.0.0/16`, one as the Kubernetes cluster CIDR and the other as the
kube-ovn `POD_CIDR`, so two allocators drew from one range.

kube-ovn owns pod IPAM and allocates flat from `POD_CIDR`. Cilium is
chained behind it (`generic-veth`, target `kube-ovn`) and allocates no
pod addresses at all, but it still carves its own per-node router IP and
Ingress IP out of `Node.spec.podCIDR`, which kube-controller-manager
slices from the cluster CIDR. Sooner or later kube-ovn hands a pod an
address cilium already holds. When that address is the Ingress IP the
agent refuses the sandbox with `IP ipv4:<addr> is already in use`, and
cilium never relents on it, so the pod proceeds only if something
reschedules it onto a different address; in the run this was filed from,
the same address was refused seven times across six pod UIDs before the
suite ran out of budget. When it is the router IP nothing complains and
only host-to-pod traffic breaks.

This moves the sandbox cluster CIDR to `100.65.0.0/16`, which no other
range in the tree uses, and adds a guard so the two cannot be brought
back together by accident.

Pod addresses do not change. They still come from kube-ovn out of
`10.244.0.0/16`, and cilium does not require a chained endpoint's
address to sit inside its own allocation CIDR, so only cilium's two
per-node infrastructure addresses move.

The quiet variant gets fixed twice over, which is worth spelling out
because only half of it was obvious. Under `ipam.mode: kubernetes`
cilium also installs a route for the node's own pod CIDR via
`cilium_host`. On the current default that route covers `10.244.a.0/24`
on a node whose pods actually live behind `ovn0`, so it shadows real pod
addresses. Moving the cluster CIDR both stops the router IP from ever
being handed to a pod and retargets that route at `100.65.a.0/24`, where
it shadows nothing.

Excluding the addresses on the kube-ovn side instead is not an option:
cilium picks its offsets by random scan over the node's range, so there
is no static set to exclude.

The collision shows up in two shapes, and both have been seen. Loudly,
on the Ingress IP, where the agent names the address in the event, which
is the failure https://github.com/cozystack/cozystack/issues/3736 was
filed for. Quietly, on the router IP, where the pod comes up and the
only symptom is `connection refused` on a kubelet probe, with `ip route
get <podIP>` on the node resolving to `local ... dev lo`; that variant
is documented in https://github.com/cozystack/cozystack/issues/3445 and
https://github.com/cozystack/cozystack/issues/3615. They were being
tracked as unrelated flakes in different suites before the shared cause
was identified.

## What the guard covers, and what it does not

`hack/sandbox-cidr-disjoint.bats` pins the configuration, not the
behaviour. It reads the values out of the tree and fails when they
overlap. It does not reproduce the collision, which needs a running
cluster.

It checks the invariant rather than the literals, so either value may
move as long as they stay disjoint, and it checks the cluster CIDR
against the service and join ranges too, because landing on one of those
instead is the next way to get this wrong. An extraction that comes up
empty, duplicated or misshapen fails the test instead of shrinking the
comparison, and comments and blank lines inside the list are skipped
rather than read as the end of it. It is not a YAML parser, though:
anything it does not read as a bare dotted quad fails the shape
validation rather than being understood, which is the safe direction but
not the same promise.

Those three are the allocator ranges and nothing more. The sandbox node
network, the load-balancer pool and the transit range of a component
that is off by default are not read, so the guard is not a complete map
of what the cluster CIDR has to avoid. An IPv6 entry fails the shape
validation rather than getting an overlap check, so dual-stack means
extending the guard rather than passing through it.

## Residual risk

I did not prove from source that cilium is the only consumer of
`Node.spec.podCIDR` in this topology. What I did check is that
kube-proxy is disabled and that `local-ccm` runs the node-lifecycle
controller only, with no route controller.

This diff selects the full e2e suite, 21 Chainsaw suites with `gateway`
and `mongodb` among them, so a consumer I missed surfaces as a red run
on this PR rather than staying an assumption.

Split of what covers what: the static checks and the unit guard cover
the configuration, and the e2e run covers the behaviour. Nothing short
of a green suite answers whether cilium's router and Ingress addresses
behave once they sit outside the range kube-ovn allocates from.

## A check that is not "the suite went green"

`hack/e2e-cilium-endpoint-leak-healer.sh` already prints a line of its
own for this exact collision, so there is a signal to read that does not
depend on the overall verdict. In the `cozyreport` artifact it lands at
`snapshots/<suite>/host/namespaces/kube-system/v1/pod/cilium-leak-healer-*/healer/current.log`,
and the lines to grep for are `IP is this node's Cilium ingress IP`. In
the run this was filed from there are 42 of them, all naming the same
address.

After this change that line cannot be produced by an address inside the
kube-ovn range, because there is no longer an address that is in both.
So the expectation is zero from the first run, not zero on average: one
occurrence contradicts the change rather than being noise, and is worth
stopping on. A green suite can be green for many reasons; this line
disappearing has only one.

## Scope

This fixes the sandbox and nothing else. For a real installation the
cluster CIDR lives in the operator's own machine configuration, outside
this repository, so the same collision is still reachable there.
https://github.com/cozystack/cozystack/issues/3740 tracks the product
side, including the finding that the install documentation does not
merely allow the overlap but instructs operators to create it, by
requiring `podCIDR` to match the cluster's pod CIDR.

There is a tradeoff in this worth naming rather than leaving to be
discovered. Talos defaults `podSubnets` to `10.244.0.0/16`
(`DefaultIPv4PodNet`, `pkg/machinery/constants/constants.go:888` at
v1.13.6, the version this tree ships), and the platform defaults
`networking.podCIDR` to the same value, so an operator who takes both
defaults gets the overlap. Until now the sandbox shared that topology
and was the one place the bug surfaced where anyone would see it. After
this change it no longer reproduces the operator-facing form, and
nothing else does either. That is the right call for CI, which should
not be wedged by a defect it did not introduce, but it does mean the
production exposure loses its only accidental detector.

Closes https://github.com/cozystack/cozystack/issues/3736

To be explicit about what closing that issue does and does not mean:
this removes the reproduction in the sandbox and leaves the product
configuration untouched. The collision is still reachable for users, and
https://github.com/cozystack/cozystack/issues/3740 is where that lives.
The issue text describes the root mechanism, which is wider than its
title, so the closure covers the e2e failure it was filed for and not
the platform default.

### Screenshots

Not a UI change.

### Downstream repositories

No box is ticked deliberately, and no follow-up PR is opened, so a human
decides.

Walking the trigger map against this diff: both
`hack/e2e-prepare-cluster.bats` entries in it are scoped to node
prerequisites, the kernel modules a node must load, the containerd
settings, the LVM `global_filter` and the cluster domain. This diff
changes none of those, it changes a network range, so no entry fires
literally.

The underlying constraint does reach several of them all the same.
`cozystack/talm` sets `podSubnets: [10.244.0.0/16]` in
`charts/cozystack/values.yaml`, the same range the platform uses for
`networking.podCIDR`, so the documented Talos bootstrap produces the
overlap this PR removes from the sandbox. `cozystack/ansible-cozystack`
reaches it from the other side: `roles/cozystack/defaults/main.yml` sets
`cozystack_pod_cidr: "10.42.0.0/16"` and renders it into
`networking.podCIDR`, while its inventories set `cluster-cidr:
10.42.0.0/16`, so those two are identical as well. And
`cozystack/website` prescribes the equality outright,
`install/kubernetes/generic.md` stating that `podCIDR` and `serviceCIDR`
must match the Kubernetes cluster configuration.

No issue is filed there yet on purpose: which side moves is not decided,
and asking talm to change `podSubnets` would be the wrong request if the
platform ends up moving instead. That decision belongs to
https://github.com/cozystack/cozystack/issues/3740.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Updated cluster networking ranges to avoid conflicts with Kubernetes,
kube-ovn, kilo, and Cilium networks.
- Added validation to detect missing, invalid, or overlapping cluster
and platform CIDR ranges.
- Preserved safeguards for environments where network ranges overlap,
with clearer diagnostic logging.

- **Documentation**
- Documented network separation requirements and clarified expected
sandbox networking behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-11 16:50:35 +03:00
Aleksei Sviridkin e0d6c6be92 fix(ci): stop labeled events from cancelling and re-running backports
A run joins its concurrency group before any job-level `if` is
evaluated, so `prepare`'s guard cannot stop a label event from
disturbing the run already backporting a PR. Labels set with the
default GITHUB_TOKEN start no run, but labels set by third-party
GitHub Apps do, and they arrive in bursts, so each one cancelled the
backport the merge had just started.

The delivery survived anyway, which is why this went unnoticed: the
old guard admitted a run on the PR's cumulative labels, so the last
label of a burst requalified and redid the work. In the bursts on
record the kill lands within seconds, well before the cherry-pick,
so the cost was a restart rather than a lost backport.

Move a label event that requests no backport into a group of its own,
and let one that does request a backport queue rather than cancel.
Both halves are needed. `backport` and `backport-previous` are
separate requests that both have to complete, so they share a group
and must not cancel each other, while every other label
does no work here and only has to stay out of the way.

Scope `prepare` to the labels each trigger is entitled to read at the
same time. Reading the cumulative label set on a `labeled` event let
any later unrelated label on a merged PR still carrying `backport`
re-enter the job to redo a backport already delivered. That redo is
also what redelivered a killed backport, so narrowing the guard alone
would turn a noisy delivery into a missing one: neither change is
safe to land without the other.

Bound both jobs with `timeout-minutes` for the same reason, since a
queued request now waits on the run ahead of it rather than replacing
it. The ceiling bounds execution and not the wait for a runner, which
no job-level key reaches, so 30 is generous rather than calculated:
every execution measured on this workflow has been seconds, against
minutes of queue that nothing here can bound.

Document the recovery path the narrowing removes: with no accidental
re-run left, re-applying whichever backport label the PR already
carries is how an operator retries a merge-triggered run that died.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-11 03:14:09 +03:00
Aleksei Sviridkin 9ace63b441 fix(e2e): stop the sandbox cluster CIDR overlapping the pod range
The sandbox Talos config and the platform install step both used
10.244.0.0/16, one as the Kubernetes cluster CIDR and the other as the
kube-ovn POD_CIDR, so two allocators drew from one range.

kube-ovn owns pod IPAM and allocates flat from POD_CIDR. Cilium is
chained behind it and allocates no pod addresses, but it still carves
its own per-node router IP and Ingress IP out of Node.spec.podCIDR,
which kube-controller-manager slices from the cluster CIDR. Sooner or
later kube-ovn hands a pod an address cilium already holds. When that
address is the Ingress IP the agent refuses the sandbox with "IP
ipv4:<addr> is already in use", and cilium never relents on it, so the
pod only proceeds if something reschedules it onto a different address.
In the failure this comes from, the same address was refused seven
times across six pod UIDs before the suite ran out of budget. When it
is the router IP nothing complains at all and only host-to-pod traffic
breaks.

Move the cluster CIDR to 100.65.0.0/16, which no other range in the
tree uses. That deliberately stops the sandbox matching the layout the
install documentation prescribes, so it is not a value to put back.
Pod addresses are unaffected: they still come from kube-ovn out of
10.244.0.0/16, and cilium does not require a chained endpoint's address
to sit inside its own allocation CIDR.

Excluding the addresses on the kube-ovn side would not work, because
cilium picks its offsets by random scan over the node's range.

The new guard checks the invariant rather than the literals, so either
value may move as long as they stay disjoint. It covers the service
and join ranges too, since landing on one of those instead is the next
way to get this wrong.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-11 02:31:27 +03:00
Andrey Kolkov d0990299ed feat(clickhouse): let the backup sidecar trust a private S3 CA, unblocking the backup e2e round-trip (#3385)
## What this PR does

Gives the ClickHouse `clickhouse-backup` sidecar a way to trust an S3
endpoint whose certificate is signed by a private CA, and uses that to
make the ClickHouse backup/restore e2e round-trip actually run in CI.
This actualizes #2600, whose feature code and example scripts have since
landed on `main` independently, and whose remaining contribution — the
e2e test — was still in the retired BATS format. #2600 can be closed
once this merges.

### 1. `backup.endpointCA` on the ClickHouse chart

The sidecar had **no TLS/CA surface at all**: bucket coordinates and
credentials arrive as `S3_*` env, and nothing let it verify an endpoint
signed by a private CA. Cozystack's own in-cluster SeaweedFS is exactly
that case — it serves `:8333` behind the self-signed "SeaweedFS CA" — so
a ClickHouse release could only ever back up to a publicly-trusted
endpoint. MariaDB (`storage.s3.tls.caSecretKeyRef`) and Postgres (barman
`endpointCA`) have had a CA field all along; this closes the gap for
ClickHouse.

`backup.endpointCA` takes `{name, key}`. When set, the chart mounts that
Secret into the sidecar and points `SSL_CERT_DIR` at it. Empty — the
default — renders no env, no mount and no volume, so existing releases
are byte-for-byte unchanged.

**Why `SSL_CERT_DIR` and not `AWS_CA_BUNDLE`.** Both are honoured by
`altinity/clickhouse-backup:2.7.4`; I measured them against a
self-signed endpoint and public AWS S3 in turn. Go reads `SSL_CERT_DIR`
*in addition to* the system bundle, so both endpoints verify.
`AWS_CA_BUNDLE` *replaces* the SDK's trust pool: the private endpoint
verifies, and public S3 then fails `x509: certificate signed by unknown
authority` — it would silently break a release that also talks to a
public endpoint. `S3_DISABLE_CERT_VERIFICATION` was rejected outright,
since it drops verification rather than trusting one more CA.

### 2. The e2e suite, and what it was really testing

Two Tests added to the existing `hack/e2e-chainsaw/clickhouse/` suite
(in name order, mirroring the `etcd`/`mariadb`/`postgres` shape — see
the layout rationale below for why they live here rather than in a new
directory):

- **`clickhouse-1-backup-contracts`** (CI, `tenant-test`, ~98s): a
backup-enabled ClickHouse asserted against the *running* cluster — the
`clickhouse-backup` sidecar on the `ch-backup-api` `:7171` port, and the
`<release>-backup-api-auth` Secret whose keys the sidecar's
`API_USERNAME`/`API_PASSWORD` actually resolve `valueFrom`. Each is a
silent break if a chart bump renames it. Needs no object store.
- **`clickhouse-2-backup-roundtrip`** (CI, `tenant-root`): `Bucket` →
`Altinity` strategy + `BackupClass` → source ClickHouse with a sentinel
row → `BackupJob` → in-place `RestoreJob` → to-copy `RestoreJob` into a
second instance, driving `examples/backups/clickhouse/run-all.sh` so
test and docs cannot drift.

The round-trip was originally gated behind `CLICKHOUSE_E2E_S3_ROUNDTRIP`
and **skipped in CI** — its `PASS (1.01s)` meant the gate fired, not
that a backup happened. `backup.endpointCA` removes the reason for the
gate, so the suite now adopts the same three preconditions
mariadb/postgres rely on: run in `tenant-root` (an isolated tenant's
Cilium egress allowlist blocks its Pods from reaching `tenant-root`'s
SeaweedFS), target the in-cluster endpoint instead of the `Bucket`'s
external ingress URL (an unroutable placeholder in CI), and trust its
certificate via the copied CA. It also pre-cleans with `cleanup.sh`,
because `tenant-root` is shared and a leftover `Succeeded` `BackupJob`
would satisfy the harness's wait without a backup ever running.

Both backup Tests live in the existing `hack/e2e-chainsaw/clickhouse/`
suite rather than a new suite directory, so no selector mapping is
added: `hack/select-e2e.sh` already derives `clickhouse` both from
`packages/apps/clickhouse/**` (the `*-application` rule) and from
`examples/backups/clickhouse/**` (the harness path), and both paths
therefore select the suite that actually runs the harness. That
alignment is the reason for the layout — a separate `clickhouse-backup/`
directory would have made an edit to the harness select a suite that
cannot exercise it. `hack/select-e2e.sh` and `hack/select-install.sh`
are otherwise untouched (the only change is a comment recording that
invariant), and the existing `backup example harness edit selects its
app suite` case in `hack/select-e2e_test.bats` covers the mapping.

### Verification

- `helm unittest` — 17 pass, including 5 new `endpointCA` cases: the
unset default renders nothing, the env/mount/volume wiring, a custom key
still projected as `ca.crt`, composition with `useSystemBucket`, and
`backup.enabled=false` renders nothing.
- The CA-trust behaviour was measured against the real image, not
assumed: without a CA knob the sidecar reports `x509: certificate signed
by unknown authority`; with `SSL_CERT_DIR` the handshake succeeds.
- `hack/select-e2e_test.bats` (16) and `hack/select-install_test.bats`
(17) pass; `select-install.sh --validate` reports graph OK.
- `make generate` is idempotent, so the pre-commit job's `git diff
--exit-code` stays clean.
- The contracts Test passed in CI on earlier runs of this branch. The
round-trip only became CI-live with this work, so it still needs a green
E2E run before merge — the branch has since been rebased onto current
`main` to get one.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff, file by file. The load-bearing
change is the new `backup.endpointCA` field in
`packages/apps/clickhouse/values.yaml` / `values.schema.json`;
everything else is `hack/e2e-chainsaw/`, `examples/backups/`, one docs
paragraph, and regenerated artifacts, none of which appear in the map.

- [ ] No downstream repository is affected by this change
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/18 —
"add, remove or rename a field in an app's `values.schema.json`" reaches
the provider's hand-maintained schema, model and expand/flatten pair, so
the new nested object is not surfaced until it is added there. Opened as
an issue rather than a PR because the natural moment is alongside that
repo's next `api/apps/v1alpha1` pin bump, which is a maintainer call.

Not ticked, with reasons rather than silence: **cozystack/website** —
the trigger is *adding, renaming or removing a package*, and this adds a
field to an existing one. `clickhouse` is already in the website
`Makefile`'s app list and its reference page is generated from this
package's `README.md`, so the regenerated `backup.endpointCA` rows are
carried by the existing release-tag automation with no manual PR.
**cozystack/external-apps-example** — vendors `hack/package.mk` and
`update-crd.sh`, neither of which this PR touches.
**cozystack/ansible-cozystack**, **ccp**, **talm**, **cozyhr**,
**cozy-proxy**, **cozystack-telemetry-server**, **examples** — no
installer values, `hack/` layout, namespace, variant, node-prerequisite,
annotation, label or metric in their trigger rows is touched here.

### Release note

```release-note
feat(clickhouse): add `backup.endpointCA` so the clickhouse-backup sidecar can verify an S3 endpoint whose certificate is signed by a private CA, such as Cozystack's in-cluster SeaweedFS. Leave it unset for publicly-trusted endpoints; existing releases are unaffected.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional private S3 endpoint CA support for ClickHouse
backup/restore sidecars via `backup.endpointCA`, including configurable
Secret name and CA bundle key (default `ca.crt`).
* Sidecars mount the CA and extend trust while leaving public endpoints
unchanged.

* **Documentation**
* Updated ClickHouse chart docs, operational backup guidance, and
backup/restore examples to cover CA setup, overrides, and cleanup.

* **Tests**
* Added helm-unittest and end-to-end coverage validating CA
rendering/mounting behavior and ClickHouse backup/restore workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-10 21:17:47 +04:00
Aleksei Sviridkin 24157ec876 chore(codeowners): keep architects as owners in every non-governance rule (#3738)
## What this PR does

CODEOWNERS applies only the **last** matching rule, and matching is not
additive: a scoped rule fully replaces the `*` catch-all instead of
extending it. As a result `@kvaps` and `@lllamnyp` were not code owners
for most scoped paths (21 and 36 of the 76 owner-bearing rules,
respectively), and several paths (`/hack/`, `/.github/`, `/Makefile`,
`/.pre-commit-config.yaml`, `/docs/`) were owned by exactly two people —
with "Require review from Code Owners" enforced on `main`, a PR authored
by one of them could only be approved by the other.

This PR:

- appends `@kvaps` and `@lllamnyp` to every rule that lists owners,
keeping existing owners and their order; the governance block
deliberately stays with `@tym83 @kvaps`, and the ownerless rules for
generated files stay ownerless;
- spells out the non-additive last-match semantics in the file header;
- adds `hack/codeowners-invariant.bats` (auto-discovered by `make
bats-unit-tests`), so a future scoped rule that omits the catch-all
owners fails unit tests instead of silently carving its path out from
under them.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(codeowners): @kvaps and @lllamnyp are now code owners for every path except governance files, removing single-approver review bottlenecks; a unit test guards the invariant
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated code ownership rules across dashboards, packages, Kubernetes
components, backups, GPU and SeaweedFS components, Keycloak,
documentation, and CI files.
* Expanded area-specific review ownership while preserving existing
governance and API review assignments.

* **Tests**
* Added automated validation to ensure ownership rules include the
required default reviewers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-10 15:49:22 +03:00
Aleksei Sviridkin f0bde76b9f docs(hack): fix a wrong claim about grep-error propagation in two contract tests
Both hack/promote-gate-contract.bats and hack/release-freeze-contract.bats
carried the same code_lines helper with the same wrong comment: it claimed
a real grep error fails the pin through the helper's own exit status.
code_lines is never the last stage of a pipeline in either file, so
ordinary (non-pipefail) pipe semantics discard whatever it returns on its
own, and hack/cozytest.sh's translator appends a return 0 to every line
that is exactly }, code_lines' closing brace included, making that exit
status even less trustworthy as a signal.

What catches a real grep error is the output, and only for one class of
pin. The error empties code_lines' stream, so a pin that requires
something to be present in that stream fails on it. A pin demanding an
absence stays blind: an exact count of zero is what an emptied stream
produces anyway, and what the pin was written to assert. Both files hold
pins of that shape.

The per-test [ -n "$block" ] guard is not the backstop either. In these
two files a block is a bare awk extraction, upstream of code_lines, so
that guard cannot see the filter fail; the comment now says so, because
reading it as the backstop gets the direction exactly wrong.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-10 12:08:02 +03:00
Aleksei Sviridkin ccd8a289ef fix(e2e): budget each linstor wait separately in post-install-prep (#3715)
## What this PR does

Both waits in `hack/e2e-post-install-prep.sh` ran off one 15-minute
deadline anchored at the script's start, so the head of the chain ate
the window and the tail got whatever was left. Two runs on 2026-08-09
failed that way. The linstor HelmRelease went Ready 452s in, and the
Deployment needed a further ~520s because its DB migration lost the
apiserver and retried four times. Neither figure passes 900s alone.
Their sum does, and the install reported the Deployment as broken.

Each wait now gets its own budget, started when the previous link
finished. Same shape as the node wait and the MetalLB CRD wait further
down the same file. What it buys is that every link gets its whole
allowance regardless of what the links ahead of it spent.

The cost is on both sides and I measured it. Worst case goes from one
15m window to 900 + 900 + 300 + 300 = 40m, against the 180m the e2e job
carries. A Deployment that never converges is now reported at 1355s
instead of 900s, from the 452s above plus the loop's 5s poll. A
HelmRelease that never goes Ready is still reported at 900s, which I
checked by running the new suite against the pre-change script.

The timeout now dumps the pods and recent events of `cozy-linstor`. The
same "Deployment not Available" has come from a crash-looping migration
init container and from a pod waiting on a client TLS Secret
cert-manager had not issued yet, and telling those apart used to need
the run's bundle. Events come from `kubectl events`, which sorts by
last-seen and carries no `--sort-by` to fail on a key some item lacks.
Both reads keep their stderr and are bounded, with the client budget
smaller than the outer one so kubectl gets to say why before the kill
lands.

`hack/post-install-prep.bats` runs the script end to end against a stub
PATH with a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact. Four of its six tests fail
against the pre-change script. The other two guard the bound that stays.

### Known gaps in the new tests

Dropping the `2>&1` from either diagnostic read leaves all six tests
green, because the harness catches the script's own stderr into the same
file the assertion reads. The property holds in production anyway, since
the caller merges both streams into one log, but nothing catches the
removal. The test for "one failing read does not suppress the other"
needs the pods read to run first; swap the two reads and it passes
vacuously. Its comment says so at the point where someone would make
that change.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walked the map against the diff. Two files, both under `hack/`. The
`ccp` trigger is about moving or renaming things under `hack/` or
changing what a make target does, and its anchors `hack/package.mk` and
`hack/common-envs.mk` are untouched. `talm` and `ansible-cozystack` key
off node prerequisites in `hack/e2e-prepare-cluster.bats`, a different
file.

### Related

Same family, found while working here and filed instead of folded in.
#3707 and #3708 for unbounded reads elsewhere in this script, #3712 for
the `--sort-by=.lastTimestamp` form in fourteen other places, #3713 for
the wait loop dropping kubectl's stderr for its whole budget, #3714 for
the MetalLB CRD wait failing without a message.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved LINSTOR readiness checks by giving each prerequisite its own
15-minute wait period.
* Added clearer timeout diagnostics, including relevant pod and event
information.
* Ensured subsequent readiness checks are skipped when an earlier
HelmRelease check fails.

* **Tests**
* Added end-to-end coverage for timeout handling, sequential failures,
diagnostic errors, and readiness behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-10 11:44:15 +03:00
Andrei Kvapil 361c729417 test(codeowners): guard the coverage invariant with a unit check
CODEOWNERS is last-match-wins, so a new scoped rule that omits the
default owners silently strips their ownership for that path. Add a
hack/*.bats unit test (auto-discovered by make bats-unit-tests) that
requires every rule listing owners to repeat the owners of the *
catch-all, exempting ownerless rules and the governance roster.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-10 10:30:30 +02:00
Aleksei Sviridkin ba15b3817a fix(cozystack-basics): normalize tenant hostname VAP comparisons with lowerAscii
The Ingress, Gateway, and Route hostname ValidatingAdmissionPolicies
compare tenant hostnames against the tenant apex with case-sensitive CEL.
The resource-side host is already lowercase — the apiserver's hostname
validation permits nothing else — but the apex operand
is the namespace.cozystack.io/host label value, which Kubernetes permits to
carry uppercase, and the Ingress policy additionally compares against the
templated root host, a free-form platform value that nothing validates for
case.

A mixed-case apex label wrongly denies a tenant its own legitimate
lowercase in-apex hostname. An uppercase root host is the dangerous
direction: the Ingress outside-root exclusion compares a lowercase host
against the uppercase literal, never matches, so the exclusion stops
excluding and the policy degenerates into admitting any non-wildcard host
— every non-wildcard platform-apex name a tenant asks for, including the
root apex itself. Only the anti-wildcard gate still holds.

Wrap both operands of every comparison in .lowerAscii() so the match
follows DNS case semantics — no behavior change for the all-lowercase
default. Each policy reads a label it does not own: the namespace host
label is settable out-of-band by a trusted caller, so none of them may
assume the apex arrives normalized.

The Gateway policy admits any listener hostname when the host label is
absent or empty, unlike its two siblings. That is the platform's one
documented fail-open exception, recorded in the threat model together with
its compensating control, so changing the posture is a security-model
decision rather than a side effect of normalizing case. This leaves the
behavior untouched and pins its current shape in the tests.

Covered by helm-unittest assertions pinning each comparison as a whole
operand pair, so dropping the normalization from any single operand is
caught; a rendering case that pins the uppercase-root-host escape shut; and
e2e probes that set a mixed-case apex on an isolated namespace and assert
both halves of the match — a lowercase in-apex HTTPRoute and Ingress are
still admitted, while a foreign apex and a hostname that ends with the apex
without a dot boundary are still denied.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-10 01:15:55 +03:00
Aleksei Sviridkin 86674c7084 fix(e2e): budget each linstor wait separately in post-install-prep
The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.

Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.

Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.

It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.

A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.

The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.

Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.

The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.

The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-10 00:10:53 +03:00
Aleksei Sviridkin 587a7b9fbd fix(ci): decide E2E coverage per changed path
The full-suite escalation for a changed package that reaches no suite was
read off the merged selection, so it depended on the whole diff rather
than on the path that needed it. Any other changed path contributing a
suite name made the selection non-empty and the escalation vanished with
nothing in the output to record it: edit a platform component, adjust the
Chainsaw test beside it, and the component ran that single suite instead
of everything. Over one values.yaml per package under
packages/{apps,system,extra}, 82 of the 159 escalate on their own and
stop escalating once an unrelated suite edit joins the diff.

Coverage is now decided per changed path. Each path resolves through its
owning sources to the suites that exist, and a path reaching none of them
forces the full run whatever the rest of the diff selected. The unit is
the path rather than the source: system/postgres-operator belongs to two
PackageSources where one reaches no suite, so deciding per source would
escalate on that half and run everything for every change to it.

The escalation names its cause on stderr, as every other escalation in
this script already does. Most packages reach no suite of their own, so
this is the branch a full run usually takes; without a line saying which
sources bought it, "why did this run everything" has no answer in the log.

The walk moves into resolve_suites() to be called once per path. POSIX sh
has no `local`, so the function writes into the caller's scope and is
safe only inside the $( ) its single call site wraps it in; a test pins
that the call count stays one.

Two entries leave src_to_suites. `postgres-application` restated what
stripping the suffix produces and `external-dns` restated what carrying
the source name through produces, so both made the exception list read as
longer than the set of real exceptions.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 22:24:37 +03:00
Aleksei Sviridkin 0980e755eb fix(ci): map E2E suites from sources not named after an app
src_to_suites() covered PackageSource names ending in -application plus
external-dns, and the filter after the graph walk dropped every other
source it reached. cozystack.kuberture and cozystack.securitygroup-
controller are neither, so both were walked to and then discarded: a CNI
change ran neither of the two controller suites most likely to regress
from it, and a change to either package reached no suite at all and
escalated to the full run instead of selecting the one that covers it.

Every reached source now maps through the table. That table is the
inverse of select-install.sh's suite_to_source(), and a test walks every
suite through both, so the next source name fitting neither convention
fails there rather than quietly making its own suite unreachable.

The cilium test measured the gap rather than hid it, asserting the
19 suites the selector could reach and naming the two it could not.
The subtraction is removed rather than inverted: kept as an exclusion
list it would stay green if the walk stopped reaching them for an
unrelated reason. What replaces it is a second assertion against a tree
holding one suite the graph reaches and one no source names, which an
escalation cannot satisfy.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 21:28:00 +03:00
Aleksei Sviridkin 677d80a479 feat(kubernetes): route tenant worker image pulls through a registry mirror (#3575)
## What this PR does

Refs #3513. It addresses one variant of that flake and does not close
it; see "Scope against #3513" below.

### Root cause of the variant this addresses

The chainsaw `kubernetes-latest` / `kubernetes-previous` suites fail
with `node-join failed: fewer than 2 tenant nodes Ready within 18m` and
zero registered tenant nodes. The in-guest Talos capture on #3548 shows
where the time goes on one class of those runs: the worker loops trying
to pull `ghcr.io/siderolabs/kubelet` from public ghcr.io and fails with
`Head "https://ghcr.io/v2/siderolabs/kubelet/manifests/...": net/http:
TLS handshake timeout host=ghcr.io`. The Talos `kubelet` service never
starts (`kubelet was not registered`), so the node never registers,
cilium-operator stays `Pending` for lack of a node to schedule on, and
the budget expires.

On those runs the tenant apiserver is healthy and the worker VMs boot
with kube-ovn IPs, so the failing link is the worker's public image-pull
egress, and the worker machine config has no
`machine.registries.mirrors`, so there is no way to route around it.
Same class as the Talos OS image, which #3231 solved with the
`talos.imageFactoryURL` cache.

The thread also carries a later analysis putting worker-VM CPU
starvation from nested-virt oversubscription at the root, with
serial-console RCU stalls showing the guest never reaching Talos
userland, and reading the TLS-handshake variant as a symptom of the same
contention. Nothing here contradicts that. Removing a live dependency on
public egress from worker bring-up is worth doing on its own terms, and
if the contention account is the dominant one, this narrows what is left
to explain rather than competing with it.

### The change

**Chart (`packages/apps/kubernetes`,
`packages/apps/kubernetes-nodes`)**, a `talos.registryMirrors`
passthrough, a map of upstream registry host to `{ endpoints: [ ... ]
}`, rendered into `machine.registries.mirrors` of the worker
`TalosConfigTemplate` that the MachineDeployment clones from, so it
applies at first boot rather than after it. Empty by default: with the
default values the rendered chart is byte-identical to main, both
content-hash object names included, so no existing worker rolls. This is
the durable half; an air-gapped, rate-limited or flaky-egress
environment can point worker image pulls at its own mirror, which is the
`machine.registries.mirrors` knob the Talos rollover left unfilled.
Talos still falls back to the upstream registry unless a host also sets
`skipFallback`, so a mirror on its own is not air-gap enforcement, and
the field doc says so.

The value lands inside the reconcile Job's unquoted heredoc, so it is
escaped for backslash, dollar and backtick and renders as a literal.
`talos.installerRepository`, `talos.schematicID` and the Talos version
render into that same heredoc one block above it and now get the same
treatment, so the file is consistent about it rather than escaping only
the newest field. Escaping the rendered value covers every byte by
construction, which enumerating known-bad shapes does not. The chain is
a no-op for values carrying none of those characters, so the default
render, content-hash Job name included, is unchanged and no existing
worker is replaced.

**e2e harness**, an in-sandbox `registry:2` pull-through cache for
ghcr.io (`hack/e2e-ghcr-mirror.yaml`), applied at install time from its
own step, with `hack/e2e-chainsaw/_lib/ghcr-mirror.sh` setting
`spec.talos.registryMirrors` on the tenant CR when the mirror is up and
its egress allow is in place, and emitting nothing otherwise so workers
pull directly. A `CiliumClusterwideNetworkPolicy` opens one
tightly-scoped hole for worker VM (`kubevirt.io: virt-launcher`) Pods in
`tenant-test` to reach the mirror, mirroring the existing
`talos-image-cache` policy.

Two properties of the helper worth calling out, because both are easy to
get wrong and both are now pinned by tests. A failed API call is never
cached: the decision is reused by every later suite in the shared
sandbox, so caching one blip would disable the mirror for the whole run.
Readiness is decided once and acted on once. `rollout status` is the
cheap way to learn it, but it also exits non-zero on a broken watch, so
when it fails the Deployment's own `Available` condition is read rather
than the watch's exit code taken as the answer. Exactly one failure is
cached, a definite `Available=False`, and that is a trade rather than an
invariant: it avoids re-paying the wait in every later suite, and costs
a mirror that recovers mid-run.

On a node-join failure the suite dumps the mirror's state alongside the
Talos image cache it already dumped. Workers being pointed at the mirror
and then joining does not establish that the mirror served anything,
because Talos falls back to public ghcr.io on its own; the registry's
access log is what separates the two, so the dump counts kubelet-image
requests across the whole log rather than tailing it. The readiness
probe writes an access line every five seconds, so by the time a run has
spent its 18m budget a fixed tail window is hundreds of lines short of
the request worth finding.

### Testing

- `helm unittest packages/apps/kubernetes`, 205 tests, 22 suites; `helm
unittest packages/apps/kubernetes-nodes`, 14 tests, 4 suites. New cases:
the default omits `machine.registries.mirrors`, and a set value renders
it with the endpoint.
- `hack/ghcr-mirror_test.bats`, 23 tests. Beyond the manifest's
two-phase split, egress identities, pull-through config and the pure
YAML builder, this covers every outcome of
`resolve_ghcr_mirror_endpoint` against a stubbed `kubectl`: an API
NotFound caches, a transient query failure does not, a shell reporting a
missing binary is not mistaken for either, a failed egress allow does
not cache and its reason reaches the log, a rollout watch that failed
for a reason the API does not confirm does not cache while a definite
`Available=False` does, and a committed endpoint is cached and
short-circuits the next call. The stub honours `--tail`, so the
diagnostic's tests tell a filtered read from a truncated one instead of
trusting the flag, and it reports errors on stderr as the real binary
does. The suite also pins that the install step excludes the Cilium
policy from its pre-Cilium apply, not only what that exclusion produces.
- `hack/talos-reconcile-heredoc_test.bats`, 5 tests, which render both
charts and run the heredoc through a real shell with hostile values
rather than regexing the rendered string. A `matchRegex` cannot catch a
heredoc the shell refuses to emit; this runs it.
- `hack/run-kubernetes-talos-spec_test.bats`, 5 tests over
`talos_spec_block`, the point where the two optional fragments are
merged. The assertions splice the result under `spec:` and read it back
with yq, so a block indented into the wrong parent fails instead of
passing.
- `packages/apps/kubernetes-nodes/tests/render-parity.sh`
byte-identical; per-chart `make generate` in both charts with no drift;
both `cozyrds` `openAPISchema` blocks equal to their
`values.schema.json`.
- The pinned `mirror.gcr.io/library/registry:2.8.3@sha256:a3d8aaa6...`
is the digest Docker Hub serves for that tag, and `mirror.gcr.io` serves
the same OCI index with a linux/amd64 manifest. Run under exactly the
security context the Deployment sets, non-root 65532, read-only root
filesystem, all capabilities dropped, writable state only on the mounted
volumes, it starts, answers `/v2/` with 200, and serves
`ghcr.io/siderolabs/kubelet` manifests and blobs as an anonymous
pull-through.
- `shellcheck` and `git diff --check` clean on the new and edited shell.

### What a CI run still has to answer

Two runtime facts the harness half rests on, both recorded in the
manifest:

1. Talos honours a plain-http mirror endpoint via
`machine.registries.mirrors[ghcr.io].endpoints=[http://...]`. If a
scheme or TLS knob is additionally needed, `machine.registries.config`
has to be added too.
2. The containerd pull from inside the worker VM leaves with the
virt-launcher Pod's own Cilium identity, which is what the policy
selects. The label half is settled: `run-kubernetes.sh` already reads
tenant worker Pods by `kubevirt.io=virt-launcher`, but whether the
in-guest pull is subject to that Pod's egress policy at all is not.

The second one bounds how strong a claim the harness can make. The gate
is a `rollout status` plus an accepted egress-allow object, and neither
observes the tenant-side datapath, so the fallback paths cannot make CI
worse while the committed path is only as safe as that assumption. The
in-repo comments say this rather than claiming the mirror can only ever
help.

### Named improvements left out

A tenant-scoped reachability probe, like the byte-level 206 check
`talos-image-cache.sh` runs from a Pod carrying the consumer's own
label, is what turns assumption 2 from an assertion into a measurement,
and would also let the suite assert the mirror served the kubelet pull
instead of the worker quietly falling back. Left out because a probe
that does not exercise the tenant network path would pass while real
workers stayed blocked, and the one that does is a change of its own
size. The same gap has a cheaper edge: both the install step and the
resolver key on the Deployment alone, so an apply that made the
Deployment but not the Service still reads as success. That one costs
little in practice, since the guest gets a fast NXDOMAIN instead of a
dropped SYN, but the gate does check less than it commits to.

`registryMirrors` could be a typed `map[string]struct{ endpoints
[]string }`, the way `nodeGroups` already is in this chart, so malformed
input is rejected at admission instead of failing late inside Talos.

The helm-unittest escape cases assert the `$` path; the backtick path is
covered by the execution-level heredoc bats and the pure-backslash path
by the escape chain's construction, so the security contract is fully
exercised across the two layers. A backtick assertion at the
helm-unittest layer would express it at the fast layer too.

The escape chain is spelled out at five interpolation sites across the
two reconcile-Job templates. A named helper (an `include` template)
would make the escape a single reusable primitive and shrink the surface
where the heredoc invariant can be forgotten; the current per-site form
is correct and covered, so this is a readability refactor rather than a
fix.

Only `machine.registries.mirrors` is passed through.
`machine.registries.config` is not, so a mirror behind a private CA or
one needing credentials still has no knob, and the platform-wide
`registries` value does not reach tenant workers. The chart's
breaking-change note says so, and also says which knob covers which
artifact: the mirror routes registry pulls, while the worker OS disk
image arrives as a raw HTTP artifact under `talos.imageFactoryURL` and
no registry mirror can touch it.

The helper's failure model is worth revisiting as a whole rather than
per call site. Every call it makes to the cluster has three outcomes
(the world says no, the world says yes, the call could not be made) and
a shell exit code carries two, so the third is re-derived at each site
from a different signal: stderr text for a NotFound, a field value for
readiness, the returned message for a failed apply. Since the resulting
decision is cached for the whole sandbox, a misclassification is
permanent for the run rather than a blip, which makes each of those
sites load-bearing. A single prober answering `yes`/`no`/`unknown`, with
the caching rule stated once over that answer, would move the question
out of the branches; the current shape reaches the same behaviour but
re-establishes it five times. Left out here because it is a visible
rework of code this PR only extends, and it would need its own
verification of every site; tracked in #3682.

One nicety left in the same helper: the diagnostic reads the mirror log
twice, once with `--tail=-1` to count kubelet-image requests across the
whole log and once with `--tail=50 --prefix` for a bounded context tail.
The second read exists for `--prefix`, which attributes each line to its
pod; collapsing to one read drops that. The attribution is worthless
while the mirror is a single replica and load-bearing the moment it is
not, in the one artifact read after a failure, and the second read is
itself wall-clock bounded, so it is left in place.

### Merge-order dependency (resolved)

#3676 landed first and this branch is rebased onto it.
`ghcr_mirror_diagnose` now runs in the node-join failure block under the
same phase gate as its neighbours, and each of its reads is wall-clock
bounded by the same `COZY_DIAG_READ_TIMEOUT`/`COZY_DIAG_READ_GRACE` the
block validates, so no single read can hold the op open. It is placed
after the guest captures and before the image-cache re-probe: the
console evidence is irreplaceable and must not be starved of budget, and
the dump is still cheaper than the Pod-creating re-probe, so the spend
order puts it between the two. When the diagnostics phase runs out of
budget the dump is declined out loud instead of attempted. An ungated
dump would spend that time anyway and take it from the captures behind
it, while the mirror's state is partly recoverable from the reads that
already ran, so gating costs less than it saves. The known cost is
#3686: cheap reads ahead of the gate can drain the whole budget on a
viscous apiserver, and the dump is then declined in exactly the runs
where the mirror is a suspect. That is a defect of the budget, not of
this dump, and it is tracked there.

### Scope against #3513

#3513 asks for three things: capture the guest side, act on the
evidence, and do not raise the 18m budget again. The capture landed in
#3548 and this consumes its output; the budget is untouched. What this
acts on is the TLS-handshake variant only. The same thread documents
runs that issue no apid CSR at all inside the whole budget, time lost
before Talos reaches the kubelet image, which no registry mirror can
affect. So #3513 stays open after this merges, and a green suite
afterwards is not evidence the flake is closed. The manifest header says
so too, for whoever reads it next.

### Both tenant suites are covered

`kubernetes-previous` selects the previous Kubernetes *minor* out of
`packages/apps/kubernetes/files/versions.yaml`, `run_kubernetes_test
'keys | sort_by(.) | .[-2]'`, not a previously-released chart. Both
suites build their tenant CR from the same in-tree chart through the
same helper, so both get the mirror.

### Screenshots

Not applicable; no UI changes.

### Downstream repositories

Walking the trigger map in `docs/agents/contributing.md` against the
diff: **cozystack/terraform-provider-cozystack is affected**. The
provider is hand-written, and the map lists "add, remove or rename a
field in an app's `values.schema.json`" and "change a default in an
app's `values.yaml`" as triggers for its schema, model and
expand/flatten pair. This adds `talos.registryMirrors` with a `{}`
default to both the `kubernetes` and `kubernetes-nodes` schemas. No
other repository matches a trigger. The website's managed-app reference
pages regenerate from each package's `README.md` on a stable tag, and
this only changes an existing package's README.

I have not opened the provider follow-up and have not ticked the box,
because a ticked box with no link claims work that does not exist.
Leaving it to a maintainer to decide whether the provider models this
field and who files it.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(kubernetes): add `talos.registryMirrors` to route tenant worker node image pulls through a registry mirror for air-gapped or flaky-egress environments
```
2026-08-09 16:57:37 +03:00
Aleksei Sviridkin 9d4bcd8d66 fix(e2e): wire the ghcr.io mirror into the tenant install and node-join flow
Deploy the pull-through mirror from its own install step rather than
folding it into the Talos image-cache step, whose early return on a
missing values key would otherwise skip it silently. Build the tenant
CR spec.talos block from both the OS image cache and the registry
mirror, each included only when its in-sandbox backend is up. On a
node-join failure dump the mirror's state alongside the Talos image
cache, under the same phase gate as its neighbours so it cannot starve
the guest captures of budget. Record the mirror's legs in the step's
budget inventory.

talos_spec_block is covered by a bats suite over all four mirror-up and
mirror-down combinations, splicing the result under spec: and reading
it back with yq so a block indented into the wrong parent fails, and
the install step is pinned to exclude the Cilium policy from its
pre-Cilium apply.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 16:17:44 +03:00
Aleksei Sviridkin 1c6ee7c047 fix(e2e): harden the ghcr.io mirror resolver and bound its diagnostics
resolve_ghcr_mirror_endpoint decides once, for the whole shared
sandbox, whether tenant workers are pointed at the in-sandbox mirror,
so a misclassified transient failure would disable it for the rest of
the run. Cache only stable answers: a genuine API NotFound and a
definite Available=False; never a transient query error, a missing
kubectl, or a failed egress-allow apply. Read readiness from the
Deployment's own Available condition when the rollout watch exits
non-zero, and commit the endpoint only once the egress allow is in
place, surfacing the apply failure reason rather than a generic blip.

The node-join diagnostic counts kubelet-image requests across the whole
access log rather than a fixed tail, and every read it makes is
wall-clock bounded by the caller's COZY_DIAG_READ_TIMEOUT so no single
read can hold the failure path open. The bats suite exercises every
resolver branch and the diagnostic's filtering, with a kubectl stub
that reports errors on stderr as the real binary does and a timeout
mock so the bounded reads stay reachable under test.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 16:17:33 +03:00
Aleksei Sviridkin 3a1292d682 fix(kubernetes): escape tenant values in the worker reconcile heredoc
The worker TalosConfigTemplate is applied from an unquoted
`cat <<EOF | kubectl apply` heredoc, so every free-form value
interpolated into it is subject to shell parameter expansion and
command substitution at Job runtime. Escape backslash, dollar and
backtick on talosVersion, installerRepository, schematicID and the
registryMirrors block so each renders as a literal, and record the
invariant above the data block in both charts: a new free-form field
here must be escaped or render-time pattern-validated, or it reopens
command injection into a host-cluster Pod. Escaping covers every byte
by construction; the chain is a no-op for ordinary values, so the
default render and its content-hash Job name are unchanged.

Tests render both charts and run the heredoc through a real shell with
hostile values, which a rendered-string regex cannot do, and the
default-omits-mirrors assertion selects the Job by kind rather than by
document index so it cannot pass against the wrong document.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 16:17:26 +03:00
Aleksei Sviridkin 18be4ee272 test(e2e): capture host-netns address attribution and a healthy-pod reference
The collector characterises the host->local-pod route for an affected pod
through ovn0 alone, so when a pod IP turns out to be a local address of the
host netns instead, nothing records which interface actually holds it. The
host netns capture now also takes `ip -o addr show` across every interface
and `ip route show table local`, so the owner can be named on the next
occurrence rather than only ruled out on ovn0.

The collector also inspects NotReady pods only, so a run has no same-window
baseline for what a normal route and conntrack state look like on the same
node. The per-pod capture becomes a reusable function and runs a second time
against one Ready, Running, non-hostNetwork pod on that node, into the same
file, so the comparison is in front of whoever opens it. hostNetwork is
excluded because such a pod's podIP IS the node's own address and would show
the same local-route fingerprint as the fault for an entirely ordinary
reason.

That baseline depends on the node and nothing else -- the candidates are the
rows of the one snapshot the affected set was computed from, and a pod
reporting Ready there is by definition not one this run is investigating --
so it is captured once per node and folded into each affected pod's file
from a cache. Several affected pods on one node is what a wedged
host->local-pod datapath looks like rather than an edge case, and capturing
the same healthy pod once per affected pod would spend the caller's envelope
re-asking a question already answered -- the cost the node-global dedup and
the lookup memo beside it exist to avoid.

This adds no read of its own. The cluster-wide pod list gains one column
instead of being asked a second time, and that single read's status is what
lets the artifact separate a node with no eligible pod from a list that
never answered: one line for both would make an unchecked baseline read as
an established absence. The two host-netns additions are best-effort execs,
like the captures they sit beside. The new row filter carries the field
count its jsonpath emits, so a list cut mid-record cannot hand over a
fragment as the pod every reading of the affected one is compared against.

The unit suite pins these on what they are rather than on what they leave
behind. Both host-netns captures are asserted through the kubectl argv the
collector issues, container included, because the section headers are
separate echoes that outlive the exec being removed. The baseline is pinned
from both sides: several affected pods on one node share a single capture,
and an affected pod is served the healthy pod from its own node rather than
from whichever node the walk reached first. The pod list's column order is
asserted against the positions its readers index, because every runtime
test answers that list from a stub written by hand and none of them can
observe the jsonpath at all.

Assisted-By: Claude <noreply@anthropic.com>

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 14:42:19 +03:00
IvanHunters 58910c0edc feat(kubernetes): route tenant worker image pulls through a registry mirror
Add a talos.registryMirrors passthrough to the kubernetes and kubernetes-nodes
charts that renders into the worker TalosConfigTemplate machine.registries.mirrors,
so an operator can point a registry such as ghcr.io at an in-cluster pull-through
mirror and a worker boot no longer depends on live public egress. Default {}
keeps the rendered worker reconcile Job byte-identical to before, content-hash
Job name included. Free-form mirror values are shell-escaped for the unquoted
apply heredoc, covered by an execution-level bats test.

Also add the e2e ghcr-mirror (registry:2 pull-through plus a Cilium egress
policy) and wire it into the kubernetes chainsaw suites, so CI exercises the
mirror path and takes the flaky public ghcr.io kubelet-image pull off live
egress.

Refs #3513 (addresses the egress variant of the tenant-worker node-join
failures; the dominant variant is CI-runner CPU starvation, tracked in the
issue).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-09 14:10:16 +03:00