Commit Graph
5549 Commits
Author SHA1 Message Date
Myasnikov Daniil 2f87cdcdfa docs(agents): correct the image-refs consumer ordinal
The routing bullet called hack/overlay-main-images.sh "a fourth
consumer" from a time when three scripts sourced the enumeration.
Four do now, so the ordinal names the wrong thing in the file every
agent reads first.

State the distinction instead of a count: overlay is the exception
because it reads refs and walks the tree itself, so a newly declared
file never reaches it, while hack/promote-packages-artifact.sh
sources nothing because it publishes the whole tree and consumes no
refs. One clause each, with the existing pointer to
docs/agents/image-refs.md carrying the rest.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil ffae003d6b fix(release): stop a failed org package listing from exiting green
The nightly sweep's own listing shared one `|| true` with the grep
that filters it, and grep legitimately exits 1 when no cozystack/*
package matches. A failed /orgs/.../packages read therefore arrived
as an empty list, took the "No cozystack/* container packages found"
branch and exited 0 -- below the nightly sweep but ABOVE the deferred
guard, so a protection set that could not be resolved reported
success having swept nothing. That bypasses the guard rather than
varying it: replaying the step against a stub gh, a failed /pulls
combined with a legitimately empty org listing exited 0 after
printing its own ::error::.

List and filter as two statements so one `|| true` stops meaning two
things, handle the listing failure like the other two (::error::,
blocked flag, emptied list), and consult the flag before the early
exit. That exit is non-zero, matching the other two listing
failures: no sweep is left to carry the run down to the guard, so
this is the last place the flag can be honoured, and an unresolved
protection set is exactly what fail-closed means. The legitimate
empty-org case still prints the same line and exits 0.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil a37a3178e5 docs(release): name all four sourcers of the image-ref enumeration
The library header still claimed three call sites. There are four:
this PR made hack/verify-promoted-packages.sh source it to compare
the candidate's refs against the rc artifact's. The historical
drift story stays with the original three, since the verifier has
only ever read the shared enumeration.

The consumer table gained hack/promote-packages-artifact.sh under a
bullet requiring every consumer of references to read all three
shapes through the library, while the exception below it still named
only hack/overlay-main-images.sh. The publisher sources nothing and
needs nothing: it hands the whole tree to `flux push artifact` and
touches one reference, the installer's own self-pin, by key with yq.
Say that, rather than implying a gap it does not have -- the real
exception remains overlay-main-images.sh, which does read references
and does walk the tree itself.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil dee398f3b3 fix(release): defer a retention listing failure to the promotion guard
Both candidate-protection sources open with a `gh api --paginate`
listing and neither handled a failure. Under `set -euo pipefail` a
transient /pulls or /branches blip therefore exited the step before
the unrelated nightly GHCR sweep had run -- the exact coupling the
deferred guard at the bottom exists to remove, and which
hack/promotion-retention-contract.bats already asserts positionally.

Route both listings into that guard instead: raise the blocked flag
and empty the list, so the promotion sweep is skipped and the job
still fails while the nightly sweep finishes. Emptying is not
cosmetic -- a half-paginated list left in place is a smaller
protected set, and a smaller protected set is the one shape that can
authorise a delete.

Replaying the step's real shell against a stub gh: with /pulls
failing it exited 5 and never reached the nightly line; it now exits
1 with the nightly sweep done and the promotion sweep skipped.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil b65b6aa3ae test(release): pin every requiredBaseFiles entry, not just the list
The promote gate refuses a target base that lacks any of five
[path, marker] pairs, and the contract test asserted the list as a
block: deleting the whole thing turned it red, deleting the
hack/lib/image-refs.sh entry alone did not. Four of five pairs were
therefore unpinned individually.

Pin the missing one. The verifier sources that library through the
same `dirname $0` as lib/promoted-packages.sh, so a base carrying
the script without it fails at exactly the same point -- after the
candidate, the staging branch and the draft release exist.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil acd0a9e0a2 test(release): cover the verifier's rc-reference scan guard
The scan treats any grep diagnostic as a failed scan, because
`find ... -exec ... +` reports the legitimate "nothing matched" as a
failure too and the `|| true` absorbing that also absorbs a file
that went unscanned. Deleting the whole guard left the suite green:
its two siblings -- a failing collector and an empty collection --
each have a dedicated test, and this one had none.

Reach the state the guard exists for by putting a grep on PATH that
answers the scan's invocation the way the real one answers a file it
cannot open. A chmod 000 fixture would stop proving anything
wherever the suite runs as root.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil 04fa8471fe docs(release): say what the gates actually guarantee
Three claims that were stronger than the code, plus one legibility fix.

The PR gate's comment said a `packages/` edit "on the PR or its base"
fails the status before merge. The base half is not true: GitHub fires
no `pull_request` event when the base branch advances, it only
recomputes refs/pull/N/merge, so a change landing on the base after this
went green is never re-verified and the PR merges on the stale result.
Checking the merge tree does make base drift visible WHEN the job runs,
which is worth keeping — but finalize is what actually catches the drift
case, after the merge and before any stable name. Say both, in the
workflow and in the reviewer checklist, so nobody reads the green as
covering more than it does.

"Transactional at every consumer-visible name" is a judgement call
dressed as an absolute, inherited from the "this dispatch performs NO
registry mutation" it replaced. The dispatch now does write to the
registry: the `promotion-*` candidate is a real, publicly listable tag.
What it creates is no *stable* name — no vX.Y.Z, no :latest, no stable
installer. Stated that way the sentence stops needing the reader to
already know which writes were meant.

The retention paragraph listed two of the conditions the workflow
enforces. It now lists all of them, including the ones the workflow
gained here: same-repository, bot-authored, `release-X.Y.Z` branch, the
`release` label, exactly one valid digest pin, and protection from a
base-branch tip as well as an open PR.

Also: `flux pull artifact` reads its credentials from
~/.docker/config.json, which is why finalize now runs `docker login` —
but its runner class is not one of the docker-using build ones and the
setup step installs everything except docker. Check for it there, where
the message can name what is missing, instead of failing with "command
not found" halfway down the release path. And the verifier's exclusion
list is a restatement of flux's `excludeOCI`, not a derivation — the CLI
does not expose it — so the coupling and the pinned version are now
written down next to the list.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 820e72d572 fix(release): protect a candidate until its retag, not until PR merge
Retention protected a packages candidate only while an OPEN bot release
PR pinned its digest. The digest stops being eligible for deletion only
when finalize's retag adds a stable tag to the manifest, because that is
what makes the `all(tags match promotion-*)` predicate stop matching —
and finalize runs after the merge that closes the PR.

The window between them is where a half-finished release sits. Finalize
creates the write-once stable git tag, then dies before the retag: a
lost runner, a 5xx publishing the draft, a non-fast-forward on the
maintenance branch. What is left pins the candidate from a merge commit
and a tag nobody can move, with no open PR and only a temporary tag on
the manifest. Thirty days later retention deletes it, after which
re-running finalize 404s at the candidate check and re-dispatching
promote is refused because the stable tag exists. The release can then
only be finished by hand — the exact class of manual cleanup this design
exists to remove.

Protect the tips of the branches a promote PR can merge into as well.
The two sources are read differently on purpose. An open promote PR's
head is supposed to carry exactly one candidate pin, so anything else is
an anomaly and blocks. A branch tip is an additional source: every
digest pin found is protected and a branch without one contributes
nothing, because a maintenance line cut before this pipeline existed
cannot pin a candidate and must not wedge retention forever.

Blocking now stops the promotion sweep alone. It used to `exit 1` inside
the resolution loop, which also killed the unrelated nightly prune
further down the same step — so a WIP commit adding a trailing comment
to a pin on a release branch would silently stop GHCR nightly pruning.
The job still fails; it fails after the nightly sweep has run.

Verified by running the step's script against a stubbed `gh`: a digest
pinned by main's tip and referenced by no open PR is now excluded while
a genuinely abandoned one is still listed; a manifest that also carries
a stable tag stays excluded by the existing predicate; and a malformed
pin on a PR head leaves the nightly sweep running, skips the promotion
sweep and exits 1.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 05b2ce90cf fix(release): close the verifier's fail-open paths and cover them
Three gaps in the guards that decide whether the published candidate is
still the tree being tagged. All three passed silently rather than
failing, which is the worst shape for a check that runs immediately
before a write-once name is created.

`normalized_refs` piped `collect_image_refs` straight into its parsing
loop, so the pipeline reported `sort -u`'s status and the collector's
was discarded. POSIX sh has no `pipefail` and this script has to stay
POSIX, so `set -e` never saw it: a failed or truncated collection became
a smaller set and the rc-vs-candidate comparison matched it happily. The
verifier then reported that no container bytes changed across promotion
having proven nothing — on the guard that carries that entire claim.
Collect into a file first, so a nonzero collector aborts before anything
is compared. An empty collection is the same failure wearing a zero exit
status and needs its own check: two empty sets compare equal, so the
proof passes having examined nothing, and every packages tree carries
image references — none means the collector understood nothing it was
handed.

The rc-leftovers scan swallowed grep's stderr, turning "could not read
this file" into "this file is clean" — the silent-skip shape
hack/promote-rewrite-tags.sh refuses by name. The `|| true` has to stay,
because `-exec … +` reports the legitimate "nothing matched" as a
failure too, so the diagnostics are the only remaining signal that a
file went unscanned. Fail on any of them. Defence in depth rather than a
live hole: a surviving rc string would also fail the per-file compare,
since the release tree has none.

The per-file content comparison had no test at all. It is the only leg
that catches a file present on BOTH sides with different bytes — exactly
the threat the job exists for, a maintainer editing packages/ on the
release PR after promotion published the candidate. The file-set compare
sees the same names, the installer compare reads one file, and the
rc-refs compare never looks at the release tree. Replacing the
comparison with a no-op left all nine other tests green.

Red phases, all observed before the fixes. The content-compare case
changes one non-image line on one side, so no other leg can fire and
claim the credit; it fails with the comparison no-op'd and passes with
it restored. The two collector cases run the verifier from a copy whose
sibling lib/ holds a stub — which is how the workflows already invoke
it, `.release-tooling/hack/…` with libraries resolved next to the
script, so the failure is reached the way production would reach it.
Against the piped version both fail on `[ "$rc" -ne 0 ]`: the verifier
exits 0 and prints its success line whether the collector returns 3 or
returns nothing.

Also pins `e2e`'s dependency on the candidate gate. The contract pinned
`e2e-report`'s, so removing `verify-release-candidate` from `e2e`'s
`needs` together with the `success` clause on its release arm left all
six tests green — and running a full-e2e promotion suite against a
candidate that never verified is the worse of the two failures.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 9fa8eeeda2 ci(release): pin the promotion toolchain and cap the candidate gate
Three review findings on the new promotion steps.

`yq` was fetched from `releases/latest/download` in all three steps this
pipeline installs it in — the promote dispatch, the promote PR gate and
finalize. That makes an unpinned moving dependency part of what decides
whether a stable release may be created, and part of what rewrites the
installer pin: two promotions of the same rc can be served different
binaries. Pin it the way FLUX_VERSION already is, version-gated so a
matching preinstalled yq is reused, on the version the e2e sandbox image
already tracks.

A version authenticates nothing about the bytes it names, though, and
these are the workflows that mint write-once stable names. yq is
therefore also pinned BY CONTENT: its SHA-256 is committed beside its
version and verified before install.

Scoped to yq deliberately. It is the one downloaded binary with write
authority over what a release contains — it stamps the candidate digest
into installer values at promote time and platformVersion into the
stable chart at finalize. skopeo, helm and oras only move bytes that are
already digest-addressed, so a substituted one is caught downstream; a
substituted yq writes the wrong pin and every later check verifies that
pin faithfully.

The Flux installer is left as it is, on purpose. It fetches
flux_${FLUX_VERSION}_checksums.txt from the pinned release and verifies
the binary against it, so the binary is not the exposure — the ~6 KB
bootstrap script from fluxcd.io is, and it is mutable by design and
fetched identically in ten other workflows here. Pinning its bytes in
these three files would break on the next regeneration of a script the
project does not control while leaving the same fetch unpinned
everywhere else. The comment names the residual risk instead.

No precedent existed either way: nothing under .github/workflows/ or
hack/ verified a downloaded tool before this, and every version-pinned
download in the tree (oras 1.2.0, cozyvalues-gen 1.6.0) is unverified.
This is a new convention rather than an application of one, confined to
the release path and justified by write authority, and the comment says
so where the next reader will be standing. The three unpinned fetches
outside this pipeline (nightly.yaml, e2e-fork.yaml) are left alone.

The checksum is trust-on-first-use, which is the honest description: it
fixes the bytes to what was checked when the version was chosen and
makes any later substitution loud. Provenance is the release's own
published checksums list cross-checked against the downloaded asset;
both agree on
fa52a4e758c63d38299163fbdd1edfb4c4963247918bf9c1c5d31d84789eded4, and
that binary self-reports v4.53.3.

Verified locally against the real asset: the version gate skips a
matching preinstalled yq and re-installs a mismatched one, the checksum
line reports OK for the genuine binary and exits 1 for a tampered copy,
which aborts the step under the runner's `bash -e`.

`verify-release-candidate` had no `timeout-minutes`, so it inherited the
6-hour default while `e2e-report` waits on it through `needs` — a
stalled `flux pull artifact` would hold the required `E2E Tests` status
pending for the whole of it. 30 minutes, matching `checks` and
`finalize`.

The contract pins that all three install steps carry both versions and
the checksum, run the checksum check, name no `latest/download`, and
agree on one yq version and one checksum. So the unpinned one-line idiom
cannot come back into a release step by copy-paste from the workflows
that still use it, and bumping the version in one file without its
checksum cannot pass. Both mutations are caught.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 2d734fb229 fix(release): preflight the rc tree's own pin before publishing
hack/promote-packages-artifact.sh pushes the candidate BEFORE rewriting
installer values, so the artifact carries the rc tree's
platformSourceUrl/platformSourceRef verbatim — and
hack/verify-promoted-packages.sh reads exactly those two fields back out
of it as the candidate's rc baseline, requiring a trusted repository and
an immutable `digest=sha256:<64 hex>` pin.

The publisher checked neither. A tag-form ref or an untrusted URL in the
rc tree published a candidate and pushed two commits regardless; the
refusal arrived later, in the promote PR gate or in finalize, once the
staging branch and the draft release already existed. Check both at
dispatch time instead, while nothing has been written.

A copied regex would drift from the guard it is bringing forward and
quietly restore the late failure, so the trusted repository and the
digest pattern move into hack/lib/promoted-packages.sh and both halves
source it. That makes the library part of what the verifier needs at run
time, so promote's base-branch precondition gains it: the list exists
because the base's own workflows run after merge, and a base carrying
the verifier without its library fails at the same two late points.

Tests: both preflights are covered, each asserting the mock registry
client was never invoked — "exited non-zero" alone does not say nothing
was written. Both fail with the preflight removed. The publisher
fixtures now pin the trusted repository the way a real rc tree does, and
still send the candidate elsewhere via REGISTRY so the rewrite
assertions keep their teeth.

Also covers a malformed PROMOTION_ID. It is not attacker-influenced, but
it is the run-uniqueness suffix of the candidate tag and retention.yaml
deletes a candidate only when every tag on the manifest matches
`promotion-v…-run-[0-9]+-[0-9]+`; a value outside that shape publishes
an artifact no cleanup pass can ever collect.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 54e8533a9d fix(release): compare executable bits unambiguously in the verifier
The mode check was written as a single chain:

  if { [ -x "$a" ] && [ ! -x "$r" ] || [ ! -x "$a" ] && [ -x "$r" ]; }

which reads like a symmetric difference and is not one. POSIX gives
`&&` and `||` equal precedence and left associativity, so it groups as
`((A && B) || C) && D` with `D = [ -x "$r" ]`. A file executable in the
candidate and not in the release tree makes `A && B` true but `D`
false, so the whole condition is false and the mismatch is accepted.
Only the mirror case was rejected — and it is the benign one. The
direction that got through is content the merge reviewer never saw made
runnable inside the very artifact the operator installs, which cmp(1)
cannot see because it compares bytes.

Resolve each side to a value and compare those, so no precedence rule
is load-bearing.

Both directions now have coverage. Before the fix the candidate-only
test failed against the shipped condition and the release-only test
passed, which is what makes the pair meaningful rather than a check
that only goes red when the whole block is deleted.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil 5d56681f8c fix(ci): put every label guard in lockstep, forks included
Promotion needs the `release` label to start work, so this PR added it
to the set of labels whose `labeled` event is not discarded. That set is
stated in four places and was updated in three.

`gate-concurrency-contract.bats` caught the shape of the problem and
then became the first casualty of it. It asserted that the concurrency
key excludes exactly ONE label from the `-label` group, by counting
occurrences of `github.event.label.name != '`. A second publishing label
made the count 2, turning the suite red — and with it `checks`, then
`finalize`, then `e2e`, then the required `E2E Tests` status. Raising
the constant to 2 would re-hardcode the same brittleness one label later
and throw away the invariant the test exists for: the set of labels
excluded from the `-label` suffix is exactly the set `plan` runs for.
Both sets are now extracted from the workflow and compared, so the
assertion holds at any number of labels.

`resolve_assets` was the guard that got missed. It sits on `e2e`'s
release arm, so a label event `plan` runs for and it skips for leaves
`e2e` with both arms false — `finalize` is force-skipped for release
PRs — while `e2e-report` still runs. On a promote PR already carrying
`full-e2e`, re-applying `release` therefore superseded the green run and
posted `E2E Tests` = failure, because `full-e2e` bypasses the "e2e not
repeated" branch and `E2E_RESULT` is `skipped`. `unlabeled` is not a
trigger, so removing the label could not clear it; only a push could.

`release` must also not publish on the fork lane. e2e-fork.yaml's
`resolve` decides "this run built nothing, so it is not a verdict" by
asking whether `Plan build` skipped. Admitting `release` at `plan` made
that false: a fork run reaches `plan`, has `build`/`finalize` gated off
by its own label, succeeds having produced no pr-patch, and the fork
lane fails closed on the missing artifact — red on a suite that was
never going to run, on a PR whose only fault is a mislabel. `plan` and
`resolve_assets` now admit `release` for same-repo PRs only, and the
concurrency key carries the complementary term, so
NOT(A OR B OR (C AND NOT F)) stays exactly (NOT A) AND (NOT B) AND
(NOT C OR F). For a same-repo PR F is false and every one of them
reduces to what it was, so none of this changes non-fork behaviour.

The same shape still reaches the fork lane from a PUSH to a fork PR
somebody mislabelled `release` — `build` is gated on the label, not on
the event, so this predates promotion using the label. Left alone; the
comment in e2e-fork.yaml now records it instead of asserting an
invariant this PR would have made false.

The contract enumerated `plan` and `e2e-report` by name, which is why it
did not catch `resolve_assets`: a named-job loop cannot notice a job
nobody thought to name. It now derives the job list from the file and
requires every job that decides from a label name to decide from the
same set, and pins the fork term on both halves of the complement. The
file's existing properties are kept — extraction runs over
executable-only lines, so a commented-out key cannot satisfy the
contract, and every comparison asserts a non-empty input first, so an
empty extraction fails closed under both bats and cozytest.sh.

Verified by mutation, each caught: excluding a label `plan` skips for,
admitting one the key does not exclude, reverting `resolve_assets`,
deleting `e2e-report`'s guard, narrowing `verify-release-candidate`'s,
renaming the label in one guard only, commenting out the group key, and
dropping the fork term from either side alone. Deleting `plan`'s
step-level fork guard is caught too, by an assertion that had to stop
matching on the bare identifier once `plan`'s own `if:` named it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil f21d277eba fix(release): retain live promotion candidates
Protect package candidates referenced by open release PRs while pruning abandoned run-specific promotion artifacts. Document the stable-candidate lifecycle and maintenance-branch rollout requirement.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil c2e145709b fix(release): publish stable packages candidate
Serialize the tag-rewritten packages tree during RC promotion and pin its immutable digest in the installer. Verify the candidate against the prospective merge and the original RC artifact before creating stable names.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Aleksei Sviridkin 1c00ddb0f3 fix(platform): give node DaemonSets memory requests and limits (#3495)
## What this PR does

Sets memory requests and limits on the node DaemonSets that had none.
Companion to the namespace-wide LimitRange; the reasoning for why a
limit specifically (and not a request, and not a `PriorityClass`) is in
that PR and in `docs/operations/system-memory-limits.md`.

Requests come from 14-day peak working set measured on a 3-node dev
cluster. Limits are deliberately loose, sized as a runaway backstop
rather than a fitted ceiling, for two reasons: several of these scale
with cluster size in ways a small dev cluster understates (cilium with
endpoint and policy count, fluent-bit with log volume, multus and
kube-ovn with pods per node, velero node-agent during a backup), and
undersizing converts a rare pressure-driven kill into a deterministic
one. This repository already carries a scar from that failure mode,
recorded in the `vertical-pod-autoscaler` values where a 110Mi limit
OOM-killed the updater and cascaded into failed HelmReleases.

Two DaemonSets are not chart-rendered and needed their own seam.
`linstor-satellite` goes through the `LinstorSatelliteConfiguration`
podTemplate, including the two plunger sidecars — every container in the
pod needs a limit for the pod cgroup to get `memory.max`, so one
limit-free sidecar would put the whole pod back in the victim set.
`virt-handler` goes through KubeVirt `customizeComponents.patches`,
because virt-operator owns that DaemonSet and chart values cannot reach
it.

`linstor-controller` is deliberately left limit-free. It is a JVM that
must not be OOM-killed mid-operation, and `cluster_test.yaml` pins that
as an invariant. The satellite is a Go binary with a bounded working
set, so the opposite call applies there.

Not covered, because upstream hardcodes the containers with no
`resources` knob: the four frr-k8s `cp-*` init containers, the kube-ovn
`hostpath-init` and `install-cni` init containers, and `cozy-proxy`,
whose vendored chart exposes no resources value at all. `hami` and
`kilo` are left alone as optional components with no usage measurements
behind them. All of these still receive a limit from the namespace
LimitRange, so they are not left exposed — they simply carry no request.

Worth stating plainly: this PR does **not** grant OOM immunity on its
own, because the init containers it cannot reach leave the pod cgroup
without `memory.max`. That only closes with the LimitRange PR in place.

### Verification

Every touched package renders and its helm-unittest suite passes. New
assertions cover the metallb speaker and all five frr-k8s containers,
the linstor satellite and plunger sidecars, and the KubeVirt patch. The
metallb suite was mutation-tested: removing the frr-k8s reloader limit
fails it.

One gap. The KubeVirt patch shape was validated against the live CRD and
pushed through the real update path with `kubectl patch
--dry-run=server` (accepted; live object confirmed unchanged), but
virt-operator actually reconciling the DaemonSet with those resources
has **not** been observed, because that means restarting virt-handler
fleet-wide. Worth confirming in e2e or on a dedicated stand before
merge.

### Downstream repositories

Walked the trigger map against the diff. No entry is triggered: this
changes chart values and cozystack-owned templates under
`packages/system/`, not the `cozy-proxy` label and annotation contract,
not node prerequisites, not an app schema or version enum.

Leaving the boxes empty rather than ticking "none affected", because the
diff surfaced a real gap in a downstream repository even though nothing
here forces a change there: the vendored `cozy-proxy` chart exposes no
`resources` value, so its DaemonSet cannot be given one from this side
at all. That wants an issue or PR in cozystack/cozy-proxy followed by a
re-vendor. Flagging it for a maintainer decision rather than filing
speculatively.

### Release note

```release-note
fix(platform): node DaemonSets (cilium, metallb speaker and frr-k8s, kube-ovn, linstor-satellite, virt-handler, fluent-bit, node-exporter, multus, velero node-agent, kubevirt-csi-node) now declare memory requests and limits, so they are no longer selected as victims by the Talos userspace OOM handler when an unrelated workload drives node memory pressure.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Resource Management**
- Added CPU and memory requests and limits across Cilium, KubeVirt CSI,
KubeVirt, Linstor, MetalLB, monitoring agents, Multus, and Velero
workloads.
- Improved workload stability with defined memory ceilings and resource
reservations.

- **Multus**
- Moved resources to the dedicated `cozy-multus` namespace and
configured Cilium as the master CNI.
  - Added optional CNI plugin staging with enhanced security settings.

- **Tests**
- Added and expanded Helm tests validating resource configurations and
Multus behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 14:13:21 +03:00
Aleksei Sviridkin b7b56530b9 docs(platform): explain system component memory limits (#3496)
## What this PR does

Adds `docs/operations/system-memory-limits.md`, recording why system
components carry memory limits — which is not recoverable from the
values alone, and is the kind of thing that gets "cleaned up" by a
well-meaning future change.

The Talos userspace OOM handler (v1.12+) selects victims by ranking
cgroups that lack `memory.max`. Three consequences are each easy to get
wrong independently: only a limit grants immunity (a request just moves
BestEffort to Burstable, which under `strictCgroupClassOrdering` means
"killed second"); every container in the pod needs one, init containers
included, because kubelet only sets pod-level `memory.max` when they all
have it; and `PriorityClass` — `system-node-critical` included — has no
effect whatsoever, because the handler walks cgroupfs and never talks to
the API server. That last one is the trap, since setting it looks like a
fix.

The page also covers the two installer knobs and the constraint that
makes them dangerous to lower (a defaulted limit below a container's own
request fails admission), how to verify a pod cgroup actually carries
`memory.max` rather than trusting the QoS class, what is deliberately
left uncovered (Kubernetes static pods are Talos-managed; a handful of
vendored containers expose no upstream resources knob), and how to relax
the global-PSI backstop clause of the trigger for a cluster that trips
it persistently.

Ordering note: this documents behaviour introduced by the LimitRange and
DaemonSet PRs, so it should not merge ahead of them.

### Downstream repositories

Walked the trigger map against the diff. This adds one file under
`docs/operations/`, alongside the existing pages there; nothing in the
map covers that path, and the website keeps its own copy of user-facing
documentation rather than generating it from this directory. A
maintainer may still want a version of this on the website, which is a
judgement call rather than a forced follow-up.

- [x] No downstream repository is affected by this change

### Release note

```release-note
docs(platform): document why system components carry memory limits, how the Talos userspace OOM handler selects victims, how to tune or disable the system-namespace LimitRange, and why PriorityClass does not influence any of it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added guidance on Talos userspace out-of-memory behavior and system
component memory limits.
* Documented namespace boundaries, installer tuning, validation
constraints, rollout behavior, and verification steps.
* Added instructions for inspecting OOM actions, identifying uncovered
workloads, evaluating trigger expressions, and configuring optional
overrides.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 13:49:46 +03:00
Aleksei Sviridkin 22ce998890 fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy (#3546)
## What this PR does

Fixes #3544.

`flux-shard-operator` clones the flux-aio `helm-controller` container
into each `helm-controller-shard<i>` Deployment. It already sanitises
the all-in-one localhost wiring, but keeps two things that stop the
standalone shard from starting behind a corporate HTTP proxy:

1. **Inherited `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`.** A standalone
shard needs no external egress (source-controller fetches artifacts).
Behind an unreachable proxy the controller makes a blocking startup
HTTPS call that never completes, the manager never serves `/healthz`,
and the liveness probe crashloops the pod forever. flux-aio survives
only because it starts once and never restarts. On an affected cluster
every HelmRelease carrying a `sharding.fluxcd.io/key` (150 of ~240) was
frozen for days, silently.

2. **No `startupProbe`.** The clone inherits only a ~30s liveness
window, so any slow start (large cache sync) is fatal and unrecoverable.

This PR extends the sanitisation to drop the proxy env, and derives a
`startupProbe` from the liveness handler with a generous budget so a
slow-but-progressing start is not killed. Verified by an A/B on a live
cluster (throwaway helm-controller with a non-matching selector): with
the proxy env it hangs; without it, it starts cleanly (`Starting
workers`).

### Downstream repositories
- [x] No downstream repository is affected by this change

Internal operator behavior only; no API/CRD/values contract change.

### Release note
```release-note
fix(flux-shard-operator): sharded helm-controller no longer crashloops at startup behind an HTTP proxy; the cloned controller drops the inherited proxy env and gains a startupProbe, so HelmReleases assigned to shards keep reconciling in proxied environments.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Prevented inherited proxy settings, including uppercase and lowercase
variants of `NO_PROXY`, from affecting Flux shard deployments.
* Automatically adds a startup health check when only a liveness check
is configured.
* Preserved the liveness check’s handler and timeout while applying
appropriate startup timing and failure settings.

* **Documentation**
  * Documented startup health checks for slow helm-controller startup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 13:28:51 +03:00
Aleksei Sviridkin b7e51d54a5 fix(backupstrategy-controller): gate the default MongoDB strategy on bootstrap (#3815)
## What this PR does

Closes #3814.

Unblocks a fresh install of `main`. Since #3562 the
`backupstrategy-controller` chart renders one default Strategy CR
ungated, and that makes the whole release fail on a cluster where its
CRDs are not yet applied:

```
Helm install failed for release cozy-backup-controller/backupstrategy-controller
resource mapping not found for name: "cozy-default-mongodb"
no matches for kind "MongoDB" in version "strategy.backups.cozystack.io/v1alpha1"
```

The chart ships its own eight `strategy.backups.cozystack.io` CRDs as
ordinary templates, since `templates/crds.yaml` globs
`definitions/*.yaml`, and Helm pre-applies only a `crds/` directory.
Helm then resolves every document in the release manifest through the
cluster's RESTMapper in `Build()` before it applies any of them, so an
ungated Strategy CR has no mapping on the first install, `Build()`
fails, and the release applies **zero** objects, including the CRDs that
would have made the mapping resolve.

Nothing recovers from that. Every retry renders the same manifest, so it
is permanent rather than a race, and Helm's kind sorter cannot help
because it orders the apply and the failure is before the apply. The
install log shows `Warning InstallFailed 43s (x36 over 20m)` and then
`Install failed (no retry)`.

### Why the seven siblings never hit this

Each of them gates on a resolved `$bucketName`, which is empty until the
COSI BucketClaim status is reconciled, so they are simply absent from
the first render. By the time `DefaultObjectsGate` forces a second
revision the CRDs from revision 1 exist and the mapping resolves.

That gate is documented as "the CR needs the bucket name", and it
silently doubles as the CRD-ordering guard. MongoDB genuinely does not
need the bucket name, the template argues that correctly, so the gate
was left off and the accidental protection went with it. This gates it
too, for the ordering reason rather than the value, and records that
reason in the template so the next bucket-independent strategy keeps it.

Convergence is unchanged. `DefaultObjectsGate` already routes
`apps.cozystack.io/MongoDB` to `cozy-default-mongodb` and forces a
revision once any routed object is missing, which is exactly how the
seven siblings materialise today.

### Why this was invisible

`tests/rendering_test.yaml` asserted `count: 1` for the MongoDB template
under a case named "bootstrap window (no resolvable bucket name)", so
the defect was codified as the expected behaviour and shipped green.
helm-unittest has no RESTMapper and cannot see the consequence. That
assertion is flipped here, with the ordering reason written next to it.

It was also masked in CI until yesterday by #3806. `StreamVisitor.Visit`
returns immediately on a validation error but only continues on a
mapping error, so the OpenAPI defect aborted the manifest stream at the
first document and the mapping check was never reached. #3808 fixed that
and this surfaced underneath it.

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Testing

Verified by rendering both windows, since the whole bug is which
documents appear in the first render and a cluster adds nothing to that.

Before, bootstrap window: one Strategy document, `MongoDB`, alongside
the eight not-yet-applied CRDs. After: none. With
`backupStorage.bucketNameOverride=b`, which stands in for a resolved
BucketClaim: all eight, MongoDB included.

```
helm template bsc packages/system/backupstrategy-controller | grep -c '^kind: MongoDB$'
```

`helm unittest packages/system/backupstrategy-controller` passes, 4
suites, 18 tests.

### Follow-up, not in this PR

The durable fix is to stop shipping these CRDs as templates and split
`definitions/` into a `backupstrategy-controller-crds` package with a
`dependsOn` edge, which is the convention six other packages already
follow. That cannot be a hotfix: none of the eight definitions carries
`helm.sh/resource-policy: keep`, so the upgrade that removes them from
this chart's manifest would have Helm delete the eight CRDs and
cascade-delete every live Strategy CR on existing clusters. It needs the
`keep` annotation shipped a release ahead.

### Release note

```release-note
fix(backupstrategy-controller): gate the default MongoDB backup strategy on the bootstrap window, so a fresh platform install no longer fails with "no matches for kind MongoDB" and applies nothing
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Prevented the default MongoDB backup strategy from rendering before
the required S3 storage configuration is available.
* Improved first-install reliability by avoiding premature strategy
resource creation.

* **Tests**
* Updated bootstrap validation to confirm the MongoDB strategy is
omitted until a bucket name is resolved.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 13:26:58 +03:00
Myasnikov Daniil 4f9c20cac0 fix(backupstrategy-controller): gate the default MongoDB strategy on bootstrap
A fresh install of packages/system/backupstrategy-controller fails
permanently and applies nothing:

  Helm install failed for release cozy-backup-controller/backupstrategy-controller
  resource mapping not found for name: "cozy-default-mongodb"
  no matches for kind "MongoDB" in version "strategy.backups.cozystack.io/v1alpha1"

The chart ships its own strategy.backups.cozystack.io CRDs as ordinary
templates (templates/crds.yaml globs definitions/*.yaml), and Helm
resolves every document in a release manifest through the cluster's
RESTMapper in Build() before it applies any of them. An ungated Strategy
CR therefore has no mapping on the first install, Build() fails, and the
release applies ZERO objects, including the CRDs that would have made
the mapping resolve. Nothing recovers: every retry renders the same
manifest, so this is permanent rather than a race, and Helm's kind
sorter cannot help because it orders the apply, not the Build.

The seven sibling strategies never hit this because each gates on a
resolved $bucketName, which is empty until the COSI BucketClaim status
is reconciled. That gate is documented as "the CR needs the bucket
name", and it silently doubles as the CRD-ordering guard. MongoDB
genuinely does not need the bucket name, so the gate was left off and
the accidental protection went with it.

Gate it too, for the ordering reason rather than the value, and record
that reason in the template so the next bucket-independent strategy
keeps the gate. Convergence is unchanged: DefaultObjectsGate already
routes apps.cozystack.io/MongoDB to cozy-default-mongodb and forces a
second revision once any routed object is missing, which is how the
seven siblings materialise today.

Verified by rendering both windows. Before: the bootstrap render emits
one Strategy, MongoDB, alongside the eight not-yet-applied CRDs. After:
it emits none, and with backupStorage.bucketNameOverride set it emits
all eight.

tests/rendering_test.yaml asserted count: 1 for MongoDB under a case
named "bootstrap window (no resolvable bucket name)", so the defect was
codified green. helm-unittest has no RESTMapper and cannot see it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 14:53:13 +05:00
Aleksei Sviridkin 15459c96e5 fix(api): declare OpenAPIModelName for core and sdn types (#3808)
## What this PR does

Fixes #3806.

Declares `OpenAPIModelName` for the `core` and `sdn` API groups the way
#3273 already did for `apps`, so every published OpenAPI definition name
is the dotted Kubernetes model name rather than a Go import path.

Worth noting the two are the same event. #3273 landed the k8s 0.35 work,
hit the loud half of this in server-side apply where no apps kind
resolved at all, fixed `apps`, and left `core` and `sdn` with the
quieter half. Its merge commit `960d85370` is exactly where the window
opens: 190 failed-E2E logs from 2026-06-25 to 07-19 carry no occurrence,
and the first one is 11 hours after it.

Before this, 20 definitions shipped under names like
`github.com/cozystack/cozystack/pkg/apis/core/v1alpha1.Option`, while
every `$ref` pointing at them spelled each slash `~1`. Clients resolve a
`$ref` by trimming `#/definitions/` and looking the remainder up
verbatim with no unescaping, so the two spellings never meet and the
reference dangles. `kubectl apply --validate` against a cluster running
a `cozystack-api` built after 2026-07-21 therefore fails on any
resource, and in e2e the platform install fails on
`cozy-backup-controller/backupstrategy-controller` and never completes.

The mechanism is a Kubernetes 0.35 change:
`DefinitionNamer.GetDefinitionName` now returns the model name verbatim,
where it previously converted the Go import-path form into the friendly
reversed-path form. So whatever name the generated map uses becomes both
the published key and the escaped `$ref`, and slash-freedom stops being
a nice-to-have and becomes the correctness condition.

### Why hand-written methods

openapi-gen can emit these with `--output-model-name-file`, but that
also rewrites `zz_generated.model_name.go` inside the read-only
apimachinery and apiextensions module-cache packages that the shared
`gen_openapi` helper always passes as inputs, which fails for any
consumer that vendors deps from the cache, CI included. The `apps` group
settled this the same way and the reasoning is repeated in each new
`model_name.go` so nobody simplifies it back into the flag.

Once a package carries `+k8s:openapi-model-package`, openapi-gen emits
`Type{}.OpenAPIModelName()` for every type it writes a schema or a
`$ref` for, so a missing method is a build failure rather than a silent
skip. That makes a future omission inside these groups loud by
construction, and the new tests make an omission on a future group loud
at the group level.

### Guard tests

`pkg/generated/openapi/definitions_test.go` adds two, both asserting the
invariant rather than today's 20 names, and both refusing to pass on an
empty or cozystack-free definition map so they cannot go vacuously
green.

`TestDefinitionNamesAreDottedModelNames` fails on any definition name
containing a slash, naming the offender and the fix.
`TestDefinitionRefsResolve` builds each `$ref` the way kube-openapi's
builder does and resolves it the way a client does, which checks the
failure directly rather than through the slash-freedom proxy, and also
catches a group whose types disagree with each other.

Nothing asserted this before, which is why the defect reached v1.6.0 and
v1.6.1 unnoticed.

### Screenshots

Not applicable.

### Downstream repositories

Walked the trigger map against the diff. Published model names change,
so I swept for consumers pinning the old form: every hit on
`github.com/cozystack/cozystack/pkg/apis` in the tree is a Go import,
which is unaffected because the package path does not change, and
`api/api-rules/cozystack_api_violation_exceptions.list` keys off the Go
package path in its own format and was unchanged by regeneration. No
string literal anywhere pins the old model-name form, and `apiPrefix` in
`pkg/cmd/server/openapi.go` is an apps-only const that this does not
touch.

- [x] No downstream repository is affected by this change

### Testing

- `go build ./... && go vet ./...`
- `go test ./...`
- `make generate && git diff --exit-code`, clean and idempotent
- `pre-commit run --all-files`
- Both guard tests run against the pre-fix tree first:
`TestDefinitionNamesAreDottedModelNames` named all 20 offenders and
`TestDefinitionRefsResolve` reproduced the reported error string with 15
dangling refs, fewer than 20 because some of the 20 are only
referenced-from and never referencing.
- An independent check driving the real
`openapi.NewDefinitionNamer(apiserver.Scheme)` reports 104 published
definitions with 0 slashes, 106 emitted refs with 0 dangling, and all 23
cozystack kinds resolving to a published definition carrying
`x-kubernetes-group-version-kind`, which also confirms the `apps`
server-side-apply fix still holds.

`make unit-tests` aborts at `hack/ghcr-mirror_test.bats`, which fails on
a pristine `origin/main` in my environment while passing in CI, so I ran
the chain standalone: the other 60 bats files pass, as do
`helm-unit-tests`, `go-unit-tests`, `rd-presets-check`,
`test-check-readiness` and `migrations-target-check`.

### Release note

```release-note
fix(api): publish OpenAPI model names for the core and sdn API groups as dotted Kubernetes names, so `$ref`s resolve and client-side validation works again against the aggregated apiserver
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Enhancements**
  * Added OpenAPI metadata for Core v1alpha1 API models.
  * Added OpenAPI metadata for SDN v1alpha1 API models.
* API schemas now provide consistent canonical model names, supporting
more reliable documentation and schema references.
* Added package-level OpenAPI markers to improve API discovery and
tooling compatibility.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 11:53:44 +03:00
Aleksei Sviridkin 1b4b6ff5e5 test(e2e): warm the ghcr.io mirror before the tenant suites need it (#3768)
## What this PR does

The most expensive e2e flake, cozystack/cozystack#3513, is a tenant
worker missing the 18m node-Ready budget with zero Nodes registered. The
in-guest capture in cozystack/cozystack#3548 diagnosed one variant: the
worker's kubelet image pull crawls to ghcr.io at 40-165 KB/s over the
runner egress and does not finish inside the budget, Talos holds the
kubelet service until its image arrives, so the node never registers and
the tenant cilium HelmRelease then fails for want of a schedulable node
rather than for any fault of its own.

The sandbox already runs an in-cluster pull-through mirror for ghcr.io,
but a pull-through cache fetches a blob only when a client first asks
for it, so the first tenant worker pays the cold fetch inside its
node-Ready budget. Measured with the same fetch script against a local
instance of the same registry image: 32m47s cold, 1.1s warm, for the
same tags the failing runs stalled on.

This PR fills the cache from a Job started by the same install step that
deploys the mirror, tens of minutes before the kubernetes suites run. It
warms the two kubelet tags the suites actually select, resolved from the
chart's version map in the same order the suites resolve it, and one
platform per tag (the one the workers run, ~61 MiB each) - warming what
nobody pulls would be a true addition to the same throttled egress the
install itself is using.

The warm-up is best-effort by construction: a Job that fails, times out
or never starts does not fail the install, and the suites behave exactly
as they did before it existed. The Job itself exits non-zero on any
missed tag, so its status stays honest.

The node-join failure diagnostics now answer the questions a red run
used to leave open: whether a worker pulled through the mirror at all
(counting GET and HEAD access lines per request, excluding the warm-up's
own requests, which name themselves via user-agent), how long the mirror
took to answer (server-side `http.response.duration` is the only
after-the-fact signal separating a warm cache from a cold one, since the
proxy stores blobs in the background and the Job reports success either
way), and whether the warm-up itself got anywhere. The diagnostic
section grows from five to seven bounded reads, moving its worst case
from roughly 100s to roughly 150s inside the existing phase budget gate;
unbudgeted diagnostic residuals stay tracked in
cozystack/cozystack#3666.

This addresses one variant of cozystack/cozystack#3513 and does not
close it: the zero-CSR class remains untouched, and a green suite after
this change is not evidence the flake is gone. The 18m node-Ready budget
is not raised.

Covered by 65 unit tests in `hack/ghcr-mirror_test.bats`: tag selection
and ordering against the suites' own expressions, rejection of malformed
and injected tag values, Job shape and restricted security context, the
readiness wait as a wall clock, every failure path of the new
diagnostics including an over-1-MiB log fixture that pins the duration
filter against real log sizes, and the exclusion of the warm-up's own
traffic from both the request count and the duration report.

### Screenshots

No UI changes.

### Downstream repositories

The diff is confined to e2e infrastructure under `hack/`: the mirror
manifest, the warm-up library and its tests, the install step, and
diagnostic labels. Walked the trigger map file by file - no chart
values, no APIs, no documentation content, nothing any downstream
repository consumes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added best-effort GHCR mirror warm-up during installation.
* Preloads selected Kubernetes image manifests and blobs to improve
cache readiness.
  * Warm-up failures safely fall back to direct image pulls.

* **Diagnostics**
* Improved request classification, timing, repository identification,
and reporting of warm-up status and logs.

* **Tests**
* Expanded coverage for warm-up behavior, validation, retries, security
settings, caching, logging, installation, and diagnostics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 11:21:10 +03:00
Myasnikov Daniil cea1703a6d test(api): guard published openapi definition names and refs
Nothing asserted that a published definition name contains no slash,
which is why the core and sdn groups shipped as Go import paths in
v1.6.0 and v1.6.1 without anyone noticing. The same omission on a
future group would be just as invisible.

TestDefinitionNamesAreDottedModelNames fails on any definition name
containing "/", names the offender and points at the fix (the
+k8s:openapi-model-package marker plus an OpenAPIModelName method).
TestDefinitionRefsResolve is the failure mode itself: it builds each
$ref the way kube-openapi's builder does, JSON pointer escaped, then
resolves it the way a client does, trimming "#/definitions/" with no
unescaping, and reports any that dangles. It also checks the declared
Dependencies against the published names, which catches a group whose
types disagree with each other rather than uniformly.

Both assert the invariant instead of listing today's 20 names, so a new
group that omits the marker is caught rather than a changed count, and
both guard against going vacuous if the definition map is ever emptied
or stops covering cozystack's own types.

Both fail on the pre-fix tree: the first names all 20 offending
definitions, and the second reproduces the reported error verbatim,
down to
"github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec".

Refs: https://github.com/cozystack/cozystack/issues/3806

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 13:12:39 +05:00
Myasnikov Daniil 5f3ad9cd58 fix(api): declare OpenAPIModelName for core and sdn types
The aggregated apiserver publishes the core.cozystack.io and
sdn.cozystack.io models under their Go import path, so a $ref to any of
them does not resolve and client-side validation fails on every
resource of those groups, not only the one named in the error:

  error validating data: SchemaError(…core/v1alpha1.Option.spec):
  unknown model in reference:
  "github.com~1cozystack~1…v1alpha1.OptionSpec"

Since Kubernetes 0.35 the apiserver's DefinitionNamer.GetDefinitionName
returns the model name it is handed verbatim; it no longer converts the
Go import-path form into the "friendly" reversed-path form. Whatever
name the generated openapi map uses therefore becomes both the
published definition key and, JSON pointer escaped, the $ref pointing
at it. A name containing "/" ships as a definition keyed on the raw
path while every reference to it spells each slash "~1", and clients
resolve a $ref by trimming "#/definitions/" without unescaping
(kube-openapi pkg/util/proto/document.go). The two spellings never
meet, the reference dangles, and validation of the whole document
fails.

Mirror b916d7475, which fixed the same disagreement for apps: add the
+k8s:openapi-model-package marker to each doc.go and declare
OpenAPIModelName for all 11 core and 9 sdn types, so
GetCanonicalTypeName, Scheme.ToOpenAPIDefinitionName and the generated
map key all agree on a slash-free dotted name. That takes the number of
slash-bearing definition names in the published document from 20 to 0.
The methods are hand-written rather than emitted by openapi-gen's
--output-model-name-file because that flag also rewrites
zz_generated.model_name.go inside the read-only apimachinery and
apiextensions module-cache packages the shared gen_openapi helper
always passes as inputs, which fails on any consumer including CI that
vendors deps from the cache. With the marker in place openapi-gen emits
Type{}.OpenAPIModelName() for every type it references, so a future
type without a method fails the build instead of silently
reintroducing a Go-path name.

The generated openapi is the output of the root `make generate`;
nothing else in the generated tree moved.

kubectl apply --validate has been broken against any cozystack-api
built after 2026-07-21 on every resource, and this shipped in v1.6.0
and v1.6.1. In e2e it surfaces as the platform install hanging on
cozy-backup-controller/backupstrategy-controller, which takes
tenant-root and everything behind cozystack-basics with it.

Refs: https://github.com/cozystack/cozystack/issues/3806

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 13:12:28 +05:00
Aleksei Sviridkin 829edfadb3 test(e2e): warm the ghcr.io mirror before the tenant suites need it
The in-sandbox pull-through cache fetches a blob only when a client
asks for it, so the first tenant worker's kubelet pull is a live fetch
from ghcr.io. That leg is what misses the node-Ready budget: measured
at 40-165 KB/s for a 61 MiB image, with two blob streams dying
mid-transfer after 7-8 MiB on an upstream HTTP/2 reset. Talos holds
the kubelet service in PreFunc until the pull completes, so the node
never registers, and the tenant cilium HelmRelease then fails for want
of a schedulable node rather than for any fault of its own.

Fill the cache from a Job started in the same install step that
deploys the mirror, tens of minutes before the kubernetes suites run.
The Job waits for the mirror to serve, then asks it for each kubelet
tag's index, walks the manifests it gets back, and discards the blob
bodies; distribution populates its blobstore on the way past.

It warms one platform per tag -- the one the workers run -- picked out of
the index by record, which keeps the walk a grep. Walking both platforms
was a true addition to the same throttled leg: arm64 is bytes nothing in
the sandbox pulls, and the digest ordering put it first for both tags, so
on the slow runs this exists for the deadline expired with the second tag
still cold.

It warms the two minors the suites select, not the whole version map.
The map would be ~300 MiB at the one-platform walk this ships (~61 MiB
per tag), three fifths of it for minors no suite can resolve, spent on
the same throttled egress the rest of the install is using. The tag
list still comes from the chart's map rather than a copy, ordered the
way the suites resolve it, so the set is theirs without restating
their expression.

The readiness wait is a wall clock, not a per-attempt count. Cilium
answers a Service with no ready endpoints by rejecting, so a probe
costs nothing when the mirror is absent and the full timeout when it
is merely slow; a constant was wrong in both directions across two
attempts at picking one.

Tags are refused by character class before the pattern is applied, so a
value cannot pass by having a valid first line: grep matches per line
and the pattern's anchor is end-of-line, not end-of-input.

The Job exits non-zero on any miss, so backoffLimit is a real bound
and a warm-up that reached nothing cannot report success. It names its
own requests, because the node-join diagnostic counts kubelet GETs in
the access log to tell a worker that pulled through the mirror from
one that fell back to public ghcr.io.

Naming them is only half of that. The same diagnostic now reports the
Job's status and result and the Job outlives the suites, because
excluding the warm-up without reporting it would leave a red run
unable to distinguish a warm cache from a warm-up that never started.
That counter also moved onto the access-line shape: registry:2.8.3
writes three lines carrying the path per request, so matching the path
counted one pull as three. It reads the repository from the same
variable the warm-up does, so a rename cannot leave the diagnostic
counting zero while the workers pull under the new name. GET and
HEAD both count: containerd resolves a manifest with a HEAD first,
so a GET-anchored count reports a pull that died before its first
GET as no request at all.

The failure path also prints how long the mirror took to answer the
worker. A warm cache and a cold one are otherwise indistinguishable in
the artifacts -- the Job reports success either way, because the proxy
stores the blob in the background and a completed GET is a different
event from a stored blob -- and server-side duration separates them by
orders of magnitude. It filters the log in the shell that read it: a
single argv element is capped at 131072 bytes on Linux regardless of
ARG_MAX, and probe traffic carries this log past that within about
twenty minutes of mirror uptime, so handing it to a helper would empty
the section on every run worth reading. Nothing is printed when the log
could not be read at all, because a heading with no lines under it
reads as a finding about the workers.

Measured against a local pull-through registry using the script the
builder emits: 32m47s cold, 1.1s warm for the same tags, and the
blobs it fetches are the same digests the failing runs stalled on.

A warm-up that fails, times out or never starts does not fail the
step, and the suites behave as they did before it existed. It is not
free, though: it pulls across the same throttled egress the install
is using, which is why the warm set is two tags and not five.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-14 11:08:49 +03:00
Aleksei Sviridkin abd331b2c8 test(system): anchor index-addressed negative asserts to what they name (#3608)
## What this PR does

Some negative assertions in these suites can't fail for the reason their
test names give. `contains` and `notContains` compare list elements for
equality, so `--pinned-tenants=` never equals a rendered
`--pinned-tenants=tenant-a=shard0`, and `{name:
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET}` never matches an env entry
carrying a `valueFrom` beside the name. Others forbid one rendered value
under a name that claims the flag itself, so `notContains
--compatibility=cilium` leaves the kilo default render free to carry
`--compatibility=flannel` off the same values key.

Flag guards become `notMatchRegex` on `args[*]`, which matches per
element, so any value fails and so does the bare form. Mapping guards
get `any: true`. flux-shard-operator gets an `equal` on the whole
default arg list instead, and `--shard-cpu-limit=` is dropped because
that list covers it. The vendored ouroboros templates keep per-flag
guards: a whole-list pin there would go red on any upstream flag change,
under a test name about one flag.

Several of these address `containers[0]` with nothing pinning that
index, so a sidecar rendered ahead of the workload turns them green
while the flag is live on the real container. Those get an `equal` on
the container name first. A Cilium deny rule has no name, so that one
pins the selector plus the rule and `toPorts` counts. Where a positive
`contains` already sits on the same path it holds the index by itself
and gets no extra anchor.

Assertions of the same shape remain in files this PR doesn't touch,
including one in a package it does.

## Tests

Every dead guard was confirmed dead first: make the forbidden content
render, watch the old form stay green. `compatibility: flannel` in
kilo's values, `--gateway-api=true` in place of the bare flag,
`--proxy-fqdn` baked beside the composition flags, the OIDC client
secret in the `mode=None` grafana render. Each one turns the new
assertion red. Anchors checked the same way, with a sidecar rendered
ahead of the workload.

`hack/helm-unit-tests.sh` is green.

### Screenshots

None. This changes test files.

### Downstream repositories

Every changed file is under a `tests/` directory. No templates, no
values, no rendered output.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```
2026-08-14 10:55:05 +03:00
Aleksei Sviridkin 9dc49de1c9 fix(apps): make the post-delete cleanup hooks report failure (#3593)
## What this PR does

The qdrant and harbor post-delete cleanup hooks could not fail. Every
delete was guarded by `|| echo "WARNING: ..." >&2`, neither script set
`-e`, and both ended on an unconditional completion message, so the
container exited 0 on every path. Helm read that as a successful hook,
`helm.sh/hook-delete-policy: hook-succeeded` removed the Job together
with the pod that held the warning, and an uninstall that reclaimed
nothing looked exactly like one that had nothing to reclaim. Written up
in #3566.

Both hooks now count failures and set the exit code at the end. `set -e`
would abort at the first failed delete and skip the rest of the sweep,
which is the opposite of what is wanted: every remaining object should
still be attempted and the outcome reported once. Already-gone objects
stay a success, since every delete keeps `--ignore-not-found`. Each
failure message names the object it left behind. `packages/apps/mariadb`
already uses this shape.

Both hooks also consumed `kubectl get` output directly: harbor by
iterating `$(kubectl get ...)`, qdrant by piping it into `grep`, whose
status the pipeline then reports. A failed listing arrived as an empty
list, and the sweep reported success without having looked. Each listing
is now captured before it is used.

#3566 attributes that shape to harbor's two loops alone. qdrant has it
too: `pvcs=$(kubectl get pvc ... | grep -E ... || true)` reports grep's
status, so a `kubectl get` that failed left `$pvcs` empty and the hook
announced "no data PVCs found; nothing to delete". Both instances are
fixed here.

### The cost

A hook that exits non-zero keeps its Job and its pods, because
`hook-succeeded` does not fire for it. That is the point, the pods hold
the log, but it means a failed uninstall now leaves objects behind where
it previously left none. Nothing collects them on its own:
`before-hook-creation` only fires when the hook runs again, and a
`post-delete` hook runs only on an uninstall, so they stay until a
release of that name is installed and uninstalled again, or until the
namespace goes.

The release itself is not stranded. Helm appends the post-delete hook
error to its error list instead of returning on it, marks the release
uninstalled, purges it from storage anyway, and only then reports
`uninstallation completed with N error(s)`. The failure surfaces once
and the app still goes away.

Both Jobs get one retry instead of the Kubernetes default of six, and
instead of none. Every delete is idempotent, so a second attempt
re-issues an API call rather than re-running any logic, and one short
backoff separates a transient apiserver error from a real one. These
failures become visible for the first time with this change, and a
signal that fires on noise stops being read, including on the day it is
right. Six is the far end of the same mistake: ten minutes of backoff
across seven pods outlasts the hook timeout, so Helm reports that
timeout instead of the hook's own error.

`packages/apps/clickhouse` and `packages/apps/mariadb` already exit
non-zero from their cleanup hooks and set no retry limit, so a real
failure in either spends the full backoff and reaches the operator as a
timeout rather than as the error the hook wrote. Adding the exit code
without the cap reproduces that.

One limitation is deliberate. harbor's ACME sweep cannot tell "this kind
is not served" from "I could not look". If cert-manager is torn down
before the release, `kubectl get challenge` fails and the uninstall is
reported failed with nothing to sweep. cert-manager is a hard dependency
of the bundle harbor ships in, so the window is narrow, and a
CRD-existence probe would add a second silent-skip path of its own: it
would swallow a discovery failure the way this change stops swallowing a
listing failure.

### Tests

`hack/app-cleanup-hooks_test.bats` renders each hook's script out of its
chart and runs it against a fake `kubectl`, because the property under
test is the container's exit code and matching against manifest text
cannot establish it. There are eight failure paths across the two hooks,
six in harbor and two in qdrant, and each has a case that goes red when
that one path alone is reverted to a swallowed warning. Also covered: a
clean delete, and an object that is already gone in both its shapes, an
empty listing and a `NotFound` on the delete itself.

The fake `kubectl` models `--ignore-not-found` instead of exiting 0
unconditionally, so dropping that flag from a hook turns the
already-gone cases red. Message assertions are anchored on the hook's
own `ERROR: ` prefix, because the fake names the object in its own
diagnostic too, and an unanchored match is satisfied by the fixture.

Two manifest fields are pinned in each chart's existing helm-unittest
suite, where manifest fields belong. The retry cap is one. The other is
`helm.sh/hook-delete-policy`: adding `hook-failed` to that annotation
hands the failed Job straight back to Helm to delete, which undoes the
whole change. I checked that before adding the assert, and the reversal
left both suites fully green.

The qdrant Chainsaw suite explained its post-delete pod collector by
quoting the guard string this change removes, and concluded that Helm
always collects the pod away. That conclusion inverts here, so the note
is rewritten, along with the sentence in the harbor suite that points at
it.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. It touches two app chart templates and adds one file
under `hack/`. No package added, renamed or removed, no
`values.schema.json`, no `values.yaml` default, no version enum, no
`ApplicationDefinition`, no namespace, no node prerequisite, and neither
`hack/package.mk` nor `hack/update-crd.sh`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(apps): the qdrant and harbor post-delete cleanup hooks now exit non-zero when a delete or a listing fails, instead of reporting success on every path. `helm uninstall` reports the failure rather than returning clean, and the hook Job and its pods stay in the namespace so the log survives to be read. Nothing collects them on its own: they remain until a release of that name is installed and uninstalled again, or until the namespace is removed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved Harbor and Qdrant cleanup reliability by detecting listing
and deletion failures.
- Cleanup jobs now retry once and report failures with a non-zero
status.
- Cleanup continues processing remaining resources after individual
failures.
- Successful cleanup remains safe and repeatable, including when
resources are already missing.
- Harbor cleanup remains limited to resources associated with the
configured Harbor host.

- **Tests**
- Added comprehensive coverage for successful, repeated, partial, and
failed cleanup scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 10:55:01 +03:00
Aleksei Sviridkin d2551b6ea7 fix(keycloak): keep KeycloakRealmUser roles when spec.roles is unset (#3531)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

The vendored keycloak-operator image is built from a pinned upstream
commit with `images/keycloak-operator/patches/*.diff` applied at build
time. This patch is a backport of epam/edp-keycloak-operator#372 onto
that pin, so it is dropped in the same commit that moves `COMMIT_REF`
past the upstream merge. The bug is epam/edp-keycloak-operator#371.

Under the default `full` strategy the operator treats `spec.roles` and
`spec.clientRoles[].roles` as authoritative sets: after adding what is
missing, it deletes every mapping the spec does not list. A resource
that manages the user record and leaves those fields out therefore loses
all of its realm roles on the first reconcile, `default-roles-<realm>`
included, and with it `offline_access`, `uma_authorization` and the
`account` client roles. The user keeps logging in and stops being able
to do anything.

Upstream fixes this in two halves that only work together. The
controller returns early when the desired list is nil, so an omitted
field means "not managed" while an explicit `roles: []` keeps its
meaning of "remove every role". The json tags on `Roles` and
`ClientRoles[].Roles` lose `omitempty`, because with it an explicit
empty list is erased the first time the operator writes the object back,
which adding its own finalizer already does, and comes back
indistinguishable from an omitted field. Take that half away and the
distinction holds only until the operator next touches the object.

All eight files from the upstream PR are here. Six apply to the pin
unchanged. `sync_user_roles.go` and its test are rewritten against the
pinned context, where the generated client package is still named
`keycloakv2` (upstream renamed it to `keycloakapi` after the pin, which
is the only reason those two do not apply as-is). Tests match upstream:
four controller cases for unset and empty realm and client roles, two in
`api/v1` for the serialization round trip.

Checked by extracting the pinned tarball, applying every `.diff` in the
patches directory in the order the Dockerfile globs them, then `go build
./...` and `go test ./internal/controller/keycloakrealmuser/...
./api/...`, all green. Reverting each of the three conditions in turn
fails exactly the tests that cover it, and nothing else.

One caveat a reader deserves up front: this patch does not reach any
cluster yet. No CI path rebuilds this package, so `hack/build-matrix.sh`
returns an empty matrix for a change confined to it, and `values.yaml`
keeps pointing at an image built by hand in March. The same happened to
`9c5fbb69a` and `5b9b420ff`, which edited this Dockerfile and left the
pin alone, so the running binary is already older than the tree says.
That gap is tracked in
https://github.com/cozystack/cozystack/issues/3416 and is deliberately
not addressed here: the fix for it is a choice between adding these
packages to the build list and dropping the in-repo build, and the first
option wakes a known mirror gap for split-host references. Landing the
patch now means it is in place the moment that is resolved.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff. It touches one build-time patch
file under `packages/system/keycloak-operator/images/`, adds no package
under `packages/apps/` or `packages/extra/`, and changes no values key,
schema, enum, default, CRD, `ApplicationDefinition`, namespace, label,
annotation, metric, release asset, variant or node prerequisite. Nothing
in the map fires.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(keycloak): a `KeycloakRealmUser` that omits `spec.roles` or leaves `roles` out of a `spec.clientRoles` entry no longer has the matching Keycloak role mappings deleted under the default `full` reconciliation strategy. Users keep `default-roles-<realm>` and the `offline_access`, `uma_authorization` and `account` roles it grants. Setting an explicit empty list still removes every role.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved Keycloak role management for realm users and client roles.
  * Leaving a role list unspecified now preserves existing roles.
* Explicitly providing an empty role list now removes all assigned
roles.
* **Documentation**
* Updated API and resource documentation to clarify role-list behavior.
* **Tests**
* Added coverage for omitted, empty, managed, and unmanaged role
configurations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 10:54:21 +03:00
Aleksei Sviridkin 3d4e8c07e9 test(tenant): anchor the operator-gate absence and refresh a spent reason (#3661)
## What this PR does

Two tests in
`packages/apps/tenant/tests/vmo_webhook_dependson_test.yaml` say the
seaweedfs and gateway HelmReleases deliberately carry no operator gate,
and assert that with `notExists: spec.dependsOn` and nothing else. A
bare absence assert also passes when the template renders no document at
all, so both stayed green with those two releases switched off entirely.
With `seaweedfs: false, gateway: false` the suite still reported 5 of 5
passing; pinning `metadata.name` next to the absence turns the same run
into 2 failures.

An earlier check on these asserts injected a `dependsOn` and watched
them redden. That proves they catch an added gate, which is the other
direction. For a negative assert the direction that matters is the
document going away, and that one was never covered.

Pairing `equal: metadata.name` with the absence is not a new idiom
introduced here. `no_upgrade_force_test.yaml`, in the same directory,
already anchors all seven of its releases that way, and this brings the
last two suites in the chart into line with it.

Both suite headers also carried a reason for their `templates:` list
that has expired: that rendering the whole chart fails because
`keycloakgroups.yaml` indexes `.Values._cluster` unguarded. #3636 added
the guard, so a no-values render of the chart now succeeds. The list is
still load-bearing, for the reason `no_upgrade_force_test.yaml` already
records — it keeps an unrelated template breaking out of this suite's
result. Checked by adding a nil dereference to a template neither suite
asserts on: both suites stay green with their lists and error without
them.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. It touches two files, both
under `packages/apps/tenant/tests/`, and no chart, template, values file
or tooling. Nothing in the map keys on test files.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```
2026-08-14 10:54:15 +03:00
Aleksei Sviridkin 340483fdf0 fix(hack): declare the shared package targets and the root test target phony (#3353)
## What this PR does

`hack/package.mk` set `.PHONY` with `=` instead of `:`:

```make
.PHONY=help show diff apply delete update image
```

That assigns a variable which happens to be named `.PHONY`; it declares
nothing. `make -p` prints it back among the variables (`.PHONY = help
show diff apply delete update image`) rather than as a special target,
so none of the targets `package.mk` defines was phony in any package
that includes it.

Nothing in the tree collides with those names today, so this is
preventive rather than a fix for present breakage. The failure it
forecloses is quiet: with a directory named `image` beside a package
Makefile, `make image` prints `make: 'image' is up to date.` and exits 0
without building. A file of that name is enough on its own where the
rule has no prerequisites; where it has one, the recipe keeps running
until that prerequisite is shadowed too.

The list changed as well. `suspend`, `resume`, `check` and `clean` are
defined in this file and were missing, and `check` is the one that
matters most, since it is the prerequisite that keeps `show`, `diff`,
`apply` and `delete` rebuilding. `update` and `image` are gone, because
`package.mk` does not define them; the including package Makefile does.
Naming an undefined target in `.PHONY` creates it as an empty target,
turning `No rule to make target` (exit 2) into `Nothing to be done`
(exit 0) wherever the rule is missing. That reaches CI too:
`hack/build-matrix.sh` parses the root Makefile `build:` target and runs
`make -C <pkg> image` per unit, so a package arriving in that list
without an `image:` rule would go from a red job to a green one that
builds nothing.

`.DEFAULT_GOAL` on the line above keeps its `=`. That one is a real
variable, so the assignment form is right there. A comment now says
which of the two lines is which, since they read alike and one of them
just changed shape.

`hack/package-mk-phony.bats` covers the declaration and is picked up
automatically by the `hack/*.bats` glob in `bats-unit-tests`. Five of
its six tests cover `package.mk`: that the declared names are really
phony with a file of each name sitting beside the Makefile; a control
that mutates a copy back to the assignment form and requires the probe
to report up-to-date, so the first cannot pass vacuously; that the
declared list and the defined targets are the same set, checked both
directions; and two pinning the `update`/`image` exclusion, one
requiring the loud failure to survive and one extending the list in a
copy and requiring it to disappear. The loud-failure test pins both the
exit status 2 and the diagnostic. A wrapper that kept make's wording
while returning something else would pass a check on the message alone.

One point decides where the guarantee actually lives. The first test
reads its names from the `.PHONY:` line, so it cannot see a name dropped
from that line at all. The set-equality test is what guards every name's
presence.

One known gap is left in place. The first test decides by matching
make's `is up to date`, which make prints only for a target that has a
recipe, and a recipe-less alias never draws it: with no prerequisite
both forms print `Nothing to be done`, and with a prerequisite that is
itself out of date, which every target in this file is for being phony,
the working form prints that prerequisite's recipe while the broken form
prints `Nothing to be done`. The probe separates neither pair, so such a
name would pass the first test either way. Measured on both shapes, not
assumed. Every target `package.mk` defines has a recipe, so nothing is
uncovered today; the control test does not close the gap either, since
an alias is a defined target and the control requires `is up to date`
from it, which turns it red against the probe rather than against the
alias. Worth closing if an alias is ever added.

One more for whoever edits the suite next: `hack/cozytest.sh` rewrites
every bare `}` at column 0 into `return 0` followed by `}`. That is
aimed at `@test` blocks, but it does not distinguish them from a plain
shell function, so a helper factored out of these tests would silently
become unable to return a non-zero status. Tests 1 and 2 repeat a loop
rather than share one for reasons of their own, but this is the trap
waiting for anyone who merges them.

Two properties of the suite are worth stating, because both are easy to
get wrong in the other direction. The set-equality test extracts the
defined targets with a pattern that accepts a rule naming several
targets at once, so `suspend resume: check` keeps both names in the set;
anchoring the colon to a single name drops every name on such a line,
and for a target nothing else depends on that goes unnoticed. And the
control test probes the targets `package.mk` defines rather than the
names on its `.PHONY:` line, because `is up to date` is only a
meaningful answer for a name that has a rule: a declared-but-undefined
name prints `Nothing to be done` in both the working and the broken
form, so probing it in the control fails against the probe, while the
set-equality test holds the message that names the defect.

Last one, about how the tests find the repo. All six read it as
`repo=$(pwd)`, which assumes the runner starts at the repo root. That
holds for `make bats-unit-tests` and for the invocation the header
documents, so nothing is broken, and the assumption is the ordinary
shape in `hack/` rather than an outlier: about half the bats files there
resolve repo-relative paths against the cwd the same way, several
stating it in their own headers, and this is not even the only one
spelling it with `$(pwd)`. What is worth naming is the failure mode:
started elsewhere, these tests go red with `no '.PHONY:' declaration
found in hack/package.mk`, which reads like a real finding rather than a
wrong directory.

The root `Makefile` had the same defect, and there it had already fired.
`test` was not on its `.PHONY:` line and a `test/` directory sits beside
it, so `make test` printed `'test' is up to date` and exited 0 without
running anything, while `docs/agents/overview.md` advertises that
command as the way to run the full suite. CI never used it: the workflow
calls `test-controllers` and drives e2e out of `packages/core/testing`,
so the cost fell on anyone following the docs by hand. One name on the
declaration closes it, and the suite gained a sixth test that goes red
without it. Every other undeclared target in that file is unshadowed
today, so none of them is reachable the way `test` was, and they are
left alone here. #3709 covers the same class in the four
`packages/core/` Makefiles, which include only `hack/common-envs.mk` and
so are out of reach of this change entirely.

### Downstream repositories

`scripts/package.mk` in cozystack/external-apps-example was a
byte-for-byte copy of `hack/package.mk` (identical sha256 before this
change), carrying the same inert `.PHONY=` line. This PR splits the two
files, and the copy was fixed separately. It landed the same nine names,
and `update` and `image` are absent there for the reason they are absent
here: `scripts/package.mk` does not define them either, and neither does
any Makefile that includes it. What separates the two files now is the
two-line comment added here, and nothing else.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [x]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up: https://github.com/cozystack/external-apps-example/pull/4
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Fixed task declaration handling so supported commands are correctly
recognized and executed.
* Updated available commands by replacing `update` and `image` with
`suspend`, `resume`, `check`, and `clean`.
* **Tests**
* Added coverage to verify command declarations, prevent regressions,
and ensure invalid or missing commands report errors correctly.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 10:54:10 +03:00
Aleksei Sviridkin d978a371a2 test(hack): cover the .PHONY declarations this tree relies on
`.PHONY=a b c` and `.PHONY: a b c` differ in the `=` against the
`:`, and only the second declares anything. The first defines an
ordinary variable named `.PHONY` that nothing reads, leaving every
listed target file-backed. A file or directory of that name beside a
package Makefile then stands in as the product of the rule: for a
target with no prerequisites make reports it up to date, skips the
recipe and exits 0, and a target whose prerequisite is itself
file-less keeps rebuilding until that prerequisite is shadowed too.
The neighbouring `.DEFAULT_GOAL=help` is a real variable and is
correct with `=`, so the two lines read alike while only one of them
works, which is what let the inert form sit there unnoticed.

Add hack/package-mk-phony.bats, six tests, picked up automatically
by the hack/*.bats glob in `bats-unit-tests`.

Three cover the declaration itself. The first reads the names off
the `.PHONY:` line rather than repeating them, plants a file named
after each beside a stub Makefile, and requires `make --dry-run` to
print the recipe instead of "is up to date". A negative control
mutates a copy back to the assignment form and requires the probe to
report up-to-date, over the targets package.mk defines rather than
the names it declares, because "is up to date" is only a meaningful
answer for a name that has a rule. The third keeps those two sets
equal: it requires the declared list and the defined targets to be
the same set. Defined but undeclared
is the direction the list rots in: check, clean, suspend and resume
sat undeclared for as long as the declaration itself was inert.
Declared but undefined is the empty-target trap in its general
form, so a future name cannot slip in the way update and image
once did.

Two more pin what the list deliberately leaves out. package.mk
defines neither `update` nor `image`; the including package Makefile
does, and most packages define neither. Naming them here would
create them as empty targets everywhere, turning "No rule to make
target" into "Nothing to be done". Since the CI build matrix is
parsed from the root Makefile `build:` target and runs
`make -C <pkg> image` per unit, a package reaching that list without
an `image:` rule would go from a red job to a green one that builds
no image. One test requires that loud failure to survive, pinning
both of the observables it is made of: make's exit status 2 and its
diagnostic. They are independent, and a wrapper that kept the
wording while returning something else would satisfy a check on the
message alone. Its control extends the declaration with the two
names in a copy and requires the failure to disappear, so the pin
tracks the list rather than some property of the harness.

Every decoy carries one fixed timestamp. Six of the nine targets take
check as a prerequisite, and apply and delete reach it through suspend
as well, so touching the decoys in list order can leave check newer
than its own dependents. Make then rebuilds them for that reason and
prints a recipe whatever .PHONY says, and the probe reports success
against a declaration that does nothing. Equal mtimes remove the
question in any list order. Measured on GNU Make 4.3: list order left
the per-target assertion inert for up to six of the nine, moving with
where the clock tick fell, while 3.81 fits the loop inside a single
timestamp and shows none of it. The control test plants its decoys
the same way for the same reason, and goes red if that is undone in
its own loop. The two loops are independent, so neither guards the
other, and the header is what keeps the arrangement from being
unpicked.

A sixth test leaves package.mk and covers the root Makefile, where this
class had already bitten: `test` is the one target in the tree with a
path of its own name beside it. It probes with --dry-run because the
recipe wants a live cluster, and fails loudly if that path ever goes
away rather than passing on a question with no subject.

`%-update` is not coverable: .PHONY matches target names literally
and does not expand patterns.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-14 01:25:34 +03:00
Myasnikov Daniil c9ba7a3cca fix(platform): give node DaemonSets memory requests and limits
The Talos userspace OOM handler (v1.12+) ranks cgroups with

  memory_max.hasValue() ? 0.0 :
    {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current

and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.

Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.

memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.

Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.

Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.

Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.

Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:47:52 +05:00
Myasnikov Daniil cf87201099 test(monitoring-agents): run this package's helm suite
hack/helm-unit-tests.sh discovers work by probing each package for a
`test` Makefile target, so a package without one is skipped in silence
however many suites sit under tests/.

monitoring-agents had tests/fluent-bit_probes_test.yaml and no target,
so that suite had never run once since it was written. It passes; the
only thing missing was the two lines that reach it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:47:35 +05:00
Myasnikov Daniil ca7d2b0182 docs(platform): explain system component memory limits
Add docs/operations/system-memory-limits.md, recording why system
components carry memory limits. That reasoning is not recoverable from
the values themselves, and is exactly the kind of thing a well-meaning
later change removes as redundant.

The Talos userspace OOM handler (v1.12+, enabled by default) selects
its victim by ranking cgroups that lack memory.max. Three consequences
are each easy to get wrong on their own: only a limit grants immunity,
since a request merely moves BestEffort to Burstable and under
strictCgroupClassOrdering that means "killed second"; every container
in the pod needs one, init containers included, because kubelet sets
pod-level memory.max only when all of them have it; and PriorityClass,
system-node-critical included, has no bearing on it whatsoever,
because the handler walks cgroupfs and holds no Kubernetes client.
That last one is the trap, since setting a priority looks like a fix.

The page also covers the two installer knobs and the constraint that
makes them dangerous to lower (a defaulted limit below a container's
own request fails admission), how to confirm a pod cgroup really
carries memory.max rather than trusting the QoS class, what is
deliberately left uncovered, and how to relax the global-PSI backstop
clause of the trigger on a cluster that trips it persistently.

Every Talos claim is checked against the siderolabs/talos v1.13.6 tag,
which is the version packages/core/talos currently ships, and every
Cozystack claim against the tree.

Ordering: this documents behaviour introduced by the system-namespace
LimitRange and the node-DaemonSet resource changes, so it should not
merge ahead of them.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-13 16:29:48 +05:00
IvanHunters 09dc7f47fb feat(backups): add a native MongoDB (psmdb) backup-strategy driver (#3562)
## What this PR does

Adds a native MongoDB (Percona Server for MongoDB / psmdb)
backup-strategy driver so `apps.cozystack.io/MongoDB` reaches feature
parity with the CNPG, MariaDB and ClickHouse native drivers — logical
backups, restore into a differently-named instance, and point-in-time
recovery — all through the unified `BackupJob` / `RestoreJob` /
`BackupClass` / `Plan` interface (closes #3545).

The driver orchestrates the operator's `PerconaServerMongoDBBackup` /
`PerconaServerMongoDBRestore` CRDs:

- **BackupJob**: creates a one-shot `PerconaServerMongoDBBackup`, polls
`status.state`, and persists the S3 destination + storage descriptor (by
reference — Secret name only, never a raw credential) onto the Cozystack
`Backup` artifact.
- **RestoreJob**: drives `PerconaServerMongoDBRestore` via
`backupSource` (read back from the source backup status, with a snapshot
fallback for a reaped operator CR) so the same artifact restores
in-place **or** into a differently-named instance. PITR is exposed as
`spec.options.recoveryTime` (RFC3339), matching the CNPG driver's option
exactly; unrecognised `spec.options` keys are surfaced as a
`UnknownRestoreOption` Warning event rather than silently dropped.

Because psmdb resolves the backup storage by name against the cluster's
`spec.backup.storages`, the driver names the chart's storage
(`s3-storage`) via the `cozy-default-mongodb` Strategy rather than
injecting it, and surfaces a clear `Ready=False` precondition when a
target has backups disabled instead of hanging until the deadline.

**Wiring**

- `api/backups/strategy/v1alpha1`: `MongoDB` strategy type + generated
CRD/deepcopy.
- `internal/backupcontroller`: `psmdbstrategy_controller` + `psmdbtypes`
/ `mongodbapp` partial-schema mirrors; BackupJob/RestoreJob dispatch,
supported-kinds, cleanup-on-delete branch, and scheme registration.
- `backupstrategy-controller`: `cozy-default` MongoDB route,
`cozy-default-mongodb` Strategy CR, and `psmdb.percona.com` RBAC
(get/list/watch, +create on backups/restores, no delete — the driver
does not own the S3 archive).
- `apps/mongodb`: a `backup.insecureSkipTLSVerify` value (psmdb's s3
storage has no CA-bundle field) so backups can target Cozystack's
self-signed in-cluster seaweedfs. Default `false`, fully
backward-compatible; regenerated schema/README/RD.

**Tests + e2e**

- Unit tests for the driver: applicationRef validation,
ensure-idempotency (backup + restore), the backups-enabled/storage
precondition gate, `backupSource` resolution (live-preferred / snapshot
fallback / no-destination), snapshot round-trip, the producer-seam
(`createMongoDBBackupArtifact`) driver-metadata, the cleanup-on-delete
no-Velero-fall-through guard, and `recoveryTime` → psmdb-UTC mapping +
unknown-option reporting.
- `examples/backups/mongodb/` + a `mongodb-2-backup-roundtrip` Chainsaw
suite that drives `run-all.sh` end-to-end (a sentinel document
round-trips through S3 into a restored copy while the source stays
untouched), per `docs/agents/e2e-testing.md`.
- `docs/operations/backup-classes.md` updated (new route, the
app-owns-the-storage model, and the `recoveryTime` PITR usage).

Verified end-to-end on a live dev cluster against the platform-default
`cozy-backups` bucket: backup via `cozy-default` reached the bucket and
the Cozystack `Backup` went `Ready`, and a to-copy `RestoreJob` restored
the sentinel into a differently-named instance with the source left
intact.

### Screenshots

N/A — no UI changes (controller, chart values, CRD, docs and tests
only).

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file by file
against the diff:

- **cozystack/website** — the only app-facing change is
`packages/apps/mongodb` (a new `backup.insecureSkipTLSVerify` value +
regenerated `README.md`). The managed-app reference page is regenerated
from the package `README.md` by the release bot for apps already in the
website `Makefile` (mongodb is one), so it refreshes automatically on
the next tag — no manual follow-up. `docs/operations/backup-classes.md`
is not a website-mirrored doc in the trigger map.
- **cozystack/terraform-provider-cozystack** — the one candidate trigger
("a field added to an app's `values.schema.json`") lands inside the
mongodb `backup` block, which the provider's
`internal/provider/mongodb_schema.go` explicitly states it does **not**
manage ("The deprecated backup block, the bootstrap restore block, and
detailed shardingConfig are not managed"). No enum, default,
`release.prefix`, Secret/Service name, or hand-typed CRD (`Plan`,
`RestoreJob`) changed; the new `strategy.backups.cozystack.io/MongoDB`
CRD is opaque to the provider. Not affected.
- All other repositories — no matching trigger (no `hack/` layout or
make-target change, no namespace rename, no `ApplicationDefinition`
semantics change, no `packages/core/platform` values, no telemetry
metric or `cozy-proxy` label rename).

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(backups): add a native MongoDB (psmdb) backup-strategy driver — logical backup to the cozy-default BackupClass, restore into a differently-named instance, and point-in-time recovery via spec.options.recoveryTime
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added MongoDB backup and restore support, including scheduled and
on-demand backups, restores to new or existing clusters, and
point-in-time recovery.
* Added configurable S3 TLS certificate verification for MongoDB
backups, defaulting to secure verification.
  * Added MongoDB support to the default backup configuration.

* **Documentation**
* Added MongoDB backup and restore examples, workflows, and operational
guidance.

* **Tests**
* Added automated coverage for MongoDB backup, restore, cleanup, and
end-to-end scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-13 11:18:46 +03:00
Aleksei Sviridkin 61a236cd96 feat(ci): auto-apply triage labels to issues (#3574)
## What this PR does

Issues that arrive through the API or `gh issue create` bypass the issue
templates, so they carry no `triage/*` label and nothing in the
repository adds one afterwards. Listing every open issue through the
same endpoint the workflow uses, `gh api --paginate
"/repos/cozystack/cozystack/issues?state=open&per_page=100"`, returns
567 entries, 171 of them pull requests, at the time of writing. That
leaves 396 open issues, and 252 of them carry no triage label at all.
Every count in this description is a point-in-time reading of a backlog
that keeps moving, so a later run of the same query will differ by a
few. The triage queue is impossible to filter on in that state, and
there is no way to see how much of the backlog nobody has looked at.

This adds a workflow that labels an issue when it is opened or reopened,
plus a daily sweep that backstops labels removed by hand and works
through the existing backlog. An issue somebody has assigned, or marked
with one of `priority/critical-urgent`, `priority/important-soon`,
`priority/important-longterm` or `epic`, gets `triage/accepted`, since a
maintainer has already looked at it. Everything else gets
`triage/needs-triage`. An issue already carrying any `triage/*` label is
left alone, so the sweep never overwrites a decision someone made by
hand, and pull requests returned by the same listing endpoint are
skipped. Against the listing above, the sweep applies `triage/accepted`
to 39 issues and `triage/needs-triage` to 213, and touches none of the
144 that are already triaged.

Deleting a triage label by hand does not undo any of this. An issue left
carrying no `triage/*` label is relabeled by the next sweep, so a triage
decision is changed by replacing the label. `AGENTS.md`'s labeling
section now says so, because that is the file somebody reads before
labeling by hand.

Two conditions decide what counts as a signal. Membership of
`stale.yaml`'s `exempt-issue-labels` is necessary: `triage/accepted` is
itself on that list, so applying it does more than reset a clock, the
issue stops being auto-closable, and a signal that is not already exempt
would hand out that permanent reprieve as a side effect of being
labeled. `priority/backlog`, the one tier `stale.yaml` deliberately
leaves reapable, is excluded for exactly that reason.

Membership is not sufficient, which is why the signal list is not simply
the exempt list. `lifecycle/frozen` is exempt and still not a signal: a
`lifecycle/*` label states what the stale bot may do with an issue and
says nothing about whether anybody reviewed it. Freezing an untriaged
issue to keep the bot off it is the case where reading it as accepted is
wrong. Of the five open issues whose only signal would have been
`lifecycle/frozen`, one is blocked on an upstream project and carries
`help wanted`, and one is an older feature request that an open epic now
covers. Neither is ready to be worked on, and stamping either accepted
would have kept it out of triage for good, since the sweep never
revisits an issue that already carries a triage label. `epic` passes the
same test from the other side: it is a maintainer-authored planning
artifact rather than an inbound report, and all eight open issues whose
only signal is `epic` are roadmap trackers that `needs-triage` would
describe falsely. A contract test pins the rule and not the instance, so
no label from the `lifecycle/*` namespace can quietly become a signal
later. `security/*` stays out on the same grounds, though the case is
closer: those labels record what is known about a vulnerability rather
than whether anyone triaged the report, and the cost of being wrong runs
the cheap way, since a `needs-triage` on a confirmed report is one click
to remove while a wrong `accepted` is permanent because the sweep never
revisits.

One caveat stated plainly: the signals stand for a maintainer decision
already recorded on the issue, which is a weaker claim than the one
`triage/accepted` makes in `.github/labels.yml`, and
`priority/important-longterm` says outright that the work may not be
staffed. Of the two labels this workflow can write it is still the
closer one, since `needs-triage` on an issue somebody already
prioritised or took asks for a triage that has happened, and a
maintainer who disagrees replaces the label in one click. An assignee is
the single signal that is neither a label nor already exempt, so an
issue whose only signal is an assignee does gain a permanent reprieve.
That is the intended reading rather than a side effect: somebody took
ownership, which makes it a known long-tail task rather than an
abandoned one, and it is the same argument `stale.yaml`'s own comment
makes for exempting `triage/accepted` in the first place.

Three operational details. First, the sweep paces its writes a second
apart and caps a run at 400, because GitHub allows 80 content-generating
requests a minute and 500 an hour, this client carries no throttling
plugin and no retries, and a throttled write answers `403`, which is
exempt from retry anyway. Unpaced, the first pass over the backlog would
not slow down when throttled, it would fail the job partway and leave
the backlog half labeled. Whatever a run leaves behind stays unlabeled,
so the next one picks it up. Second, adding a label bumps an issue's
`updated_at`, which `stale.yaml` reads as activity, so the first sweep
restarts the 60-day staleness clock on every issue it writes to, all 252
of them, and un-marks the 32 that carry `lifecycle/stale` today. Both
are deliberate and one-time: the backlog gets triaged now and ages again
from there, rather than being closed unread while it is still unlabeled.
Third, a run writes a job summary with the counts, the issue numbers
under each label, and both remainders it can leave behind: the write
cap's deferral, and how much of the listing it never reached when a
refusal stopped it early. A sweep over the backlog logs a line per
issue, and the summary is what makes the outcome readable without paging
through the 252 the current backlog produces.

A write that fails does not take the rest of the run with it, but the
two failure shapes are told apart rather than counted together. Any
status other than a `403` or `429` lets the run carry on, most often a
`404` or `410` from an issue transferred or deleted mid-sweep, since a
failure about one issue says nothing about the next. A `403` or `429`
means the token lost `issues: write` or the pacing stopped being enough,
which applies to every write left in the run, so the sweep stops at the
first one and the job fails loudly. Repeating a rejected request up to
the cap is what gets an integration throttled harder.

`workflow_dispatch` takes a `dry_run` input that logs what would be
labeled without writing anything. It is on by default, so a manual run
writes nothing until somebody unchecks the box, and `schedule` is the
one event that writes without being asked. The cap applies in dry-run
too, so a dry run predicts what a real one would do instead of printing
a longer list nobody gets. The job asks for `issues: write` and
everything else is read-only, the action it uses is pinned by digest,
and a 20-minute timeout keeps a hung call from holding the queue for the
default six hours.

No pull request lane can exercise this workflow, so its first real
execution is a sweep over every open issue with nothing having run it
before. `hack/issue-triage-contract.bats` pins the executable lines that
bound that blast radius: the pull-request skip, the already-triaged
skip, the pacing rate and the write cap, the job timeout, the failure
handling, the digest-pinned action, the token scopes, the serialised
queue, the dry-run default and the derivation that does not depend on
it, the cron ordering against `stale.yaml`, that both labels it writes
exist in `.github/labels.yml`, that no accepted-signal label sits
outside `stale.yaml`'s exempt list or inside the `lifecycle/*`
namespace, and that the summary is written with the `await` that makes
it flush and accounts for every outcome a run can end on. Each assertion
was checked against a mutated copy of the workflow to confirm it goes
red rather than merely existing, with the mutations run through
`hack/cozytest.sh` rather than `bats`. The two runners disagree about
negated assertions, so a pin that `bats` reports as live can be dead
under the runner CI actually uses, and every negative assertion in the
file captures grep's status instead of leading with `!`.

Suggested first step after merge: run the workflow by hand and read the
summary before the 05:53 cron arms itself. The box arrives checked, so
that run costs nothing.

### Screenshots

Not a UI change, so there is nothing to show.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff. The diff adds two files, `.github/workflows/issue-triage.yaml` and
`hack/issue-triage-contract.bats`, and appends one line to `AGENTS.md`.
No row matches: it adds and renames no package, changes no chart values,
no schema, no version enum, no `ApplicationDefinition`, no CRD, no
namespace, no variant or bundle, no label or annotation any downstream
repository matches on, no metric and no release asset. Its `hack/` rows
name `hack/upload-assets.sh`, `hack/e2e-prepare-cluster.bats`,
`hack/package.mk`, `hack/update-crd.sh` and `hack/common-envs.mk`, which
downstream repositories copy, anchor on or mirror by hand, plus one row
for moving or renaming anything under `hack/`; a new bats file added
next to them is none of those. The map has no row for the agent
documentation, and no downstream repository copies `AGENTS.md`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(ci): new and reopened issues now get a `triage/*` label automatically, and a daily sweep labels any open issue that has none: `triage/accepted` when it is assigned or carries an `epic` or a priority above `priority/backlog`, `triage/needs-triage` otherwise.
```




<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added automated issue triage triggered by new or reopened issues,
scheduled reviews, or manual runs.
* Issues can be labeled as needing triage or accepted, with pull
requests and already-triaged items excluded.
  * Added dry-run support and reporting for triage results.

* **Documentation**
* Added guidance for replacing existing triage labels when making
decisions.

* **Tests**
* Added coverage for workflow behavior, permissions, scheduling, failure
handling, and safety limits.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-13 02:41:54 +03:00
Aleksei Sviridkin 59878850f9 test(e2e): capture worker CPU throttling counters on node-join failure (#3723)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

When a tenant worker misses the node-Ready deadline, the failure bundle
currently cannot tell two different causes apart: a worker held at its
own CFS ceiling, and a worker losing host CPU it was entitled to. Both
look identical from outside the Pod, and node-level utilisation answers
neither, because it shows what the node used rather than what the
container was allowed.

This adds a collector that reads the CFS counters and the ceiling itself
from the kubelet's cAdvisor endpoint, for every tenant worker on every
node in the sandbox. The counters are the only place that difference is
recorded, and nothing else in the tree collects them for these workers.

The motivating measurement is on nightly run 31290999592: the `compute`
container reports 0.968 and 0.920 core against a `limits.cpu` of 1,
which is 92-97% of a one-core ceiling. Whether the container ever
actually met that ceiling is exactly what the artifact cannot say, and
what these counters answer.

Most of the change is the failure taxonomy rather than the read. An
empty capture is the dangerous outcome here, because zero bytes reads as
a container that never hit its ceiling -- the conclusion the collector
exists to stop a reader reaching by default. So every way the capture
can come up short gets its own sentence: the kubelet was not read, the
read was cut short part way, the metric stream could not be read back on
this runner, the kubelet answered and carried no series for this
namespace, the namespace was there and the worker Pods were not. The one
non-failure outcome is written down too: a container with no CPU limit
puts a single line on the wire, which is also what a truncated read
leaves behind, so it says which one it is.

### Ordering

The collector runs first among the gated collectors, ahead of the guest
serial console. Its ceiling is four bounded reads, about 100s of a 480s
phase budget, so on a slow run the console capture starts that much
later. That is a deliberate trade and not a side effect of putting the
cheapest first: cost is not what settles the order, survivability of the
answer is. The console evidence cannot be re-taken, and this collector's
question has no other answer in the tree, so both go ahead of the
captures whose state is partly recoverable from reads above them.

### Known limits, named rather than left to be found

- The outcome ladder enumerates combinations of read status, filter
status and result emptiness. Every new source of partial output needs a
new rung, and that has already happened twice while this branch was in
review. Deriving the label from the three axes instead would make a new
source land in an existing axis.
- The uncapped note is decided over the whole per-node file, so one
quota line anywhere in it suppresses the note. That is the conservative
direction -- it never claims uncapped over a file that shows a ceiling
-- but it makes the note rare.
- The messages say `see read-error.log` while the file on disk is
`<node>.read-error.log`. That is the convention the neighbouring
serial-console collector already uses, and changing one side alone would
make the two inconsistent.
- The `mkdir -p` for the report directory is unchecked, as it is in four
neighbouring collectors in the same file. Worth fixing across the file
rather than in one collector.
- The section letters in the failure block no longer track emission
order. They already did not before this change; this adds one more entry
to the sequence.
- The count of bats files sourcing the library, in
`docs/agents/e2e-testing.md`, is corrected here from three to seven.
Nothing pins it, so it will drift again.
- The node walk iterates `for node in ${nodes}` unquoted, so pathname
expansion applies alongside word splitting. Node names are RFC1123 so it
cannot bite today; `while IFS= read -r node` would make that structural
instead of incidental.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(e2e): capture tenant worker CPU throttling counters and their CFS ceiling when a node-join fails, so a worker held at its own limit can be told apart from one losing host CPU
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added worker CPU-throttling diagnostics to Kubernetes troubleshooting
reports.
* Reports now identify CPU quota gaps, partial reads, filtering issues,
and incomplete results.
* Added safeguards for bounded node traversal and configurable
collection budgets.

* **Bug Fixes**
* Improved failure handling when diagnostic commands, timeouts, or
metric reads are unavailable.

* **Tests**
* Expanded coverage for CPU-throttling capture, node-join diagnostics,
timeout behavior, configuration validation, and partial failures.

* **Documentation**
* Updated diagnostic collector counts and collection-order
documentation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-13 02:24:50 +03:00
Aleksei Sviridkin 3ad4d452fc fix(tests): fail when a chart runs no Helm unit test suites (#3632)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

`helm unittest` is fail-closed on a suite it cannot parse and on one
declaring `tests: []`, and fail-open on suites that are not there at
all: it prints `Test Suites: 0 passed, 0 total` and exits 0.
`hack/helm-unit-tests.sh` judges each package by that exit code alone,
so a chart whose suites were deleted, moved, or renamed past the
`tests/*_test.yaml` glob reports success having asserted nothing.

That is measurable in both directions rather than argued. Move
`packages/system/velero/tests/velero_test.yaml` aside and run the script
as it exists on main today:

```text
Running tests in packages/system/velero
helm unittest .

### Chart [ cozy-velero ] .


Charts:      1 passed, 1 total
Test Suites: 0 passed, 0 total
Tests:       0 passed, 0 total
```

The run ends with `All Helm unit tests passed.` and exit 0. The chart
passed, and the count of things it checked was zero. With this PR the
same tree exits 1 and names the package and the reason. Neither
`--strict` nor an explicit non-matching `-f` glob changes this on its
own; all three forms exit 0, checked against plugin 1.1.1.

So the script now reads the captured output and fails the package when
the zero-suite line appears.

Matching that line names a cause and a remedy where it applies, but the
set of ways to run nothing is not closed. A recipe that never invokes
`helm unittest`, and a `test:` rule that make finds nothing to do for,
print no marker of any kind and exit 0, so no additional match reaches
them; enumerating absences leaves a fresh hole every time a new shape
turns up. So the script also requires the line a real run always emits,
a `Test Suites:` count of at least one. That turns the question round:
every way of asserting nothing exits 0, and "what did this run assert"
is the question that has an answer. All 76 packages the script visits
emit that line today. The tradeoff is that a `test` target which
legitimately runs no `helm unittest` would need renaming or an opt-out,
which is the same bargain the zero-suite check already strikes.

The discovery gate has the same shape from the other side. `make -C dir
-n test` succeeds against a file-backed target, so a path named `test`
beside a package Makefile whose rule is not phony would make both the
gate and the run exit 0 without the recipe ever firing. The check keys
on what make reports rather than on the path existing, because a package
that declares the target phony runs its recipe whatever sits next to it,
and refusing that would be a false failure. The quoting around the
target name differs between make 3.x and 4.x, so it accepts either
opening quote; anchoring on the trailing quote alone would also match a
sub-make reporting some other target whose name ends in `test`.

That second check is guarding a live gap rather than a hypothetical one.
`hack/package.mk` line 2 reads `.PHONY=help show diff apply delete
update image`, which is a variable assignment and not a target
declaration, so it makes nothing phony. Of the packages the script
visits that define a `test` rule, 18 declare it phony in their own
Makefile; the rest are unprotected if a path of that name ever appears.
No such path exists today. Fixing the shared include is #3353 and is not
this PR.

One deliberate non-change, stated so it is not rediscovered as an
oversight. Running the suite now captures output instead of streaming
it, because a pipeline's exit status in POSIX `sh` reports the last
command rather than `make`. Streaming reads better and loses the status,
so it stays buffered.

The run is pinned to `LC_ALL=C`, since both checks match English wording
and a localized `make` would disarm one of them silently, which is the
failure mode the whole change exists to remove.

The guard is all-or-nothing, which is worth stating plainly. A chart
that loses four of its five suites still reports `Test Suites: 1 passed`
and passes; only total disappearance is caught. Tying the expected count
to what each chart actually has would be a per-chart number to maintain,
and a number maintained in one place while the suites move in another is
the failure this repo has been paying for elsewhere, so the cheap check
that catches the total loss is the one worth having.

One behaviour worth stating because the message does not: a suite in
which every test carries `skip:` reports `Test Suites: 0 passed, 1
skipped, 1 total` and exits 0, and the positive-evidence check refuses
it. That refusal is intended, since a wholly skipped suite asserted
nothing, but the message talks about suites expected under `tests/`,
which in that case are present and skipped on purpose. No package is in
that state today. Partial skips are unaffected: `2 passed, 1 skipped, 3
total` satisfies the check.

Three known gaps in what the change ships, none of them a wrong
statement. The positive-evidence message explains its cause but stops
short of naming a remedy, where the other two messages both end in an
action; the remedy for the legitimate case, renaming the target or
giving it an opt-out, currently lives only in the code comment. That
same message enumerates two ways a run reports nothing, and the
skipped-suite case above is a third, so it reads as a diagnosis where it
is really a list of the common causes. And no document in the tree
states the convention this tightens: `docs/agents/overview.md` still
describes the script accurately as running over every package that
defines a `test` target, but the contract is now that such a package
must also report at least one passing suite. All three are improvements
to make on the next touch of these files rather than reasons to hold the
change.

One adjacent gap stays open, and it is worth naming rather than leaving
someone to assume otherwise. This catches a chart whose suite *files*
vanished; it does not catch a package that loses its `test:` rule along
with them. Nothing ties the presence of `tests/*_test.yaml` to the
presence of a `test` target, so a change that removes both is skipped in
silence, and the script only complains when no package in the tree has a
`test` target at all. The test named `a package with no test target is
skipped, not failed` pins that permissiveness deliberately, because the
many packages that legitimately define no `test` rule would otherwise
turn every run red. Closing it needs an instrument keyed on the suite
files rather than on the Makefile target, which is a separate change.

`hack/helm-unit-tests.bats` adds eight tests, picked up automatically by
the `hack/*.bats` glob in `make unit-tests`. Three pin the new refusals,
and each matches its own message so none can stay green on a refusal
that came from another. Two pin what must not be refused: a phony target
with a colliding path, and a sub-make reporting an unrelated target up
to date. One pins that a chart which does run suites still passes, which
is what stops the refusals being satisfied by rejecting everything. Two
pin behaviour that was already there, namely that a failing package is
still reported by name and that a package with no `test` rule is skipped
rather than failed. That last one builds a tree carrying a second
package that does run a suite: with only the target-less package
present, the script takes an early exit above the failure summary and
the assertion would hold whatever the script had done. The fixtures stub
`make` output instead of invoking helm, so they need no plugin
installed.

Relates to #3453.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(tests): `make helm-unit-tests` now fails a chart that runs no Helm unit test suites, instead of reporting success for a chart whose suite files were deleted, moved, or renamed out of the `tests/*_test.yaml` glob
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved Helm test validation to detect skipped test recipes, zero
discovered suites, and runs without evidence of a passing suite.
* Helm test failures now provide clearer diagnostics, including affected
directories and failed commands.
* Test output is consistently captured and replayed for easier
troubleshooting.

* **Tests**
* Added comprehensive coverage for successful, failed, skipped, empty,
and invalid Helm test scenarios.
* Expanded coverage across package test targets, including phony and
unrelated sub-make behavior.

* **Documentation**
  * Clarified requirements for package test targets and suite reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-12 23:37:43 +03:00
Aleksei Sviridkin 3c009509ba test(cozystack-controller): guard the Secret grant by effective permission (#3627)
## What this PR does

`packages/system/cozystack-controller/tests/rbac_test.yaml` has a test
named "does not grant a blanket wildcard verb on secrets" that does not
guard that. Its only assert is a `notContains` on `path: rules` holding
a rule object. helm-unittest compares every field it is given by value
equality, and `any: true` only changes which fields get compared, so
nothing in that assert bounds what a verbs list may hold. `verbs:
["get", "*"]` passes every spelling of it.

Each row below is a mutation of `templates/rbac.yaml`, confirmed visible
in `helm template` output before I read any verdict.

| # | widened Secret grant | whole suite | the test named as the guard |
| --- | --- | --- | --- |
| 1 | `verbs: ["*"]` on `""`/secrets, the exact object it names | red |
red |
| 2 | `resources: ["*"]`, `verbs: ["*"]` in the core group | red
(positive pin) | green |
| 3 | `apiGroups: ["*"]`, secrets, `verbs: ["*"]` | red (positive pin) |
green |
| 4 | `"*"` appended to the declared verb list | red (positive pin) |
green |
| 5 | the exact named object added as a second rule | red | red |
| 6 | `verbs: ["*"]` plus a `resourceNames` field | red (positive pin) |
green |
| 7 | `deletecollection` spelled out | red (positive pin) | green |
| 8 | declared rule kept, `apiGroups:['*'] resources:['*'] verbs:['*']`
appended | green, 2 passed 2 total | green |
| 9 | declared rule kept, `secrets: [deletecollection]` appended |
green, 2 passed 2 total | green |

Rows 8 and 9 are the bad ones. The declared rule stays untouched, so the
positive pin is happy, the suite reports a clean pass, and the
controller holds every verb on every Secret in every tenant namespace.

The replacement is `hack/cozystack-controller-secret-rbac.bats`, which
reads effective permission instead of manifest text. It renders the
chart, unions the verbs every ClusterRole and Role rule confers on core
Secrets, expands a `*` on any of the three axes, and compares that union
against the declared set. All nine rows above are red under it, and so
are widening the namespaced `Role`, bundling `secrets` with another
resource under `verbs: ["*"]`, and dropping `delete` from the set. That
last one is deliberate: the comparison is an equality rather than a
ceiling, because both reconcilers need every verb in it and the
controller is cluster-scoped, so a widening reaches every tenant.
Reflowing the rule to block style changes the text and nothing else, and
stays green. Green on `main` under both `bats` and `hack/cozytest.sh`.

Expanding wildcards is what makes it hold at all. The chart ends with a
blanket `apiGroups: ['*'] resources: ['*'] verbs: [get, list, watch]`
rule that does reach Secrets, so a guard phrased as "no wildcard near
secrets" would be red on `main` today. That rule predates this branch
and is tracked separately in #3625. This guard stays green either way,
whether the blanket rule is narrowed or left alone, as long as Secret
access matches the declared set.

The positive `contains` pin stays. Value equality is the right primitive
for pinning a rule as written and the wrong one for bounding what the
rule may become.

The same shape exists in
`packages/system/cozystack-api/tests/rbac_test.yaml` and
`packages/extra/seaweedfs/tests/cleanup_client_test.yaml`, and more
weakly in
`packages/system/etcd-operator/tests/selector-fix-hook_test.yaml`, which
matches a scalar inside `rules[0].verbs`. Out of scope here, listing
them so they are not lost.

`BATS_UNIT_FILES` picks the new file up on its own (`hack/*.bats`, no
`e2e-` prefix), so `make unit-tests` runs it with no Makefile change. It
needs helm, yq and jq, the same three
`hack/admin-kubeconfig-invariant.bats` already needs.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. The only entry that looks at
`hack/` is cozystack/ccp, which triggers on moving or renaming something
there, or on changing what a make target does. This adds a file and
touches neither `hack/package.mk`, `hack/common-envs.mk`, nor any
target.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```
2026-08-12 23:37:39 +03:00
Aleksei Sviridkin 94c41d100f fix(hack): fail when a chart runs no Helm unit test suites
helm-unittest is fail-closed on a suite it cannot parse and on one
declaring `tests: []`, and fail-open on suites that are not there
at all: it prints "Test Suites: 0 passed, 0 total" and exits 0.
Measured on plugin 1.1.1 against a throwaway chart, all three
forms.

hack/helm-unit-tests.sh judges each package by that exit code
alone, so a chart whose suites were deleted, moved, or renamed past
the tests/*_test.yaml glob reports success having asserted nothing.
Moving packages/system/velero/tests aside reproduces it: the script
prints "All Helm unit tests passed" and exits 0.

Check the captured output for the zero-suite line and fail the
package when it appears. Neither `--strict` nor an explicit
non-matching `-f` glob does this on its own; both exit 0.

Matching that line names the cause where it applies, but the set of
ways to run nothing is not closed: a recipe that never invokes
helm-unittest, and a `test:` rule make finds nothing to do for,
both print no marker at all, so no further match reaches them. So
also require the line a real run always emits, a `Test Suites:` count
of at least one. Every way of asserting nothing exits 0, and asking
what the run asserted is the question that has an answer. All 77
packages the script visits emit that line today. The tradeoff is
that a `test` target which legitimately runs no helm-unittest would
need renaming or an opt-out, which is the bargain the zero-suite
check already struck. That is a contract contributors have to know
about, so docs/agents/overview.md states it where it describes the
sweep.

Because the check matches helm-unittest's own summary wording, a
plugin upgrade that reworded it would fail every package at once
while blaming each package's Makefile. The error text says so, so
the reader of a tree-wide red has the real cause in front of them
rather than 77 copies of the wrong one.

The discovery gate has the same shape from the other side.
`make -C dir -n test` succeeds against a file-backed target, so a
path named `test` beside a package Makefile whose rule is not phony
would make both the gate and the run exit 0 without the recipe
firing. Of the packages the script visits that define a `test` rule,
18 declare it phony, and no such path exists today. Match what make
reports rather than looking for the path, because a package that
declares the target phony runs its recipe whatever sits next to it,
and refusing that would be a false failure. The
quoting around the target name differs between make 3.x and 4.x, so
the match accepts either opening quote; the trailing quote alone
would also match a sub-make reporting another target whose name
ends in "test".

Running the suite now captures output rather than streaming it,
because a pipeline's exit status in POSIX sh reports the last
command rather than make. The capture file is allocated once and
removed by the EXIT trap that already owns the failure list, and the
run is pinned to LC_ALL=C: all three checks match English wording,
one on make's and two on helm-unittest's, so a localized make would
disarm the first of them silently, which is the failure mode this
guard exists to remove. The locale reaches every package's recipe,
not only make.

Extend hack/helm-unit-tests.bats with eight tests. Three pin the new
refusals; two pin what they must not refuse, a phony target with a
colliding path and a sub-make reporting an unrelated target up to
date; one pins that a chart which does run suites still passes; two
pin behaviour that was already there, namely that a failing package
is still reported by name and a package with no `test` rule is
skipped rather than failed. That last one carries a second package
that does run a suite, because with only the target-less package
present the script takes its early exit above the failure summary
and the assertion holds whatever the script did. The fixtures stub
make output instead of invoking helm, so they need no plugin
installed.

Both tests that assert a package is named in the failure summary
match the summary's "  - " prefix rather than the bare path. The
script announces "Running tests in $dir" before it runs anything, so
the bare path is present in the output whatever the summary does;
matching it asserted nothing, and the summary could stop naming
packages entirely with the suite still green.

The failing-package fixture reports a passing suite count alongside
its non-zero exit, so that the exit status is the only thing that
can decide the outcome. A fixture that fails without printing the
marker is refused by the positive-evidence check first, which leaves
the exit-status branch unpinned. That shape occurs in the tree: a
`test` target that runs helm-unittest and then a second step, a
render-parity script or a vendored-version guard or go tests, emits
the marker and still fails.

The two tests already in that file cover a package reached under
packages/tests, and their fixture printed a marker and nothing else.
That is the shape the positive-evidence check refuses, so the
fixture now also emits the line a real run emits; without it both
would fail for a reason unrelated to what they assert.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:45:22 +03:00
Aleksei Sviridkin b8a5b9ba1d test(e2e): capture worker CPU throttling counters on node-join failure
When a tenant worker misses the node-Ready deadline, the sandbox node
it runs on is routinely at half its capacity with a clean kernel log.
In that state "the host starved the VM" and "the VM was held at its
own CFS ceiling" produce identical evidence: node-level utilisation
reports what the node used, never what the container was allowed. The
CFS counters are the only place the difference is recorded, and
nothing in the tree collected them for the tenant workers, so every
diagnosis of this failure so far had to be inferred from outside.

Read them from the kubelet's cAdvisor endpoint on the node hosting
each worker. Five series carry the whole answer and cAdvisor publishes
all of them per container, so nothing is computed here: the scheduled
and throttled period counts, the time spent throttled, and the quota
and period that form the ceiling. The throttled counters alone say a
container hit some ceiling, not which one, and a VM capped at one core
and a VM capped at eight are the same number without the quota.

Absence of the quota series is itself a reading rather than a gap.
cAdvisor emits it only when the quota is non-zero, so a container
reporting the period and no quota is one running uncapped - which is
one of the two answers this collector exists to distinguish, arriving
without being derived.

Reading the kubelet rather than the container matters beyond
convenience. A reader running inside the container shares the cgroup
it measures, so it slows down in proportion to how much the answer
matters and would need a budget sized for the pathological case; the
kubelet is outside that cgroup. It also drops any dependency on what
the container image ships and on how the host lays cgroups out.

The namespace is not a worker filter and is not used as one: it also
holds the tenant control plane, whose apiserver and etcd carry CPU
limits of their own and can be genuinely throttled. The Pod name
carries the second half of the filter, so a throttled apiserver cannot
answer under a heading that says worker.

The walk is over nodes rather than Pods, because the endpoint is per
node and one read covers every worker it hosts. It is gated on the
diagnostics phase budget, and placed first among the five gated
collectors: the budget declines whatever has not started when it runs
out, its own ceiling is four bounded reads rather than the minutes the
four after it can spend, and it is the only one whose question has no
other answer in the tree. That order is a trade, stated here because
it is a decision rather than a detail - it puts the guest serial
console one place further from the budget, and that capture is the
only one describing a worker which never reached apid.

Both reads take the block's own read-budget knob rather than a literal
equal to it, so lowering that knob lowers these too. Each keeps its own
exit status rather than routing through the shared helper, which always
returns zero by design: this collector has to tell "the kubelet was not
read" from "the kubelet answered and carried no worker", and only the
status separates them. The stream is captured whole and filtered after,
since a pipeline reports its last command's status and would otherwise
hand that decision to grep.

Every way a read can come up short is named rather than left as zero
bytes: a kubelet that was not read, one that answered with nothing for
the namespace, a stream cut short, and exit 124 - which cannot
distinguish this collector timing out from the read exiting 124 on its
own - each get their own line, and each says whether an error log
exists to explain it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:43:11 +03:00
Aleksei Sviridkin c681ae1243 test(tenant): anchor the operator-gate absence and refresh a spent reason
The two "carries no operator gate" tests asserted only
`notExists: spec.dependsOn`. A bare absence assert passes on a template
that renders no document, so toggling `seaweedfs` and `gateway` off left
both green while the releases they guard were gone. Measured: all 5 tests
passed with the templates rendering nothing, and 2 of them fail once the
name is pinned alongside the absence.

An earlier check on these two asserts injected a `dependsOn` and watched
them redden, which proves only that they catch an added gate. That is the
other direction; the one that matters for a negative assert is the
document disappearing.

Both suite headers also justified their `templates:` list with a reason
that has expired: keycloakgroups.yaml now guards its `.Values._cluster`
index, so a no-values render of the whole chart succeeds. The list is
still load-bearing, for the reason the sibling suite already records. It
keeps an unrelated template out of this suite's result: with a nil
dereference added outside the guard of a template neither suite asserts
on, both suites stay green with their list and error without it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:43:08 +03:00
Aleksei Sviridkin e9411d6b8c fix(apps): make the post-delete cleanup hooks report failure
The qdrant and harbor post-delete hooks guarded every delete with
`|| echo "WARNING: ..." >&2`, set no `-e`, and ended on an
unconditional completion message, so the container exited 0 on
every path. Helm read that as a successful hook and the
hook-succeeded delete policy removed the Job together with the pod
that carried the warning, leaving an uninstall that reclaimed
nothing indistinguishable from one that had nothing to reclaim.

Count the failures instead and set the exit code at the end, so a
failed delete does not skip the deletes that follow it. An object
that is already gone stays a success: every delete keeps
--ignore-not-found, which exits 0 in that case. This is the shape
packages/apps/mariadb already uses for the same problem.

Both hooks also consumed the output of `kubectl get` directly --
harbor by iterating it, qdrant by piping it into grep, whose status
the pipeline then reports. A listing that failed therefore arrived
as an empty list and the sweep was skipped rather than reported.
Each listing is now captured before it is used, so a listing that
failed and one that returned nothing are different events.

A hook that exits non-zero keeps its Job and pods, which is the
point: they hold the log. Give the Job one retry rather than the
Kubernetes default of six or none at all. Every delete is
idempotent, so a second attempt re-issues an API call rather than
re-running any logic, and one short backoff is what separates a
transient apiserver error from a real one -- these failures become
visible for the first time here, and a signal that fires on noise
stops being read, including on the day it is right. Six is the far
end of the same mistake: ten minutes of backoff across seven pods
outlasts the hook timeout, and Helm reports that timeout instead of
the hook's own error.

The qdrant Chainsaw suite explained its post-delete pod collector
by quoting the guard string removed here and concluding the pod is
always collected away. That is now backwards: a failed cleanup is
exactly when the pod survives and the collector is the evidence.
Rewrite the note, and the sentence in the harbor suite pointing at
it.

hack/app-cleanup-hooks_test.bats renders both scripts and runs them
against a fake kubectl. There are eight failure paths across the two
hooks -- six in harbor, two in qdrant -- and each has a case that
goes red when that one path alone is reverted to a swallowed
warning. Message assertions are anchored on the hook's own `ERROR: `
prefix, because the fake kubectl names the object in its own
diagnostic too and an unanchored match is satisfied by the fixture.
The helm-unittest suites pin the retry limit and the hook delete
policy: adding hook-failed to that annotation hands the Job back to
Helm to delete on failure and destroys the evidence this change
exists to keep, and until these asserts that reversal left both
suites green.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:43:02 +03:00
Aleksei Sviridkin 5d436fad43 fix(build): declare the root test target phony
The root Makefile declares nine of its targets phony and `test` is not
among them, while a `test/` directory sits beside it. Make therefore
takes that directory for the target's product: `make test` prints
"'test' is up to date", exits 0, and the recipe -- which installs the
platform and runs the e2e suite -- never executes.

docs/agents/overview.md advertises the command as the way to run the
full suite, so someone following it by hand gets a green exit and no
tests. CI is unaffected: the workflow calls test-controllers, and the
e2e path is driven out of packages/core/testing directly.

Every other undeclared target in this file is unshadowed today, so this
is the only one that misbehaves, and one name closes it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:42:59 +03:00
Aleksei Sviridkin 91b9f58057 fix(hack): declare the shared package targets phony
package.mk set .PHONY with `=` rather than `:`, which assigns a
variable named .PHONY instead of declaring anything phony, so none
of the targets package.mk defines was phony in any package that
includes it.

Declare exactly the concrete targets package.mk defines: help, show,
diff, apply, delete, suspend, resume, check and clean. The earlier
list named update and image, which package.mk does not define —
naming an undefined target in .PHONY creates it as an empty target,
so `make image` and `make update` would report success without
running anything in the many packages that lack those recipes. It
also omitted check and clean, the targets a same-named file or
directory could actually shadow.

The neighbouring .DEFAULT_GOAL is a variable and correctly keeps its
equals sign. A comment now records which of the two adjacent lines is
which, so the corrected one does not pull the other into its shape by
analogy.

The trigger map in docs/agents/contributing.md called the two files a
byte-for-byte copy, which was exact rather than loose: they were the
same object. They stop being one here, so the entry now records that
they were identical and no longer are, and asks for a diff-and-port,
the wording the map already uses for the other copy that has
diverged. It deliberately does not enumerate which lines differ:
that set moves as the copy catches up, and an entry that names it
would be wrong again by the next port.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:42:58 +03:00
Aleksei Sviridkin 5f8a820494 fix(keycloak): keep KeycloakRealmUser roles when spec.roles is unset
The operator reconciles user roles as an authoritative set. Under the
default `full` strategy it deletes every realm role mapping missing
from spec.roles, and every client role mapping missing from the
matching spec.clientRoles entry. A consumer that manages only the user
record and leaves those fields out therefore has all of its roles
stripped on the first reconcile, `default-roles-<realm>` included,
which costs the user offline_access, uma_authorization and the account
client roles.

Treat an omitted list as "not managed" and leave Keycloak untouched
for that field, the same convention spec.identityProviders already
uses. An empty but non-nil list keeps its meaning, so revoking every
role is still possible with an explicit `roles: []`.

Drop omitempty from the two json tags as well. Without that an
explicit empty list is erased the first time the operator writes the
object back, adding its own finalizer is enough, and returns
indistinguishable from an omitted field, so the distinction does not
survive a round trip.

The patch mirrors epam/edp-keycloak-operator#372 file for file, so it
can be dropped in one move once COMMIT_REF moves past the upstream
merge.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:42:14 +03:00
Aleksei Sviridkin 62204d4903 test(cozystack-controller): guard the Secret grant by effective permission
The suite's "does not grant a blanket wildcard verb on secrets" assert
was a notContains over a rule object. helm-unittest compares every field
it is handed by value equality, and `any: true` changes which fields are
compared rather than how, so no spelling of that assert bounds what a
verbs list may hold: `verbs: ["get", "*"]` satisfies all of them.

Measured against the rendered ClusterRole, seven realistic spellings of
a widened Secret grant left that assert green: a wildcard in resources,
a wildcard in apiGroups, a "*" appended to the declared verb list, an
extra field beside the verbs, deletecollection spelled out, and two
where the grant is added as a separate rule. Those last two leave the
whole suite green - appending apiGroups:['*'] resources:['*']
verbs:['*'] keeps the positive pin satisfied while the controller gains
every verb on every Secret cluster-wide.

Replace it with a bats guard over the rendered manifests. It unions the
verbs that every ClusterRole and Role rule confers on core Secrets,
expanding "*" on each of the three axes, and compares that union with
the declared set. Equality rather than an upper bound: the controller is
cluster-scoped, so a widening reaches every tenant namespace, and a
narrowing breaks the two reconcilers that share the grant.

The positive pin stays in tests/rbac_test.yaml. Value equality is the
right primitive for pinning a rule as written and the wrong one for
bounding what the rule may become.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:37:09 +03:00
Aleksei Sviridkin ea78d09dd0 feat(ci): auto-apply triage labels to issues
Issues created via the API or gh bypass the issue templates and arrive
without any triage/* label; 252 of the 396 open issues carry none.
Label issues on arrival (opened/reopened) and run a daily sweep as a
backstop: issues already prioritised, assigned, or marked as epics get
triage/accepted, the rest triage/needs-triage.

A signal has to be on stale.yaml's exempt-issue-labels, because
triage/accepted is itself exempt and a signal that is not would grant a
permanent stale reprieve as a side effect of being labeled. That is
necessary and not sufficient: lifecycle/frozen is exempt and still not
a signal, because a lifecycle/* label says what the stale bot may do
with an issue and nothing about whether anybody reviewed it.

The sweep paces its writes and caps how many it makes per run, so a
pass over a large backlog stays inside GitHub's secondary rate limits
instead of failing partway through. Each run writes a job summary with
the counts, the issue numbers per label, and both remainders it can
leave behind, because a backlog pass logs a line per issue and nobody
reads 252 of those to find out what happened.

The manual dispatch defaults to dry-run, so the one entry point a human
can reach writes nothing until somebody unchecks the box. The schedule
is the one event that writes without being asked.

Deleting a triage label does not undo a decision: an issue left with
none is relabeled by the next sweep, so AGENTS.md's labeling section
says to change a decision by replacing the label instead.

No PR lane can exercise this workflow, so the first real run is a sweep
over every open issue. A contract test pins the executable lines that
bound that: the pull-request skip, the already-triaged skip, the pacing
and the cap, the dry-run default, the pinned action, the token scopes,
that no signal comes from the lifecycle/* namespace, and that the
summary is written with the await that makes it flush.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 16:35:36 +03:00
myasnikovdaniil e944619e98 docs(release): add changelog for v1.6.1 (#3551)
This PR adds the changelog for release `v1.6.1`.

✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.1.md`.
2026-08-12 18:27:03 +05:00
Aleksei Sviridkin d73c102945 test(system): anchor index-addressed negative asserts to what they name
notContains passes whenever its path does not resolve, and also
whenever it resolves to something that simply lacks the forbidden
content. A different container at the same index qualifies, so these
guards go green the moment a sidecar is placed ahead of the workload
they were written for, while the flag or env var they forbid comes
back on the real container.

Assert the identity at that index first, in each affected case: the
container name for the Deployment-shaped ones, and for the Cilium
policy its selector, since a deny rule has no name. The selector
identifies that rule only while there is one rule to identify, so it
is paired with a length assertion, and with a second one on the
rule's toPorts, because that path carries two indexes and anchoring
the outer one alone leaves the inner free.

The anchor goes only where nothing else holds the index. A positive
contains on the same path fails on its own once a sidecar arrives
ahead of the workload, and an equal on the whole argument list fails
the same way, so neither carries a separate anchor. A negative does,
including the per-element form below: a wildcard resolving to the
sidecar's own arguments matches nothing and passes.

Several of the guards could not fail for the reason they name, and
the anchor alone would have left that hidden behind a comment saying
they were covered. contains and notContains compare elements for
equality, so a mapping is compared whole: `{name: GF_...}` never
matched an entry carrying a value beside the name, and `{port: 7473}`
would not match that port re-added with a protocol beside it, which
is how the neighbouring entry in the same list is written. Those get
`any: true`.

Equality against a rendered argument is the same failure worn
differently, and it arrives in three disguises. A prefix that never
equals an element: `--pinned-tenants=` never equalled
`--pinned-tenants=tenant-a=shard0`, and `--target-host=` never
equalled a rendered `--target-host=<value>`. One value out of many
under a name that claims the flag: `--compatibility=cilium` left the
kilo default render free to carry `--compatibility=flannel` off the
same values key, and the two `--proxy-fqdn` negatives forbade only
the FQDN the default cluster-domain produces. And a form that holds
by luck: `--gateway-api` still fails today only because that template
renders it bare, so an upstream switch to `--gateway-api=true` would
retire the guard in silence.

All of those become notMatchRegex over args[*], which matches the
flag name per element and fails on any value and on the bare form
alike. A length assertion would not have covered the prefix cases,
since an upstream bump that adds the flag while consolidating another
argument keeps the count. A wildcard that resolves to nothing still
passes, so the index anchors stay.

The whole argument list is a different instrument, answering what the
default invocation is rather than whether one flag is absent. It
fails on an argument added, removed or swapped instead of only on the
ones someone thought to enumerate, which is worth having where the
template is first-party and a failure is a deliberate change asking
to be read. flux-shard-operator gets it in `renders the default
Deployment shape`, replacing eleven individual contains that listed
exactly the same eleven elements; `--shard-cpu-limit=` was dead for
the prefix reason and is dropped rather than repaired, because that
list now covers it. The vendored ouroboros templates get no such pin:
upstream churn arrives there on its own schedule, and a whole-list
assertion would fail a test named for one flag for a reason that has
nothing to do with that flag.

Anchor every index on a path, not the outermost one. The Cilium
assertion reads spec.ingressDeny[0].toPorts[0].ports, and pinning the
rule list alone would still let the port return on a second toPorts
entry of the same rule.

Each guard was checked by rendering what it forbids onto the element
it names, and each anchor by rendering the shift it exists to catch.

The form is already used in this tree, in opensearch-operator and
dashboard. Assertions of the same shape remain in files this commit
does not touch, including one in a package it does, and stay out of
scope here.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-12 15:28:37 +03:00