Commit Graph
7777 Commits
Author SHA1 Message Date
Aleksei Sviridkin d1c6a24bf4 test(nats): state precisely what a label stamp on a key-bearing Secret reaches
The comment claimed a visibility label on either cert-manager Certificate
would hand its private key to every tenant subject. It would not, today:
tenantsecrets serves only what the lineage webhook marked
tenantresource=true, and lineage resolves through ownerReferences or the
Flux helm label, neither of which a cert-manager Secret carries while
enableCertificateOwnerRef stays false.

The assertions stay -- they become load-bearing as soon as either gate
moves, and one of them is not this chart's to hold.

Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-15 22:40:58 +03:00
Aleksei Sviridkin 7a8f27c77b test(nats): compare both tenant grant surfaces against their whole form
The guard filtered each surface down to the entries that name a Secret
and compared the subset. Every such filter admits the spelling it did
not anticipate: on the ResourceDefinition a label selector names no
resource, and controller.cert-manager.io/fao is stamped on both
key-bearing Secrets, so a selector on it grants the CA private key; on
the dashboard Role a rule with no resourceNames names no Secret, a rule
with resources ["*"] names no resource, and a ClusterRole is not a
Role. All of them passed green.

Compare the whole ResourceDefinition include list, the rendered RBAC
kinds and the Role's complete rule list instead. Aggregating with
eval-all also drops the dependence on the Role being the first rendered
document. The header claimed both surfaces are out of reach of
helm-unittest for want of a chart template; both are rendered, and what
is missing is a suite that asserts on them.

Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-15 22:40:58 +03:00
Aleksei Sviridkin 736d44674f test(nats): pin the trust-anchor sentinel to a single projection entry
hasDocuments counts rendered documents, not list entries, so a second
projection naming tls.key as its source rendered inside the same
document and left the suite green. The controller refuses a sentinel
carrying more than one entry, which makes that a silent loss of the
tenant's trust anchor.

Assert the list length alongside the entry it pins.

Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-15 22:40:58 +03:00
Aleksei Sviridkin ff3294e02d fix(e2e): size tenant workers by resources, not by u1.large (#3840)
## What this PR does

Follow-up to #3839, which moved the e2e tenant workers to `u1.large` to
give them a second vCPU. That instancetype doubles memory along with the
vCPUs, and four 8Gi workers do not fit on the sandbox beside the rest of
the suite. On the first runs after it landed, the second worker of each
cluster stayed Pending with `0/3 nodes are available: 3 Insufficient
memory`, so the node group never reached two Ready nodes and node-join
failed for a reason unrelated to the CPU ceiling the change was meant to
test. The memory estimate in that PR body counted the workers against
total node memory and did not account for what the rest of the suite
already occupies.

This sizes the group with `resources.cpu` and `resources.memory`
instead, which keeps the two vCPUs at the 4Gi the workers had before.
The chart omits the instancetype from the VM whenever a group sets both
fields, since KubeVirt rejects a VM that references an instancetype and
also overrides CPU or memory, and the values schema still lists
`instanceType` as required, so it stays in the values as the fallback it
now is.

The measurement from those runs is worth keeping even though the runs
were spoiled. Where the workers did start, the two-scrape capture read
1.46 to 1.93 against the new ceiling of 2.0, with 2 to 5 percent of CFS
periods throttled, against 0.93 to 0.98 of a 1.0 ceiling at 8 to 13
percent throttled before. The ceiling was binding, and raising it does
loosen it; whether that is enough to make the join deadline is what the
next runs measure.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

The diff is one values block inside
`hack/e2e-chainsaw/_lib/run-kubernetes.sh`. Nothing under `hack/` is
moved or renamed, no make target changes behaviour, no package, schema,
default or CRD changes. Nothing in the trigger map is touched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(e2e): tenant workers in the kubernetes e2e suites keep their two vCPUs but go back to 4Gi of memory, since the larger instancetype left workers unschedulable on the sandbox
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration**
  * Updated tenant worker nodes to use `u1.medium` instances.
  * Set each worker’s resources to 2 CPUs and 4 GiB of memory.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-15 18:37:10 +03:00
Aleksei Sviridkin 2b3c163d15 fix(e2e): size tenant workers by resources, not by u1.large
Sizing the workers with u1.large doubled their memory along with the
vCPUs, and four 8Gi workers do not fit on the sandbox beside the rest of
the suite: the second worker of each cluster stayed Pending with
Insufficient memory on all three nodes, so the node group could never
reach two Ready nodes and node-join failed for a reason that has nothing
to do with the CPU ceiling it was meant to test.

Setting resources.cpu and resources.memory keeps the two vCPUs at the
memory the workers had before. The chart drops the instancetype from the
VM when a group sets both, and the values schema still requires the
field, so it stays in the values as the sizing fallback it now is.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-15 18:34:13 +03:00
Myasnikov Daniil 73bc4c0ae3 fix(ci): build the backstop's reason list without a blind pipeline
`unmatched=$(echo … | tr … | sort -u | grep -v '^$' | paste -sd ' ' -)`
takes its exit status from paste, so a failure in tr, sort or grep is
invisible under set -e. Demonstrated with a sort stub that exits 1: the
pipeline reports rc=0 and yields an empty string, so the line becomes
`no runnable suite is named by ''`.

That is the same last-command blindness this script already handles
twice -- for the suite-list find and for the two yq indexes -- and the
comments there explain why each was a bug. Selection stays fail-safe
because the escalation is decided before the message is built, so the
only casualty is the reason line; but the reason line is a contract as
of this change set, and a contract that degrades in silence is what the
change set exists to remove.

Replaced with the shell's own split plus a `case` membership test: no
external command, nothing that can half-succeed, dedup preserved, and
the same idiom resolve_suites already uses a few lines up.

A test feeds two unresolved directories with one repeated and pins the
whole line, so a regression to a form that can emit a partial or empty
name turns the file red rather than printing a worse message.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:08 +05:00
Myasnikov Daniil c28d7804af fix(ci): enumerate the inert .gitattributes instead of matching by name
The rule was broader than its justification. It read `(^|/)\.gitattributes$`,
so it claimed every present and future .gitattributes anywhere, while
the argument for it was about content: every one in this tree holds
nothing but `linguist-generated` markers, which reach no build, chart or
test.

The filename does not carry that argument. .gitattributes can also set
`filter`, `eol`, `working-tree-encoding` and `export-subst`, each of
which changes what lands in the working tree and therefore what gets
built, so a by-name rule would eventually make a live file inert with
nothing to notice it.

Nothing is lost by narrowing, because only three files ever reached the
rule: api/.gitattributes and internal/crdinstall/manifests/.gitattributes
are escalated by full_suite_pattern first, so entries for them would be
dead. Measured against the base selector, those three are exactly the
paths whose classification this branch changes, and they still select
nothing.

Same trade as the workflow enumeration a few lines up, and the same
upkeep, which the comment now states: after adding or moving one,
re-check with `git ls-files '*.gitattributes' '.gitattributes'` and
confirm the contents are still only linguist markers. A .gitattributes
not on the list is classified by whatever rule its path falls under --
the graph inside a package, the unclassified fall-through at the root --
and both fail safe. A test pins that root case, so a future widening
back to a by-name match turns the file red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:08 +05:00
Myasnikov Daniil 2d6a7ec777 test(ci): pin the unclassified-path reason line
The header claims "a test asserts each line" about the reason-line
contract. It was false for one branch: deleting the `unclassified path`
echo at the fall-through left `hack/cozytest.sh
hack/select-e2e_test.bats` exiting 0.

That is the worst one to leave unpinned. It is #3392's own guard, its
cause is the one a reader cannot infer from the selection ("a path
nobody has classified" is not expressible as a suite list), and the
only action it asks for is to classify the path it names. The word
"unclassified" appears in seven comments in the test file while pinning
it zero times, so the branch read as covered.

Verified by mutation both ways: with the assertion added, deleting the
echo turns the file red, and a sweep that mutes each of the eleven
`select-e2e:` messages in turn now reddens the file for every one of
them, so the header's claim holds for the whole set rather than for
most of it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:08 +05:00
Myasnikov Daniil b2702621d3 fix(ci): decide the per-app bats rule per path, not after the merge
Two defects in the hack/e2e-apps/ arm added by the previous commit.

The arm set trigger_any=1 for whatever basename it derived and left the
verdict to the final intersection. That makes the escalation depend on
the REST of the diff: hack/e2e-apps/monitoring-oidc-system.bats alone
emptied the selection and the backstop escalated, but the same file
beside packages/apps/redis/templates/redis.yaml selected `redis` with
nothing on stderr, where the pre-change selector ran all 21 suites.
That is the merge-before-escalate shape #3330 removed from the graph
walk, reintroduced one rule over. The basename is now
membership-tested against the suite list on the spot and an unmatched
one sets trigger_full=1 and names the file.

POSIX case matches `/` with `*`, so the arm also sees nested paths, and
the unanchored capture turned hack/e2e-apps/fixtures/postgres.bats into
the "suite" fixtures/postgres -- right outcome by accident, reason line
printing a path where a suite name belongs. The capture is anchored to
one segment, and what the arm does with anything that is not a
top-level <name>.bats is now stated rather than left to fall out of a
regex: it is shared material for those suites the way
hack/e2e-chainsaw/_lib/ is for the Chainsaw ones, so it escalates for
the same reason.

Both claims that said otherwise are corrected -- the arm's own comment
and docs/agents/e2e-testing.md both asserted the deferred escalation
was equivalent to the old fall-through, which held only for a
single-file diff. The backstop comment drops the e2e-apps clause it
gained last commit, since that route no longer reaches it. The header's
"four of the seven" becomes count-free: this commit adds two escalating
branches, so a hardcoded total was already stale.

Tests: the mixed-diff case is the regression pin (the isolated case
passes with the escalation deferred and cannot see the bug), plus a
nested-path case and a non-.bats case. Both fixes are
mutation-verified -- restoring either defect turns the file red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:08 +05:00
Myasnikov Daniil 4d4f5edf92 fix(ci): escalate only the e2e bats files, not the whole unit lane
full_suite_pattern took every hack/*.bats, so all 63 of them ran the
whole Chainsaw suite -- including the 60 that are the unit lane and
that the e2e sandbox never executes.

The root Makefile is the authority on the split and draws it at
exactly the prefix this narrows to:

  BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))

The 60 files it keeps run under `make unit-tests`; the three it
filters out -- e2e-prepare-cluster, e2e-install-cozystack,
e2e-test-openapi -- are the ones packages/core/testing's recipes
execute inside the sandbox, and those still escalate. So no Chainsaw
suite can regress from an edit to one of the 60.

Inert here is not a green gate with nothing behind it. `make
unit-tests` is gated on pull-requests.yaml's `code` output, computed
in the workflow as "any changed path outside docs/" and never from
select-e2e.sh, so a bats-only pull request still runs the whole unit
lane. The `e2e` job reads the same output, so it still installs the
platform and runs the OpenAPI tests before skipping Chainsaw.

Two tests: one pins the rule on both sides of the prefix and asserts
an inert bats file beside an app path does not mask the selection, and
one walks every hack/*.bats in the tree and requires the selector's
verdict to agree with the Makefile's split, so a file named to fit
neither lane surfaces here rather than on the next PR that touches it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:07 +05:00
Myasnikov Daniil 33428671fa fix(ci): classify hack/e2e-apps/ and packages/tests/ in select-e2e
Three path classes matched no rule in hack/select-e2e.sh and escalated
to all 21 Chainsaw suites through the unclassified fall-through -- 7
of the last 150 merged pull requests between them.

hack/e2e-apps/<name>.bats now maps to the <name> suite, off the
basename exactly as the per-suite rule takes the name off a
hack/e2e-chainsaw/<app>/ directory. Not marked inert: what is left in
that directory after the Chainsaw migration is wired to nothing, and
inert would bake that orphan status into the rule and go quietly wrong
the day a lane runs those files again. A basename no suite carries
drops out of the intersection at the bottom of the script and
escalates through the backstop, which is the outcome those paths
already had, so mapping optimistically loses nothing and gains the
correct answer for every name that is a suite.

packages/tests/ and every .gitattributes join inert_config_pattern,
which is what the script's own header says to do with a genuinely
inert path rather than widening the fall-through. packages/tests/ is a
helm-unittest fixture chart: changing a test OF cozy-lib does not
change cozy-lib, no PackageSource lists those paths as a component,
and nothing installs them -- a change to the library itself still
escalates through packages/library/. .gitattributes is matched by name
wherever it lives because every one in this tree is a
linguist-generated marker, which reaches no build, chart or test; one
under a tree full_suite_pattern already covers still escalates, since
that pattern is checked first.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:07 +05:00
Myasnikov Daniil 42bd45a87c fix(platform): make etcd-application depend on etcd-operator
extra/etcd renders kind: EtcdCluster from
etcd-operator.cozystack.io/v1alpha2, but cozystack.etcd-application
declared no dependsOn edge to cozystack.etcd-operator -- the only
operator-backed application source in the tree that did not. The
edge has been absent since the sources were first added.

Two consequences, one per side of the graph.

On install this is a latent ordering bug. Without the edge the
etcd-rd HelmRelease registers the ApplicationDefinition as soon as
cozystack-engine is up, so a tenant can create an Etcd before the
operator exists and its HelmRelease then fails with `no matches for
kind "EtcdCluster"`. hack/select-install.sh's forward closure for the
etcd suite omitted the operator for the same reason, which would
install the suite against a cluster with no CRD to apply to.

On test selection cozystack.etcd-operator reached no runnable suite,
so every change to the operator escalated to all 21 suites -- 5 of
the last 150 merged pull requests. The etcd suite exists; only the
edge that makes it reachable from the operator was missing.

Scope of the ordering change: both Packages are emitted
unconditionally by the same bundle (templates/bundles/system.yaml,
under the single bundles.system.enabled guard, in every variant), so
no supported configuration has one without the other. An admin who
lists cozystack.etcd-operator in bundles.disabledPackages while
keeping the app now leaves it DependenciesNotReady -- the same
exposure postgres, mariadb, kafka and redis already carry, with
Package.spec.ignoreDependencies as the escape hatch.
select-install.sh --validate confirms the edge adds no cycle and
nothing dangling.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:07 +05:00
Myasnikov Daniil 3c3f9450ae fix(ci): name the cause on every select-e2e escalation
Seven branches in hack/select-e2e.sh escalate to the full Chainsaw
suite and four of them did it in silence: the shared
hack/e2e-chainsaw/_lib/ and .chainsaw.yaml case, the
full_suite_pattern match, a packages/ path no PackageSource claims,
and the empty-selection backstop.

"Run everything" is the same 21 suite names whichever rule produced
it, so a reason line is the only thing that tells the seven apart.
full_suite_pattern is the commonest cause of a full run by a wide
margin -- 88 of the last 150 merged pull requests, 68 of them with no
other cause -- so the usual answer to "why did this pull request run
the whole suite" was the one the log never carried, recoverable only
by re-deriving the selection by hand.

Each new line names the path that caused it rather than only the rule,
and the backstop names the directly-selected names it could not
resolve. They go to stderr, never stdout: stdout is the suite list and
both e2e lanes parse it, so a reason line there would be read as a
suite name.

A unit test asserts each of the four messages. A reason that regresses
to silence changes no selection, so it is invisible to every other
test in the file.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:58:07 +05:00
Myasnikov Daniil 302b55cc38 docs(agents): defer the image-refs routing edit to its own PR
This reverts commit 2f87cdcdfa.

AGENTS.md sits at the repository root, where no scoped CODEOWNERS
rule matches it, so it falls through to the `*` catch-all owned by
@kvaps and @lllamnyp alone. Every other path this PR touches
(/.github/, /docs/, /hack/) lists @lexfrei, whose review already
satisfies them. That single file is the only reason this PR still
gates on an architect review, which is exactly the deadlock
hack/codeowners-invariant.bats was written to describe.

No substance is lost. docs/agents/image-refs.md carries the full
consumer table and the paragraph explaining why
overlay-main-images.sh and promote-packages-artifact.sh both stand
outside the shared enumeration, and it stays in this PR. What goes
away is the one-line routing summary in AGENTS.md that points at
that file. It leaves AGENTS.md briefly describing
overlay-main-images.sh as "a fourth consumer" while the detail file
lists five rows; the detail file is the authoritative one, and the
summary is re-landed on its own branch where a catch-all review
blocks nothing else.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:55:04 +05:00
Myasnikov DaniilandAleksei Sviridkin 6181493d4f refactor(dashboard): validate the schema form before page-level checks
In the four backup create pages (Backup, BackupJob, BackupPlan, BackupRestoreJob), run the RJSF SchemaForm.validate() call right after the name check and before the page-level required-field alerts. The submit button lives outside RJSF and bypasses its validation, so each page triggers validate() explicitly; it previously ran only after the manual alert checks, so a schema-required field left empty exited via an alert without RJSF ever rendering its inline errors. Validating first makes inline errors show consistently; the page-level alerts then cover only page-only requirements.

Re-applied from cozystack/cozystack-ui#26 onto the in-tree console after the UI was vendored (#2963); the source repo is being archived. Scope is the validation-ordering change only — the original PR's VMDisk/StorageClass parts were obsoleted by the DynamicOptionsWidget refactor (#30) already in the vendored snapshot. Includes the original per-page tests.

Co-authored-by: Aleksei Sviridkin <1366514+lexfrei@users.noreply.github.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-15 17:35:40 +05:00
Aleksei Sviridkin 71c0fc185b test(e2e): give tenant workers two vCPUs (#3839)
## What this PR does

One line in the kubernetes e2e suite values: tenant workers move from
`u1.medium` to `u1.large`, one vCPU to two.

The reason is measured, in two independent ways on
cozystack/cozystack#3513. A single-scrape reading on a quiet branch run
and the two-scrape instrument from #3836 on a loaded run with six
parallel e2e workflows agree: the worker's compute container sits at 93
to 98 percent of its one-core ceiling for the whole node-Ready budget,
the sandbox nodes under it show ~0.05 percent steal, and raising the CPU
request changes nothing because the workers never lose a scheduling
fight. The binding constraint is capacity: one vCPU has to boot Talos
and unpack the kubelet image, and its quota also covers the QEMU
emulator and IO threads.

This is an experiment as much as a fix: the suites run the same
assertions, and the next waves of runs under parallel load measure
whether the join flake documented in cozystack/cozystack#3513 disappears
with the ceiling doubled. The diagnostics landed in #3804 and #3836 stay
aboard, so the same instrument reports the new profile.

Memory math on the sandbox: two suites with two workers each move from
4Gi to 8Gi guests, and the virt-launcher pods spread across three 24Gi
nodes, so the placement still fits.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

The diff is one value inside `hack/e2e-chainsaw/_lib/run-kubernetes.sh`.
Nothing under `hack/` is moved or renamed, no make target changes
behaviour, no package, schema, default or CRD changes. Nothing in the
trigger map is touched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
test(e2e): tenant workers in the kubernetes e2e suites run with two vCPUs, since measurements showed the single-vCPU worker saturating its CPU ceiling for the entire node-join budget
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the tenant worker node configuration to use a larger instance
type, improving available capacity for end-to-end Kubernetes testing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-15 15:23:16 +03:00
Aleksei Sviridkin 3b49be07fa test(e2e): give tenant workers two vCPUs
The tenant worker boots Talos and unpacks the kubelet image on a single
vCPU whose CFS quota also covers the QEMU emulator and IO threads, and
measurements on both quiet and loaded runs show the compute container
pinned at 93-98% of that one-core ceiling for the whole node-Ready
budget. The join deadline is then decided entirely by how much CPU work
the boot happens to need, and under parallel e2e load it does not fit.

Doubling the worker to u1.large raises the ceiling instead of the
weight, which measurements showed changes nothing since the workers
never lose a scheduling fight.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-15 15:08:31 +03:00
Aleksei Sviridkin ef96292b64 fix(ci): give the nightly e2e runner headroom over its guest vCPUs (#3833)
## What this PR does

The e2e sandbox boots three QEMU nodes at `-smp 8`, so 24 guest vCPUs
are committed before any tenant VM exists. Four jobs boot that sandbox.
Three run on the 32-vCPU class; nightly ran on the 24-vCPU one, where 24
guests against 24 host vCPUs leaves nothing for QEMU's own emulation and
IO threads, containerd, the kernel and the runner agent. The other lanes
moved to the larger class at some point and nightly was not moved with
them.

The sandbox keeps its 24 vCPUs. That guest count is part of what e2e
claims to test, and a smaller nightly cluster stops matching the one PRs
run against.

This is hygiene, not a fix. Measured boot-phase data across 18 runs says
the load a workflow puts on the host dominates guest starvation, not the
runner class: the lanes already on 32 vCPU boot guests slower than
nightly did on 24, and class and workflow kind are fully confounded
there, so neither separates out. Do not read this as making anything
faster or as addressing a particular red run. What is left is the
arithmetic. A 1:1 commit has no headroom at all, and two lanes running
the identical sandbox on different classes is drift nothing was
watching.

The rest is a check, because the guest demand and the runner classes
live in different files and nothing compared them.
`hack/sandbox-runner-headroom.bats` reads `-smp` and `-m` times the loop
count out of `hack/e2e-prepare-cluster.bats`, reads the runner class of
every job that calls the sandbox make targets, and wants the runner
strictly larger on both axes. Neither side is a hardcoded list, so
bumping `-smp`, adding a management node, or adding a fifth lane on a
smaller runner turns it red on the same commit. Memory is in there
because the same argument applies to it and `-m 24576` sits on the same
line as `-smp 8`, not because a class that violates it exists today.

It stops instead of guessing when it cannot read something, since a
guess comes out smaller than the real demand and passes a runner that
cannot hold the sandbox. Refused: a missing or non-literal `-smp` /
`-m`, a launch written outside any `@test` block or inside one left open
at end of file, a `for` opener the strict pattern does not match, a
`for` list holding anything but literal words, a `while` sharing the
block, two launches or two loops in one block, and a `runs-on` with no
`oracle-vm-<n>cpu-<m>gb` in it. Unparsed loop openers are counted
separately from parsed ones, so moving `do` to the next line is refused
rather than silently multiplied by one. There is also a cross-check that
every workflow calling the sandbox make targets yields at least one
matched job, so a renamed target turns the parser red instead of quietly
shrinking what it covers.

The derivation runs against the live tree, which passes whether or not
those refusals still work, so each one is pinned by a fixture that feeds
a synthetic file to the same program. That matters more than it sounds:
two of the refusals above exist because the first version of this file
did not have them, and a launch moved into a file-level helper would
have halved the measured demand and stayed green.

The numbers come from:

```bash
sed -n '98,107p' hack/e2e-prepare-cluster.bats    # three nodes, -smp 8, -m 24576
grep -rn 'prepare-env' .github/workflows/         # the four jobs that boot it
grep -rn 'runs-on: oracle-vm-' .github/workflows/ # their runner classes
```

The rule also goes into `docs/agents/e2e-testing.md` next to the other
environment invariants, so whoever adds the fifth lane picks its runner
class from the docs instead of from a red unit suite.

Checked that it bites, twice over. Fourteen mutations of the inputs:
reverting the runner class, `-smp 11`, a fourth and fifth management
node, `-smp cpus=8`, `-m size=24576`, `do` on the next line, `$(seq 1
3)` as the list, a `while` loop, the launch moved into a file-level
helper, renaming the make target, `ubuntu-latest`, a matrix expression
in `runs-on`, a hypothetical `32cpu-64gb` class, and a new lane on a
4-vCPU runner. Then eleven mutations of the parser itself, one per
refusal, each of which kills exactly the fixture that names it and
nothing else. Green on this tree, and the same numbers out of gawk, mawk
and BSD awk.

Closes #3831.

### Screenshots

No UI change.

### Downstream repositories

Walked the trigger map against the diff. It changes one `runs-on` line
plus a comment in `.github/workflows/nightly.yaml` and adds one file
under `hack/`. The `ccp` trigger is moving or renaming something under
`hack/`, or changing what a make target does; `hack/package.mk` and
`hack/common-envs.mk` are untouched, and the new test is picked up by
the existing `hack/*.bats` wildcard without the target changing. The
`ansible-cozystack` and `talm` triggers are node prerequisites in
`hack/e2e-prepare-cluster.bats`, which this reads and does not modify.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(ci): run the nightly e2e suite on a runner class larger than the sandbox's guest vCPU demand, and add a unit check that fails when an e2e lane loses that headroom
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Updated nightly end-to-end testing to run on a higher-capacity runner,
improving support for multiple QEMU guests and required resource
headroom.
* Added validation to prevent sandbox runners from being undersized or
oversubscribed.

* **Documentation**
* Documented the resource requirements and deterministic conventions for
end-to-end sandbox testing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-15 12:59:55 +03:00
Aleksei Sviridkin 0cf82ebfe2 test(e2e): measure tenant worker CPU as a rate, and the sandbox nodes under it (#3836)
## What this PR does

Two instruments for the tenant worker that misses the node-Ready
deadline. They are instruments, not a fix: the failure they instrument
is cozystack/cozystack#3513, and this PR does not close it. Neither
changes what the suites assert.

The worker CPU capture now reads `container_cpu_usage_seconds_total`
beside the CFS counters it already read, and it is read twice a fixed
interval apart instead of once. The throttled counters say a container
met its ceiling; they cannot say how much CPU it got, and the two
failures behind a stalled worker are exactly those, a guest held at a
one-core ceiling and a guest whose vCPU thread is rarely scheduled at
all. Both leave throttled periods behind. Every counter involved is
cumulative since the container started, so a single reading divides out
to an average over the whole uptime and says nothing about the window
the deadline covered, which matters here because the profile under
investigation is a guest that freezes for tens of seconds and then runs
flat out: the average comes out low, the instantaneous figure comes out
at the cap, and only the second distinguishes it from a guest that is
simply capped.

The second instrument is a new subject. The worker captures describe a
cgroup, so they are blind to the layer under the sandbox, and a guest
that is runnable and never scheduled looks from the cgroup like a guest
that asked for nothing. Whether the sandbox node itself was given its
turn is recorded only in the steal column of its own `/proc/stat`. No
kubelet endpoint carries it, and crust-gather cannot reach it because
PodSecurity refuses its node-shell Pod in this cluster, so the capture
goes over the Talos API, which is already how the report reads those
nodes for dmesg. It is captured verbatim with its column legend beside
it rather than reduced to a percentage: `/proc/stat` has no header,
steal and guest sit next to each other, and guest is large on a node
running VMs and already counted inside user, so reading one as the other
turns a fraction of a percent into twenty-odd.

Both subjects are read on either side of one shared wait, so the pair
pays it once. That wait is guarded like every other external in the
block: it is the one call there outside a `|| true`, and under `set -eu`
a missing `sleep` would exit 127 and take the collectors after it, two
of which the phase's own missing-`timeout` warning promises keep
collecting. Without it the two readings are taken back to back and the
block says so, which is honest here because the interval is read off the
stamps rather than off the knob. The default is 12 seconds and that is
the whole wall-clock cost this adds to a failing run. The wait is not
the interval either subject's counters span, though: each pass walks its
own nodes and the other subject's pass falls between a subject's two
readings, and on the run these diagnostics exist for those passes are
the slow part rather than the wait. Each capture therefore stamps both
ends of its read, so the real interval is subtracted off the pair
instead of assumed from the knob, and the width of those brackets is the
uncertainty. This matters more on the sandbox side than on the worker
side, since a cAdvisor row carries its own sample time and a
`/proc/stat` row carries none. That wait is a measurement interval
rather than a stand-in for a condition nobody wrote a wait for, so
`docs/agents/e2e-testing.md` carves it out by name instead of leaving a
reader to decide whether the `TODO(e2e-replace-fixed-timeouts):` marker
applies to a gap between two readings of a cumulative counter.

The pair is gated below the serial console rather than above it, and its
ceiling rather than its worth is the reason. Reading two collectors
twice is sixteen bounded reads plus the interval, which at the read
bound and the three-node cap is most of the phase budget, and admission
gates only when a collector may start. Above the console, a run that hit
those bounds would leave the console never started rather than merely
cut short, and the console is the only capture that survives a worker
which never reached apid, which is the shape this failure usually takes.
That overrun is inside the term the budget derivation already allows for
one collector admitted a moment before the deadline, and a guard in
`hack/run-kubernetes-node-join_test.bats` holds the pair's ceiling
against it. Two more guards there hold what may sit ahead of the
console: nothing read more than once, and nothing whose ceiling could
spend the budget on its own.

Failure marking follows the existing collectors rather than inventing a
shape: no collector here may leave an empty directory, because an empty
directory reads as a healthy node, so a missing binary, a missing
talosconfig, a listing that never answered, a node with no address and a
read that was cut off each land as a named note or a
`COLLECTION-FAILED.txt`.

The phase budget is unchanged at 420s and the derivation guard still
holds.

Covered by `hack/run-kubernetes-sandbox-cpu-time_test.bats` (new) plus
additions to the cpu-throttle, node-join and network-counters suites.
Each collector was checked against a deliberate break of its own guard:
dropping the usage family, ignoring the sample number, removing the
interval, accepting a zero interval, swallowing the missing-talosconfig
marker, reading the wrong `/proc` file, taking a dual-stack node's
second address for a node, counting addresses instead of nodes in the
truncation marker, stamping only one end of a read, letting the stamp
claim counters were sampled on an arm that read nothing, telling a
capture with no counters to subtract a sibling, leaving the sampling
wait unguarded, and starving the budget guard of its input. Every one is
caught by a named test.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

The diff touches `hack/e2e-chainsaw/_lib/run-kubernetes.sh`, two
`chainsaw-test.yaml` files, four BATS files and one agent doc. Walking
the trigger map: nothing under `hack/` is moved or renamed,
`hack/package.mk` and `hack/common-envs.mk` are untouched, no make
target changes what it does, no package is added or removed, no schema,
default, enum, CRD or namespace moves. The one entry that mentions this
area is the `ccp` rule about moving or renaming things under `hack/`,
and no file is moved or renamed here.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
test(e2e): the node-join failure diagnostics now read the tenant worker CPU counters twice a fixed interval apart, so consumed CPU and throttling are both available as a rate over the same window instead of as an average over the container's uptime, and they add the sandbox node's own /proc/stat over the Talos API, which is the only place the steal time under the sandbox is recorded. Each reading carries the time it was taken, so the interval a rate divides by comes from the captures rather than from the configured wait.
```
2026-08-15 10:53:51 +03:00
Aleksei Sviridkin 527d4ea00f test(e2e): measure tenant worker CPU as a rate, and the sandbox nodes under it
Two instruments for the node-join failure the kubernetes-* suites keep
hitting. Neither changes what the suites assert.

The worker CPU capture now reads container_cpu_usage_seconds_total
beside the CFS counters, and reads them twice a fixed interval apart
instead of once. The throttled counters say a container met its ceiling
and cannot say how much CPU it got, which are the two candidate
explanations for a stalled worker: a guest held at a one-core ceiling,
and a guest whose vCPU thread is rarely scheduled at all. Both leave
throttled periods behind. Every counter involved accumulates from
container start, so a single reading is an average over the whole uptime
and no rate at all, and the profile under investigation is a guest that
freezes for tens of seconds and then runs flat out: the average comes
out low, the instantaneous figure comes out at the cap, and only the
second separates it from a guest that is simply capped.

The sandbox nodes are a new subject. The worker captures describe a
cgroup, so a guest that is runnable and never scheduled looks from there
like a guest that asked for nothing. Whether the sandbox node itself was
given its turn is recorded only in the steal column of its own
/proc/stat: no kubelet endpoint carries it, and crust-gather cannot
reach it because PodSecurity refuses its node-shell Pod here, so the
read goes over the Talos API, which is already how the report reaches
those nodes for dmesg. The rows are captured verbatim with their column
legend beside them rather than reduced to a percentage, because
/proc/stat has no header, steal and guest sit next to each other, and
guest is large on a node running VMs and already counted inside user.

Both subjects are read on either side of one shared wait, so the pair
pays it once. The wait is not the interval either subject's counters
span, though: each pass walks its own nodes and the other subject's pass
falls between a subject's two readings, and on the run this exists for
those passes are the slow part. Each capture therefore stamps both ends
of its read, so a rate comes from subtracting stamps rather than from
assuming the knob, and the width of those brackets is the uncertainty.
That wait is a measurement interval rather than a stand-in for a
condition nobody wrote a wait for, so the e2e conventions carve it out
by name instead of marking it as a fixed timeout to be replaced.

The pair is gated below the serial console rather than above it. Its
ceiling is sixteen bounded reads plus the interval, which is most of the
phase budget, and admission gates only when a collector may start, so
above the console a run that hit those bounds would leave the console
never started rather than merely cut short. The console is the only
capture that survives a worker which never reached apid, which is the
shape this failure usually takes. That overrun is inside the term the
budget derivation already allows for one late admission, and a guard
holds the pair's ceiling against it.

No collector here may leave an empty directory, since an empty directory
reads as a healthy node: a missing binary, a missing talosconfig, a
listing that never answered, a node with no address and a read that was
cut off each land as a named note or a COLLECTION-FAILED.txt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-15 10:19:19 +03:00
Aleksei Sviridkin 3dd142a718 fix(e2e): capture both ends of the worker byte path on node-join failure (#3804)
## What this PR does

A tenant worker that misses the node-Ready deadline while its kubelet
image pull crawls leaves two mechanisms behind one symptom, and the
failure artifact could not tell them apart: either the link is slow, or
the pull keeps restarting and discarding what it fetched. From inside
the guest both are a progress line that barely moves. This adds the
host-side counters that separate them, and the guest-side reads that
explain a kubelet which never got as far as pulling at all.

**Host side.** A per-worker network capture reads the eight
`container_network_*` families from the same kubelet cAdvisor endpoint
the CPU throttling capture already uses. cAdvisor builds them from the
interface counters inside the Pod's own network namespace and publishes
one row per interface, so what arrived at the worker is counted there in
full, including the bytes a restart threw away, set against the progress
the guest reported, that separates a pull which kept restarting from a
slow link.

Which row answers is not obvious, and getting it wrong inverts the
reading, so the collector states it rather than leaving it to the
reader. A bridge-bound worker presents four interfaces in its Pod netns:
the one still named `eth0` is a dummy that holds the Pod IP so the
kubelet's check passes and carries no traffic, so it reports zero, and
it is the row a reader reaches for first, by name, where zero reads as a
worker that received nothing. The number that answers is the renamed
NIC's receive; the bridge counts the same bytes again, and the tap's
counters are named from the host's end. The filter deliberately carries
no interface predicate, so all four reach the artifact and can be told
apart.

Two further properties of the family fail by capturing nothing rather
than loudly, so both are stated where the filter is: the dropped-packet
series are spelled `packets_dropped` and there is no `*_drops_total`,
and every row carries an empty container label because cAdvisor
publishes them on the Pod sandbox, which means a filter borrowed from
the CPU counters beside it matches nothing at all.

The fixture's row shape and values are real kubelet output with the
identifying labels rewritten. Its four interface names are derived
rather than observed: no virt-launcher Pod was available to sample, so
they are read from the name generators in the KubeVirt version this repo
ships, and the counters attached to them illustrate direction rather
than magnitude.

**Guest side.** The Talos capture gains the service state machine and
the link table. Talos creates a service's log buffer when its runner
opens the log writer, as it starts the process, so a kubelet still
waiting on its volumes, on time sync, or on the container runtime
reporting healthy never reached its runner and has no log at all, and
`talosctl logs kubelet` then fails naming a log that was never
registered, which is true about the log and silent about the service.
`talosctl services` answers with the state and the condition being
waited on, in one call for every service, so the runtime a kubelet is
blocked behind arrives in the same file. The link table is read as YAML
because MTU lives in the `LinkStatus` spec and not among the resource's
printed columns, and it is the value that decides whether a path-MTU
reading of a stalled transfer survives.

Both new reads are made with the short-lived `os:reader` credential the
capture already mints; neither needs a wider role.

**A worker that answered nothing now says so once, for the worker.**
Each read already carried its own error and exit code, which is accurate
per file and adds up to a directory indistinguishable from a worker that
simply had little to report.

### What this does not do

This collects evidence; it fixes nothing and is not expected to change
any suite's outcome. The comparison it enables is not symmetric either,
and the source says so: the host side reports bytes on an interface,
while the guest side reports an image pull's own progress, which is
unpacking rather than arrival. They bound each other; they are not two
measurements of one quantity.

### Cost

The diagnostics phase budget drops from 8m to 7m. That is forced rather
than chosen: the inequality that keeps the tenant crust-gather snapshot
reachable had ten seconds of slack, and the two added guest reads cost
sixty across the minimum worker pool. A shorter budget declines more,
and what it declines first is what is gated last: the mirror capture,
whose subject is partly recoverable elsewhere, and the image-cache
re-probe, which has no substitute and was already first to go. The new
network capture also spends a listing and one read per node ahead of the
guest captures, so those reach their gate with less headroom than
before. All of it is priced where the number lives, as a ceiling the
read bounds impose rather than a duration measured on a live cluster,
and a collector the budget declines says so in the log rather than going
quiet.

The remaining slack is ten seconds, which is worth knowing before the
next collector is added rather than after: it buys no bounded read at
any bound this file uses, so adding one means re-deriving the budget
again.

### Testing

`hack/run-kubernetes-network-counters_test.bats` is new (15 tests). The
node-join, Talos-diagnostics and CPU-throttle suites gain coverage for
the phase ordering, the per-worker marker boundary, the spend-order
comments in both Chainsaw suites, and the enumeration of collectors that
survive a missing `timeout`. Every claim the two Chainsaw comment blocks
make about spend order is derived from the source rather than restated,
so a collector added without a line there fails.

The two cAdvisor captures now run one shared body. They previously
carried near-identical copies, and the copies cost more than duplication
normally does: the block-level bound audit pins each collector by
something only it produces, and two collectors issuing identical reads
left neither with anything of its own, so removing one from the audit
went unnoticed. Each is now pinned by the report directory it writes.

Verified with `hack/cozytest.sh` across eleven suites, checking that the
number of tests executed matches the number declared, because an exit
code answers "did anything fail", never "did anything run". Each
behavioural change was confirmed by mutating the source and observing
the intended test fail.

### Screenshots

No UI changes.

### Downstream repositories

Walked the trigger map file by file against the diff. All seven changed
files live under `hack/`, and the change adds one file and modifies six;
no repository file is moved, renamed or deleted. The `ccp` entry is the
only one that keys on `hack/` at all, and it keys on `hack/package.mk`,
`hack/common-envs.mk` and `hack/update-crd.sh` as anchors for `make
generate`; none of those is touched, and no make target changes
behaviour. Nothing here is a package, a chart value, an
`ApplicationDefinition`, a CRD, a release asset, a namespace, a variant,
or a node prerequisite, so no other entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added worker network-counter diagnostics with interface-level and
cumulative traffic data.
* Expanded Talos diagnostics with service-state and link-state
information.
* **Bug Fixes**
* Improved reporting for partial captures, timeouts, unavailable data,
and zero-traffic workers.
* Clarified failure reporting and fallback messages when timeout
controls are unavailable.
* **Documentation**
* Updated diagnostic timing and collector documentation for the
seven-minute budget and new metrics.
* **Tests**
* Added comprehensive coverage for network counters, filtering,
truncation, and expanded Talos diagnostic scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-15 03:44:05 +03:00
Aleksei Sviridkin db2ddfab2b fix(ci): give the nightly e2e runner headroom over its guest vCPUs
The e2e sandbox boots three QEMU management nodes at `-smp 8`, so 24
guest vCPUs are committed before any tenant VM starts. The nightly lane
ran that sandbox on a 24-vCPU runner, a 1:1 commit that leaves the host
nothing for QEMU's emulation and IO threads, containerd, the kernel and
the runner agent. The tag, fork and pull-request lanes already run the
same sandbox on the 32-vCPU class, so move nightly there too.

This aligns headroom and claims no performance effect. Measured boot
data shows the load a workflow puts on the host, not the runner class,
dominates guest starvation, and the two are confounded in that data; no
red run is attributed to the class here. The case is the arithmetic: a
1:1 commit leaves no headroom, and two lanes running the identical
sandbox on different classes is drift nothing was watching.

Add hack/sandbox-runner-headroom.bats to keep the two sides paired. It
derives the guest vCPU and memory demand from the sandbox bats file, and
the runner class of every job that drives the sandbox make targets, then
requires the runner to be strictly larger on both axes. Neither side is
a literal list, so raising `-smp`, adding a management node or adding a
lane on a smaller runner turns it red instead of drifting apart
silently.

Input the derivation cannot read in full is refused rather than guessed:
a guess produces a demand smaller than the real one, which passes a
runner that cannot hold the sandbox, while a refusal names the line.
That covers a launch written outside any `@test` block, a block left
open at end of file, a `for` opener the strict pattern misses, a
non-literal loop list, and a `while` loop sharing the block.

The derivation runs against the live tree, which passes whether or not
those refusals still work, so each one is pinned by a fixture feeding a
synthetic file to the same program. Loosening the parser turns a fixture
red rather than quietly widening what the check accepts.

Record the rule alongside the other environment invariants in
docs/agents/e2e-testing.md, so a new lane picks its runner class from
the documentation rather than from a failing unit suite.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-15 02:41:19 +03:00
Aleksei Sviridkin e06932f496 test(e2e): stop the mongodb backup suite before the restore leg (#3834)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

The `mongodb-2-backup-roundtrip` suite now drives its harness with
`SKIP_RESTORE=1`, so it stops once the ad-hoc `BackupJob` reaches
`Succeeded` instead of continuing into the to-copy restore.

A to-copy restore replays the source cluster's system users into the
target. The target's own generated credentials stop authenticating
afterwards, so the harness cannot read its sentinel document back out of
the restored copy, and the psmdb operator loses access to the cluster it
just restored. That behaviour is tracked in #3832 and is not addressed
here; this change only stops the suite from asserting a flow that cannot
currently succeed.

What the suite still proves is unchanged and is the larger half: bucket
provisioning, a source MongoDB coming up with backups enabled, a
sentinel document written into it, and an ad-hoc `BackupJob` against the
`cozy-default` BackupClass completing against S3.

The skip is pinned rather than assumed. Exiting zero is also what a
completed round-trip does, so a `SKIP_RESTORE` that stopped reaching the
harness would be indistinguishable from one that worked. The step
therefore also asserts the skip branch's own log line, which the harness
writes to stderr.

Two places already told the reader that CI covers the restore, and this
change is what makes them wrong, so both now say where coverage ends:
the header of `30-restorejob-to-copy.yaml` and the example README. The
asserted log line is annotated where it is printed, since rewording it
would fail the suite on a bare match with nothing pointing at the cause.

Coverage returns by dropping `SKIP_RESTORE=1` together with the
assertion on that line, once #3832 is fixed. Dropping only the variable
would trade this failure for a marker mismatch, so the suite comment
spells out both. The test keeps its round-trip name, matching
`etcd-2-backup-roundtrip`, which is gated out of CI under the same
convention.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked against the diff, which is one Chainsaw suite
(`hack/e2e-chainsaw/mongodb/chainsaw-test.yaml`) and three files under
`examples/backups/mongodb/` (`run-all.sh`, `30-restorejob-to-copy.yaml`,
`README.md`). No package is added, renamed or removed; no
`values.schema.json`, version enum, chart default,
`ApplicationDefinition` or CRD changes; nothing under `hack/` moves or
is renamed and no make target changes behaviour; no telemetry metric
changes; and the only entry for `cozystack/examples` is the node and
network requirements, which are untouched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(e2e): the MongoDB backup suite now stops after a successful BackupJob; the to-copy restore leg stays uncovered until the credential handling behind it is fixed
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated MongoDB backup and restore guidance to clarify current restore
limitations.
  * Documented how to skip restore operations when needed.
* Clarified which restore scenarios are automated and which require
manual handling.

* **Tests**
* MongoDB end-to-end coverage now validates successful backups while
restore testing is temporarily skipped.
* Added verification that the test harness stops after a successful
backup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 23:44:24 +03:00
Aleksei Sviridkin e441edc2aa test(e2e): stop the mongodb backup suite before the restore leg
A to-copy restore replays the source cluster's system users into the
target, so the target's own generated credentials stop authenticating
and the harness cannot read the sentinel back out of the restored copy.
The suite has never passed for that reason, holding the in-tree e2e run
red on every branch that selects it.

Drive run-all.sh with SKIP_RESTORE=1 so the suite still proves the
backup half end to end, and pin the skip by asserting the skip branch's
own log line: exit zero alone is also what a completed round-trip
returns, so without that assertion the step could not tell a skipped
flow from a full one, nor notice SKIP_RESTORE failing to reach the
harness.

The to-copy manifest and the example README both stated that CI covers
the restore, which stops being true here, so both now say where
coverage ends. The asserted log line is annotated at its source, since
rewording it would red the suite with a bare match failure.

Coverage returns by dropping SKIP_RESTORE together with the assertion
on that line, once the credential handling behind the restore is
fixed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-14 21:18:09 +03:00
IvanHunters 31fac3c4f1 fix(kubernetes): guard on child-release ownership, not parent release-name
The worker-adoption render guard required meta.helm.sh/release-name == the
parent release as a precondition to refuse a control-plane-only render. But
Helm's prune gates solely on helm.sh/resource-policy=keep and never consults
the live release-name (migration 54 documents this and its foreign-owner branch
depends on it). A same-cluster, unpinned worker MachineDeployment whose
release-name is empty (Helm annotations stripped out-of-band, or a DR restore
of the CAPI objects) or foreign therefore slipped past the guard, yet Helm
still entered it into the parent's prune set and deleted it on the transition
upgrade: the exact worker-VM data loss the guard exists to prevent, in the
exact case (migration 54 never ran for this cluster) it was written for.

Refuse when a same-cluster, unpinned MachineDeployment is NOT owned by its
kubernetes-nodes child release (release-name lacks the kubernetes-nodes-<cluster>-
prefix), which catches parent, empty and foreign owners alike while exempting
adopted and natively-created child pools. Add a regression fixture with an
empty release-name that reddens under the old parent-release-name predicate,
and keep every conjunct non-vacuous by mutation.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 20:06:05 +03:00
IvanHunters adf7913f79 test(kubernetes): isolate the clusterName predicate in the adoption guard
Complete the per-conjunct coverage of the guard: add a synthetic fixture where
the MachineDeployment carries the parent release-name and no keep pin but names
a different cluster, so spec.clusterName is the sole predicate keeping it out of
the refusal set. Deleting any one of the guard's three predicates now reddens
exactly one fixture (clusterName, release-name, resource-policy=keep).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 17:37:51 +03:00
IvanHunters b1ce856c47 test(platform): cover the 53-char child-name overflow skip branch
Every warn-pin-skip hardening branch in migration 54 had a regression fixture
except the >53-char child-name overflow guard. Add one: a 33-char cluster name
plus pool md0 yields a 54-char child release name, so the overflow branch must
warn, pin the live pool objects with resource-policy=keep, skip adoption, and
let the run stamp. Without the guard the flow reaches kubectl apply of an
over-long HelmRelease name, the apiserver rejects it, and set -e exits before
the version stamp, deadlocking every tenant's platform upgrade. Verified
non-vacuous: neutralising the length guard reddens this fixture.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 17:37:51 +03:00
IvanHunters c7d3d40810 test(platform): assert the non-scalar-field skip branch pins live pool objects
The existing bad-scalar test ran with the pool objects absent, so it pinned
the warn+skip+no-child-HR contract but not the pin itself. Add a companion that
presents the live MachineDeployment, MachineHealthCheck, WorkloadMonitor and
KubevirtMachineTemplate and asserts each is annotated resource-policy=keep with
no ownership transfer, mirroring the non-object-nodeGroups companion. Verified
non-vacuous: removing the branch's pin_pool_objects call reddens it.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 17:21:14 +03:00
IvanHunters 8046144494 docs(kubernetes): scope the adoption guard comment to the etcd-present branch
The guard lives in the else of 'if not $etcd', so it does not run during the
awaiting-etcd window, where the chart renders only the status beacon. Note that
an owned+unpinned MachineDeployment is not refused in that transient window.
This is a pre-existing exposure (the pre-split chart also rendered pools only in
this branch) that closes once etcd resolves and a real upgrade re-renders; the
comment previously claimed an unconditional refuse-to-render.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 17:21:14 +03:00
IvanHunters f9065a9455 fix(migrations): correct the non-scalar-field skip remedy
The non-scalar scalar-field skip branch told the operator to restore the field
and 're-run adoption', but migration 54 stamps the version unconditionally and
does not re-run, so that remedy is a dead end. Align it with the sibling skip
branches: restore the scalar, then recreate the pool, and state plainly that
the migration has already stamped and will not re-run.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 17:21:14 +03:00
IvanHunters e68784b3b6 test(kubernetes): cover the release-name predicate in the worker adoption guard
The adoption guard in cluster.yaml refuses a control-plane-only render only
when a MachineDeployment matches all three predicates: spec.clusterName,
meta.helm.sh/release-name owned by this parent release, and not pinned with
resource-policy=keep. The suite left the release-name predicate untested: the
adopted-pool case carried both a child release-name and a keep pin, so the keep
predicate alone exempted it and deleting the release-name predicate kept every
case green.

Add a natively-created (greenfield) fixture: child release-name, no keep pin.
Its clusterName and unpinned state are both true, so the release-name predicate
is the sole reason it renders. This is the routine post-split parent upgrade of
every cluster, the guard's highest-blast-radius path.

Per-conjunct mutation now: dropping release-name reddens the greenfield case,
dropping keep reddens the pinned case, dropping clusterName reddens nothing (an
MD from another cluster cannot carry this release name, so clusterName is
belt-and-braces). Baseline 6 of 6 green.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 17:17:30 +03:00
Myasnikov Daniil 2f87cdcdfa docs(agents): correct the image-refs consumer ordinal
The routing bullet called hack/overlay-main-images.sh "a fourth
consumer" from a time when three scripts sourced the enumeration.
Four do now, so the ordinal names the wrong thing in the file every
agent reads first.

State the distinction instead of a count: overlay is the exception
because it reads refs and walks the tree itself, so a newly declared
file never reaches it, while hack/promote-packages-artifact.sh
sources nothing because it publishes the whole tree and consumes no
refs. One clause each, with the existing pointer to
docs/agents/image-refs.md carrying the rest.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil ffae003d6b fix(release): stop a failed org package listing from exiting green
The nightly sweep's own listing shared one `|| true` with the grep
that filters it, and grep legitimately exits 1 when no cozystack/*
package matches. A failed /orgs/.../packages read therefore arrived
as an empty list, took the "No cozystack/* container packages found"
branch and exited 0 -- below the nightly sweep but ABOVE the deferred
guard, so a protection set that could not be resolved reported
success having swept nothing. That bypasses the guard rather than
varying it: replaying the step against a stub gh, a failed /pulls
combined with a legitimately empty org listing exited 0 after
printing its own ::error::.

List and filter as two statements so one `|| true` stops meaning two
things, handle the listing failure like the other two (::error::,
blocked flag, emptied list), and consult the flag before the early
exit. That exit is non-zero, matching the other two listing
failures: no sweep is left to carry the run down to the guard, so
this is the last place the flag can be honoured, and an unresolved
protection set is exactly what fail-closed means. The legitimate
empty-org case still prints the same line and exits 0.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil a37a3178e5 docs(release): name all four sourcers of the image-ref enumeration
The library header still claimed three call sites. There are four:
this PR made hack/verify-promoted-packages.sh source it to compare
the candidate's refs against the rc artifact's. The historical
drift story stays with the original three, since the verifier has
only ever read the shared enumeration.

The consumer table gained hack/promote-packages-artifact.sh under a
bullet requiring every consumer of references to read all three
shapes through the library, while the exception below it still named
only hack/overlay-main-images.sh. The publisher sources nothing and
needs nothing: it hands the whole tree to `flux push artifact` and
touches one reference, the installer's own self-pin, by key with yq.
Say that, rather than implying a gap it does not have -- the real
exception remains overlay-main-images.sh, which does read references
and does walk the tree itself.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil dee398f3b3 fix(release): defer a retention listing failure to the promotion guard
Both candidate-protection sources open with a `gh api --paginate`
listing and neither handled a failure. Under `set -euo pipefail` a
transient /pulls or /branches blip therefore exited the step before
the unrelated nightly GHCR sweep had run -- the exact coupling the
deferred guard at the bottom exists to remove, and which
hack/promotion-retention-contract.bats already asserts positionally.

Route both listings into that guard instead: raise the blocked flag
and empty the list, so the promotion sweep is skipped and the job
still fails while the nightly sweep finishes. Emptying is not
cosmetic -- a half-paginated list left in place is a smaller
protected set, and a smaller protected set is the one shape that can
authorise a delete.

Replaying the step's real shell against a stub gh: with /pulls
failing it exited 5 and never reached the nightly line; it now exits
1 with the nightly sweep done and the promotion sweep skipped.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil b65b6aa3ae test(release): pin every requiredBaseFiles entry, not just the list
The promote gate refuses a target base that lacks any of five
[path, marker] pairs, and the contract test asserted the list as a
block: deleting the whole thing turned it red, deleting the
hack/lib/image-refs.sh entry alone did not. Four of five pairs were
therefore unpinned individually.

Pin the missing one. The verifier sources that library through the
same `dirname $0` as lib/promoted-packages.sh, so a base carrying
the script without it fails at exactly the same point -- after the
candidate, the staging branch and the draft release exist.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil acd0a9e0a2 test(release): cover the verifier's rc-reference scan guard
The scan treats any grep diagnostic as a failed scan, because
`find ... -exec ... +` reports the legitimate "nothing matched" as a
failure too and the `|| true` absorbing that also absorbs a file
that went unscanned. Deleting the whole guard left the suite green:
its two siblings -- a failing collector and an empty collection --
each have a dedicated test, and this one had none.

Reach the state the guard exists for by putting a grep on PATH that
answers the scan's invocation the way the real one answers a file it
cannot open. A chmod 000 fixture would stop proving anything
wherever the suite runs as root.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:59 +05:00
Myasnikov Daniil 04fa8471fe docs(release): say what the gates actually guarantee
Three claims that were stronger than the code, plus one legibility fix.

The PR gate's comment said a `packages/` edit "on the PR or its base"
fails the status before merge. The base half is not true: GitHub fires
no `pull_request` event when the base branch advances, it only
recomputes refs/pull/N/merge, so a change landing on the base after this
went green is never re-verified and the PR merges on the stale result.
Checking the merge tree does make base drift visible WHEN the job runs,
which is worth keeping — but finalize is what actually catches the drift
case, after the merge and before any stable name. Say both, in the
workflow and in the reviewer checklist, so nobody reads the green as
covering more than it does.

"Transactional at every consumer-visible name" is a judgement call
dressed as an absolute, inherited from the "this dispatch performs NO
registry mutation" it replaced. The dispatch now does write to the
registry: the `promotion-*` candidate is a real, publicly listable tag.
What it creates is no *stable* name — no vX.Y.Z, no :latest, no stable
installer. Stated that way the sentence stops needing the reader to
already know which writes were meant.

The retention paragraph listed two of the conditions the workflow
enforces. It now lists all of them, including the ones the workflow
gained here: same-repository, bot-authored, `release-X.Y.Z` branch, the
`release` label, exactly one valid digest pin, and protection from a
base-branch tip as well as an open PR.

Also: `flux pull artifact` reads its credentials from
~/.docker/config.json, which is why finalize now runs `docker login` —
but its runner class is not one of the docker-using build ones and the
setup step installs everything except docker. Check for it there, where
the message can name what is missing, instead of failing with "command
not found" halfway down the release path. And the verifier's exclusion
list is a restatement of flux's `excludeOCI`, not a derivation — the CLI
does not expose it — so the coupling and the pinned version are now
written down next to the list.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 820e72d572 fix(release): protect a candidate until its retag, not until PR merge
Retention protected a packages candidate only while an OPEN bot release
PR pinned its digest. The digest stops being eligible for deletion only
when finalize's retag adds a stable tag to the manifest, because that is
what makes the `all(tags match promotion-*)` predicate stop matching —
and finalize runs after the merge that closes the PR.

The window between them is where a half-finished release sits. Finalize
creates the write-once stable git tag, then dies before the retag: a
lost runner, a 5xx publishing the draft, a non-fast-forward on the
maintenance branch. What is left pins the candidate from a merge commit
and a tag nobody can move, with no open PR and only a temporary tag on
the manifest. Thirty days later retention deletes it, after which
re-running finalize 404s at the candidate check and re-dispatching
promote is refused because the stable tag exists. The release can then
only be finished by hand — the exact class of manual cleanup this design
exists to remove.

Protect the tips of the branches a promote PR can merge into as well.
The two sources are read differently on purpose. An open promote PR's
head is supposed to carry exactly one candidate pin, so anything else is
an anomaly and blocks. A branch tip is an additional source: every
digest pin found is protected and a branch without one contributes
nothing, because a maintenance line cut before this pipeline existed
cannot pin a candidate and must not wedge retention forever.

Blocking now stops the promotion sweep alone. It used to `exit 1` inside
the resolution loop, which also killed the unrelated nightly prune
further down the same step — so a WIP commit adding a trailing comment
to a pin on a release branch would silently stop GHCR nightly pruning.
The job still fails; it fails after the nightly sweep has run.

Verified by running the step's script against a stubbed `gh`: a digest
pinned by main's tip and referenced by no open PR is now excluded while
a genuinely abandoned one is still listed; a manifest that also carries
a stable tag stays excluded by the existing predicate; and a malformed
pin on a PR head leaves the nightly sweep running, skips the promotion
sweep and exits 1.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 05b2ce90cf fix(release): close the verifier's fail-open paths and cover them
Three gaps in the guards that decide whether the published candidate is
still the tree being tagged. All three passed silently rather than
failing, which is the worst shape for a check that runs immediately
before a write-once name is created.

`normalized_refs` piped `collect_image_refs` straight into its parsing
loop, so the pipeline reported `sort -u`'s status and the collector's
was discarded. POSIX sh has no `pipefail` and this script has to stay
POSIX, so `set -e` never saw it: a failed or truncated collection became
a smaller set and the rc-vs-candidate comparison matched it happily. The
verifier then reported that no container bytes changed across promotion
having proven nothing — on the guard that carries that entire claim.
Collect into a file first, so a nonzero collector aborts before anything
is compared. An empty collection is the same failure wearing a zero exit
status and needs its own check: two empty sets compare equal, so the
proof passes having examined nothing, and every packages tree carries
image references — none means the collector understood nothing it was
handed.

The rc-leftovers scan swallowed grep's stderr, turning "could not read
this file" into "this file is clean" — the silent-skip shape
hack/promote-rewrite-tags.sh refuses by name. The `|| true` has to stay,
because `-exec … +` reports the legitimate "nothing matched" as a
failure too, so the diagnostics are the only remaining signal that a
file went unscanned. Fail on any of them. Defence in depth rather than a
live hole: a surviving rc string would also fail the per-file compare,
since the release tree has none.

The per-file content comparison had no test at all. It is the only leg
that catches a file present on BOTH sides with different bytes — exactly
the threat the job exists for, a maintainer editing packages/ on the
release PR after promotion published the candidate. The file-set compare
sees the same names, the installer compare reads one file, and the
rc-refs compare never looks at the release tree. Replacing the
comparison with a no-op left all nine other tests green.

Red phases, all observed before the fixes. The content-compare case
changes one non-image line on one side, so no other leg can fire and
claim the credit; it fails with the comparison no-op'd and passes with
it restored. The two collector cases run the verifier from a copy whose
sibling lib/ holds a stub — which is how the workflows already invoke
it, `.release-tooling/hack/…` with libraries resolved next to the
script, so the failure is reached the way production would reach it.
Against the piped version both fail on `[ "$rc" -ne 0 ]`: the verifier
exits 0 and prints its success line whether the collector returns 3 or
returns nothing.

Also pins `e2e`'s dependency on the candidate gate. The contract pinned
`e2e-report`'s, so removing `verify-release-candidate` from `e2e`'s
`needs` together with the `success` clause on its release arm left all
six tests green — and running a full-e2e promotion suite against a
candidate that never verified is the worse of the two failures.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 9fa8eeeda2 ci(release): pin the promotion toolchain and cap the candidate gate
Three review findings on the new promotion steps.

`yq` was fetched from `releases/latest/download` in all three steps this
pipeline installs it in — the promote dispatch, the promote PR gate and
finalize. That makes an unpinned moving dependency part of what decides
whether a stable release may be created, and part of what rewrites the
installer pin: two promotions of the same rc can be served different
binaries. Pin it the way FLUX_VERSION already is, version-gated so a
matching preinstalled yq is reused, on the version the e2e sandbox image
already tracks.

A version authenticates nothing about the bytes it names, though, and
these are the workflows that mint write-once stable names. yq is
therefore also pinned BY CONTENT: its SHA-256 is committed beside its
version and verified before install.

Scoped to yq deliberately. It is the one downloaded binary with write
authority over what a release contains — it stamps the candidate digest
into installer values at promote time and platformVersion into the
stable chart at finalize. skopeo, helm and oras only move bytes that are
already digest-addressed, so a substituted one is caught downstream; a
substituted yq writes the wrong pin and every later check verifies that
pin faithfully.

The Flux installer is left as it is, on purpose. It fetches
flux_${FLUX_VERSION}_checksums.txt from the pinned release and verifies
the binary against it, so the binary is not the exposure — the ~6 KB
bootstrap script from fluxcd.io is, and it is mutable by design and
fetched identically in ten other workflows here. Pinning its bytes in
these three files would break on the next regeneration of a script the
project does not control while leaving the same fetch unpinned
everywhere else. The comment names the residual risk instead.

No precedent existed either way: nothing under .github/workflows/ or
hack/ verified a downloaded tool before this, and every version-pinned
download in the tree (oras 1.2.0, cozyvalues-gen 1.6.0) is unverified.
This is a new convention rather than an application of one, confined to
the release path and justified by write authority, and the comment says
so where the next reader will be standing. The three unpinned fetches
outside this pipeline (nightly.yaml, e2e-fork.yaml) are left alone.

The checksum is trust-on-first-use, which is the honest description: it
fixes the bytes to what was checked when the version was chosen and
makes any later substitution loud. Provenance is the release's own
published checksums list cross-checked against the downloaded asset;
both agree on
fa52a4e758c63d38299163fbdd1edfb4c4963247918bf9c1c5d31d84789eded4, and
that binary self-reports v4.53.3.

Verified locally against the real asset: the version gate skips a
matching preinstalled yq and re-installs a mismatched one, the checksum
line reports OK for the genuine binary and exits 1 for a tampered copy,
which aborts the step under the runner's `bash -e`.

`verify-release-candidate` had no `timeout-minutes`, so it inherited the
6-hour default while `e2e-report` waits on it through `needs` — a
stalled `flux pull artifact` would hold the required `E2E Tests` status
pending for the whole of it. 30 minutes, matching `checks` and
`finalize`.

The contract pins that all three install steps carry both versions and
the checksum, run the checksum check, name no `latest/download`, and
agree on one yq version and one checksum. So the unpinned one-line idiom
cannot come back into a release step by copy-paste from the workflows
that still use it, and bumping the version in one file without its
checksum cannot pass. Both mutations are caught.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 2d734fb229 fix(release): preflight the rc tree's own pin before publishing
hack/promote-packages-artifact.sh pushes the candidate BEFORE rewriting
installer values, so the artifact carries the rc tree's
platformSourceUrl/platformSourceRef verbatim — and
hack/verify-promoted-packages.sh reads exactly those two fields back out
of it as the candidate's rc baseline, requiring a trusted repository and
an immutable `digest=sha256:<64 hex>` pin.

The publisher checked neither. A tag-form ref or an untrusted URL in the
rc tree published a candidate and pushed two commits regardless; the
refusal arrived later, in the promote PR gate or in finalize, once the
staging branch and the draft release already existed. Check both at
dispatch time instead, while nothing has been written.

A copied regex would drift from the guard it is bringing forward and
quietly restore the late failure, so the trusted repository and the
digest pattern move into hack/lib/promoted-packages.sh and both halves
source it. That makes the library part of what the verifier needs at run
time, so promote's base-branch precondition gains it: the list exists
because the base's own workflows run after merge, and a base carrying
the verifier without its library fails at the same two late points.

Tests: both preflights are covered, each asserting the mock registry
client was never invoked — "exited non-zero" alone does not say nothing
was written. Both fail with the preflight removed. The publisher
fixtures now pin the trusted repository the way a real rc tree does, and
still send the candidate elsewhere via REGISTRY so the rewrite
assertions keep their teeth.

Also covers a malformed PROMOTION_ID. It is not attacker-influenced, but
it is the run-uniqueness suffix of the candidate tag and retention.yaml
deletes a candidate only when every tag on the manifest matches
`promotion-v…-run-[0-9]+-[0-9]+`; a value outside that shape publishes
an artifact no cleanup pass can ever collect.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:58 +05:00
Myasnikov Daniil 54e8533a9d fix(release): compare executable bits unambiguously in the verifier
The mode check was written as a single chain:

  if { [ -x "$a" ] && [ ! -x "$r" ] || [ ! -x "$a" ] && [ -x "$r" ]; }

which reads like a symmetric difference and is not one. POSIX gives
`&&` and `||` equal precedence and left associativity, so it groups as
`((A && B) || C) && D` with `D = [ -x "$r" ]`. A file executable in the
candidate and not in the release tree makes `A && B` true but `D`
false, so the whole condition is false and the mismatch is accepted.
Only the mirror case was rejected — and it is the benign one. The
direction that got through is content the merge reviewer never saw made
runnable inside the very artifact the operator installs, which cmp(1)
cannot see because it compares bytes.

Resolve each side to a value and compare those, so no precedence rule
is load-bearing.

Both directions now have coverage. Before the fix the candidate-only
test failed against the shipped condition and the release-only test
passed, which is what makes the pair meaningful rather than a check
that only goes red when the whole block is deleted.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil 5d56681f8c fix(ci): put every label guard in lockstep, forks included
Promotion needs the `release` label to start work, so this PR added it
to the set of labels whose `labeled` event is not discarded. That set is
stated in four places and was updated in three.

`gate-concurrency-contract.bats` caught the shape of the problem and
then became the first casualty of it. It asserted that the concurrency
key excludes exactly ONE label from the `-label` group, by counting
occurrences of `github.event.label.name != '`. A second publishing label
made the count 2, turning the suite red — and with it `checks`, then
`finalize`, then `e2e`, then the required `E2E Tests` status. Raising
the constant to 2 would re-hardcode the same brittleness one label later
and throw away the invariant the test exists for: the set of labels
excluded from the `-label` suffix is exactly the set `plan` runs for.
Both sets are now extracted from the workflow and compared, so the
assertion holds at any number of labels.

`resolve_assets` was the guard that got missed. It sits on `e2e`'s
release arm, so a label event `plan` runs for and it skips for leaves
`e2e` with both arms false — `finalize` is force-skipped for release
PRs — while `e2e-report` still runs. On a promote PR already carrying
`full-e2e`, re-applying `release` therefore superseded the green run and
posted `E2E Tests` = failure, because `full-e2e` bypasses the "e2e not
repeated" branch and `E2E_RESULT` is `skipped`. `unlabeled` is not a
trigger, so removing the label could not clear it; only a push could.

`release` must also not publish on the fork lane. e2e-fork.yaml's
`resolve` decides "this run built nothing, so it is not a verdict" by
asking whether `Plan build` skipped. Admitting `release` at `plan` made
that false: a fork run reaches `plan`, has `build`/`finalize` gated off
by its own label, succeeds having produced no pr-patch, and the fork
lane fails closed on the missing artifact — red on a suite that was
never going to run, on a PR whose only fault is a mislabel. `plan` and
`resolve_assets` now admit `release` for same-repo PRs only, and the
concurrency key carries the complementary term, so
NOT(A OR B OR (C AND NOT F)) stays exactly (NOT A) AND (NOT B) AND
(NOT C OR F). For a same-repo PR F is false and every one of them
reduces to what it was, so none of this changes non-fork behaviour.

The same shape still reaches the fork lane from a PUSH to a fork PR
somebody mislabelled `release` — `build` is gated on the label, not on
the event, so this predates promotion using the label. Left alone; the
comment in e2e-fork.yaml now records it instead of asserting an
invariant this PR would have made false.

The contract enumerated `plan` and `e2e-report` by name, which is why it
did not catch `resolve_assets`: a named-job loop cannot notice a job
nobody thought to name. It now derives the job list from the file and
requires every job that decides from a label name to decide from the
same set, and pins the fork term on both halves of the complement. The
file's existing properties are kept — extraction runs over
executable-only lines, so a commented-out key cannot satisfy the
contract, and every comparison asserts a non-empty input first, so an
empty extraction fails closed under both bats and cozytest.sh.

Verified by mutation, each caught: excluding a label `plan` skips for,
admitting one the key does not exclude, reverting `resolve_assets`,
deleting `e2e-report`'s guard, narrowing `verify-release-candidate`'s,
renaming the label in one guard only, commenting out the group key, and
dropping the fork term from either side alone. Deleting `plan`'s
step-level fork guard is caught too, by an assertion that had to stop
matching on the bare identifier once `plan`'s own `if:` named it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil f21d277eba fix(release): retain live promotion candidates
Protect package candidates referenced by open release PRs while pruning abandoned run-specific promotion artifacts. Document the stable-candidate lifecycle and maintenance-branch rollout requirement.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Myasnikov Daniil c2e145709b fix(release): publish stable packages candidate
Serialize the tag-rewritten packages tree during RC promotion and pin its immutable digest in the installer. Verify the candidate against the prospective merge and the original RC artifact before creating stable names.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-14 18:37:57 +05:00
Aleksei Sviridkin 61edf33ef4 fix(e2e): capture both ends of the worker byte path on node-join failure
A tenant worker that misses the node-Ready deadline while its kubelet
image pull crawls leaves two mechanisms behind one symptom, and the
artifact could not tell them apart. Either the link is slow, or the pull
keeps restarting and discarding what it fetched; from inside the guest
both are a progress line that barely moves.

Per-worker network counters, read from the kubelet's cAdvisor endpoint
the CPU throttling capture already uses, count every byte that crossed
into the Pod including the ones a restart threw away, so the gap
between that and the progress the guest reported separates the two.

Which row carries it is not obvious and getting it wrong inverts the
reading, so the collector states it. cAdvisor publishes one row per
interface in the Pod netns, and a bridge-bound worker has four: the one
still named eth0 is a dummy holding the Pod IP, carries no traffic, and
reports zero -- the row a reader reaches for first, where zero reads as
a worker that received nothing. The renamed NIC's receive is the number
that answers; the bridge counts the same bytes again and the tap is
named from the host's end. No interface predicate is applied, so all
four reach the artifact and can be told apart.

Two further properties of the family fail by capturing nothing rather
than loudly: the dropped-packet series are spelled packets_dropped, and
every row carries an empty container label because cAdvisor publishes
them on the Pod sandbox, so a filter borrowed from the CPU counters
beside it matches nothing at all. Both are stated where the filter is.
The fixture's rows are real kubelet output relabelled; its interface
names are read from the KubeVirt version this repo ships rather than
sampled, since no virt-launcher Pod was available.

The guest side gains the service state machine and the link table. Talos
creates a service's log buffer when its runner opens the log writer, as
it starts the process, so a kubelet still waiting on its volumes, on
time sync, or on the container runtime reporting healthy never reached
its runner, has no log at all, and the capture recorded an rpc error
naming a log that was never registered: true about the log, silent about
the service, while the state and the awaited condition were the
finding. The link table is read as yaml because MTU lives in the spec
rather than among the printed columns, and it is the value that decides
whether a path-MTU reading of a stalled transfer survives. Both carry a
tighter bound than the two reads beside them, being single tabular RPCs
rather than streams, and the walk they sit in is the term the phase
budget sizes itself against.

A worker whose reads all failed now says so once for the worker. Each
log already carried its own error and exit code, which is accurate per
file and adds up to a directory indistinguishable from a worker that had
little to report.

The phase budget drops from 8m to 7m to pay for the two added guest
reads, because the inequality keeping the tenant snapshot reachable had
ten seconds of slack. A shorter budget declines more, and what it
declines first is what is gated last: the mirror capture, whose subject
is partly recoverable from its own request counts and the host snapshot,
and the image-cache re-probe, which has no substitute and was already
the first to go.

The second cost lands on the guest captures, which are gated after the
new counters and reach their gate with both the shortened budget and a
listing plus a read per node already spent -- up to 100s at the read
bound and the three-node cap, a ceiling those numbers impose rather than
a duration measured on a live cluster. Sharing that read would mean the
two captures becoming one walk, which is left out for scope rather than
merit: the phase gate is a deadline rather than a per-collector
allowance, so one walk admitted at the first of the two gates would cost
less and collect more. The cost is stated rather than netted away, and
the source says which way the argument runs.

Both captures run one shared body, which also settles what the two
copies had already broken once: the audit that catches an unbounded read
pins each collector by something only it produces, and two collectors
issuing byte-identical reads left neither with anything of its own, so
stubbing one out of the audit went unnoticed. It is pinned by the report
directory each writes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-14 14:34:37 +03:00
Aleksei Sviridkin 1c00ddb0f3 fix(platform): give node DaemonSets memory requests and limits (#3495)
## What this PR does

Sets memory requests and limits on the node DaemonSets that had none.
Companion to the namespace-wide LimitRange; the reasoning for why a
limit specifically (and not a request, and not a `PriorityClass`) is in
that PR and in `docs/operations/system-memory-limits.md`.

Requests come from 14-day peak working set measured on a 3-node dev
cluster. Limits are deliberately loose, sized as a runaway backstop
rather than a fitted ceiling, for two reasons: several of these scale
with cluster size in ways a small dev cluster understates (cilium with
endpoint and policy count, fluent-bit with log volume, multus and
kube-ovn with pods per node, velero node-agent during a backup), and
undersizing converts a rare pressure-driven kill into a deterministic
one. This repository already carries a scar from that failure mode,
recorded in the `vertical-pod-autoscaler` values where a 110Mi limit
OOM-killed the updater and cascaded into failed HelmReleases.

Two DaemonSets are not chart-rendered and needed their own seam.
`linstor-satellite` goes through the `LinstorSatelliteConfiguration`
podTemplate, including the two plunger sidecars — every container in the
pod needs a limit for the pod cgroup to get `memory.max`, so one
limit-free sidecar would put the whole pod back in the victim set.
`virt-handler` goes through KubeVirt `customizeComponents.patches`,
because virt-operator owns that DaemonSet and chart values cannot reach
it.

`linstor-controller` is deliberately left limit-free. It is a JVM that
must not be OOM-killed mid-operation, and `cluster_test.yaml` pins that
as an invariant. The satellite is a Go binary with a bounded working
set, so the opposite call applies there.

Not covered, because upstream hardcodes the containers with no
`resources` knob: the four frr-k8s `cp-*` init containers, the kube-ovn
`hostpath-init` and `install-cni` init containers, and `cozy-proxy`,
whose vendored chart exposes no resources value at all. `hami` and
`kilo` are left alone as optional components with no usage measurements
behind them. All of these still receive a limit from the namespace
LimitRange, so they are not left exposed — they simply carry no request.

Worth stating plainly: this PR does **not** grant OOM immunity on its
own, because the init containers it cannot reach leave the pod cgroup
without `memory.max`. That only closes with the LimitRange PR in place.

### Verification

Every touched package renders and its helm-unittest suite passes. New
assertions cover the metallb speaker and all five frr-k8s containers,
the linstor satellite and plunger sidecars, and the KubeVirt patch. The
metallb suite was mutation-tested: removing the frr-k8s reloader limit
fails it.

One gap. The KubeVirt patch shape was validated against the live CRD and
pushed through the real update path with `kubectl patch
--dry-run=server` (accepted; live object confirmed unchanged), but
virt-operator actually reconciling the DaemonSet with those resources
has **not** been observed, because that means restarting virt-handler
fleet-wide. Worth confirming in e2e or on a dedicated stand before
merge.

### Downstream repositories

Walked the trigger map against the diff. No entry is triggered: this
changes chart values and cozystack-owned templates under
`packages/system/`, not the `cozy-proxy` label and annotation contract,
not node prerequisites, not an app schema or version enum.

Leaving the boxes empty rather than ticking "none affected", because the
diff surfaced a real gap in a downstream repository even though nothing
here forces a change there: the vendored `cozy-proxy` chart exposes no
`resources` value, so its DaemonSet cannot be given one from this side
at all. That wants an issue or PR in cozystack/cozy-proxy followed by a
re-vendor. Flagging it for a maintainer decision rather than filing
speculatively.

### Release note

```release-note
fix(platform): node DaemonSets (cilium, metallb speaker and frr-k8s, kube-ovn, linstor-satellite, virt-handler, fluent-bit, node-exporter, multus, velero node-agent, kubevirt-csi-node) now declare memory requests and limits, so they are no longer selected as victims by the Talos userspace OOM handler when an unrelated workload drives node memory pressure.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Resource Management**
- Added CPU and memory requests and limits across Cilium, KubeVirt CSI,
KubeVirt, Linstor, MetalLB, monitoring agents, Multus, and Velero
workloads.
- Improved workload stability with defined memory ceilings and resource
reservations.

- **Multus**
- Moved resources to the dedicated `cozy-multus` namespace and
configured Cilium as the master CNI.
  - Added optional CNI plugin staging with enhanced security settings.

- **Tests**
- Added and expanded Helm tests validating resource configurations and
Multus behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 14:13:21 +03:00
Aleksei Sviridkin b7b56530b9 docs(platform): explain system component memory limits (#3496)
## What this PR does

Adds `docs/operations/system-memory-limits.md`, recording why system
components carry memory limits — which is not recoverable from the
values alone, and is the kind of thing that gets "cleaned up" by a
well-meaning future change.

The Talos userspace OOM handler (v1.12+) selects victims by ranking
cgroups that lack `memory.max`. Three consequences are each easy to get
wrong independently: only a limit grants immunity (a request just moves
BestEffort to Burstable, which under `strictCgroupClassOrdering` means
"killed second"); every container in the pod needs one, init containers
included, because kubelet only sets pod-level `memory.max` when they all
have it; and `PriorityClass` — `system-node-critical` included — has no
effect whatsoever, because the handler walks cgroupfs and never talks to
the API server. That last one is the trap, since setting it looks like a
fix.

The page also covers the two installer knobs and the constraint that
makes them dangerous to lower (a defaulted limit below a container's own
request fails admission), how to verify a pod cgroup actually carries
`memory.max` rather than trusting the QoS class, what is deliberately
left uncovered (Kubernetes static pods are Talos-managed; a handful of
vendored containers expose no upstream resources knob), and how to relax
the global-PSI backstop clause of the trigger for a cluster that trips
it persistently.

Ordering note: this documents behaviour introduced by the LimitRange and
DaemonSet PRs, so it should not merge ahead of them.

### Downstream repositories

Walked the trigger map against the diff. This adds one file under
`docs/operations/`, alongside the existing pages there; nothing in the
map covers that path, and the website keeps its own copy of user-facing
documentation rather than generating it from this directory. A
maintainer may still want a version of this on the website, which is a
judgement call rather than a forced follow-up.

- [x] No downstream repository is affected by this change

### Release note

```release-note
docs(platform): document why system components carry memory limits, how the Talos userspace OOM handler selects victims, how to tune or disable the system-namespace LimitRange, and why PriorityClass does not influence any of it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added guidance on Talos userspace out-of-memory behavior and system
component memory limits.
* Documented namespace boundaries, installer tuning, validation
constraints, rollout behavior, and verification steps.
* Added instructions for inspecting OOM actions, identifying uncovered
workloads, evaluating trigger expressions, and configuring optional
overrides.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 13:49:46 +03:00
Aleksei Sviridkin 22ce998890 fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy (#3546)
## What this PR does

Fixes #3544.

`flux-shard-operator` clones the flux-aio `helm-controller` container
into each `helm-controller-shard<i>` Deployment. It already sanitises
the all-in-one localhost wiring, but keeps two things that stop the
standalone shard from starting behind a corporate HTTP proxy:

1. **Inherited `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`.** A standalone
shard needs no external egress (source-controller fetches artifacts).
Behind an unreachable proxy the controller makes a blocking startup
HTTPS call that never completes, the manager never serves `/healthz`,
and the liveness probe crashloops the pod forever. flux-aio survives
only because it starts once and never restarts. On an affected cluster
every HelmRelease carrying a `sharding.fluxcd.io/key` (150 of ~240) was
frozen for days, silently.

2. **No `startupProbe`.** The clone inherits only a ~30s liveness
window, so any slow start (large cache sync) is fatal and unrecoverable.

This PR extends the sanitisation to drop the proxy env, and derives a
`startupProbe` from the liveness handler with a generous budget so a
slow-but-progressing start is not killed. Verified by an A/B on a live
cluster (throwaway helm-controller with a non-matching selector): with
the proxy env it hangs; without it, it starts cleanly (`Starting
workers`).

### Downstream repositories
- [x] No downstream repository is affected by this change

Internal operator behavior only; no API/CRD/values contract change.

### Release note
```release-note
fix(flux-shard-operator): sharded helm-controller no longer crashloops at startup behind an HTTP proxy; the cloned controller drops the inherited proxy env and gains a startupProbe, so HelmReleases assigned to shards keep reconciling in proxied environments.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Prevented inherited proxy settings, including uppercase and lowercase
variants of `NO_PROXY`, from affecting Flux shard deployments.
* Automatically adds a startup health check when only a liveness check
is configured.
* Preserved the liveness check’s handler and timeout while applying
appropriate startup timing and failure settings.

* **Documentation**
  * Documented startup health checks for slow helm-controller startup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 13:28:51 +03:00