Commit Graph
7777 Commits
Author SHA1 Message Date
Andrey Kolkov ff52aeb44e chore(etcd-operator): bump to v0.5.6 (#4425)
## What this PR does

Bumps the vendored `etcd-operator` from **v0.5.5** to
**[v0.5.6](https://github.com/cozystack/etcd-operator/releases/tag/v0.5.6)**.

v0.5.6 is a controller-only bug-fix release; the two changes are:

- **Terminal-phase `EtcdMember` Pods are replaced**
([etcd-operator#370](https://github.com/cozystack/etcd-operator/pull/370)).
A graceful node shutdown SIGTERMs etcd, which exits 0, and the kubelet
parks the Pod in `Succeeded` without deleting it. The operator manages
bare Pods and only recreated on `NotFound`, so after a whole-cluster
reboot every member stayed down until the Pods were deleted by hand. The
reconciler now deletes an owned Pod in a terminal phase and lets the
next pass recreate it against the same PVC (or, for memory-backed
members, through the existing pod-loss re-add path).
- **`EtcdCluster` status stays truthful while etcd is unreachable**
([etcd-operator#369](https://github.com/cozystack/etcd-operator/pull/369)).
On a converged cluster the learner-promote and auth reconcile paths
returned their transient requeue before `updateStatus`, so
`Available`/`Degraded` and `readyMembers` froze at the last healthy
value and a fully down cluster kept advertising `Available=True`. The
status write now always happens, with the requeue carried through it.

Package changes are pin-only:

- **`etcd-operator`**: `appVersion` → v0.5.6 (image tag derives from
it), `ETCD_OPERATOR_REF` → v0.5.6, `tests/deployment_test.yaml` image
pins → v0.5.6.
- **`etcd-operator-crds`**: `ETCD_OPERATOR_REF` → v0.5.6. `make update`
at the new ref produces an empty diff: the five CRDs are byte-identical
to v0.5.5, and `charts/etcd-operator/files/manager-role-rules.yaml` did
not change, so `templates/rbac.yaml` is untouched.

Verified locally: `helm unittest` 18/18 (`etcd-operator`) and 1/1
(`etcd-operator-crds`), `helm lint` clean on both, `helm template`
renders `ghcr.io/cozystack/etcd-operator:v0.5.6`, and that tag resolves
in ghcr.io.

### Screenshots

n/a — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. The change is confined to
`packages/system/etcd-operator{,-crds}` and touches only version pins:
etcd-operator is a system component (not a
`packages/apps`/`packages/extra` package the website and TF provider
track), no CRD, schema, values, `hack/`, installer or platform values
changed.

- [x] No downstream repository is affected by this change

### Release note

```release-note
chore(etcd-operator): bump to v0.5.6, which replaces EtcdMember Pods left in a terminal phase after a node shutdown and keeps EtcdCluster Available/Degraded conditions truthful while etcd is unreachable
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Updated the etcd operator to version 0.5.6. This version is used
consistently across the operator deployment and its associated custom
resource definitions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 22:14:16 +04:00
IvanHunters 380c446cda fix(monitoring): pin the scrape size rendering and sharpen the alert
The per-job limit replaces the global flag downward as well, so an
install that raised promscrape.maxScrapeSize past it through extraArgs
has to raise this value too; say so where the value is set. Pin that a
numeric value still renders as a string, which the CRD field requires,
and pin the alert severity. Name the cluster in the alert and say where
the apiserver value lives, since a guest cluster cannot set it.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:56:07 +03:00
Andrey Kolkov 88073bb209 fix(backups): fail Velero RestoreJob rename for non-VMInstance kinds
A RestoreJob that set targetApplicationRef.name to a new name and
options.targetNamespace to another namespace reported Succeeded for any
Velero-backed application, but only a VMInstance was actually renamed.
postRestoreRename looks up the HelmRelease under the vm-instance- prefix,
gets NotFound for every other kind and returns nil, so a Postgres,
MariaDB, ClickHouse, MongoDB or standalone VMDisk copy came up under the
source name with no error surfaced.

Renaming those kinds through the Velero strategy cannot work: DataUpload
restores volumes under their original names, and only a VMInstance keeps
using them after its HelmRelease is renamed, because its disks keep their
names and CDI adopts the restored PVCs. A renamed database Cluster or
VMDisk renders fresh volumes and comes up empty.

Reject the rename up front for every kind other than VMInstance, before
the RestoreJob enters Running, with a message that points at omitting
targetApplicationRef.name or using the engine-native strategy. The
existing same-namespace guard keeps precedence, so its message is
unchanged.

Closes #3537

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-23 21:30:12 +04:00
IvanHunters a17c5b2667 feat(monitoring): alert when vmagent refuses an oversized scrape
A target whose response exceeds the scrape size limit reports up == 0,
which is indistinguishable from the target being down: TargetDown and
KubeAPIDown fire and nothing names the real cause. Alert on
vm_promscrape_max_scrape_size_exceeded_errors_total so the first alert
says what is wrong. The counter has no target label, so the description
points at the vmagent targets page to find the target.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:28:29 +03:00
IvanHunters 4a268f6c01 fix(monitoring): give the kube-apiserver scrape its own size limit
kube-apiserver's /metrics grows with the number of resources and CRDs,
and on a full install it outgrows the global 32MB
-promscrape.maxScrapeSize. vmagent then refuses every scrape of it, so
up is 0 and KubeAPIDown and TargetDown fire against a healthy API
server while every rule over API server metrics goes blind.

Set max_scrape_size on the apiserver endpoint instead of raising the
global flag. vmagent uses it in place of the flag for that job only, so
the 32MB guard for every other target stays as it is. vmagent reads the
whole uncompressed body into memory even with streamParse, so the
per-job value is a memory ceiling and stays finite (128MB), exposed as
vmagent.apiserverMaxScrapeSize.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:27:00 +03:00
Andrey Kolkov e60fbc3f37 chore(etcd-operator): bump to v0.5.6
Controller-only release: EtcdMember Pods that reach a terminal phase
are replaced instead of being left behind, and EtcdCluster status is no
longer rewritten while etcd itself is unreachable. No CRD, RBAC or
values changes; re-vendoring the CRDs produces an empty diff.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-23 21:22:43 +04:00
IvanHunters 70ea242e1b docs(vm-instance-rd): trim the disable-wait comment and correct the mechanism
The annotation comment repeated the PR description and the test header;
two lines on why are enough. The hang comes from the kstatus generation
check in cli-utils, not the CEL status evaluator, since vm-instance sets
no healthCheckExprs. The config-hash rollout runs in cozystack-controller,
not cozystack-operator.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:18:06 +03:00
IvanHunters 22f2d76ead test(controller): pin the invalid-annotation and kind-selector paths of the wait carry-over
The reconciler skips a disable-wait annotation it cannot parse, and it
selects releases by application kind. Removing either left the suite
green; each now has a test that fails without it.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:18:06 +03:00
IvanHunters 5f379577d5 fix(controller): carry the readiness-wait setting to existing releases
A HelmRelease created before its ApplicationDefinition carried
release.cozystack.io/helm-install-disable-wait keeps waiting on readiness,
because cozystack-api only reads that annotation while building a release
and an existing release is not rebuilt until its Application is written
through the apps API again. Where the wait is what broke, that is every
existing instance of the kind at once, each looping upgrade and timeout,
and rewriting them by hand is the operator's only way out.

The definition already drives chartRef, valuesFrom and labels on releases
that exist, so drive the wait from there too, in both directions: the API
rebuilds the whole spec on every write and emits DisableWait=false once the
annotation is gone, so a release left disabled would drift back anyway. An
absent install or upgrade block behaves as DisableWait=false already and is
only materialised when the definition asks to disable the wait, which keeps
a release nothing asks to change from being rewritten.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:18:06 +03:00
IvanHunters 1ed32aec58 fix(controller): roll cozystack-api when a release annotation changes
The ApplicationDefinition config hash was built from name and spec alone, so
a definition whose whole change is a release.cozystack.io/* annotation left
the hash untouched and cozystack-api kept running with the configuration it
read at its last start-up. Those annotations are only ever read there, once,
in buildResourceFromCRD, which makes the config-hash rollout the single
mechanism that delivers a change in one: without it the annotation is inert
until the api pod restarts for an unrelated reason, and an inert annotation
is indistinguishable from one that was never delivered.

Hash the release.cozystack.io/* annotations alongside the spec. Only that
prefix, because kubectl and Helm rewrite annotations of their own on every
apply and hashing those would roll the api Deployment on every reconcile.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:18:05 +03:00
Aleksei Sviridkin d918887aac fix(e2e): collect the CDI controller log and CDI pods in cozyreport
CDI deletes an importer pod once, right after it marks the PVC import
Succeeded, and never retries that delete. When that one delete
does not land, the Completed pod keeps pvc-protection on the PVC and a
later DataVolume delete hangs. The report had no evidence for why: the
broken-pod walk skips Completed pods and the cdi-deployment log was
never read.

Read the cdi-deployment log (current and previous) and every pod
carrying CDI's app label as YAML, across namespaces and phases. Only
the log reads wait on the cdi-deployment probe: an importer pod is
owned by its PVC, so it can outlive the Deployment, and a missing
controller must not also drop the pod it left behind.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 19:11:28 +03:00
Aleksei Sviridkin 45f5698bce fix(cdi): validate kubevirt-cdi values against a schema (#4417)
## What this PR does

This gives the kubevirt-cdi chart a values schema, so a bad clone
strategy fails when the chart renders instead of when the CDI object is
applied. Today a typo like `csi_clone` renders fine, the CDI CRD refuses
it on apply, and the operator finds out from a failed HelmRelease.

The schema is generated by `cozyvalues-gen` from annotations in
`values.yaml`, through a new `make generate` target, the same way
`talos-log-collector` does it. There is no README here, so the target
only writes the schema.

The clone strategy is checked with a pattern, not with the CRD's enum.
The chart uses the empty string to omit the field and let CDI pick a
strategy per StorageProfile, and the CRD enum has no member for that.
`cozyvalues-gen` also can't emit an empty enum value. So the accepted
values are `copy`, `snapshot`, `csi-clone` and `""`. A null still works
as before. The key has a chart default, so Helm drops a null for it
before validation, and the template omits the field. `uploadProxyURL`
only has to be a string.

The new test suite refuses `csi_clone` and values of the wrong type, and
accepts each of the allowed values and null.

Closes #4141

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): kubevirt-cdi now validates its values against a schema, so an invalid `cloneStrategyOverride` such as `csi_clone` fails at render time instead of when the CDI object is applied.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration**
* Chart values are now validated to ensure the upload proxy URL is a
string, importer resources are provided as an object, and clone strategy
uses a supported value: `copy`, `snapshot`, `csi-clone`, or an empty
value.
  * Existing defaults for these settings remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 20:07:06 +04:00
Aleksei Sviridkin 5e460605c2 fix(cdi): drop the ExpandDisks gate from the CDI object (#4415)
## What this PR does

This drops a feature gate from the CDI object that CDI never reads.
`ExpandDisks` sits in `spec.config.featureGates` in
`packages/system/kubevirt-cdi/templates/cdi-cr.yaml`, but it is a
KubeVirt gate.

CDI v1.66.1 has no gate with that name. It looks a gate up by exact
name, so an extra entry is read and ignored. The CRD types the list as
plain strings, so nothing rejects it either. Disk expansion comes from
KubeVirt, where `ExpandDisks` is GA at v1.9.0, the release the platform
ships. I left the KubeVirt CR as is; #4029 tracks the GA gates in that
list.

The gate list now has its own test case, pinned whole to what upstream's
`cdi-cr.yaml` ships. Adding `ExpandDisks` back turns it red. `make
update` only checks the upstream file against its pinned hash and writes
nothing under `templates/`, so it can't bring the line back.

Closes #4112

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): stop listing the KubeVirt `ExpandDisks` feature gate on the CDI object, where CDI ignored it. Disk expansion is unaffected.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration**
* CDI now enables only the `HonorWaitForFirstConsumer` feature gate. The
`ExpandDisks` gate is no longer included in the configuration.
* **Tests**
* Updated configuration checks to verify the exact feature-gate list and
ensure it matches the gate CDI supports.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 20:06:31 +04:00
Aleksei Sviridkin 25d4a8c951 fix(kubevirt)!: evict non-migratable VMs instead of blocking the drain
The cluster-wide evictionStrategy was LiveMigrate. For a VMI that is
not live-migratable (a ReadWriteOnce disk, a passthrough GPU, a host
device), virt-api's eviction webhook answers every eviction with 429,
so a node drain retries forever and never completes.

LiveMigrateIfPossible keeps live migration for every VMI that can
migrate and lets the eviction through for the rest. The VM is shut
down and started again according to its runStrategy. A VM whose disk
is bound to the drained node stays down until the node returns.
KubeVirt writes the cluster strategy into a VMI's spec when the VMI
is created, so a VMI started before the upgrade keeps LiveMigrate and
still blocks the drain until it is restarted once.

A drain and a launcher update after a KubeVirt upgrade go through the
same eviction, so the workload updater's Evict fallback shuts down
non-migratable VMs started under the new strategy on every later
KubeVirt upgrade, where it used to get the same 429. Evict stays
anyway: for kubernetes-nodes workers, which use evictionStrategy
External, that eviction marks the VMI for evacuation, CAPK drains the
tenant node and deletes the VMI, and the VM starts it again on the new
launcher. The test pins the method list so the fallback is not dropped
to avoid the shutdowns.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 17:36:56 +03:00
Aleksei Sviridkin 1f30ac4598 fix(tests): let the mariadb waits cover a first boot (#3769)
## What this PR does

Closes #3741

Several waits for a database HelmRelease to become Ready gave up before
the release could: the mariadb, mongodb, rabbitmq and clickhouse backup
walkthroughs waited 300 seconds, and both mariadb Chainsaw suites gave
their HelmRelease-ready assert 6 minutes. A generated release carries a
10-minute install timeout (`pkg/cmd/server/start.go:91`), so a failed
install was reported as a bare timeout, because the release had not yet
recorded why it failed, and for mariadb a first boot the startup probe
was still willing to wait for failed the harness instead.

A wait on such a release has two floors and has to beat the larger. The
first is the app's own first boot: cozystack-api leaves
`spec.waitStrategy` unset and helm-controller defaults it to `poller`,
so the install waits through kstatus, and kstatus maps the operator CR's
`Ready=False` onto `InProgress`. For mariadb that boot may take the
chart's 310-second startup budget (`startupProbe.failureThreshold: 30`
in `packages/apps/mariadb/templates/mariadb.yaml` with the operator's
probe defaults of 20 and 10); mongodb, rabbitmq and clickhouse set no
startup probe. The second is the install timeout. At or below it the
waiter is the one that expires, because its clock starts when the step
runs and the install's only once the Application has been reconciled and
the chart artifact fetched. The install timeout binds in every case
here, so the walkthrough waits go to 660 seconds and the mariadb asserts
to 11 minutes.

Raising a wait inside a walkthrough spends the Chainsaw op that runs the
script, and an op that runs out SIGKILLs the process group and loses the
script's error. Each op therefore grows by exactly what the waits it
reaches gained: mariadb's round-trip from 30 to 42 minutes, rabbitmq's
from 30 to 42, mongodb's from 30 to 36 (its round-trip runs with
`SKIP_RESTORE`, so only the source wait counts), and clickhouse's from
60 to 72, which still sits above the sum of every inner ceiling. That
bounds a failure at a single wait, not a run that is slow at every stage
and then fails late; bounding that needs a phase budget, not a larger
literal. postgres keeps its 360-second waits and every walkthrough keeps
its 300-second bucket wait: whether a suite pays the extra minutes on
every red run is a per-suite call.

The mariadb endpoint asserts keep their job. A freshly applied CR
carries no conditions and kstatus reads it as current, so the install
wait can end before any pod is up; an endpoint address closes that
window. That is also why the `wait_for_field` on the CR stays in the
walkthrough.

`wait_hr_ready` now dumps every condition and the release history on
timeout and on `Stalled=True`, instead of the single Ready message a
scheduled retry can overwrite before anyone reads it. It also stops
printing one `(none recorded)` line for different answers: a release
that never appeared, a release deleted during the wait, a lookup kubectl
could not answer, a failed history read and an empty history each say
so, with kubectl's own stderr kept, and a release seen earlier still
gets its dump when only the last lookup failed. The change goes to every
byte-identical copy of the helper, in
`examples/backups/{clickhouse,mariadb,mongodb,postgres,rabbitmq}/00-helpers.sh`,
and they stay identical.

`hack/mariadb-first-boot-waits.bats` pins the mariadb budgets against
the tree. It derives both floors from their sources instead of pinning
literals, so raising `failureThreshold` past the install timeout, or
changing the install-timeout default, reds it; each source has a test
that refuses to proceed when it cannot be read, including the
ApplicationDefinition, whose two override annotations would each
invalidate the derivation. It holds the round-trip op at a fixed
remainder over the ceilings up to and including the source wait, bucket
stage and S3 preflight included, and at a fixed slack once both mariadb
waits are paid, so raising a wait without the op reds. The extractors
are also tested against fixtures, because the tree cannot tell a correct
extractor from a broken one. Beyond mariadb, the file runs every exit
branch of `wait_hr_ready` in each copy against a stubbed kubectl, and
pins the copies byte-identical. The population is every definition in
the tree found by grep, minus two named exceptions held to still differ,
so a new copy joins without editing the test and an exception whose copy
converges goes red:

- `examples/backups/redis/00-helpers.sh` is written in the compact
one-line style of the rest of that file, so converging it rewrites its
shape rather than applying the shared body.
- `examples/backups/kafka-metadata/00-helpers.sh` is a helper of its own
with the same diagnostic gap on timeout (it prints no reason at all),
and closing that is a change to kafka-metadata.

`docs/agents/e2e-testing.md` replaces the reviewer checklist's 5-6m
standard and its list of exceptions, which the tree had outgrown, with
the floor a wait has to beat and the rule that a longer wait carries its
reason in-line. The derivation gets a section of its own, opening with
the rule in one line. For a release an app chart renders itself the
floor is `spec.install.timeout`, then `spec.timeout`, then
helm-controller's 5m default, and the nested `<name>-system` releases
never take cozystack-api's 10m. The harbor comments name the floor each
of their asserts sits against, and the opensearch comment stops citing
the removed standard.

Adding `hack/mariadb-first-boot-waits.bats` does not escalate the E2E
selection: `hack/[^/]+\.bats$` sits in `inert_config_pattern` in
`hack/select-e2e.sh`, and `full_suite_pattern` escalates only
`hack/e2e-*.bats`. On this changeset `hack/select-e2e.sh` selects
`clickhouse harbor mariadb mongodb opensearch postgres rabbitmq`.

What this costs is paid only on a failing run; the raised numbers are
ceilings. Summing the timeouts stated on each suite's own steps, mariadb
goes from 92m to 114m, clickhouse from 116m to 128m, rabbitmq from 67m
to 79m and mongodb from 95m to 101m, before the untimed steps that fall
back to the defaults in `.chainsaw.yaml`. clickhouse is the most
expensive suite by that sum. All of them compete for the 215m
`timeout-minutes` of the whole `e2e` job, which also covers image
download, sandbox preparation and the Cozystack install, and a job
cancelled at the cap loses both its log and `cozyreport.tgz`.

This does not fix the failure that led here. In the run that prompted
the investigation the mariadb replica never opened port 3306 at all,
spending its whole 310-second budget inside `mariadb-install-db` before
being killed, and no wait budget would have saved it. That defect is
#3744. What this fixes is a second, independent defect found while
reading the same code.

One thing here is argued rather than measured: the mechanism is checked
against helm-controller and kstatus, and every number is derived from
the tree, but no run ID pins that it is what any particular red run hit.
Both floors justify 660s whatever the cause was, so being wrong about
the cause leaves slack rather than breakage.

### Not in this PR

- This PR retires the mariadb, mongodb, rabbitmq and clickhouse
walkthrough waits and the two mariadb asserts. The rest is tracked in
#4404: the other Chainsaw HR-Ready asserts under their floor, the harbor
asserts, the 300-second bucket waits, the postgres waits, and the redis
and kafka-metadata diagnostic gap.
- The clickhouse round-trip comment says every wait fails fast,
`wait_hr_ready` on `Stalled=True`, "so a genuinely broken backup reports
in seconds". A release whose install is still running is neither Ready
nor Stalled, so that holds for a stalled release and not for an install
that has not finished. The sentence predates this change.
- `hack/mariadb-first-boot-waits.bats` is large for what it guards (two
floors, two ratchets on the op, and the helper identity pin), with many
fixture tests on its own extractors. Trimming it to that core is a
separate decision.
- Two comments in that file state facts about the mariadb suite that are
true today and that nothing checks: three script ops carry a minute
timeout, and the only one ahead of the round-trip is the 8m pre-clean.
- The mariadb op comment says a bound on the whole flow "needs a phase
budget (docs/agents/e2e-testing.md)"; the doc describes it as bounding
the collection phase as a phase (`COZY_DIAG_PHASE_BUDGET`), so the
pointer would resolve faster if it named that bullet.
- Cosmetic: the behavioural test's loop body is not indented under its
`for`, one line of the bats header runs past the width of the rest, and
the checklist's sentence under the example assert ("what this block
demonstrates least") reads awkwardly.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file, for all eighteen files. The five
`examples/backups/*/00-helpers.sh` files change the diagnostics of
`wait_hr_ready`, and the mariadb, mongodb, rabbitmq and clickhouse
walkthrough scripts change wait budgets; the manifests the walkthroughs
apply and the flows they document are unchanged, so no website page
restates anything that moved. Four Chainsaw suites (mariadb, clickhouse,
mongodb, rabbitmq) change timeouts and comments, and the harbor and
opensearch suites change comments only. `docs/agents/e2e-testing.md`
changes the reviewer checklist entry and adds the sizing section.
`hack/mariadb-first-boot-waits.bats` is a new file picked up by the
existing `BATS_UNIT_FILES` wildcard. `hack/package.mk`,
`hack/common-envs.mk` and `hack/update-crd.sh` are untouched, and no
`values.schema.json`, `values.yaml` default, version enum,
`ApplicationDefinition`, CRD, namespace or release asset is touched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tests): the MariaDB, MongoDB, RabbitMQ and ClickHouse backup examples and the MariaDB Chainsaw suites now wait for a HelmRelease past the release's own install timeout, so a failed install reports its own reason instead of a bare timeout, and the examples' HelmRelease wait now tells a release that never appeared, one deleted mid-wait and a kubectl that could not answer apart.
```
2026-09-23 18:23:04 +04:00
Aleksei Sviridkin 9c97a7a7ac fix(cdi): validate kubevirt-cdi values against a schema
Without a schema, a cloneStrategyOverride outside the CDI enum, such as
csi_clone, renders a CDI object the API server refuses on apply, so the
mistake shows up as a failed HelmRelease rather than at the value.

The field is constrained by a pattern, not by the CRD's enum. The empty
string is the chart's way to omit the field and let CDI pick a strategy
per StorageProfile, the CRD enum has no member for it, and
cozyvalues-gen cannot emit an empty enum value. The key has a chart
default, so Helm drops a null for it before validating, and a null
keeps rendering as an omitted field.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 17:06:56 +03:00
Andrey Kolkov 7ce85a2f7f test(kafka): pin the strategy script's guards with a hermetic bats suite
The in-Pod script had grown eight fail-closed guards across as many
rounds, and the only thing exercising any of them was a 40-minute
cluster test that drives the happy path plus one refusal. The partition
listing guard in particular - the whole of the previous round's second
fix - could be deleted with nothing in the tree going red. Add
hack/kafka-topic-backup-strategy.bats on the shape of
hack/kafka-backup-script.bats: render the strategy through a stub
kubectl exactly as step 01 applies it, lift the container body out of
it, and execute it against stubbed kafka-*.sh, a stubbed curl and a
directory standing in for S3, with fault injection for the broker
behaviours the guards exist for. Deleting that listing guard now fails
one case; the script gains BIN/SCRATCH/CA_FILE overrides so it can run
outside the Pod, each defaulting to the Pod's value.

The same rounds' smaller findings: the manifest's topic name is held to
Kafka's charset like the topics parameter, since it reaches unquoted
expansions with globbing on; the shape check after --create compares
the replication factor as well as the partition count, and refuses a
replication factor below the topic's effective min.insync.replicas
before replaying, because kafka-console-producer only logs a rejected
acks=all write and the run would otherwise end at the count check with
the cause unnamed; the manifest's replication factor must agree across
a topic's lines; topic_records carries its own || return 1 on each
command, as a function called as the operand of || runs with errexit
off; the CA lookup in step 03 distinguishes a denied read from an
absent Secret, and the README's persona column says the copy needs the
admin; the e2e comment no longer claims the 2-label endpoint fails the
serving cert's SANs, which include '*.tenant-root'.

Rebased onto main, which moved the chart to KRaft: the demo's Kafka CRs
drop the zookeeper block the chart no longer has.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
2026-09-23 18:00:15 +04:00
Aleksei Sviridkin 0c358addfb fix(cdi): drop the ExpandDisks gate from the CDI object
ExpandDisks is a KubeVirt feature gate. CDI declares no gate by that
name and matches gate names exactly, so the entry was accepted and
ignored. Disk expansion is a KubeVirt feature, GA there at the release
the platform ships, so listing the gate on the CDI object points a
reader tracing where expansion comes from at the wrong component.

The feature-gate list is pinned whole in its own case, so a gate added
beside the one upstream ships turns the suite red.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 16:58:56 +03:00
Aleksei Sviridkin 2b5770cb4a fix(vm-instance)!: refuse resources that cannot size the VM (#4405)
## What this PR does

A vm-instance could ask for half a core and get one, with no error
anywhere. `resources.cpu` and `resources.sockets` are typed as
quantities, but they end up in `domain.cpu.cores` and
`domain.cpu.sockets`, which KubeVirt declares as unsigned integers. The
chart cast them with `int64`. That turns `500m` or `1.5` into `0`
without an error, and KubeVirt boots a zero core count as one core.

The chart now reads these two fields the way an integer field reads
them. It decodes the raw value as YAML and stops the render unless the
result is a positive whole number that fits in uint32. So `2`, `"2.0"`,
`"+2"` and `"1e1"` render as the integer they decode to. `500m`,
`2000m`, `1k`, `"1.5"`, `0`, `"0"`, `"-2"`, `"010"` and `"4294967296"`
stop the render with an error that names the field. A leading zero is
refused because YAML reads `010` as octal 8. The check looks at
presence, not truthiness, so an unquoted `0` is refused instead of being
read as unset. It is the same rule #4386 uses for kubernetes-nodes.

Memory gets a presence check too, which covers the case from the issue
thread. A quoted `"0"` was truthy and rendered `requests.memory: "0"`,
while an unquoted `0` was falsy and got dropped. Both spellings of zero,
and any negative amount, now stop the render. Any other value passes
through unchanged, because KubeVirt takes memory as a quantity. That
includes an amount below one byte, such as `500m` or `1e-3`, which
renders as it does on `main` today; this change does not refuse it.

The check lives in `virtual-machine.domainResources`. The VM spec, the
instance type decision and the update hook's patch all take their values
from that helper. The tests that pinned `cores: 0` as the old behaviour
now expect the refusal, and the helper comments about the truncation are
trimmed.

This is a breaking change. A release that carries one of these values
fails its next upgrade until the value is fixed: a cpu or sockets that
is not a positive whole number within uint32 or has a leading zero, and
a zero or negative memory. This includes a cpu or sockets set on its own
next to an `instanceType`, which used to be ignored whatever it held.

Some values change meaning instead of failing. A value like `"1e1"` or
`"2."` was truncated to 0 by the `int64` cast, so the VM booted with one
core. Now it gets the count the value spells, so that VM changes size on
upgrade.

I refuse these values instead of rounding them, because silent coercion
is how `500m` became one core in the first place. The issue asks for a
deprecation path, and I did not add a migration that rewrites stored
values. It would have to guess what `500m` meant, and both options (one
core, or dropping the field) hide the same mistake again. The
HelmRelease upgrade retries without uninstalling, so the running VM
stays up until someone fixes the value. #4386 makes the same trade-off.

The check runs at render time, not at admission. The schema still
accepts `500m`, so the API takes the object and the error shows up in
the HelmRelease status. Narrowing the schema to integers, as the issue
also proposes, would move the error to admission, but it needs a
migration of stored values. I left it for a separate change.

The field descriptions now say which values are accepted, and `make
generate` carried them into the README, the schema, the Go types and the
ApplicationDefinition.

The new `tests/resources_values_test.yaml` covers the accepted and
refused spellings for cpu, sockets and memory.
`tests/instancetype_resources_test.yaml` gets a case for a lone cpu next
to `cpuModel`. The whole vm-instance suite passes. Every refusal case in
the new file, and the `"1e1"` case, fails against the current `main`
template.

Closes #4104

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map against the diff. No field is added, removed or
renamed, and no default or enum changes. Only three field descriptions
change, and the website reference page is regenerated from the README at
release time.

### Release note

```release-note
fix(vm-instance)!: resources.cpu and resources.sockets must now be positive whole numbers, and resources.memory a positive quantity. A cpu or sockets that is not a plain whole number within uint32 (for example 500m, 1k or 1.5), or a zero or negative memory, now fails at render time with an error naming the field, instead of silently becoming a one-core VM or a zero memory request. A release carrying such a value fails its next upgrade until the value is fixed. A value such as 1e1 or 2., which used to be truncated to one core, now sizes the VM with the count it spells.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* VM sizing now accepts only positive whole-number CPU and socket values
within the supported range. Fractional, millicore, zero, negative, and
out-of-range values are rejected during rendering.
* Memory values must be positive quantities. A CPU or socket value
specified alone does not size the VM and is ignored; incomplete sizing
is rejected where required.

* **Documentation**
* Updated VM instance resource guidance to clarify CPU and socket
requirements, memory validation, and how socket values define vCPU
topology.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 17:58:36 +04:00
Aleksei Sviridkin 99de79ff52 fix(cdi): pin the CDI release kubevirt-cdi is reconciled against and stop update overwriting it (#4107)
## What this PR does

`packages/system/kubevirt-cdi` resolved its CDI version at run time from
GitHub's `releases/latest` redirect and wrote the result straight over
`templates/cdi-cr.yaml`. That file is not vendored. Upstream ships a
bare CDI object; ours adds Helm templating for `cloneStrategyOverride`,
`uploadProxyURL` and the worker-pod `podResourceRequirements`, the
`ExpandDisks` feature gate, a `CriticalAddonsOnly` toleration, and the
two RBAC documents that let a tenant clone from the golden-image
namespace. I diffed the two at v1.66.1 to write that list, so it is
complete as of that release rather than only hedged: upstream ships 283
bytes with one feature gate, two nodeSelectors and an `imagePullPolicy`,
and we are a strict superset of it. Every entry arrived by hand, so
`wget -O templates/cdi-cr.yaml` was not refreshing a vendored file, it
was overwriting a first-party one. Same loss as the `rm -rf templates`
that #4097 removes, arriving through the fetch instead of the wipe.

`update` now pins the release the object is reconciled against, and
fetches it only to confirm the bytes still hash to a digest I measured.
Nothing is written under `templates/`. At the next CDI bump whose asset
differs, that check fails, which is the point: upstream moving its own
defaults is when the local copy needs a human to reconcile it, not an
overwrite. The download is kept on a mismatch, since reconciling one
starts by reading it.

`RELEASE` is `v1.66.1`, which is the release `kubevirt-cdi-operator`
vendors, so the pin and the deployed version agree. What the platform
deploys is decided by that package, and it can move to another release
without this file changing, so this line is pointed at it by hand.
`update` then checks the two agree: it compares `RELEASE` with the
`cdi-operator` image tag that package vendors and fails naming both
versions when they differ, because otherwise its closing line would
report a release the platform may no longer run. That is a check and not
a derivation, so `RELEASE` stays a pin a human moves.

Upstream publishes no checksum for the asset, so the pin comes from a
download rather than from a published file:
`a497f90de608c1df26f9ee4095289373f17132d74a2042ca03e00c17c964f8a7`. It
catches an asset that moved under a tag that should be immutable. It is
not authenticity verification, and the comment beside it says so.

The digest command is chosen when `update` expands it, `sha256sum` when
it is on PATH and `shasum -a 256` otherwise. Which one ran here is worth
stating, because the usual claim about macOS is out of date: this
machine picked `sha256sum`, and that binary is `/sbin/sha256sum`, a
native Darwin one rather than GNU coreutils. It reports `sha256sum
(Darwin) 1.0` and prints the same `<digest> <path>` format.

The package already had a `test:` target, from `0418577f5`, so what it
gains here is the `.PHONY` line that commit did not add.
`hack/package.mk` declares only the nine targets it defines itself, so
without it a file or directory named `test` or `update` beside the
Makefile makes make consider that target up to date and skip the recipe
in silence. The new suite pins the `ExpandDisks` gate, the toleration,
`podResourceRequirements`, the `uploadProxyURL` override and the clone
RBAC, and renders the upload-proxy Ingress and TLSRoute, each under the
`gateway-enabled` value that selects it. The clone-strategy override is
pinned by `tests/clone_strategy_test.yaml`, which is already on `main`.

One consequence of the pin move is a claim rather than a number, so it
gets its own line. Upstream carried `WebhookPvcRendering` from v1.63.0
through v1.65.0 and dropped it by v1.66.0; the local object never
carried it, so the divergence ran both ways across those three releases
and one way since. The comment says to re-derive the delta at each bump
instead of trusting it, because that was a fact about a span of releases
and not about CDI.

`packages/system/kubevirt-cdi/Makefile` carried two identical `test:`
targets, one from `0418577f5` and one appended by a later merge, so make
printed `overriding commands for target 'test'` on every invocation in
the package. The second sat inside the span this recipe rewrites, so it
goes with the rewrite rather than being re-authored into it. The comment
kept above it is the reason a reader should not add another. Closes
#4157.

### Scope

An earlier revision also pinned and tested `kubevirt-cdi-operator`.
#3920 has since landed the two real bugs in that recipe, the patch strip
level and `patches/tolerations.diff`, so that half is dropped here
rather than merged twice. Pinning and testing the operator package is
#4190.

Named improvements left alone, so they read as choices.

- The two upload-proxy tests each render their route under the
`gateway-enabled` value that selects it and neither asserts the other is
absent; the templates already settle that with `ne "true"` and `eq
"true"` on the same variable, and `tests/standalone_render_test.yaml`
already asserts each route absent under the values that select the
other. So the package does cover it; these two cases just do not, and a
third copy would not add anything.
- The CDI object carries no version string for a rendered-output
assertion, so `RELEASE` is checked by `make update` rather than by `make
test`.
- Neither route case pins the backend service port, 443 in both
templates; that is outside what the suite claims.
- The template-loss guard still counts `cdi-cr.yaml` as named, because
the download line in `update` mentions the stem, so re-adding a wipe
here would have it report two files instead of three; the guard's header
already says a mention is not proof of a write, and the case needs a
wipe this recipe no longer has.
- The suite pins the local additions by design, and most of the
upstream-derived half is pinned by nothing: `imagePullPolicy` and the
two `nodeSelector` blocks have no assertion anywhere under `tests/`.
`HonorWaitForFirstConsumer` is the exception and only as a side effect:
`featureGates` is pinned whole, so that gate rides along with
`ExpandDisks`. The rest are maintained by the same hand reconciliation
the pin forces whenever the upstream asset changes, so a carry-over that
drops one of them is green. Three asserts would close what is left.
- One more, remote enough to name rather than code around: if the digest
tool is missing entirely, `$$actual` comes out empty, the comparison
against `CR_SHA256` fails, and the error text diagnoses a moved asset
when the real cause is a machine with neither `sha256sum` nor `shasum`.
Busybox ships the first and macOS ships the second, so nobody has hit
it. Closing it means an executable line, which would then owe mutation
coverage of its own, and that is disproportionate to a case with no
known instance. The mechanism is written here so the next reader does
not have to re-derive it from an error message that points the wrong
way.
- Separately, the per-package `.PHONY` gap this Makefile's own line
argues against is not specific to this package: most package Makefiles
do not declare their own `test:` or `update:` phony, and #4214 tracks
that.
- A key that does not exist yet is not held either: every item in the
delta is pinned whole, but a NEW key appearing at `spec` or
`spec.config` level passes unnoticed, and such a key can change
behaviour rather than sit inert. I confirmed it on a copy extracted from
this commit, appending `spec.config.filesystemOverhead.global: "0.99"`
and getting a green suite. That is the shape to recognise at the next
bump: not a changed value, which is caught, but a new sibling key beside
the ones we pin.
- And `spec.config.podResourceRequirements` is now pinned whole here and
by four leaves in `tests/importer-resources_test.yaml`: the whole-map
form is the stronger one and the leaves are a subset of it, so changing
the default takes two edits and the second failure reads as a surprise
until you know why both exist.
- If the operator's image reference ever gains a digest
(`:v1.66.1@sha256:...`), `${deployed##*:}` takes the digest's hex
instead of the tag, and `update` fails with a loud false mismatch rather
than passing silently.
- Deriving `RELEASE` with `$(shell ...)` from the sibling manifest was
suggested and declined on merits: a derived value is not a pin, the
point is that a human chooses which CDI release this object is
reconciled against, and computing it from whatever the neighbour last
vendored hides that choice from the diff.

The operator comparison lives in `update` and not in `test:`, even
though `kubevirt-operator` ties its own manifest to the tree from
`test:`. A mismatch calls for a hand reconciliation, and `update` is the
recipe that asks for one; it is also where the success line that would
otherwise name the wrong release is printed. I checked it on a copy
extracted from this commit. With the operator's image tag moved to
v1.67.0, `make update` exits 2 and names both versions. With the tags
equal it passes. With the check's `exit 1` removed, the moved tag passes
again, so the red comes from the check.

Worth knowing when reading the mismatch path: `kubevirt-operator`
deletes its download on a hash mismatch and this package keeps it. That
is deliberate here, since reconciling a moved asset starts by reading
it, but the two packages do read oppositely and neither file says so.

### Base and ordering

This stacks on #4097 and should land after it. Both edit
`packages/system/kubevirt-cdi/Makefile`, and this change builds directly
on that one's edit to the same recipe, so it is based on the head of
`fix/cdi-update-keeps-local-templates` rather than on `main`. GitHub
computes mergeability against the PR's base branch, which here is that
stack branch and not `main`. Both branches are rebased onto current
`main`, which has not touched these paths since the previous merge-base.


### Screenshots

No UI changes.

### Downstream repositories

Walked the trigger map against the diff. Two entries come close and
neither fires.

The website triggers on developer tooling and package Makefiles, but
`content/en/docs/next/development.md` documents `make update` only as
"Update Helm chart and versions from the upstream source", which still
holds, and its worked example is cilium, which this does not touch. Its
`make test` line refers to the e2e sandbox in `packages/core/testing`,
not per-package unit tests.

`ccp` triggers on `hack/` layout and on what a make target does, and
nothing under `hack/` is touched. Its `package-bump` skill already
assumes a package's `update` recipe hardcodes the upstream version
instead of resolving it at run time, and names cilium, kafka-operator,
opensearch-operator and postgres-operator as examples. `kubevirt-cdi`
was one of those counterexamples and now is not. Pinning
`kubevirt-cdi-operator` the same way is #4190 and out of scope here. The
skill handles vendored Helm charts, in-repo image builds and version
enums, and a raw manifest download is none of those. Everything else in
the map needs an app, a schema, an installer value, a variant or a
release asset.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): `make update` in kubevirt-cdi no longer overwrites the hand-maintained CDI object. It pins the CDI release that object is reconciled against instead of resolving GitHub's releases/latest at run time, and verifies the fetched asset against a pinned sha256 without writing under templates/.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Reliability**
* Updates now use verified, pinned CDI releases to prevent unintended
manifest changes.
* Failed or checksum-mismatched downloads no longer overwrite existing
configuration.

* **Bug Fixes**
  * Preserved local CDI customizations during release updates.
* Added validation for CDI resources, access controls, upload-proxy
routing, and TLS configuration.

* **Tests**
* Added automated checks for incomplete template refreshes and generated
Kubernetes resources.
* Added chart tests covering CDI behavior, clone permissions, and
upload-proxy networking.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 17:53:23 +04:00
Andrey Kolkov 5bd029c46a fix(kafka): restore with the captured replication factor and a complete listing
Record each topic's ReplicationFactor in the backup manifest, off the
same --describe line the partition count already comes from, and
recreate the topic with it on restore. The value used to be a
BackupClass parameter defaulting to 1, and the Cozystack Kafka chart
sets the broker-level min.insync.replicas to 2 as soon as kafka.replicas
reaches 2: kafka-console-producer writes with acks=all, the broker
rejects every write into a topic with fewer replicas than that, and the
producer logs the rejection and exits 0. A restore onto any multi-broker
cluster therefore landed zero records and failed the post-replay count
with a message naming the wrong cause; in place, the documented delete
had already taken the live topic with it. replicationFactor stays as an
override for restoring into a smaller cluster, and a value the target
cannot host now fails at --create in Kafka's own words.

Hold the pre-replay emptiness check to the topic's partition count, as
the backup side already does. kafka-get-offsets skips a partition it
could not read and still exits 0, so a leader election in flight dropped
the populated partitions from the listing, the surviving empty ones
summed to zero, and the replay appended a full copy into a live topic -
the exact outcome the check was moved ahead of the replay to prevent.
Both restore-side counts now go through one helper that refuses a short
listing. The data-file-versus-manifest check runs both ways before
anything is created, and the replay walks the manifest topic by topic,
announcing each one, so a retried Pod's log says which topics landed.

The demo now runs two brokers with two-replica topics, which is the
shape the chart puts min.insync.replicas=2 on; the restore steps assert
the restored topic's replication factor, and step 06 submits a second
RestoreJob against the populated topic and requires it to be refused
with the topic untouched.

Smaller findings from the same round: the topics parameter is held to
Kafka's name charset (a literal "*" no longer path-expands) and refused
on a repeated name, which produced a Succeeded backup no restore could
balance; the strategy verifies S3 TLS against the seaweedfs CA projected
into the <app>-backup-s3 Secret instead of curl -k, following the
rabbitmq example, so the e2e endpoint moves to the .svc name the serving
cert covers; the image lookup keeps digest-pinned references; the
default namespace is tenant-root, the only one the demo ever ran in;
step 05 caches the resolved Backup name for the restore steps; the decoy
count subtracts begin offsets like the topic count; the bats test name
carries its count again; and the scenario walk-throughs the sibling
examples ship are added.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov b141166cde fix(kafka): key begin-offset lookups on topic:partition
The begin-offset lookups matched on the partition index alone, so a line
belonging to another topic could supply this partition's begin. The cut
then shifts silently and the drain check cannot see it, because n is
derived from that same wrong begin. Match on the whole topic:partition
key in all four lookups, which holds whatever order the listing arrives
in and no longer depends on the topic filter being pinned.

Hold the tarball's file names to the manifest as well. The manifest is
validated as untrusted input, but the replay derived its topic from the
data file names, so a file naming a topic the manifest never listed was
replayed into a topic that skipped the create, shape, emptiness and
post-replay checks while the restore still reported success.

Correct what the decoy is documented to prove. It reliably covers the
offset listing, the in-place delete and the two record counts; it does
not cover the two PartitionCount reads, because kafka-topics --describe
yields topics in the admin client's result-map order rather than sorted,
so which one their head -1 sees is not something a fixture can pin down.
The previous text claimed every pin reddens the run, and explained the
decoy's name by a sort order that does not govern that output.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 17475f1ee5 fix(kafka): refuse a non-empty restore target before replaying
The replay appends and the driver retries a Pod that died mid-replay, so
checking the record count only afterwards meant every attempt had already
written another full copy into a live topic and the check merely reported
the damage. Read the target's count before replaying and refuse a topic
that already holds records, leaving it as the failed attempt left it; the
post-replay check still verifies what landed.

Name the decoy so it sorts before the demo topic. Both PartitionCount
reads pipe --describe through head -1, so an unpinned describe reads
whichever matching topic comes first: with the decoy sorting after the
topic those two pins were never exercised and could be removed with the
suite staying green. "-" precedes ".", so orders-v1 now sorts first and
an unpinned describe reads its partition count instead.

Also state in the README that an empty backup is refused on restore
rather than merely uploaded, so the empty-tarball note does not read as
if such an artifact were usable.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 2c9b3dfeea fix(kafka): validate the restore manifest and pin topic matching in the suite
The manifest arrives inside the downloaded tarball but was consumed as
trusted input. A non-numeric partition field slipped through the numeric
test that computes a topic's partition count: an if condition is exempt
from errexit, so test exits 2, the term reads false, the count comes out
short, the topic is created smaller, and the shape check then compares
that short value against itself and passes. The per-topic total still
added up, so the restore reported Succeeded with every keyed record
re-placed. Validate each manifest field on read, refuse an empty manifest
whose verification loops would otherwise run zero times and report a
successful restore of nothing, and guard the restore-side PartitionCount
read the way the backup side already guards its own.

The delete-wait still matched the topic name as a regex; use grep -qxF.
The suite now pins that class of bug instead of relying on review: the
demo topic is orders.v1 and a decoy ordersXv1 sits beside it, matching the
first as a regex but not as a literal. The decoy is never named in the
BackupClass, so dropping any \Q...\E pin fails the run on its own - the
partition-set guard counts the decoy's partitions, or the in-place delete
takes the decoy with it.

Also: the host-side record count skipped a partition missing from the
begin-offset listing instead of defaulting it to 0, undercounting the
topic; the two preflight invariants now cover the kafka flow they already
describe; and the README names the disabled TLS verification.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 9cb9c70083 fix(kafka): match topic names literally, not as regexes
kafka-topics and kafka-get-offsets compile --topic into a Java regex and
full-match it, so any name carrying a metacharacter also selects its
siblings - "audit.events" matches "audit-events". Two consequences here:

- The in-place step's `--delete --topic "$TOPIC"` deleted every topic the
  name happened to match, which for a demo meant destroying a bystander
  topic to "simulate data loss".
- The partition-set guard read another topic's partitions into `ends` and
  another topic's PartitionCount into `pc`, so it either refused a legal
  backup or, when the counts coincided, wrote duplicate manifest lines and
  overwrote a data file.

Pin every filtering call with \Q...\E, the same fix the kafka-metadata
example carries. --create and the console producer/consumer take a literal
name and stay unquoted.

Also: verify the restore target's partition count matches the backup, since
--if-not-exists leaves an existing topic's shape alone and the post-replay
total cannot see re-placed records; stop wrapping the whole kafka_run in
2>/dev/null, which hid pod and API failures rather than in-pod noise; and
point the warned image fallback at the newest entry of the operator's map,
the one the resolver itself picks, instead of the oldest.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov d405a36e3c fix(kafka): validate the backup's partition set against the topic
The drain guard only covers a short read within a partition it was told
about. kafka-get-offsets lists just the partitions whose lookup
succeeded - GetOffsetShell logs a per-partition KafkaException to stderr,
skips it and still exits 0 - so a partition lost to a leader election got
no manifest line, no data file and no drain check, and the tarball was
uploaded with the Backup reporting Succeeded. The restore-side count
check could not catch it either, since it sums the same short manifest,
and a missing highest-numbered partition made restore recreate the topic
with fewer partitions, re-placing every keyed record.

Read the topic's own PartitionCount before the drain and fail closed when
the offset listing is short. A short "begins" needs no separate check:
its per-partition default of 0 can only overstate n, which the existing
drain check then catches.

Also run the shared BucketAccess preflight in the e2e - accessGranted
does not prove the S3 server reloaded the new identity, and a fresh grant
answering the SigV4 PUT with 403 reds the run as a backup defect - and
state the .chainsaw.yaml namespace override as the rule it is rather than
as a claim about which Tests match a name pattern.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 7d8dee66a5 fix(kafka): verify restore landed and namespace-scope the S3 key
- Restore verification: kafka-console-producer, like the consumer on
  backup, exits 0 without every record being accepted, so a replay that
  dropped records still reported Succeeded. After the replay, compare
  each restored topic's record count (sum of end-begin over partitions)
  against the manifest and fail on any shortfall or overshoot - the
  latter being a restore into a non-empty topic, which this appending
  DATA restore does not support.
- Namespace-scope the S3 object key (<namespace>/<app>/kafka-topics.tar)
  so two same-named apps in different namespaces sharing one bucket do
  not collide. The namespace is .Release.Namespace in both modes and
  always names the source, since a to-copy target shares the source's
  namespace (a TypedLocalObjectReference).
- Warn instead of failing open when the Kafka image cannot be resolved
  from the operator: the pinned public literal is unreachable on an
  air-gapped cluster, so tell the caller to set KAFKA_IMAGE.
- Word the .chainsaw.yaml namespace note as "most" (etcd-2-backup-
  roundtrip sets no namespace) and document the empty all-topics backup.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 9673af6aca fix(kafka): guard short backup drains and settle example teardown
- Short-drain guard: kafka-console-consumer exits 0 after a timed-out
  short read, so a truncated drain was tarred and reported Succeeded.
  Compare the drained line count against end-begin per partition and
  fail the backup on any shortfall, making the README's "fails closed"
  promise real for compacted/transactional topics.
- Settling teardown: cleanup.sh returned before the Kafka apps and
  Bucket had uninstalled and could not report failure, letting the
  neighbouring chainsaw/kafka Test contend on PVC reclaim. Add a
  wait_deleted helper and a teardown_rc gate that waits for the heavy
  objects and exits non-zero when the teardown does not settle.
- Fully qualify the ambiguous delete kinds (backup/bucket/backupclass/
  restorejob/backupjob) so a bare shortcut cannot target a foreign group
  and silently leave the Cozystack artifact behind.
- Default KAFKA_IMAGE from the running operator (newest STRIMZI_KAFKA_
  IMAGES entry), falling back to the literal only when the operator
  cannot be read; drop the duplicate resolution note.
- Make the strategy and throwaway CLI Pods restricted-clean
  (runAsNonRoot + JSON-patch securityContext on the CLI Pods).
- gitignore .source-dump.txt; replace the deprecated "small" preset with
  c1.small; capture the to-copy cluster in the e2e catch; document the
  all-topics risk; de-enumerate the decaying suite-list comments.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 21f5b2db2d fix(kafka): correct internal-topic filter and harden the delete-wait
Two accuracy fixes in the all-topics path and the in-place delete-wait.
The all-topics default filtered `^_`, which also drops single-underscore
user topics such as `_schemas`, silently excluding user data from a
backup that still reports success; filter `^__` (Kafka's internal topics
are double-underscore). The in-place delete-wait ran `--list | grep` in a
pod snippet under `set -eu` without pipefail, so a transient `--list`
failure read as empty output and false-positived "deleted"; capture
`--list` on its own and check its exit before trusting it.

Sharpen the docs: the frozen-end-offset cut is per-topic, not
cross-topic; restore uses the configured replication factor (default 1),
not the source's; compacted and transactional topics are out of scope;
and the endpoint parse is demo-scope (no IPv6 / path-carrying forms).

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 34a04e6b37 chore(kafka): document copy-safety caveats and harden all-topics loop
Document the caveats that make the example safe to copy - a null key
round-trips as the literal string "null" (changing key and compaction
identity), the drain stages the topic twice under /tmp with no
ephemeral-storage bound, every backup of an app overwrites the same S3
key, and the Pods target warn-only Pod Security. Harden the all-topics
partition-count loop: the final `[ ... ] && parts=...` returned non-zero
under `set -e` when the last test was false, so use `if ... then ... fi`.
Align the default KAFKA_IMAGE with the vendored operator (0.45.1-rc1) and
list kafka-2- in the .chainsaw.yaml tenant-root header.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov cc8c56849d test(kafka): assert restore content and exact record count
The backup roundtrip only summed offset counts, so a restore that
replayed garbage, scrambled keys, or ran twice (the strategy Job's
backoffLimit retry re-appends from the start) still passed. Both
verifications also compared with `< MESSAGE_COUNT`, so an inflated
count slipped through.

Snapshot the source topic in step 04 as a per-partition, offset-ordered
partition/key/value dump, and diff the restored topic against it in the
in-place and to-copy steps: byte-equality proves keys, values, partition
placement and per-partition ordering survived, not merely the count.
Make both count checks exact.

Document that restore appends into a non-empty topic and is safe only
against an absent or empty one. Raise the helper Pods'
--pod-running-timeout to 5m so a cold image pull on a first run outside
CI does not abort before seeding.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov d38b538d98 test(kafka): fail fast on terminal backup phases and capture strategy Job logs
Two CI-quality fixes, both diagnostics/wall-clock only (neither can turn a
run falsely green):

- The chainsaw comment claimed run-all.sh fails fast on terminal
  BackupJob/RestoreJob phases and Stalled HelmReleases, but the kafka helpers
  never implemented it (carried over from mariadb). A BackupJob that settled
  phase=Failed was polled the full 600s and then mislabelled 'Timeout',
  burning ~10 of the 40 chainsaw minutes per already-red run. Port the mariadb
  variants: wait_for_field gains a terminal fail_value arg (call sites pass
  Failed), and a new wait_hr_ready fails fast on Stalled=True with an existence
  backstop. HR waits in steps 03/04/07 now use it; the Strimzi Kafka CR waits
  keep their genuine Ready condition. run-all.sh runs under set -e, so a
  non-zero step aborts the flow immediately, matching the comment.

- The catch block only captured the Strimzi broker Pods
  (app.kubernetes.io/instance=kafka-kafka-test); the generic Job strategy runs
  the drain/replay in controller-created Pods labelled
  job.strategy.backups.cozystack.io/mode=backup|restore (never that instance
  label) and not TTL-reaped, so the actual S3/SigV4 error left no log in the
  report. Add a second podLogs selector keyed on that label.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov b099f8dde4 fix(kafka): fail closed when listing topics for an all-topics backup
In the empty-TOPICS (back up all non-internal topics) branch, the topic
list was resolved with a single pipeline under set -eu without pipefail:

  TLIST=$(kafka-topics.sh --list | grep -v '^_' || true)

The pipeline's exit status is grep's alone, so a real broker error
(unreachable / auth) that emits no stdout makes grep exit 1, || true swallows
it, and TLIST is empty - indistinguishable from 'no non-internal topics'. The
backup then drains nothing, tars an empty dir, PUTs it and reports success: a
silent empty backup a later restore would treat as authoritative.

Split the --list into its own command substitution so a broker error aborts
the Pod under set -e; filter internal topics only afterwards, so an empty
TLIST means genuinely no user topics.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov c84f77ac16 fix(kafka): sign S3 host without non-default port in backup strategy
The Strimzi image ships curl 7.76.1, whose --aws-sigv4 omits a
non-default port from the SigV4 canonical Host while still sending it in the
Host header (a bug fixed in curl 7.87.0). seaweedfs recomputes the signature
from the received host:port, so every PUT/GET to the :8333 in-cluster S3
Service the e2e harness targets was rejected with HTTP 403, failing
kafka-2-backup-roundtrip at the backup upload. It passed on dev7 only because
that run used a :443 ingress, where the port is absent from the Host anyway.

Strip the port from the signed and sent URL and redirect the connection to the
real port with --connect-to, so the signed, sent and server-side Host all
agree on the bare host. Verified against the image's curl 7.76.1: the emitted
SigV4 signature matches a portless canonical Host, and the request still
reaches the :8333 listener. Correct on curl >= 7.87 too, which drops the
default port from the canonical Host as well.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 00943392cc fix(kafka): resolve a single Kafka image for backup e2e harness
STRIMZI_KAFKA_IMAGES is a newline-separated version=image map with one
entry per supported Kafka version. The old 'sed | tr -d [:space:]' collapsed
every line into one string, producing an invalid multi-image reference
(quay.io/...3.8.0quay.io/...3.8.1...) that failed 'kubectl run' with
'invalid reference format' and broke kafka-2-backup-roundtrip.

Split on whitespace, keep image lines, strip the version prefix and take the
last (newest) image so a single valid reference is exported.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 7842b88676 feat(kafka): add topic-data backup/restore example and e2e roundtrip
Add examples/backups/kafka, a worked backup/restore of a Cozystack-managed
Kafka application's topic data via the generic Job backup strategy - the
same pattern as examples/backups/nats, with no purpose-built backup image:
a stock Strimzi Kafka image (kafka-*.sh CLI + bash/curl/tar) runs one
shell script that branches on .Mode.

Consistency is a frozen-end-offset cut: kafka-get-offsets --time -1 is read
once per partition at backup start and the consumer drains only up to that
offset, so anything produced during the run is excluded and every partition
is captured as of the same instant. Restore recreates each topic with its
original partition count and replays every partition file; keyed records
reproduce into their original partition via the default partitioner.

This is deliberately a DATA-only backup: topic configs, ACLs, SCRAM users
and consumer-group offsets are not captured, and re-produced records get
fresh offsets. The README spells out the scope and points at a
volume-snapshot strategy for faithful, offset-preserving backups.

Add kafka-2-backup-roundtrip to hack/e2e-chainsaw/kafka, driving the
example run-all.sh as the harness (Bucket -> source Kafka + seeded topic
-> BackupJob -> in-place restore -> to-copy restore + verify) so the test
and the documented flow cannot drift, mirroring mariadb-2-backup-roundtrip.
The image is resolved from the running operator so the backup Pod reuses
the cached Kafka image.

Verified end to end on a live cluster: freeze cut orders:0 [0,8),
orders:1 [0,11), orders:2 [0,11) = 30 records; in-place and to-copy
restores both return 30 with the original 8/11/11 partition distribution.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Aleksei Sviridkin b1519acfe7 fix(e2e): cap the bucket S3 reads so the verify step honors its timeout (#4281)
## What this PR does

Refs #4268 — a plain reference, not `Fixes`: the issue asks for a
`script` op to be cut off at its declared ceiling, and that is Chainsaw
behaviour this repo cannot change. Chainsaw runs the op's shell under
Go's `exec.CommandContext`, whose default cancel SIGKILLs the shell PID
alone and then blocks on the inherited stdout/stderr pipe until an
orphaned `mc` closes it, so the op returns only when `mc` exits on its
own rather than at the ceiling. This PR mitigates that where it bit —
per-command caps so no descendant outlives its own budget — but the
op-timeout gap itself lives in Chainsaw, so the link stays a reference
rather than a `Fixes`. The sibling step that produced the 606s,
`verify-every-bucket-takes-a-write` in the `bucket-1-capacity` Test, is
on `main` and already carries its own per-call caps; this PR applies the
same shape to `verify-s3-access`.

On run 34948279363 the bucket suite's
`verify-every-bucket-takes-a-write` step declared `timeout: 5m` but ran
606s and ended on `mc`'s own non-zero status rather than being cut at
the ceiling. `verify-s3-access` in
`hack/e2e-chainsaw/bucket/chainsaw-test.yaml` is the same-shaped step in
this suite. This PR wraps every S3 (`mc`) and apiserver (`kubectl`) read
in `verify-s3-access` in its own `timeout`, so a hung read dies at a
known budget instead of holding the step open, and corrects the two
places (`hack/e2e-chainsaw/.chainsaw.yaml` and
`docs/agents/e2e-testing.md`) that explained the op ceiling with a
mechanism that does not hold. It caps `verify-s3-access` for sharing the
shape, not for producing the 606s number.

### Which of the issue's two candidate mechanisms holds

The issue named two candidates and stated the log could not separate
them. The Chainsaw v0.2.15 source separates them, and it is the second
one, with the exact refinement the issue proposed.

**Candidate 1 — "an operation `timeout` does not bound a `script` op" —
is refuted.** The op timeout is applied and does reach the process:
`pkg/runner/operations/script.go:24` copies the step's `timeout` into
`Timeouts.Exec`, and `:38` wraps the op context with `ctx, cancel :=
context.WithTimeout(ctx, tc.Timeouts().Exec)` before calling
`op.Exec(ctx, …)`. So a 5m step produces a real 300s deadline on the
context the command runs under.

**Candidate 2 — "it bounds it and kills the shell, but Chainsaw waits on
a pipe a grandchild (`mc`) still holds open" — is confirmed.** In
`pkg/engine/operations/script/operation.go` the command is built with
`exec.CommandContext(ctx, shell, …)` (`:105`), and in `execute` its
stdout/stderr are set to `*bytes.Buffer` (`:128`–`:129`, the buffers are
`internal.CommandOutput` in
`pkg/engine/operations/internal/command_output.go:12`–`13`) and run via
`cmd.Run()` (`:130`). Two Go `os/exec` properties then produce the
overrun: `exec.CommandContext`'s default cancel is `Process.Kill()` on
the shell PID alone — not a process-group kill, and the operation sets
no `SysProcAttr.Setpgid` — and because stdout/stderr are not `*os.File`,
`os/exec` allocates OS pipes and copier goroutines, so `cmd.Wait()`
blocks until those pipes reach EOF. EOF needs every inherited write-end
closed. When the shell forks `mc`, `mc` inherits those fds; at 300s the
deadline SIGKILLs the shell but `mc` is orphaned, keeps retrying the S3
PUT, and holds the pipe, so the op does not return until `mc` exits on
its own at 606s. `cmd.Run()` then returns `signal: killed`, which is
stored in the `$error` binding and never printed — only the `($error ==
null): true` check failure is logged — which is why grepping the run for
`signal: killed` found nothing.

**Empirical confirmation.** A standalone Go program reproduces it
exactly. `exec.CommandContext` with a 1s timeout, `*bytes.Buffer`
stdout/stderr, running `sh -c 'sleep 2 & sleep 4'` (a descendant that
inherits the pipe): `cmd.Run()` returns after ~4s, not 1s, with
`err=signal: killed` and `ctx.Err()=context deadline exceeded` — the
ceiling fired at 1s, control came back only when the descendant closed
the pipe. Redirecting the descendants' stdout/stderr away from the pipe
(`… >/dev/null 2>&1`) returns at 1.00s. That is the whole mechanism: the
ceiling cuts the shell but not a descendant that still holds the pipe.

### The fix

Per-command caps, the remedy the issue itself points to ("every read
carries its own cap") and the shape already used elsewhere in the suite.
The bucket step now defines a `k()` wrapper (`timeout 30 kubectl`) and
an `mc()` wrapper (`timeout 15 mc`) for the reads, and calls `timeout 45
mc cp` directly for the two PUTs; a hung read is killed at its own
budget, releasing the inherited pipe promptly, so even when the op
ceiling fires the residual is one inner cap rather than `mc`'s full
retry exhaustion. The terminal precedents for these budgets are the
redis suite's terminal `k()` (30s, `redis/chainsaw-test.yaml:298`) and
the sibling `verify-every-bucket-takes-a-write` step's terminal `mc cp`
(45s, `:390`, sized for a volume grow); the 15s cap on `mc` in
`hack/e2e-chainsaw/_lib/backup-access-preflight.sh` is not a precedent,
because it sits inside a `while :;` retry loop where a 124 costs one
attempt, whereas here under `set -e` a 124 ends the step. The inner
budgets — 2×30 for the two `k` reads + 2×45 for the two `mc cp` PUTs +
at most 7×15 for the other `mc` calls (eight other call sites, but the
two cleanup `mc rm` are mutually exclusive) + the 30s port-forward
readiness wait = at most 285s — sit under the 5m op ceiling, which stays
as a backstop, per the "op above the sum of inner budgets" rule in
`docs/agents/e2e-testing.md`. The long-lived `kubectl port-forward`
keeps no cap by design (it must persist for the whole step, and its
stdout/stderr are already redirected to `/dev/null`, so it cannot hold
the step's pipe). The `mc()` wrapper is non-recursive because `timeout`
execs its argument through a PATH lookup that bypasses shell functions —
verified in `dash`, `bash` and `sh`. The `catch` block's diagnostic read
gets the same treatment — `timeout 30 kubectl` under a 1m op `timeout` —
for the same pipe reason.

### The readonly deny probe fails on a timeout

The readonly-upload probe used to branch on `mc` directly (`if mc cp …;
then`), which suspends `set -e`, so a `timeout` there (exit 124) was
indistinguishable from the access-denied non-zero the probe expects: a
45s hang on the deny path would exit 124, take the false branch, reach
cleanup and exit 0 — a long hang turned green, the exact window this PR
exists to bound. This is the first `ro-user` write in the step, so no
earlier call proved that path healthy. The probe now captures the
status: exit 0 is the readonly regression, 124 is a timed-out probe,
both fail; any other non-zero is the expected denial.

### No unit surface; why the evidence is the proof

This is an e2e-config and doc-correctness change. The embedded Chainsaw
script is not extracted into a unit harness anywhere in the tree, so
there is no red-to-green unit test to add without inventing a bespoke
script-linter (which would be a new checker standing in for the fix).
The regression is instead pinned by the mechanism proof above: the
source citations show the ceiling cannot cut a descendant that holds the
pipe, and the caps remove the only way a descendant outlives its budget.
The deny-probe status handling and the pipe-vs-file secret reads were
exercised against stub exit codes (0 → regression-fail, 124 →
timeout-fail, other non-zero → expected denial) and the whole script
passes `sh -n` and `bash -n`.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The change touches only `hack/e2e-chainsaw/` suite content and
`docs/agents/e2e-testing.md`; it moves or renames nothing under `hack/`,
and touches no node-prerequisite or `package.mk` path. No downstream
repository is reached.

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved end-to-end S3 access verification so stalled Kubernetes API
and S3 operations fail promptly.
- Correctly distinguishes timed-out access checks from expected
permission denials.
- Preserves partial command output and allows subsequent test steps to
continue after individual timeouts.
- Applies separate time limits to bucket reads, uploads, and access
checks for more predictable test behavior.

- **Documentation**
- Clarified operation timeout behavior and recommended per-command
limits within the overall operation limit.
- Documented timeout handling for orphaned processes and updated
worst-case timing guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 17:14:56 +04:00
Aleksei Sviridkin ec706c49b1 docs(e2e): correct the Chainsaw Test-ordering claim in etcd and clickhouse suites (#4277)
Fixes #4269

## What

`hack/e2e-chainsaw/etcd/chainsaw-test.yaml` explained its layout with a
comment claiming Chainsaw executes Tests in **name order, not file
order**, and used that claim to justify why teardown lives inside each
scenario. The clickhouse suite had picked up the same claim. This
removes the ordering claim from both — it is false at the pinned version
— and re-grounds the etcd teardown justification in a reason that does
not depend on order.

## Why the claim was wrong (evidence, Chainsaw v0.2.15 — the pinned
version)

At v0.2.15 Chainsaw guarantees **no order between Tests at all**: not
file, not document, not name. Restating "name order" as "document order"
is wrong for the same reason. Verified against the pinned source:

- `pkg/runner/runner.go` — the runner registers each Test through
`testing.T.Run` and the per-Test body calls `t.Parallel()` whenever
`spec.concurrent` is nil or true. Neither suite sets `concurrent`, so
both take that path. Go pauses those subtests and releases them together
once the parent returns; they then race for the `-test.parallel`
semaphore.
- `pkg/runner/flags/flags.go` — `execution.parallel` maps only to
`-test.parallel`, a cap on how many Tests run at a time, not an order.
- `hack/e2e-chainsaw/.chainsaw.yaml` sets `execution.parallel: 1`, so
Tests are serialised — one at a time — but not ordered.

The E2E run on this PR's own head demonstrates it: discovery loaded the
five selected Tests in document order, then ran `clickhouse-2` before
`clickhouse-1` with both etcd Tests interleaved between the clickhouse
pair. Serial, non-overlapping, and in an order that is neither file,
document, nor name.

## The fix

The teardown decision the false claim justified is still correct, for a
reason that does not touch order. Each etcd scenario shares the
singleton `etcd` release and its cluster-scoped Kamaji `DataStore`, runs
with `skipDelete: true` (opting out of Chainsaw auto-cleanup), and
v0.2.15 has **no test-level `finally`** — so a scenario is reclaimed
only if it reclaims itself, and that is true of whichever Test runs last
regardless of order. The etcd comment now says exactly this, with the
ordering clause dropped rather than restated. The clickhouse header's
ordering sentence is removed outright: the three Tests use distinct
releases and share no state, so nothing there needed an ordering claim.

This is a comment-correctness change with no runtime surface, so there
is no unit test to add — the load-bearing claim was verified empirically
against the pinned Chainsaw source, as above. A repo-wide grep confirms
no other ordering claim survives in `hack/e2e-chainsaw/` or
`docs/agents/e2e-testing.md`.

Per `docs/agents/e2e-testing.md`: comment-only, no new retries, no
traps, no assertion or teardown-timing changes.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Corrected end-to-end test comments to drop an incorrect test-ordering
claim; Chainsaw v0.2.15 guarantees no order between Tests (not file, not
document, not name).
* Clarified that teardown behavior is independent of test execution
order.
* Documented the relationship between ClickHouse backup tests and the
corresponding backup test suite.
  * No test behavior or user-facing functionality changed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 16:40:30 +04:00
Myasnikov Daniil bc8c54d6e6 feat(kubernetes): boot tenant worker disks from a shared golden Talos image
Every tenant worker VM imports the same Talos raw image from the Image Factory
over HTTP, once per worker: a 20-worker cluster fetches the identical ~4.15 GiB
artifact 20 times, and each fetch is a live dependency on the Factory at exactly
the moment a node joins.

packages/system/kubernetes-worker-image is a new opt-in catalog that holds one
golden DataVolume per (schematicID, version) in cozy-public, named
talos-worker-<schematicID>-<version>. A worker pool opts in with osImage.builtin
on its own KubernetesNodes resource, and its disk-system DataVolume then carries
source.pvc pointing at that golden, so CDI performs a storage-layer CSI clone
instead of an HTTP import -- one import per flavor, shared by every worker of
every tenant. Confirmed on a LINSTOR-backed cluster: the worker disk is a CSI
clone plus a resize, with no host-assisted upload pod.

Both halves are off unless asked for. The catalog is registered through the
platform's optional-package helper and disabled by default, so a cluster that
does not enable it creates no golden and keeps importing over HTTP. The disk
source is derived entirely from osImage, which is absent by default, so a pool
that does not set it renders byte-identical to the pre-feature chart and
adopting the feature never rolls existing workers. Editing osImage on an
existing pool re-images that pool, as changing any other image reference does.

What the render refuses, rather than degrading into a silent fallback:

- A pool whose golden has no catalog entry, or whose golden terminally failed to
  import. A golden that is merely mid-import is not an error; CDI holds the
  worker disk until the source populates.
- A pool whose StorageClass differs from its golden's. CDI cannot CSI-clone
  across classes and falls back to a host-assisted copy over the pod network,
  which is the transfer this feature exists to remove. An empty class on either
  side means the cluster default and is resolved before the comparison, and a
  pool that has already cloned reads its class from its own bound PVC rather
  than from today's default -- the golden's side is frozen the same way, and an
  admin moving storageclass.kubernetes.io/is-default-class must not fail the
  release of a pool whose disks are all on the right class. That resolved class
  is then written onto the worker disk, so the disk binds where the guard
  checked instead of on whatever the default has become by then -- reading one
  class and binding another is how a pool at an empty storageClass passed the
  comparison and still took the host-assisted copy. The HTTP path leaves an
  empty class to the cluster as before, which is what keeps its render, and the
  KubevirtMachineTemplate hash named after it, byte-identical.
- A diskSize below the golden's storage size, which a clone target cannot be.
- A schematicID that is not a plain lowercase alphanumeric identifier, or a
  version that is not a vMAJOR.MINOR.PATCH release. Both are interpolated into
  a DataVolume name and an image URL, and values.schema.json types them as bare
  strings, so a malformed one otherwise reaches the render. The catalog applies
  the same patterns, since both sides build the same name from the same fields.
- An images[] entry edited after its golden exists. A DataVolume spec cannot be
  patched in place, so the message names the entry and the supported move -- a
  new (schematicID, version) -- instead of leaving the API server to reject the
  patch on the next upgrade.

Goldens carry helm.sh/resource-policy: keep, so dropping an entry stops
managing a golden rather than deleting it: a missing golden aborts the whole
Kubernetes HelmRelease of every tenant pinned to it, not just the disk. Each
cloned worker disk annotates the golden it came from and the pool that owns it,
so "does anything still clone this?" is answerable before a manual delete
instead of after.

A builtin pool needs a StorageClass whose volumes are reachable from any node,
in practice a replicated/DRBD one. The golden in cozy-public has no consumer and
binds as soon as it imports, wherever CDI's importer landed, while a worker disk
binds where its VM is scheduled, and nothing here arranges for those to be the
same node. The platform ships cloneStrategyOverride: csi-clone, which rules out
the host-assisted strategy that would cross nodes. What a node-pinned class does
in that situation is not measured, so values.yaml states the requirement and not
a symptom.

ComputePlane pools forward only the per-pool shape fields, and osImage is
excluded there by name: the worker OS image, its Talos installer and its Image
Factory are operator decisions about the platform's own images.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 15:35:44 +03:00
Aleksei Sviridkin 9e0e1239da fix(kubevirt-cdi-operator): pin the vendored CDI release (#4383)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This pins the CDI release the platform deploys, so it moves only when
someone edits a version line. Until now `make update` in
`packages/system/kubevirt-cdi-operator` took the release from GitHub's
`releases/latest` redirect. Any refresh could move CDI and its image
tags, and the diff had no version line to review. The fix follows the
shape `packages/system/kubevirt-operator` already uses.

- `RELEASE = v1.66.1` names the release. It is the one already vendored,
so `templates/` does not change, and `make update` reproduces it byte
for byte.
- `MANIFEST_SHA256` pins the downloaded asset and matches the digest
GitHub publishes for it. On a mismatch the recipe writes nothing.
- The download is transformed and patched outside `templates/` and moved
in at the end. A refresh that fails halfway leaves the vendored file as
it was.
- A new `test` target runs a helm-unittest suite over the vendored
manifest, then greps it for the operator image at `$(RELEASE)`. The grep
is there because helm-unittest cannot see a Makefile variable. Moving
the pin without re-running `update` fails here.

One thing is left for later. The suite asserts three of the eight image
references directly, and the digest keeps the rest in step on `update`,
but a hand edit to one of the other image env entries would pass.

Closes #4190

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Nothing on the trigger map applies. The diff touches one package's own
Makefile, `.gitignore` and tests, and no shared tooling, values, CRD,
namespace or release asset.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubevirt-cdi-operator): pin the vendored CDI release to v1.66.1 and verify the downloaded manifest against a digest, so refreshing the package no longer moves the deployed CDI. The deployed version stays the same.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* The KubeVirt CDI Operator is now pinned to version 1.66.1, providing a
consistent operator version across deployments.
* **Reliability**
* Operator manifest updates are checked before they’re applied, helping
prevent incomplete or mismatched manifests from replacing the current
version.
* Deployment configuration is validated to ensure it matches the pinned
release.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 16:20:04 +04:00
Myasnikov Daniil 5df219b37a fix(api): keep a suspended HelmRelease suspended across an app update
Update rebuilds the HelmRelease from the Application and writes it back
whole, and the Application carries no suspension, so every edit made
through the API resumed a release that an operator or a controller had
suspended.

The CNPG restore driver depends on that suspension. It suspends the
target release, patches the Postgres app through this API, purges the
Cluster, and resumes only once the purge is done, so the chart's next
render lands bootstrap.recovery on an empty namespace. Resumed by the
patch instead, helm-controller upgrades straight away: the chart can
render the recovery Cluster while the purge is still running, the
driver then deletes it as the Cluster being replaced, and nothing
renders it again because the values have not changed since. The
RestoreJob sits in Running until its deadline.

On main this has been masked by timing rather than prevented. The same
write also dropped the Flux finalizer, so helm-controller's first pass
after it only put the finalizer back and requeued, which held the
upgrade back by at least its 750ms minimum retry delay. A branch that
keeps the finalizer removes that delay, and the backup round-trip
failed on both of its runs there.

The suspension is re-read on a conflict retry as well, since the write
that caused the conflict may be the suspend itself.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-23 16:40:17 +05:00
Aleksei Sviridkin b96c822bde docs(agents): give the HR-Ready budget derivation its own section
The checklist entry had grown into a derivation and buried the rule it
states. The rule stays in the checklist and the derivation moves to a
section of its own.

The derivation under-budgeted in two ways. It stated the 10m install
timeout as universal, while releases an app chart renders itself take
their template's spec.install.timeout, then its spec.timeout, or
helm-controller's 5m where neither is set. And it restated which kinds
kstatus gates on, which is easy to get wrong, so it now points at the
upstream table instead.

The harbor comments now name the floor each of their asserts actually
sits against, and the opensearch comment stops citing a 5-6m standard
the checklist no longer states.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-23 11:56:12 +03:00
Aleksei Sviridkin 4373c8c0f2 test(e2e): pin the mariadb first-boot wait budgets against the tree
The waits are right only relative to numbers that live elsewhere: the
chart's failureThreshold, the install timeout cozystack-api stamps, and
the Chainsaw op that runs the walkthrough. Deriving the floors from the
tree rather than pinning literals means a change to any of them reds
the contract instead of silently invalidating the waits.

The extractors are tested against fixtures as well, because the tree
cannot tell a correct extractor from a broken one: the chart carries a
single failureThreshold and every wait clears every floor, so a reader
of the wrong operand stays green.

The wait_hr_ready exit branches are tested by behaviour against a
stubbed kubectl, one stub per answer they have to tell apart, because
a static check cannot see what errexit does to a failing read. That
test and a byte-identity check cover every definition of the helper
in the tree, found by grep rather than listed, so a new copy is picked
up without editing the test. The redis and kafka-metadata definitions
are named exceptions with their reasons in the header, and each is
held to still differ, so an exception whose copy converges goes red
instead of lingering.

The suite runs under /bin/sh, which is dash on the runner, so the
behavioural test runs its body under an explicit bash.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-23 11:56:11 +03:00
Aleksei Sviridkin 957ddd7f25 fix(tests): raise the mariadb suite HR asserts past the install timeout
At 6m the HelmRelease-ready asserts expired before the release's own
600s install timeout, so a failed install was reported as a bare
Chainsaw timeout with no reason. 11m lets the release record why it
failed first.

The round-trip op grows by exactly what the two walkthrough waits
gained. Otherwise a failure at either wait could run past the op,
which SIGKILLs the process group and loses the script's error. That
bounds one failure, not the whole flow: a run slow at every stage that
fails late still reaches the op, and bounding that needs a phase
budget rather than a larger literal.

The endpoint asserts stay. A freshly applied CR has no conditions and
kstatus reads it as current, so the HelmRelease wait can end before
any pod is up; an endpoint address closes that window.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-23 11:56:11 +03:00
Aleksei Sviridkin f629280944 fix(tests): raise the mongodb, rabbitmq and clickhouse HR waits
Those walkthroughs gave their database HelmReleases 300s, under the
600s install timeout the generated releases carry, so a failed install
surfaced as a bare timeout rather than its own reason. None of these
charts sets a startup probe, so 660s clears both floors, as it does
for mariadb.

Each Chainsaw op that runs one of the scripts grows by exactly what
the waits it reaches gained, so it keeps the margin it had. The
clickhouse op still sits above the sum of every inner ceiling. The
rabbitmq one covers a failed install at the source wait, while the
ceilings up to its target wait add up to more than the op, as they
did before. The mongodb round-trip runs with SKIP_RESTORE, so only its
source wait counts there.

postgres keeps its 360s: whether a suite pays the extra minutes on
every red run is a per-suite call, and this one is not made here.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 11:56:11 +03:00
Aleksei Sviridkin 289acb2ed6 fix(tests): let the mariadb backup example cover a first boot
The walkthrough gave each MariaDB HelmRelease 300s, less than the
release can legitimately take, so a slow first boot failed the harness
and a failed install surfaced as a bare timeout with no reason.

A wait here has two floors and must beat the larger. The chart grants a
310s startup budget, and the install can span it: helm-controller
defaults to the poller strategy, and kstatus reads the operator CR's
Ready=False as InProgress. The generated release also carries a 600s
install timeout, and a wait equal to it expires first, because its
clock starts at the apply and the install's only once the chart is
fetched. The install timeout is the binding floor, hence 660s.

On timeout and on Stalled, wait_hr_ready now dumps every condition and
the release history: a scheduled retry can overwrite the Ready message
before anyone reads it. It also stops folding different answers into one
"(none recorded)" line. An absent release, a release deleted during the
wait, a lookup kubectl could not answer, an empty history and a failed
history read each say so, with kubectl's own stderr kept, because that
message is usually the cause. A release seen earlier still gets its dump
when only the last lookup failed.

Every backup example whose wait_hr_ready carries this body takes the
same change, so the copies stay byte-identical.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-23 11:56:10 +03:00
Aleksei Sviridkin eec069b94f docs(kubevirt): name every payload line the tail checks do not cover (#4340)
## What this PR does

Follow-up to a review comment on #4254 that was right and landed after
the merge.

The sentence that PR left in `packages/system/kubevirt/Makefile` says
the device and CPU payload lines stay outside the marker checks, which
reads as if the rest are handled. The `- {{ . }}` entries inside the
`extraFeatureGates` and `disabledFeatureGates` ranges are neither device
nor CPU and are not covered either. Deleting either of them from a copy
of the CR and running `make update CR_FILE=<copy>` exits 0 and leaves
the range wrapped around nothing, with no check firing; dropping `-
Template` from a clean copy fails the target as a control.

The wording now names every uncovered payload line: the device tables,
the `cpuAllocationRatio` value and both feature-gate entries, and keeps
`migrations` as the one block validated whole. Covering those payloads
in code is a wider change; happy to do it separately if that is wanted.

### Screenshots

Not a UI change.

### Downstream repositories

Comment-only change in one Makefile.

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Clarified update validation guidance for disabled-feature markers.
* Documented partial-block recovery requirements and the workflow for
restoring missing markers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 12:37:10 +04:00
Aleksei Sviridkin c290dee945 fix(vm-instance)!: refuse resources that cannot size the VM
resources.cpu and resources.sockets are typed as quantities, but they
land in domain.cpu.cores and domain.cpu.sockets, which KubeVirt
declares as uint32. The chart cast them with int64, which turns 500m,
1.5 or 1e1 into 0 without an error, and KubeVirt boots a zero core
count as one core. The user asked for half a core and got a one-core
VM with nothing reporting it.

The raw spelling is now decoded as YAML, the way an integer field
reads it, and the render stops unless the result is a positive whole
number that fits in uint32. A leading zero is refused, because YAML
reads 010 as octal 8. The check keys on presence rather than
truthiness, so an unquoted 0 is refused instead of being read as
unset.

Memory gets the same presence check. A quoted "0" was truthy and
rendered requests.memory: "0", while an unquoted 0 was falsy and
dropped, so one quantity took two paths. A zero or negative amount
now stops the render in both spellings.

BREAKING CHANGE: values that used to render now either fail or
render differently. A cpu or sockets that does not decode to a plain
whole number within uint32 (a millicore, suffixed such as 1k,
fractional, zero, negative or leading-zero value), and a zero or
negative memory, fail the next upgrade of the release until the
value is fixed. This includes cpu or sockets set on its own beside an
instanceType, which used to be ignored whatever it held. A spelling
that the int64 cast truncated to 0 but that YAML decodes to a whole
number, such as the exponent 1e1 or the trailing dot 2., used to boot
one core and now sizes the VM with the count it spells, so such a VM
changes size on upgrade.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 11:31:07 +03:00
Aleksei Sviridkin 89da4a121c fix(cdi): stop make update from deleting first-party templates (#4097)
## What this PR does

`make update` in `packages/system/kubevirt-cdi` began with `rm -rf
templates` and then refetched one file, `cdi-cr.yaml`. Nothing in that
directory is purely vendored. Two of the three manifests, the
upload-proxy Ingress and TLSRoute from the Gateway API work, are ours
and no fetch recreates them, so the next CDI bump would have dropped
upload-proxy exposure from the chart and still exited 0. The third is
`cdi-cr.yaml` itself, which despite its name carries Helm templating,
the `ExpandDisks` gate, a toleration and the two RBAC documents that
preauthorize cloning from `cozy-public`; the fetch overwrites it, and
removing the wipe does not change that.

That recipe now opens with `mkdir -p templates` and deletes nothing,
which is where `packages/system/kubevirt-operator` landed at the last
KubeVirt bump. The wipe bought nothing there anyway: `wget -O` truncates
the file it writes.

`kubevirt-cdi-operator` has the same recipe shape, and the same edit
there would be a regression rather than a fix. It applies two patches,
and since #3920 moved them to `-p4` they resolve, so a hunk upstream has
moved out from under leaves `templates/cdi-operator.yaml.rej`; its `awk
-i inplace` step leaves `templates/cdi-operator.yaml.gawk.XXXXXX` if
gawk dies abnormally. Helm renders every file under `templates/`, and I
reproduced both: the reject fails the package with a YAML parse error,
and the gawk temp renders a duplicate object at exit 0. The wipe is what
takes both, so removing it for symmetry would trade a latent loss for a
live one, and the quieter of the two is the one that survives.

Deleting the wipe there wants the recipe restructured to transform
outside `templates/` and move the result in once, which is what
`kubevirt-operator` does today and what its own comment gives this exact
reason for. That is a larger change than this one and belongs on its
own; #4383 makes it. Meanwhile the scan below covers the package: I
added a first-party file to its `templates/` and the guard reported it,
so the day that package grows one, this fails rather than the file being
lost.

`hack/package-update-templates.bats` guards this across the tree rather
than in these two packages. It scans every package Makefile and fails
when a recipe that clears its own `templates/` leaves behind a tracked
file it never names. Put the removal back into `kubevirt-cdi` and it
goes red naming both upload-proxy files. A file counts as named only
outside the Makefile's own `NAME` and `NAMESPACE`, however the
assignment is written. The match is on the variable name rather than on
a list of spellings: the tree already carries three of them, `export
NAME=`, the bare `NAME=`, and `NAME := ` in `packages/core/testing`, and
the hole is identical in each. Without that, a package called `foo-bar`
satisfies the check for its own `templates/bar.yaml` through its
identity alone. Nothing the scan reaches is exempted that way as the
tree stands, so the exclusion is there for the first package that would
be: rename the asset in `kubevirt-cdi-operator`'s recipe and put the
removal back, and the guard reports the orphaned template only with the
exclusion in place. The two variables are matched by name rather than by
a `[A-Z_]+=` shape, because a shape that broad is one unspaced
assignment away from swallowing a variable that does name templates,
which is how `etcd-operator-crds` names all five of its CRDs. A control
test runs the same helpers over a fixture broken the same way, over one
whose name swallows a template stem, then over that fixture rewritten to
spell the removal with a trailing slash, to take one file instead of the
directory, and to drop the removal entirely. Both cases pass under
`hack/cozytest.sh hack/package-update-templates.bats`.

Two limits, both written into the file's header. The check fires on the
consequence rather than the shape, so `rm -rf templates` in a package
that does refetch everything stays green, which is what keeps
`kubevirt-cdi-operator` green for as long as its recipe wipes. And it
only looks at a recipe clearing its own directory: `cert-manager` clears
`../cert-manager-crds/templates/*` across the package boundary, which
needs the removal's target resolved rather than the wiping package's
own, so that directory is unguarded. The other packages that clear their
own `templates/` refetch all of it and are untouched here.

Two more limits the header names. The scan reads the text of each
package's own `update:` recipe, so a removal made elsewhere is out of
range: in a prerequisite target, behind an include, or behind a
recursive make. `capi-operator`, `clustersecret-operator`, `openbao` and
`reloader` write `update: clean <name>-update` and take both recipes
from `hack/package.mk`, which removes nothing under `templates/` today.
And the glob stops at `packages/*/*/Makefile`. The package Makefiles off
that depth are the group aggregators such as `packages/system/Makefile`
and the vendored kamaji subcharts, and none of them declares `update`.

One behaviour of the matcher worth naming, and deliberately not changed
here. `wipe_command` evaluates `/^update:/` ahead of its terminator, so
a second `update:` in the same Makefile re-arms the scan instead of
ending it, and what that means depends on what sits between the two:
adjacent or separated by a blank line, both recipes are read; separated
by another target, only the first is. The direction that matters is a
false positive rather than a miss. A Makefile whose first `update:`
clears `templates/` and whose second does not would be reported by this
scan, while make runs only the second, so the package fails over a
recipe that never executes. No package in the tree defines `update:`
twice today.

Three more notes, all non-blocking and none acted on. The header says
what the scan leaves uncovered is a recipe clearing someone else's
`templates/`; that is the uncovered set for *deletion*, and
`kubevirt-cdi` is green under the scan while still losing
`cdi-cr.yaml`'s content to the in-place `wget -O` on every run, which
the Makefile comment records but the header does not. The identity strip
has a false-positive direction the header does not argue: a wiping
package writing `templates/$(NAME).yaml` would be reported even though
the recipe writes it, which no package does today. And the fixture
stages `.gitkeep` with `git add --all`, so a developer whose global
excludes list that name would get a red for environment rather than
defect; `--force` would remove the dependency.

The stem match is generous, and the helper's comment now names the two
live shapes. A sibling name: `templates/crds.yaml` next to the existing
`templates/crds-experimental.yaml` on `gateway-api-crds`. A mention from
an unrelated context: `templates/sidecar.yaml` on
`objectstorage-controller`, whose Makefile says `sidecar` across its
image targets while its recipe clears `templates/` and writes only
`templates/controller.yaml`. I fed both to the helper and both come back
exempted; a stem the Makefile never mentions is still reported.

The duplicate `test:` target in this Makefile is #4157 and stays there.
It sits outside the hunks this diff touches, so removing it here would
be reaching past what this change is about.

One more note, not a defect. Where the comment points at
`kubevirt-operator` as the shape that closes the transient-file class,
`packages/system/kubevirt/Makefile` is the closer analogue for the
`cdi-cr.yaml` half specifically: its `update` downloads nothing and
re-applies the parameterization onto a hand-maintained CR with a
per-marker self-check. And the failure message offers a third option,
staging into a tmpdir the way `etcd-operator-crds` does, in a
parenthetical that reads as a gloss on the option before it.

What this PR does not fix, and what does. `make update` here still
overwrites `templates/cdi-cr.yaml`, which the Makefile comment says
plainly. A wholesale overwrite is loud: upstream's object carries
neither `podResourceRequirements` nor `cloneStrategyOverride`, and
`tests/importer-resources_test.yaml` and
`tests/clone_strategy_test.yaml` pin both, so `helm unittest` goes red.
The silent case is a partial hand reconciliation that keeps those two
and drops something else, because nothing here pins the `ExpandDisks`
gate, the `CriticalAddonsOnly` toleration, the `uploadProxyURL` override
or the two RBAC documents that preauthorize tenant cloning from
`cozy-public`. #4107, stacked on this branch, stops `update` writing the
file at all and adds `tests/local_templates_test.yaml`, which pins
those.

One limit the header does not name, left alone here. The comment strip
is anchored to the start of a line, so a trailing comment on a recipe
line survives into the searched text and counts as naming the file it
mentions. I confirmed it by feeding `wget ... -O templates/cdi-cr.yaml #
local-extra.yaml is written by hand` through the filter. Nothing in the
tree is written that way, and the header only claims comment lines are
dropped, which is what it does. `sed 's/#.*//'` in place of the `grep
-v` closes it if it ever matters.

Closes #4028.

### Screenshots

No UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two entries come close: the website's "developer tooling (the
package `Makefile`s)" line, and ccp's "move or rename anything under
`hack/`". Neither fires. `make update` still behaves as the website's
development page describes, nothing under `hack/` moved or was renamed,
and the two files ccp's skills key on (`hack/package.mk`,
`hack/common-envs.mk`) are untouched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): `make update` in kubevirt-cdi no longer deletes its first-party templates
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Updated Kubernetes CDI package updates to preserve existing template
files while refreshing downloaded manifests.
- Prevented update operations from clearing first-party and vendored
template content.

- **Tests**
- Added safeguards to detect package update recipes that could remove
tracked templates without restoring them.
  - Added Helm unit test support for the Kubernetes CDI package.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 12:23:58 +04:00
Timofei Larkin 8f2a6bcc87 ci(api-gate): report tightened Application name limits
An application declares the limits on its own metadata.name at the
root of its schema, beside properties, where the gate's walk over spec
never looked. A chart lowering its name cap or adding a pattern rejects
names its previous release accepted, which is the same break as
tightening a spec field, so the gate reports it the same way.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-09-23 10:59:51 +03:00