Commit Graph
7777 Commits
Author SHA1 Message Date
Aleksei Sviridkin f4a6eb809c fix(nats): enable JetStream for the generated account (#4675)
## What this PR does

With `users` configured and `jetstream.enabled: true`, the application
chart enables JetStream on the server but generates account `A` without
it. Authentication and ordinary NATS messaging work, while
`$JS.API.INFO` returns error 10039, "JetStream not enabled for account".

The generated account now gets `jetstream: true` when persistent
messaging is enabled and the account does not set JetStream itself.
Tenant-supplied limits, explicit disablement, other accounts and the
existing generated-user merge are kept. nats-server matches config keys
case-insensitively and walks the account map in no fixed order, so a
`JetStream` key in any casing counts as set. Otherwise the account would
carry both keys and the server would apply either one, changing from
restart to restart.

Existing instances with `users` pick this up on upgrade. The rendered
NATS config changes, so the StatefulSet rolls its pods once, and account
`A` gets JetStream without account limits unless
`config.merge.accounts.A.jetstream` sets them.

### Validation

- `helm unittest` passes all 75 application-chart tests. Eight cases
cover this change. Against the original template the generated-account
and null-override cases fail. The other six pin what the change must
keep, and the mixed-case one fails as soon as the check is
case-sensitive.
- Dropping the `jetstream.enabled` check, the case folding or the
explicit-setting check turns at least one case red each time.
- The application chart and the vendored system chart were rendered, and
the resulting account configuration was run on NATS Server 2.11.10 in an
isolated standalone setup. The original template reproduces error 10039.
With this change a file-backed stream can be created, published to and
read, and explicit disablement and a one-stream limit stay enforced.
- `make generate` with cozyvalues-gen v1.7.0 leaves the generated files
unchanged.

### Screenshots

Not a UI change.

### Downstream repositories

The diff changes an application template and chart tests only. Public
values, schemas, output resource names and tooling contracts are
unchanged. The website NATS documentation contains no account-level
workaround to remove. No downstream trigger applies.


- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note



```release-note
fix(nats): enable JetStream on the account generated for users when jetstream.enabled is true, keeping any explicit account-level JetStream setting. Instances with users roll their NATS pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Generated NATS accounts now enable JetStream when configured, without
overriding an existing JetStream setting. Configured passwords are added
while preserving existing account settings.

* **Tests**
* Added coverage for enabled and disabled JetStream, explicit limits and
false or empty values, case-insensitive setting names, null overrides,
and manually configured accounts when generated users are absent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 18:00:37 +03:00
Aleksei Sviridkin 0cae0e87ad test(e2e): spell the find negation as -not in the suite-list helper
The negated-assertion audit splits a line on `(` and reads the `!`
after `\(` as a shell negation that errexit would exempt, so the
helper's find predicate fails the audit and the unit tests on main.
`-not` is the same predicate in GNU, BSD and busybox find and does not
look like one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 17:57:05 +03:00
Aleksei Sviridkin c854e054b9 ci(workflows): install flux in the isp-hosted lane through the pinned action
The isp-hosted lane fetched fluxcd.io/install.sh and ran it as root,
which the CI tool-install contract forbids for every workflow: it runs
whatever the site serves at that moment. The lane landed next to the
contract and kept the old step, leaving the unit tests red on main.

It now uses the same SHA-pinned fluxcd/flux2 action and version as the
other workflows.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 17:57:05 +03:00
Aleksei Sviridkin f527ef17ee test(fluxcontract): accept a timeout-bounded history read
The monitoring suite bounds its HelmRelease history read with
`timeout 30 kubectl`, which keeps a hung API call from eating the
step's whole budget and the diagnostic with it. The shared-read pin
admitted kubectl only bare or through the bounded wrapper, so the two
landed side by side and left the unit tests red on main.

The pin now takes an optional `timeout [-k N] N` in front of kubectl,
the forms the suites already use for bounded reads. The bound is
spelled out instead of allowing any prefix, since
`$(echo kubectl ...)` would name the expansion without running it;
that form, bare and behind a timeout, is now a refuse case. So is a
timeout in front of _cozy_guard_kubectl: timeout runs a program, the
wrapper is a shell function, and the read would fail every time. The
command name has to end at a blank, so `kubectlx` or
`_cozy_guard_kubectl_v2` is not taken for either one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 17:57:04 +03:00
Aleksei Sviridkin 7bf6906a80 fix(platform): keep the legacy etcd-operator stopped after etcd adoption (#4738)
## What this PR does

Migration 50 scales the legacy etcd-operator to 0, adopts every legacy
`etcd.aenix.io` cluster with `etcd-migrate`, and then scales the
operator back to 1. At that point the Deployment still runs the legacy
operator, and after a successful adoption it has nothing left to manage.
Bringing it back undoes part of the adoption.

Observed on a production cluster upgraded from 1.4.3 to 1.6.4. The API
audit log and the Helm history of a tenant's `etcd` release show this
sequence, in UTC on the same day.

- 13:21:12 migration 50 scales
`cozy-etcd-operator/etcd-operator-controller-manager` to 0.
- 13:22:59 `etcd-migrate` updates `etcd-headless`, pointing its owner
references at the adopted EtcdMembers, and orphan-deletes the legacy
EtcdCluster and StatefulSet.
- 13:23:04 migration 50 scales the operator back to 1. It is still the
legacy operator, the v1alpha2 one replaced it only at 13:40.
- 13:23:42 Flux upgrades the tenant's `etcd` release with the old chart
(Helm revision 87), because the new chart artifact appeared only at
13:24:40. The old chart re-creates the legacy EtcdCluster at 13:23:43.
- 13:23:43 to 13:23:45 the legacy operator reconciles it, updates
`etcd-headless` twice and re-creates the legacy StatefulSet `etcd`.
- 13:36:49 Flux upgrades the release to the new chart (revision 88),
which prunes the legacy EtcdCluster.
- 13:36:50 the garbage collector deletes `etcd-headless` and the
re-created StatefulSet.

The same happened in all five tenants that run an etcd. The pods
survived because EtcdMembers own them, but they keep `spec.subdomain:
etcd-headless`, so their `etcd-<i>.etcd-headless.<ns>.svc` names stopped
resolving. From then on every member logged `no such host` for its
peers, the v1alpha2 EtcdCluster had no status conditions, and the
`etcd-defrag` CronJob failed every hour with `context deadline
exceeded`. The transitional `etcd-headless` Service the etcd chart ships
for this window (#3270) does not cover it. Helm patched the existing
Service 202 ms before the garbage collector removed it, and did not
create it again because it was already part of the release.

With this change the operator stays at 0 after a successful adoption.
The etcd-operator HelmRelease replaces the Deployment with the v1alpha2
operator later in the same upgrade, and no later migration waits for the
operator. A failed or interrupted adoption still restores it to 1
through the existing EXIT trap, so a failed run does not leave the
tenant etcds without an operator. If that HelmRelease cannot reconcile,
the adopted clusters wait for it with or without this change, because
the legacy operator does not reconcile v1alpha2 EtcdClusters.

Clusters already upgraded with this bug can be repaired without
restarting anything. Creating the Service from the tenant's own `etcd`
release is enough.

```
helm -n <ns> get manifest etcd | awk 'BEGIN{RS="\n---\n"} /\nkind: Service\n/ && /\n  name: etcd-headless\n/' | kubectl -n <ns> create -f -
```

On the cluster above the v1alpha2 EtcdCluster reported `Available` with
reason `QuorumHealthy` within 11 seconds of the Service being created,
in every tenant, and the members stopped logging `no such host`.

### Validation

- `hack/cozytest.sh hack/migration-50-etcd-adopt.bats` passes all 29
tests. A new test asserts that a successful adoption scales the operator
to 0 and never back to 1. Against the current script it fails with
`FAIL: the legacy operator was scaled back up after a successful
adoption`.
- The existing test for a failed adoption still passes, so the EXIT trap
still restores the operator to 1.
- The new test checks the absence with `if grep ...; then echo "FAIL:
..."; false; fi`, the form the other suites use. A bare `! grep`
assertion cannot fail a test under `hack/cozytest.sh`, so it could not
have caught this. The other negated assertions in this file had the same
weakness, and #3790 has rewritten them on main. With all of them made
able to fail in a scratch copy, the suite still passes against this
change.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. It changes a migration script and its bats test. No package,
values, schema, CRD, make target or `hack/` layout changes, so no entry
matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): keep the legacy etcd-operator stopped after migration 50 adopts the legacy etcd clusters. Scaling it back up let it reconcile the legacy EtcdCluster that Flux re-created from a not yet upgraded etcd chart, re-create the legacy StatefulSet and take the legacy etcd-headless Service away from the adopted members, which the garbage collector then deleted. The adopted members lost their DNS names, the EtcdCluster never reported a status and etcd defrag failed every hour. Clusters already upgraded can recreate the Service from the etcd release manifest.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Migration Updates**
* After successful etcd adoption, the legacy operator remains scaled to
zero replicas for replacement by the etcd operator; it is no longer
scaled back up before the migration is stamped.
* If adoption fails or exits before completion, the legacy operator is
restored to one replica.
* The dry-run message now describes the zero-replica outcome of
successful adoption.
  * Successful adoption continues to record the migration version stamp.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 17:53:30 +03:00
Aleksei Sviridkin 415430bcc7 fix(e2e): fail on a failed release of cilium and csi as on the parent, and report addon remediation (#3594)
## What this PR does

The history guard in the tenant Kubernetes e2e suite checked only the
parent `kubernetes-<test>` HelmRelease. A failed action on one of the
addon releases it installs left nothing the suite looked at, so a run
showed only the downstream symptom, usually the node-join deadline, and
never named the release behind it.

The guard now reads every addon release of the parent. It selects them
from the cluster by name prefix, so a release added to the chart later
is covered too. The rule:

- A `failed` or `uninstalled` Snapshot fails the run on any release
whose install strategy is `RetryOnFailure`. That is the parent (the
aggregated apiserver stamps it on every Application HelmRelease) and the
tenant `cilium` and `csi`.
- The same footprint on the other addons, which run the default strategy
with `retries: -1`, is printed as a note and the run stays green. For
the CRD addons that configuration is tracked in #3555.

A transient `cilium` or `csi` failure that later recovers on a retry
still turns the run red, by design, as long as its Snapshot still reads
`failed`. One recovery leaves no such Snapshot: when an install fails
and the very next attempt succeeds, Helm marks the failed install
`superseded`, because there is no deployed release to supersede, and the
history reads like an ordinary upgrade. The guard cannot see that case
from the statuses alone. The code and the docs now say so, a test pins
it, and detection is left to a follow-up.

An earlier revision of this PR tried to tell a teardown apart from a
retried failure and made only `uninstalled` fatal. That split does not
hold. In helm-controller v1.5.0, the version the platform installs, a
release with an active retry strategy never remediates.
`atomic_release.go` returns an upgrade for a failed release before it
reaches the remediation switch, and the two other paths that uninstall
(a changed release target, and deletion) both clear history afterwards.
So on these releases `uninstalled` never shows up, and `failed` is the
only signal. The parent keeps the rule it already had on main, and
`cilium` and `csi` now follow it too.

The failure paths get the same reading. The guard runs after the
assertions, while the deadlines a failed addon breaks end the script
earlier, so the same classification is also printed, without a verdict,
from the node-join collector and from the exit handler. That handler is
now armed at the tenant CR apply, so the management-cluster waits before
the tenant answers are covered too. Smaller changes that came with this:

- Every guard read is bounded, and a failed read is never folded into
"no history".
- The parent must read back a non-empty strategy. An empty value there
means the jsonpath stopped resolving, and without this check every
footprint would quietly turn into a note.
- The history read uses `HELMRELEASE_HISTORY_JSONPATH`, and the
upstream-type pin in `internal/fluxcontract` now accepts the bounded
wrapper the read goes through.

`docs/agents/e2e-testing.md` §6 and checklist item 8 describe the new
rule.

On related issues: this adds the `local` that #3774 reports missing in
`helmrelease_has_remediation_cycle`, but not the sweep over the other
`_lib/` helpers that the issue also asks for. For #3773 it rewrites the
guard comment the issue quotes. The waits without a backstop are not
touched.

Closes #3553.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff. It touches
`hack/e2e-chainsaw/_lib/`, `hack/*.bats`, one Go test under
`internal/fluxcontract`, two spend-order comments in the kubernetes
suites, and `docs/agents/e2e-testing.md`. No package, schema, CRD, make
target or user-facing document changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): the HelmRelease history guard now also covers the tenant addon releases. A failed or uninstalled Snapshot fails the run on the parent, cilium and csi, which retry in place; other addons get a note. The same classification is printed on the failure paths where the guard itself cannot run.
```
2026-10-05 17:12:38 +03:00
IvanHunters 0639ce39aa fix(kamaji): drop any subchart nested under the vendored kamaji chart (#4459)
## What this PR does

A subchart nested under the vendored kamaji chart would deploy no matter
what the values say, so this PR removes that directory in `make update`
and adds a test that fails if anything renders from it. The vendored
chart declares a `kamaji-etcd` dependency gated on `kamaji-etcd.deploy`,
and the subchart itself is not vendored. The issue proposed pinning
`kamaji-etcd.deploy: false` in the package values. I tested that, and it
would not have helped: through this package the condition is never
evaluated at all.

Helm skips the condition because the umbrella chart declares no
dependencies. `processDependencyEnabled` returns immediately when the
chart it starts from has no `dependencies` in its `Chart.yaml`, and
`cozy-kamaji` has none. The vendored `kamaji` chart is loaded from
`charts/` without being declared, so Helm never recurses into it and
never reads its `kamaji-etcd.deploy` condition. The same early return is
in Helm v3 and v4. I put `kamaji-etcd-0.15.0.tgz` into
`charts/kamaji/charts/` and rendered the package three ways:

- default values: StatefulSet, Job, DataStore and four Certificates from
`kamaji-etcd`
- `kamaji.kamaji-etcd.deploy: false` in the package values: the same
etcd
- `deploy: false` edited into the vendored `charts/kamaji/values.yaml`:
the same etcd

Rendering `charts/kamaji` on its own with `kamaji-etcd.deploy=false`
renders no etcd, which confirms the umbrella is what skips the
condition. So any chart that ends up under `charts/kamaji/charts/`
deploys whatever the values say, and no values change can close the
hazard.

This PR changes two things:

- `make update` removes `charts/kamaji/charts` after extracting the
upstream chart. Upstream does not ship that directory today, so the
recipe still reproduces the current tree exactly. I checked that by
running it on a copy of the package. I also ran it against a fake
upstream tarball that carried the subchart: the directory was extracted
and then removed.
- The package gets its first helm-unittest suite. It asserts that no
template of a subchart nested under `charts/kamaji/charts/` renders a
document, and a `test` target lets `hack/helm-unit-tests.sh` pick it up
in `make unit-tests`. With the subchart tgz dropped in, the suite fails.
With the `rm` line removed from the recipe, the fake-upstream update
leaves the tgz behind and the suite fails too. `/tests` is anchored in
`.helmignore`, so the packaged chart's file list is unchanged.

I left the `kamaji-etcd` block in the vendored values alone. It is not
quite dead: `certmanager_issuer.yaml` reads it to decide whether to keep
the self-signed issuer. That only matters when
`certManager.issuerRef.name` is set, and this package does not set it.
`helm template` output is byte-identical to main.

Closes #4397

### Screenshots

Not a UI change.

### Downstream repositories

The change touches only the kamaji package's update recipe, its test
suite and its `.helmignore`. The closest trigger is package `Makefile`s
→ the website's `development.md`, but that page describes `make update`
and `make test` in general terms that still hold, and the rendered chart
is unchanged.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kamaji): `make update` now drops any subchart nested under the vendored kamaji chart, and a unit test fails if one appears, because such a subchart (for example the bundled kamaji-etcd) would deploy regardless of `kamaji-etcd.deploy`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Helm chart updates now avoid including nested charts that could deploy
unexpectedly.
* Helm chart packages exclude test files, keeping them out of
distributed chart artifacts.

* **Tests**
  * Added a chart check to verify nested charts render no resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:48 +03:00
IvanHunters 2188f16461 test(vminstance): cover VMDisk clones from an image and from a disk (#4586)
## What this PR does

Closes #3482.

The `vminstance` suite now covers both VMDisk clone paths, which had no
e2e test at all. One disk is cloned from a golden image with
`source.image`, and a second disk is cloned from the first with
`source.disk`. For each clone the test checks that the DataVolume points
at the expected source PVC and becomes `Ready`, that its PVC is `Bound`,
and that the VMDisk reports `Ready` and `WorkloadsReady`, the same way
the standalone VMDisk test does. The `Bound` check is separate because
CDI derives `Ready` from the DataVolume phase alone, and can reach
`Succeeded` from the claim's annotations before the claim binds; a
`phase: Succeeded` assert would only repeat `Ready`.

The golden image is a fixture the suite creates itself.
`vm-default-images` is an opt-in package and the e2e install does not
enable it, so `alpine-3.21` from the default collection is not there to
clone. The suite imports the same Alpine 3.21 image into `cozy-public`
under the name `e2e-alpine-3.21`, so it can never collide with an image
the package owns. This is the only download in the new test, and the
`source.disk` case reuses the first clone as its base disk.

The long wait sits on the DataVolume, not on the HelmRelease. Since
#4380 a vm-disk release goes Ready once the DataVolume exists and the
clone's progress is reported through the VMDisk `WorkloadsReady`
condition, so the test does not assert the HelmRelease at all. Chainsaw
deletes the clones before the golden image at cleanup.

The clone itself is only proven by this PR's E2E run, on the `local`
class of the PR lane.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/vminstance/`, and nothing in
the trigger map covers it.

### Release note

```release-note
test(vminstance): e2e now covers VMDisk clones from a golden image (source.image) and from another disk (source.disk).
```
2026-10-05 15:14:38 +03:00
IvanHunters 4b1a75a332 test(vminstance): dry-run evict the VM launcher pod (#4585)
## What this PR does

Part of #4422.

The e2e suite now asks KubeVirt whether it would let a VM launcher pod
be evicted, without evicting it. Until now nothing in e2e evicted a
launcher pod, so the drain behaviour set up in #4403 was known only from
reading KubeVirt. Once the `vminstance` VM is running, the suite sends a
dry-run Eviction (`dryRun=All`) for its virt-launcher pod and checks the
answer. Nothing gets deleted.

A dry run still goes through the KubeVirt eviction webhook, which is
registered with `sideEffects: NoneOnDryRun` in v1.9.0. With
`LiveMigrateIfPossible` its answer follows the VMI's `LiveMigratable`
condition. That condition depends on storage, so the test reads it
instead of assuming it:

- `LiveMigratable=True`: the eviction must be refused with the webhook's
`Eviction triggered evacuation of VMI "<ns>/<name>"` message (HTTP 429).
A drain retries on that while the VM migrates away.
- `LiveMigratable=False`: the eviction must be allowed. The old
`LiveMigrate` value refused this case with "is not live-migratable".
KubeVirt v1.9.0 creates no PodDisruptionBudget for a VMI, so no PDB can
refuse it either.
- Anything else, including a missing condition, fails the test.

The allowed answer alone proves little, because the webhook also lets an
eviction through under `None`. So the test also asserts
`evictionStrategy: LiveMigrateIfPossible` in the VMI spec, where
virt-api copies the cluster value at creation. After the dry run it
checks that the launcher pod is the same object and not terminating, and
that the VMI has no `evacuationNodeName`.

On the PR lane only the `False` branch runs. Post-install prep pins
`StorageProfile/local` to RWO because DRBD is off there, and an RWO disk
makes the VMI non-migratable. Nightly and tag runs use `replicated`,
where I expect the `True` branch, but I have not seen this suite run on
that lane yet. A real drain and a workload update after a KubeVirt bump
are still open in #4422.

I ran the script against a stub `kubectl`. Both expected answers pass,
and it fails on every wrong answer I tried, such as a refusal for a
non-migratable VMI, a PDB-style 429, a missing condition, two launcher
pods, or a replaced pod after the dry run. `chainsaw lint` (v0.2.15) and
shellcheck pass.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/vminstance/` and the suite
README, and nothing in the trigger map covers either.

### Release note

```release-note
test(vminstance): the e2e suite now dry-run evicts the VM launcher pod and checks the KubeVirt eviction decision against the VMI's LiveMigratable condition.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added end-to-end coverage for dry-run eviction of a running virtual
machine’s launcher pod. The test checks the VM’s live-migration policy
and `LiveMigratable` condition, expecting eviction to be refused when
the VM is live-migratable and accepted otherwise.
* Verified that the dry run leaves the launcher pod unchanged and does
not record an evacuation node on the VM.
* **Documentation**
* Documented the test’s behavior and the local `jq` requirement for
running it.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:32 +03:00
IvanHunters 6a98acb61d fix(api): leave failed golden images out of the image options (#4568)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The VM image picker no longer offers golden images whose import failed.
A golden keeps its PVC in `cozy-public` after the import fails, so a
picker built on the PVC list offered an image whose clone never
completes.

The `image` option source in cozystack-api now reads the
`vm-default-images-*` DataVolumes next to the PVCs and drops names whose
DataVolume is in phase `Failed`. I filter rather than mark them, because
such an option can never be used and `OptionItem` has no disabled state.
A golden that is still importing stays in the list, since CDI holds a
clone until the source is populated, and so does a PVC with no
DataVolume. When the DataVolumes cannot be read, the PVCs go out
unfiltered. The Option list drops a source whose provider errors, and an
empty picker would be worse than the old behaviour. cozystack-api gets a
read grant on `cdi.kubevirt.io` DataVolumes for this.

The console did not use that source at all. `SourceField` listed the
PVCs itself, as the tenant, so the fix above would not reach the picker
users see. It now reads the same Option list the other dynamic dropdowns
use. With that, nothing in the console lists `cozy-public` PVCs, so I
removed the read grant on them from the `cozy:tenant:dashboard` Role.
That is a visible change: a tenant loses `kubectl get pvc --namespace
cozy-public`, which was only ever granted for this picker in #2843.

One limitation: in CDI v1.66.1, which cozystack ships, a golden import
reaches `Failed` only on a checksum mismatch, and a broken URL keeps it
in `ImportInProgress`. So this filter covers less than the issue
assumed. I filed #4564 for that.

Closes #4553

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

I have no screenshot. The picker looks the same; only the list behind it
changes. `apps/console/src/components/SourceField.test.tsx` covers it:
the picker shows the images the option source serves and not the raw PVC
names.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(api): the VM image picker no longer offers golden images whose import failed. The console now reads images from the image option source, and tenants no longer get read access to PersistentVolumeClaims in cozy-public.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* The image picker now displays available images from the tenant’s image
options.
* **Bug Fixes**
* Images whose imports have failed are excluded from the picker. Images
remain available when import status is still in progress, has succeeded,
or cannot be retrieved.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:25 +03:00
IvanHunters 86bf2ba8ec fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI (#4563)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This lets the kubevirt-csi controller detach a volume from a VM instance
that no VM owns. Right now that call is always denied. Closes #4452.

The driver has a fallback for a volume that is still attached to the VMI
but is no longer in the VM spec. It calls the VMI-level `removevolume`
subresource. The `-kcsi` Role grants only the two VM-level subresources,
so the call ends in 403. Nothing else reaches this service account with
that grant. Only its own Role and ClusterRole name it as a subject, and
the group bindings that cover every service account grant nothing on
this subresource. At v1.9.0, `kubevirt.io:default` grants only
`kubevirts`, `version` and `guestfs`. `kubevirt.io:edit` and
`kubevirt.io:admin` do include the VMI subresource, but they aggregate
only into the built-in `edit` and `admin` roles, and no tenant binding
gives those to this account. The cozystack tenant roles aggregate on
their own labels and don't list it.

The fallback is still needed after the flag in #4453 goes to false. The
flag only skips VMIs that a VM owns. When a VM is deleted with
`propagationPolicy=Orphan`, the KubeVirt VM controller drops its
finalizer and leaves the VMI running (`vm.go` at v1.9.0). The tenant
node stays up with its volumes attached. When a pod moves off that node,
the driver has to detach through the VMI, and on a 403 the hot-plug pod
keeps holding the infra volume. Until #4453 lands, the fallback also
runs for VM-owned VMIs and gets the same 403.

The grant is `update` on `virtualmachineinstances/removevolume` and
nothing more. The account can already remove a volume from any VM in the
namespace, so it gets no new reach. virt-api also rejects the VMI-level
call on a VM-owned VMI once `HotplugVolumes` is off.
`virtualmachineinstances/addvolume` stays out because the driver only
attaches through the VM.

The new helm-unittest pins the Role's `subresources.kubevirt.io` rule to
these three subresources with `update`. I haven't reproduced the 403 on
a cluster. This shows it on a tenant namespace, before and after:

```bash
kubectl auth can-i update virtualmachineinstances.subresources.kubevirt.io --subresource=removevolume \
  --namespace <tenant-ns> --as system:serviceaccount:<tenant-ns>:<release>-kcsi
```

Upstream kubevirt/csi-driver has the same gap in
`deploy/infra-cluster-service-account.yaml`.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi controller can now hot-unplug a volume from a VMI that no VM owns, instead of failing with 403 Forbidden.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Kubernetes CSI volume removal for virtual machine instances,
including cases where a volume is not listed in the virtual machine
specification. This helps the driver complete volume removal in
scenarios where the Kubernetes virtualization API accepts the
instance-level request. Existing permissions for adding and removing
volumes from virtual machines remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:19 +03:00
IvanHunters ec4d26df54 fix(kubernetes): accept a golden worker image that exists only as a PVC (#4578)
## What this PR does

A worker pool on `osImage.builtin` refused a golden image that exists in
`cozy-public` as a PVC without its DataVolume, and the whole
KubernetesNodes release failed to render. CDI clones the PVC, not the
DataVolume, so that golden is a valid source, and vm-disk already
accepts it. This can happen once CDI garbage-collects DataVolumes
(`dataVolumeTTLSeconds`) or when someone restores a golden PVC on its
own.

Now the render fails only when neither the DataVolume nor the PVC
exists. The StorageClass and size checks already fall back to the PVC,
so a PVC-only golden is still checked against the pool, and the new test
suite pins that too.

Closes #4565

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): a worker pool on osImage.builtin now accepts a golden image that exists in cozy-public only as a PVC, matching vm-disk; it is refused only when neither the DataVolume nor the PVC exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Built-in golden images can now be used when available as either a
DataVolume or a PVC.
* **Bug Fixes**
* Missing golden images now produce an error that identifies the missing
image and suggests ways to resolve the issue. The error is suppressed
when the pool already has a successful clone.
* PVC-based golden images are checked against the pool’s storage class
and minimum disk size requirements.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:45:50 +03:00
IvanHunters f9f7562e56 fix(kamaji): resolve Gateway API availability once at startup (#4501)
## What this PR does

Carries the fix proposed upstream in clastix/kamaji#1327 (not yet merged
there; issue clastix/kamaji#1326) as a build patch on the 26.9.4-edge
tag that cozystack builds Kamaji from.

In Kamaji 26.9.4-edge, every TenantControlPlane reconcile checks whether
the Gateway API is available. The discovery call ignores the reconcile
context, so it is bounded only by the client's 32s default timeout,
which is longer than the 30s reconcile budget. The watches, however,
read availability only once, at startup.

With the patch, Kamaji resolves availability once when the manager
starts. The watches, the indexer, the reconciled resources and the
admission webhook's Gateway API check all use that one value. A
discovery error at startup now stops the manager, so it is no longer
mistaken for "Gateway API not installed". A Gateway API installed after
Kamaji starts needs a Kamaji restart. On this platform that case does
not come up: the kamaji package depends on cert-manager, and
cert-manager depends on gateway-api-crds.

The webhook's separate TLSRoute check still runs discovery on each
request, as before, and is listed as a follow-up in clastix/kamaji#1327.
Kamaji's chart sets no `timeoutSeconds` on the webhook, so the
apiserver's 10s default applies, and a discovery call that can take up
to the client's 32s can make that admission time out.

The patch is the upstream commit applied unchanged except for the
dropped docs hunk, which is not part of the image. It applies cleanly to
26.9.4-edge with no offsets. It modifies different files than the
existing `fix-kubelet-config-compat.diff`, so the two apply in either
order. The upstream tests stay in the patch so that it matches the
upstream commit; the image build does not run them. On the patched tree,
`go build ./...` passes, and `go test -race` passes for
`./controllers/...`, `./internal/webhook/...`,
`./internal/utilities/...`, `./internal/kubeadm/...` and `./cmd/...`. I
also checked that the new tests can fail: they go red when the
availability check is inverted and when the old per-reconcile lookup is
put back.

If clastix/kamaji#1327 changes before it merges, this patch should be
updated to match. It is dropped once Kamaji ships the change.

Refs #4489. This does not close #4489, because this change does not
explain the long stall reported there.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kamaji): Kamaji now checks for the Gateway API once at startup instead of on every TenantControlPlane reconcile, so a slow discovery endpoint can no longer use up the reconcile timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Gateway API availability is now checked at startup and consistently
applied to Gateway-related setup and configuration validation.
* Gateway configuration is rejected if the Gateway API was unavailable
when Kamaji started. If you install the required CRDs afterward, restart
Kamaji before configuring Gateway resources.
* Discovery errors now stop startup rather than allowing Kamaji to
continue with an uncertain Gateway API status.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:51 +03:00
IvanHunters b0e07cadbb fix(operator): rebuild each changed ArtifactGenerator generation once (#4594)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

After an upgrade, an ArtifactGenerator can report Ready on its new spec
while its artifacts were built from the old one, and source-watcher
never rebuilds them. That is how the bucket-system chart in #4593
shipped without cozy-lib.

The cause is upstream (fluxcd/source-watcher#384). source-watcher builds
from the spec it read when the reconcile started, but the final status
patch stamps Ready with the live generation. An upgrade changes the
packages source and the copy rules within seconds, so a spec change
often lands during a build.

The generator status has nothing that tells a stale build from a good
one. So the operator now rebuilds every changed generation once, after
source-watcher first reports it Ready, through the force-drift path the
operator already uses for fluxcd/pkg#934. An unchanged spec rebuilds to
the same digest, so ExternalArtifacts and HelmReleases do not move.
Generation 1 is skipped, since the race needs a spec change during a
build.

On the first rollout every generator past generation 1 gets one extra
rebuild, and its PackageSource shows
`Unknown/AwaitingSourceWatcherRecovery` until that rebuild lands.
Nothing waits on that condition today. The same rebuild repairs clusters
already stuck like #4593.

The recovery messages on the PackageSource no longer blame
fluxcd/pkg#934, because the same path now serves both cases.

### How it was tested

Unit tests in `internal/operator/packagesource_reconciler_test.go` cover
the forced rebuild on generation 2 and on a later generation, every case
where no rebuild must happen, the rebuild settling without a second one,
both failure orders (force fails, recording the generation fails), and a
stale recovery counter. Each condition of the new predicate was mutated
once to confirm a test fails.

I also ran this commit's operator image on an arm64 test cluster running
v1.7.0-alpha.3, the one where #4593 was found. Before that I had
repaired its bucket generator by hand: deleting the stale
ExternalArtifact and requesting a reconcile produced an artifact with
`charts/cozy-lib` and both bucket HelmReleases went Ready. With the
patched operator, the only generator past generation 1 got exactly one
forced rebuild and the verified-generation annotation. source-watcher
reported all its ExternalArtifacts up to date. The digests of all 214
ExternalArtifacts and the revisions of all 104 HelmReleases matched a
snapshot taken before the swap, and every PackageSource and HelmRelease
stayed Ready. The race itself was not reproduced on the cluster, since
that needs a spec change landing during a build.

Closes #4593

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(operator): rebuild an ArtifactGenerator once after each spec change, so an upgrade no longer leaves a component on artifacts built from the previous spec (for example a chart missing a library it now needs).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved PackageSource recovery when an artifact generator reports
ready: the system now verifies the generated output and triggers a
rebuild when needed.
* Failed rebuild or verification attempts can be retried during a later
reconciliation, and recovery tracking is cleared once the rebuilt
generation settles.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:44 +03:00
IvanHunters 50dec5d46a fix(platform): resume a retried migration Job from the stamped version (#4583)
## What this PR does

A retried pod of the platform migration Job no longer runs migrations
that already finished in the same upgrade. The Job gets
`CURRENT_VERSION` once, at render time, and a failed pod is replaced by
one with the same environment. By then every finished migration has
stamped the `cozystack-version` ConfigMap, but the new pod ignored that
and started from the rendered version again.

`run-migrations.sh` now starts from the version stamped in the
ConfigMap. If reading it fails, the pod fails and the Job retries it,
because falling back to the rendered version is exactly the re-run this
fixes. The rendered value is used only when the ConfigMap holds no
integer version, which is also when the chart itself could not get a
real version from it. Without that fallback a value like `v58` gave a
green Job that ran nothing.

The new `hack/run-migrations.bats` runs the script in the migrations
image's alpine base against a fake kubectl that keeps the stamp in a
file, so one pod's stamps are what the next pod reads. It needs docker,
like the other migration suites.

I also found that the ConfigMap existence check at the top of the script
treats any API error as a missing ConfigMap and skips every pending
migration. That is older than this change and is filed as #4579.

Closes #4555

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a retried platform migration Job now resumes from the version stamped in the cozystack-version ConfigMap instead of re-running migrations that already finished in the same upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Migration startup now resumes from a valid stamped version, avoiding
reruns of migrations that have already completed.
* Empty or non-numeric stamps fall back to the rendered version.
Out-of-range stamps or failed stamp reads stop migrations before they
run.
* If the migration ConfigMap is missing, migrations do not run and the
target version is stamped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:39 +03:00
IvanHunters bb2946990d fix(vm-instance)!: refuse guest memory below KubeVirt's 1M floor (#4569)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The vm-instance resource guard now refuses guest memory below 1M, the
same floor KubeVirt admission enforces. Before, a value such as `500m`
or `1Ki` rendered and then failed when the VirtualMachine was applied.

This is a breaking change. A memory above zero and below 1M rendered
before and is now refused at render time. KubeVirt v1.9.0 would not
admit such a VM either: `validateMemoryRequestsNegativeOrNull` refuses a
non-zero request below 1M
(`pkg/virt-api/webhooks/validating-webhook/admitters/vmi-create-admitter.go:1135`).
The VirtualMachine admitter runs the same check on its template.

The cpu, sockets and memory descriptions now list the values the guard
refuses, so the schema states the same rule as the error.

The error cannot quote an unquoted number as it was written, because
YAML has decoded it before the chart sees it. A large one also printed
in exponent form, so `4294967296` came out as `4.294967296e+09`. A
string is still quoted exactly as written. A number is now printed in
plain digits and marked "read as", which is the closest honest wording I
found.

Closes #4413

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(vm-instance)!: refuse `resources.memory` below 1M at render time, since KubeVirt does not admit a VM with less. A value above zero and below 1M rendered before and is now refused. The cpu, sockets and memory descriptions list the refused values, and the guard error marks an unquoted number as read rather than written.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* VM resource values are now checked when rendering: CPU cores and
sockets must be positive whole numbers no greater than 4294967295.
Unit-suffixed, fractional, zero, negative, and quoted values with
leading zeros are rejected.
* Memory values below `1M` are rejected, with clearer messages showing
the value as entered or interpreted.

* **Documentation**
* Updated resource guidance to describe these limits and validation
behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:15 +03:00
IvanHunters ed20c53449 fix(vm-disk): refuse to clone a golden image that does not exist (#4567)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Stacked on #4392, so the base is `fix/vm-default-images-health-check`.
Retarget to `main` once #4392 merges.

A new vm-disk with `source.image` now fails to render when the golden
image it names does not exist, and the error names the image and
`cozy-public`. Before this the chart cloned
`cozy-public/vm-default-images-<name>` without looking, and CDI waited
for that PVC forever while the VMDisk stayed Ready. That happens on a
typo, and on every cluster where the opt-in
`cozystack.vm-default-images` package is off, so the message also says
to add it to `bundles.enabledPackages`.

The golden counts as missing only when neither its DataVolume nor its
PVC exists. CDI clones the PVC, so a PVC with no DataVolume is still a
usable source, and a DataVolume whose PVC is not created yet is an
import about to start.

Under `helm template` every lookup is empty, so the check runs only when
the release namespace itself is visible. The same trick is already used
in the kubernetes and seaweedfs charts. Existing disks are exempt the
same way #4392 exempts them, because their DataVolume spec is immutable.
A disk refused while the catalog is off renders by itself once the
golden appears, since app HelmReleases retry on failure.

Tests are in `packages/apps/vm-disk/tests/dv_golden_image_test.yaml`:
refusal of a missing golden, a PVC without a DataVolume, a DataVolume
before its PVC, rendering with no cluster, and an existing disk whose
golden was removed.

Closes #4548

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(vm-disk): a new disk with source.image now fails with a clear error when the golden image does not exist in cozy-public, instead of waiting forever for a clone that never starts.
```
2026-10-05 14:42:52 +03:00
IvanHunters c7b8009964 fix(kubernetes): log kubevirt-csi temp NFS mount dir removal failure (#4562)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The kubevirt-csi node plugin now logs a warning when it cannot remove
the temp dir it uses for NFS publish. Before, the error was dropped, so
a dir left behind after a failed unmount was never reported.

The publish path temp-mounts the export root to create `/data`, then
removes the temp dir in a defer. The failure is logged, the same way the
unmount failure next to it already is, and not returned. A leftover temp
dir does not break the volume, and failing a publish that otherwise
worked only makes kubelet retry it.

A new unit test publishes an NFS volume through the fake mounter, where
the removal always fails, and checks that a warning naming the dir is
logged.

This is independent of #4451. That PR touches the same Go module but not
`node.go`.

A failed unmount still leaves the NFS root mounted. That is older than
this change and is tracked in #4561.

Closes #4466

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi node plugin now logs a warning when it cannot remove its temporary NFS mount dir instead of dropping the error
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Failed cleanup of a temporary mount directory now produces a warning
with the directory path, making the issue easier to identify. The volume
publish operation continues to succeed when cleanup fails.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:39:58 +03:00
Aleksei Sviridkin 2ddd55ce5b fix(build): honour PUSH and LOAD from the environment and in image-talos (#4654)
## What this PR does

A local build that asks to load images into docker no longer pushes
them. `hack/common-envs.mk` set `PUSH` and `LOAD` with plain `:=`, and a
makefile assignment beats the environment, so `LOAD=1 PUSH=0 make image`
quietly pushed a multi-arch build. Both now take the environment value,
and an empty one falls back to the default. I also made `PUSH` default
to 0 when `LOAD=1`, so `LOAD=1` alone only loads. Before this, `make
image LOAD=1` pushed and loaded at once. An explicit `PUSH` from the
environment or the command line still wins, and `OCI_EXPORT_DIR` still
forces both off. Every buildx target gets the values through
`BUILDX_ARGS`, so the fix lives in that one file.

The Talos installer was the one image that ignored `PUSH`. It is
published with skopeo, not buildx, so it never saw `--push=$(PUSH)`. Its
registry copies now run only when `PUSH=1`.

A build that pushes nothing no longer writes `PUSHED_TAGS_LOG`. That log
tells the multi-arch stitch which tags to move, so it must hold only
tags that were really pushed.

The cache helper lost an argument nobody used. No caller passes the
second argument of `cache-args`, and an explicit tag there would bypass
`CACHE_TAG`, so the amd64 and arm64 builds of that image would share one
cache ref.

CI sets neither `PUSH` nor `LOAD`, so CI builds do the same thing as
before. Each fix has a `make --dry-run` test in
`hack/common-envs_test.bats`.

Closes #4499

Closes #4504

Closes #4551

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The ccp skills pass `PUSH=1` explicitly, which still works. The website
docs do not mention `PUSH`, `LOAD` or `cache-args`.

### Release note

```release-note
fix(build): `LOAD` and `PUSH` set in the environment are now honoured, `LOAD=1` no longer pushes unless `PUSH=1` is also set, and the Talos installer is no longer published when `PUSH=0`.
```
2026-10-05 14:15:37 +03:00
Aleksei Sviridkin 2af8b0c229 ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.

Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.

Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.

The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.

`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.

Closes #4591

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:47 +03:00
Aleksei Sviridkin 9c025fa2b1 fix(installer): skip the packages artifact push and stamp with PUSH=0 (#4663)
## What this PR does

A local build with `PUSH=0` no longer publishes the packages artifact.
Before this change, `make image-packages` in `packages/core/installer`
ran `flux push artifact` no matter what `PUSH` was set to, and then
wrote the pushed digest into `values.yaml`. The `chart` target in the
same Makefile already respects `PUSH`.

With `PUSH` set to anything other than 1, the target now skips both the
push and the digest stamp. It prints one line saying so and leaves the
committed `platformSourceUrl` and `platformSourceRef` as they are. It
still writes `packages/core/platform/.build-revision`, which is
gitignored.

I chose to skip the stamp instead of computing a local digest, because
every reader of that value needs a digest that exists in the registry.
The chart passes `platformSourceRef` to the operator as
`--platform-source-ref`, and Flux then pulls
`cozystack-packages@<digest>` from the registry. The release scripts
`verify-promoted-packages.sh`, `promote-packages-artifact.sh`,
`stitch-multiarch.sh` and `nightly-mirror.sh` also treat it as a
registry digest. A digest of an artifact that was never pushed points at
nothing. On main an unpushed run would not even get that far, since the
recipe dies at `test -n "$DIGEST"`.

The gate is a make-level `ifeq ($(PUSH),1)`, so it reads whatever `PUSH`
ends up as after `hack/common-envs.mk`. On main that means `PUSH=0` on
the command line. I also applied the `common-envs.mk` change from #4654
locally, which takes `PUSH` from the environment and makes `LOAD=1`
default it to 0. With that applied, a dry run shows no push and no stamp
for `PUSH=0` in the environment, for `LOAD=1`, and for `PUSH=0` on the
command line, while `LOAD=1 PUSH=1` and an empty `PUSH` still push.

CI is not affected, because no workflow sets `PUSH` or `LOAD` and `PUSH`
stays at 1. I dry-ran the three CI invocations of this Makefile before
and after the change, on main and again with #4654 applied:

- `tags.yaml`, release stitch step: `make -C packages/core/installer
image-packages chart` with `REGISTRY`, a release tag as `IMAGE_TAG`, and
`PUSHED_TAGS_LOG`
- `pull-requests.yaml`, installer step of a same-repo PR: `make -C
packages/core/installer image` with a `pr-N-sha` tag,
`PLATFORM=linux/amd64`, `BUILDER`, and an empty `OCI_EXPORT_DIR`
- `build-main.yaml` and `build-release.yaml`: `make build`, which runs
the installer `image` target with defaults

```text
$ diff --recursive before/ after/     # tags.txt pr.txt build.txt
$ echo $?
0
```

There is one side effect. `OCI_EXPORT_DIR` forces `PUSH := 0`, so
running `image-packages` with it set no longer pushes, where before it
would have tried an anonymous push. No CI path does that. The fork PR
build already runs only `pre-checks image-operator` for the installer,
and the `make -n ... image OCI_EXPORT_DIR=/oci` scan in `e2e-fork.yaml`
greps only the `.oci.tar` names, which this change does not touch.

The new test dry-runs the target in both modes. With `PUSH=0` there must
be no `flux push artifact` and no `platformSourceRef` stamp; with `PUSH`
at its default, both must be present. The first test fails on main. I
mutated the guard three ways, and each time only the matching test went
red:

- condition forced true: the `PUSH=0` test fails
- compared against 2: the default-path test fails
- `endif` moved up so only the push is gated: the `PUSH=0` test fails on
the stamp check

One limitation is left as a follow-up. The test's `env -u PUSH -u LOAD`
only isolates it once #4654 lands, because on main a file assignment
wins over the environment anyway. It does not cover `PUSH=0` coming in
through `MAKEFLAGS`, so `make bats-unit-tests PUSH=0` would fail the
default-path test here and the ordering check in
`hack/stitch-multiarch_test.bats`. Nothing runs the suite that way
today. If that changes, passing an explicit `PUSH=1` on the default-path
`make` lines, or unsetting `MAKEFLAGS` the way
`hack/bats-unit-keep-going.bats` does, would fix it.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website's `development.md` tells developers to run `make
image-packages` with `REGISTRY` set to their own registry. That is the
default `PUSH=1` path, and it works as before. Neither the website nor
ccp mentions `PUSH=0` for this target.

Closes #4656

### Release note

```release-note
fix(installer): `make image-packages` no longer pushes the cozystack-packages artifact or stamps its digest into values.yaml when PUSH=0.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* You can run `image-packages` with `PUSH=0` to skip publishing the
artifact and keep the committed platform source reference unchanged.
* When pushing is enabled, the command publishes the artifact and
updates the platform source URL and digest-based reference.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:42 +03:00
Aleksei Sviridkin 33cf6972bb test(qdrant): exercise the tenant trust anchor end to end (#4701)
## What this PR does

The qdrant e2e suite now runs with TLS on and checks the tenant trust
anchor end to end. Before this, the fixture left `tls` unset with
`external: false`, so the chart rendered no CA and no
`TenantProjection`, and nothing in e2e reached the controller or
`tenantsecrets`. Setting `tls.enabled: true` turns TLS on without the
LoadBalancer that `external: true` would allocate.

The first new step checks that the anchor is published. The
`qdrant-test-ca` sentinel has to report `Ready=True/Projected`, and
`qdrant-test.tenant-ca` has to carry both the `tenant-ca` and
`tenantresource=true` labels. The projection must hold only `ca.crt`
with the same fingerprint as the chart's CA, and `tenantsecrets` must
serve it. The key-bearing `qdrant-test-ca` and `qdrant-test-tls` must
answer NotFound there, and any other error fails the step.

The second step checks that the anchor is what decides trust. It reads
the anchor through `tenantsecrets`, port-forwards the qdrant Service and
runs `openssl s_client` with that file as the only CA, hostname check
included. A CA with the same subject and another key must fail, and so
must the runner's default trust store. A handshake failure counts only
when the output shows a verification error, so a refused connection does
not pass as a rejection. Plaintext HTTP must get no response. Curl exits
7 and 28 fail the step, and a second verified handshake afterwards
proves the forward was still alive.

TLS does not break the existing steps, because none of them talk to
qdrant. `create-qdrant`, `wait-helmreleases-ready`,
`verify-statefulset-and-storage` and `verify-pvc-reclaim-on-delete` only
read Kubernetes objects: no `http://`, no port 6333 or 6334, no curl or
exec, and the suite has no backup round trip. The release now also
renders two Issuers, two Certificates and the sentinel. The chart
switches its probes to HTTPS with `enable_tls`, so `readyReplicas: 1`
now covers a pod serving TLS.

Local checks: `chainsaw lint test` (v0.2.15) says the file is valid.
`helm template` with the fixture's spec renders 2 Certificates, 2
Issuers and 1 TenantProjection, and none without TLS. I ran the
handshake logic against a local `openssl s_server` with a CA and leaf
shaped like the chart's. The anchor passes, the wrong CA and the default
store fail on verification, a wrong hostname fails, plaintext gets
`000`, and a closed port is not counted as a rejection.

A sentinel with a missing source (`SourceNotFound`) is not tested here,
the reconciler unit tests cover it.

Review notes I left as they are:

- Through `kubectl port-forward` the local socket is always accepted, so
curl exits 7 and 28 only catch a dead forward process. A false pass
would need qdrant to answer plaintext and a pod-side blip on that one
request at the same time.
- qdrant may reset the socket on the plaintext request, and some kubectl
versions drop the whole port-forward on a pod-side error. Then the
handshake after it turns the step red, not green. If CI shows `lost
connection to pod` there, the plaintext check should get its own
port-forward.
- The suite's `catch` has no `describe` of the `TenantProjection`. The
sentinel assert prints its conditions when it fails, but a red Secret
assert or script after it would not.
- The vendored chart's `servicemonitor.yaml` hardcodes `scheme: http`,
so metrics scraping fails for any qdrant with TLS on. That was already
the case before this PR, and the suite does not check metrics. Tracked
in #4703.

cert-manager does not set owner references here, so `qdrant-test-ca` and
`qdrant-test-tls` stay in the namespace with their private keys after
the app is deleted. That is tracked in #4704. The nats half of #3706 is
tracked in #4702.

Closes #3706

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/qdrant/`, which no entry in the
trigger map covers.

### Release note

```release-note
test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused.
```
2026-10-05 14:14:36 +03:00
Aleksei Sviridkin 7ea71042b7 ci(e2e): add a daily isp-hosted install lane on kind (#4712)
## What this PR does

This adds a daily CI lane that installs the `isp-hosted` platform
variant through the `hosted` installer variant. No lane ran either of
them before. The PR, nightly and tag lanes all install `isp-full` on
Talos, where Cozystack brings its own CNI and storage, and that is how
#3376 shipped unnoticed.

What `isp-hosted` changes is what it leaves to the host. The system
bundle swaps kube-ovn and Cilium for a `noop` networking Package and
drops multus, LINSTOR, MetalLB, cozy-proxy, snapshot-controller and the
rest of the shared data-plane set. The iaas bundle refuses to render.
Migrations are not a difference, `isp-full` enables them too.

So the lane runs on kind. kindnet, kube-proxy and local-path come from
the cluster, and there is no Cilium, no LINSTOR and no `replicated` or
`local` StorageClass. It installs the latest GHCR nightly instead of
building anything, at 05:00 UTC after the nightly mirror publishes, and
on `workflow_dispatch`. There is no pull request trigger, because a run
would install the nightly images and not the PR's.

A run passes when every HelmRelease is Ready, `cozystack-platform` and
`tenant-root` included. It also checks that the platform emitted the
`noop` networking Package and none of LINSTOR, MetalLB, multus or
KubeVirt, and that the cluster has no Cilium CRD and no `replicated` or
`local` class.

The lane reuses the e2e sandbox container for tooling and
`collect-report`, attached to the kind network.
`hack/isp-hosted-lane_test.bats` pins the wiring: the Package selects
`isp-hosted`, the install uses `variant=hosted`, the kind DNS domain
matches `networking.clusterDomain`, and kind is pinned by checksum and
digest. Six of its seven tests were red before the implementation
existed.

`hack/sandbox-runner-headroom.bats` gets a named exemption. That check
treats any workflow touching `packages/core/testing` as a lane that
boots the QEMU guests, and this one boots none. The exemption is checked
after the job match, and removing it turns the test red again.

The pins come from the kind v0.32.0 release: the binary sha256
`50030de2…` from its `kind-linux-amd64.sha256sum`, and
`kindest/node:v1.33.12@sha256:3f5c8443…` from its release notes.
v1.33.12 is the Kubernetes version the Talos lanes use, so the variant
is the only thing that differs. All three are plain `env` values in the
workflow, where a Renovate custom manager can match them.

### Real runs

The first real run on this branch,
[36961209755](https://github.com/cozystack/cozystack/actions/runs/36961209755),
went red on `tenant-root`:

```text
tenant-root  False  Helm install failed for release tenant-root/tenant-root with chart tenant@0.0.0+629b61300902: unable to build kubernetes objects from release manifest: resource mapping not found for name: "allow-external-communication" namespace: "tenant-root" from "": no matches for kind "CiliumNetworkPolicy" in version "cilium.io/v2"
```

This is #3387 reproduced live: `packages/apps/tenant` renders Cilium
policies on a cluster without Cilium. The other releases went Ready,
except `cozystack-basics`, which was still in an upgrade at the last
listing. The fix landed in #4710.

The run after it,
[37089699259](https://github.com/cozystack/cozystack/actions/runs/37089699259),
is green on the first attempt. It installed nightly
`0.0.0-nightly.20261003.37089458387.1`, built from 309875ab6, which
contains #4710. All four install tests passed, `tenant-root` included,
with no `cilium.io` CRD on the cluster. The install took about 9
minutes.

### Notes

The OpenAPI suite is not run. Its "Test kinds" case needs an Ingress
application, and kind has no LoadBalancer, so it would fail for a reason
unrelated to the variant.

The red run's job log is 347 MB. cozytest's live trace prints the whole
HelmRelease snapshot JSON from `cozy_wait_all_helmreleases_ready` on
every 2-second poll, and `gh run view --log` breaks at that size. The
isp-full lanes use the same helper, and it is tracked separately.

The red run spent about 31 minutes in install, most of it waiting out
the 30-minute gate; the green one took about 9 minutes on
`oracle-vm-24cpu-96gb-x86-64`. Moving to a smaller runner needs a
measured run, so it is a follow-up and not part of this PR.

Follow-ups from review, none blocking:

- No `vars.NIGHTLY_ENABLED` gate, so with the nightly switched off the
lane keeps testing a stale `nightly` tag. A `workflow_run` trigger on a
successful mirror would fix that too.
- The Package endpoint `cozy-hosted-control-plane` has to agree with
`KIND_CLUSTER: cozy-hosted`, and no test checks that yet.
- The operator registration check greps a Go literal in
`cmd/cozystack-operator/main.go`, so reformatting that table breaks it.
- The oras download has no checksum check, same as in `nightly.yaml`.
- cozytest stops at the first failing test, so on a red install the "no
Cilium" check never runs. Running it before the install would keep that
evidence in red runs.

Closes #3390

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds CI files, test files and one new make target. Nothing
under `hack/` is moved or renamed and no existing target changes, so the
ccp trigger does not apply.

### Release note

```release-note
ci(e2e): add a daily lane that installs the isp-hosted platform variant on kind
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added scheduled and manually triggered end-to-end checks for hosted
installations using the latest nightly packages.
* Validation covers platform readiness, package deployment, and expected
networking and storage behavior.
* Test runs collect diagnostic reports and attempt cleanup after
failures to support investigation.
* Added checks to verify hosted installation settings and ensure
prohibited network and storage components are absent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:05 +03:00
IvanHunters 14eab3cfe3 fix(backups): restrict the kafka-metadata CLI Pod and correct demo personas (#4695)
## What this PR does

This fixes two problems in the Kafka and NATS backup demos under
`examples/backups/`.

The Kafka metadata demo ran its CLI Pod without any securityContext, so
a namespace that enforces the restricted Pod Security level would reject
it. The Pod now gets the same restricted context as the CLI Pod in the
topic-data demo: `runAsNonRoot`, the `RuntimeDefault` seccomp profile,
no privilege escalation, all capabilities dropped. The new bats case
reads the overrides passed to `kubectl run` and fails if any of those
fields is missing or the patch is not sent as `--override-type=json`. It
is red on main, and it goes red on each of seven single-field mutations
of the patch while the other five cases stay green.

One known limit: `runAsNonRoot` needs an image that runs as a non-root
user. The default Strimzi image runs as UID 1001 and is fine. If
`KAFKA_IMAGE` is overridden with an image that runs as root, the kubelet
refuses to start the container, and the helper fails only when its 5
minute `kubectl wait` times out.

The kafka and nats walkthroughs told a tenant to run scripts that the
tenant roles forbid. Steps 03, 04, 06, 07 and `cleanup.sh` were in the
tenant column, and the scenario docs and several script comments had the
tenant create the backup Secret. Those rows now say admin and name the
part a tenant admin can do on its own, and the scenario docs say the
same. After this change the persona column in both tables matches the
tenant roles. kafka-metadata has no persona table, so I didn't change
anything there.

These are the tenant rules on pods and backup objects, plus one example
of the per-app secret rules. Each app chart's dashboard Role allows
`get`, `list` and `watch` on a few named Secrets and no other verb on
secrets. Nothing aggregated into the tenant roles grants `pods/exec`,
`pods/log`, `helmreleases`, `objectstorage.k8s.io`, `statefulsets` or
`kafkas.kafka.strimzi.io`.

```yaml
# packages/system/cozystack-basics/templates/clusterroles.yaml:100-112 (cozy:tenant:view:base, every tenant level)
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - persistentvolumes
  - endpoints
  - events
  - resourcequotas
  verbs:
  - get
  - list
  - watch
# packages/system/cozystack-basics/templates/clusterroles.yaml:238-246 (cozy:tenant:admin:base)
- apiGroups: [""]
  resources:
  - pods
  - services
  - persistentvolumes
  - endpoints
  - events
  verbs:
  - delete
# packages/system/backup-controller/templates/tenant-clusterroles.yaml:35-44 (cozy:backups:admin)
- apiGroups: ["backups.cozystack.io"]
  resources:
  - plans
  - backupjobs
  - restorejobs
  verbs:
  - create
  - update
  - patch
  - delete
# packages/apps/bucket/templates/dashboard-resourcemap.yaml:6-14 (bound at the tenant "use" level)
- apiGroups:
  - ""
  resources:
  - secrets
  resourceNames:
  {{- range $name, $user := .Values.users }}
  - {{ $.Release.Name }}-{{ $name }}-credentials
  {{- end }}
  verbs: ["get", "list", "watch"]
```

How each changed row runs into those rules:

| Row | What the script needs | Why a tenant can't |
| --- | --- | --- |
| 03 (kafka, nats) | waits on the `HelmRelease` and COSI objects, `get`
on the `bucket-<name>-backup` Secret, kafka also reads the CA Secret in
`tenant-root` | no `helmreleases` or `objectstorage.k8s.io` rule; the
bucket Role covers only `<release>-<user>-credentials` |
| 04 (kafka, nats) | waits on the `HelmRelease` and the Strimzi `Kafka`
or the `StatefulSet`, `create` on `secrets`, `create` on `pods`,
`pods/exec` | pods are `get`/`list`/`watch`/`delete` only, and nothing
grants `create` on secrets |
| 06 (kafka, nats) | `create` on `pods` and `pods/exec` for the CLI Pod,
kafka also `get` on `pods/log` for the refused restore | same pod rules,
and nothing grants `pods/log` |
| 07 (kafka, nats) | same as 04 | same as 04 |
| `cleanup.sh` (kafka, nats) | `delete` on `secrets`, on `backups`, and
on the cluster-scoped `BackupClass` and strategy | `cozy:backups:admin`
deletes only plans, backupjobs and restorejobs, and no tenant rule
covers secrets or cluster-scoped objects |

Step 05 stays a tenant step. Its `BackupJob` comes from
`cozy:backups:admin`, so in practice that means a tenant admin. The nats
demo's CLI Pod has the same missing securityContext as the
kafka-metadata one, and I left it for a separate issue.

Closes #4625

Closes #4636

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): the kafka-metadata backup demo now runs its CLI Pod with the restricted Pod Security context, and the kafka and nats backup demos now list steps 03, 04, 06, 07 and cleanup as admin steps, because they need Pod, exec, Secret and cluster-scoped permissions that tenant roles do not grant.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Security**
* Kafka metadata backup CLI Pods are recreated with restricted security
settings when they are not running or lack the required non-root
configuration.
* **Documentation**
* Kafka and NATS guides clarify which setup, restore, and cleanup steps
require administrator access and which job actions tenants can perform.
* Updated examples clarify that the demo provides required storage
Secrets and certificates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:54 +03:00
IvanHunters 1621c471f2 ci(dashboard): run eslint on the console and clear its errors (#4692)
## What this PR does

The console now passes its own eslint config, and CI runs it. On main
`pnpm lint` exits 1 with 54 problems, 51 of them errors, and
`ui-test.yaml` never ran it. On this branch it exits 0 with no errors,
and a Lint step sits between Typecheck and Test.

How each rule was handled:

- `no-explicit-any` (38 errors): replaced with real types. These commits
build a vite bundle byte-identical to main's.
- `react-refresh/only-export-components` (6 errors): the hooks, context
objects, `resolveSource` and the RJSF registry moved out of component
files into their own modules. `useTenantContext` stays in
`tenant-context.tsx`, so its importers and the tests that mock that
module are untouched.
- `no-unused-vars` (1 error): dropped the unused `ad` binding in
`OverviewTab`.
- `react-hooks/set-state-in-effect` (6 errors): relaxed per line with a
reason, not in the config.

The last rule is the only relaxation. Each of these effects re-syncs
local state to something that changes outside the component. The rule
wants the state derived during render, which changes when the value is
committed or what the first render shows, so it would be a behaviour
change. Per file:

- `components/ResourceQuotasField.tsx` re-syncs the inputs after the
form value is reset from outside.
- `components/SourceWidget.tsx` re-syncs the selected option when the
value loads asynchronously in edit forms.
- `components/command-palette/command-palette.tsx` resets query and
level whenever the palette closes.
- `components/command-palette/use-keyboard-nav.ts` moves the highlight
back to the top when the item list changes.
- `hooks/use-is-mac.ts` would return `true` on the first render on macOS
with a lazy initial state. Today it returns `false`.
- `lib/tenant-provider.tsx` falls back to an existing tenant once the
tenant list has loaded.

The eslint config itself is unchanged, and `no-explicit-any` stays on
everywhere.

`pnpm typecheck` and `pnpm test` (442 tests) pass at every commit. After
the file moves the bundle has the same chunk set, and only the entry
chunk differs, by reordered and renamed minified functions.

Three `react-hooks/exhaustive-deps` warnings from main are left as they
are, because fixing them changes when the effects run. One is in
`SourceWidget.tsx`, where the effect reads `options` but depends only on
`value`. The other two come from `ads = adList?.items || []` in
`use-command-items.tsx`, which gives two `useMemo` hooks a new
dependency on every render.

Things I did not do here:

- The step has no `--max-warnings`, so a new warning will not fail CI.
Once the three warnings are fixed, `--max-warnings 0` can go in.
- `SourceField.tsx` and `SourceWidget.tsx` each declare the same small
`SourceOptionSchema` interface. I kept them local since the two widgets
share nothing else.
- The job name stays `Typecheck and test`, so a required check with that
name keeps matching.
- `pnpm -r lint` only reaches `apps/console`.
`packages/{k8s-client,ui,types}` have no `lint` script, same as on main.

Closes #3911

Closes #4566

### Screenshots

No visual change: types, module layout and CI only.

### Downstream repositories

The diff touches only the console workspace and `ui-test.yaml`. The
console has no downstream copy since `cozystack-ui` was archived, and no
trigger map entry matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(dashboard): the console's eslint now runs in the UI test job, and the console passes it with no errors.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Tenant selection now uses available tenant namespaces, restores a
saved selection when possible, and falls back to the root tenant or the
first available tenant.
* Tenant options are sorted, and the selected tenant is kept in sync
with the available choices.
* **Chores**
* Automated checks now include linting alongside type checks and unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:31 +03:00
IvanHunters 9239555e6a ci(workflows): pin flux, yq and helm-unittest installs (#4645)
## What this PR does

Workflows no longer run the flux install script as root. Every step that
needed the flux CLI downloaded `https://fluxcd.io/install.sh` and ran it
with `sudo`, so whatever that host served at the moment ran on runners
that later get registry credentials. PR #4442 pinned the version, but
the script was still fetched and executed. All 12 of those steps now use
`fluxcd/flux2/action`, pinned to the v2.8.6 commit with `version:
"2.8.6"`. The action downloads the release tarball, checks it against
that release's checksums file, and adds the binary to `PATH`.

Two workflows still took yq from the floating latest release, nightly
and the fork e2e lane. They now use the same install the promotion steps
already have: yq v4.53.3 checked against a committed SHA-256. I verified
that hash against the release's `checksums` file and against the binary
itself.

The helm-unittest plugin was installed at whatever version was newest.
The plugin embeds its own Helm, so a plugin release could change chart
test results on a PR that touched no chart. It's pinned to v1.1.2 now,
the version CI already runs. A runner that carries a different version
gets it replaced.

A new contract test scans every workflow for these three tools, because
an unpinned install usually comes back by copying an older step. On main
it fails on the install script, the missing setup action, the
latest-release yq and the unpinned plugin. I ran one mutation per guard,
and each one failed only the test meant to catch it. The existing
contract tests that checked `FLUX_VERSION` inside the old steps now
check for the action step.

The e2e sandbox image still runs `install.sh` in its Dockerfile. That's
an image build, not a workflow step, so I left it for a separate change.

Closes #3619

Closes #3640

Closes #4510

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only workflow tool-install steps and `hack/*.bats`
contract tests. Nothing in the trigger map reaches them.

### Release note

```release-note
ci(workflows): install flux through the commit-pinned upstream setup action instead of running fluxcd.io/install.sh as root, pin yq with checksum verification in the nightly and fork e2e workflows, and pin the helm-unittest plugin version.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Reliability**
* CI workflows now use a consistent, pinned Flux version. The yq and
Helm plugin setup also verifies or enforces specific versions.
* **Tests**
* Added checks to ensure workflow tool versions are pinned and
consistent, and that downloaded yq releases are checksum-verified.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:02 +03:00
IvanHunters 40638429bd fix(release): collect refs stored as registry/image/tag/digest keys (#4697)
## What this PR does

The shared image-ref enumeration skipped refs stored the way upstream
ingress-nginx stores them: `registry`, `image`, `tag` and `digest` as
separate keys, with the repository in `image` and a digest that has no
`@`. Two refs in the tree use this layout, the ingress-nginx controller
and its default backend, both in
`packages/system/ingress-nginx/values.yaml`. Every tool that reads refs
through `hack/lib/image-refs.sh` missed both of them: the promote retag,
the nightly mirror, the multi-arch gate, the stitch step and the
promoted-packages verifier.

I added a sixth YAML shape that joins `<registry>/<image>@<digest>`. It
fires only on a complete `sha256:` digest, so upstream's unpinned
default (`digest: ""`) and image maps without a digest emit nothing. It
checks that `digest` is a string before `test()`, same as shape 3 does.
The library header and `docs/agents/image-refs.md` now say "the YAML
sub-shapes listed" instead of a number, so the next shape doesn't make
the count wrong again.

The real tree now gives 93 refs instead of 91, and the two new ones are
exactly the ingress-nginx pair. The retag and mirror output does not
change today, because both refs sit outside the owned registry and the
ownership filters drop them. Both digests are multi-arch indexes, so
`hack/verify-multiarch.sh --refs-file` with just those two passes and
the release gate stays green.

To show the consumers really pick the shape up, I ran both dry-runs on a
copy of the tree with the controller moved under the build registry, the
host being the whole `registry` value. In that copy the controller is
the only build-registry ref.

Before:

```text
$ REGISTRY=iad.ocir.io/idyksih5sir9/cozystack hack/promote-retag.sh 1.99.0 --dry-run
No cozystack-owned digest-pinned image refs found under iad.ocir.io/idyksih5sir9/cozystack/ — is this the rc's baked tree?
exit 1
$ hack/nightly-mirror.sh 1.99.0 <tree>/packages --dry-run
No cozystack-owned digest-pinned image refs found under iad.ocir.io/idyksih5sir9/cozystack/ in '<tree>/packages' — is this the baked main tree?
exit 1
```

After:

```text
$ REGISTRY=iad.ocir.io/idyksih5sir9/cozystack hack/promote-retag.sh 1.99.0 --dry-run
▸ iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller  sha256:a8ddcee59b63f6cbf93f4d3c9ffa1d1dda3c1d39497748d4484655356c68b3f6
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller@sha256:a8ddcee5… docker://iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller:1.99.0
exit 0
$ hack/nightly-mirror.sh 1.99.0 <tree>/packages --dry-run
▸ iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller  ->  ghcr.io/cozystack/cozystack/ingress-nginx-controller  sha256:a8ddcee5…
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/…/ingress-nginx-controller@sha256:a8ddcee5… docker://ghcr.io/cozystack/cozystack/ingress-nginx-controller:1.99.0
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/…/ingress-nginx-controller@sha256:a8ddcee5… docker://ghcr.io/cozystack/cozystack/ingress-nginx-controller:nightly
exit 0
```

The red test is a new case in `collect_refs_from_file understands every
YAML shape`. It has a fixture in the ingress-nginx layout, with a
non-string digest in front of it, and three near-misses that must emit
nothing: no digest, an empty digest, and a digest without `image`. On
main the positive case fails with `got: <nothing>`. Removing the rule,
the registry join, the full-digest check, or the type guard on `image`
or on `digest` turns that test red, and only that one. The bats suites
of all library consumers pass with both the old and the new library.

Two gaps remain in the tests and I left them out on purpose. No fixture
covers the new rule without a `registry` key, and none pins its type
guard on `registry`. Shapes 2 and 3 have the same unpinned `registry`
guard on main.

The overlay script has no such gap. `hack/overlay-main-images.sh` does
not use the library, and its line classifier already accepts `image:`,
`registry:`, `tag:` and `digest:` lines.

Closes #4518

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff. None of the changed files are
in it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(release): the image-ref enumeration used by promotion, the nightly mirror and the multi-arch gate now reads refs stored as separate registry/image/tag/digest keys, the layout of the ingress-nginx controller and default backend.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Image references can now be read from configurations that store the
image name and SHA-256 digest separately, with an optional registry.
* **Bug Fixes**
* Incomplete or invalid image and digest values are ignored rather than
producing malformed references.
* **Documentation**
  * Documented the additional image storage format.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:10:58 +03:00
IvanHunters 9196b9798e fix(e2e): bound the dataplane capture collector by wall clock (#4640)
## What this PR does

The e2e data-plane collector limited itself by counting pods and
LoadBalancers, but both of its callers stop it by the clock.
`hack/cozytest.sh` gives it 600s and the Chainsaw catch gives it 300s.
One affected pod could already run past the smaller limit, and the
LoadBalancer capture, which runs last, was the part that got lost. I
moved the collector to a wall-clock budget and fixed the other issues
filed against it.

- Every kubectl call now goes through `dp_run`. It cuts the call's
timeout to what is left of `COZY_DATAPLANE_BUDGET` and starts nothing
once the budget is spent. The default is 270s, and `cozytest.sh` passes
570s. The pod section stops 90s early, so the LoadBalancer section
always gets time to start. A call the deadline cut says so, a call cut
by its own bound names that bound, and a refused call says it was not
started and names its section; exec blocks carry that line inside the
block. The node capture and the LoadBalancer probe used to blame an
unanswered lookup or a missing cni-server when the budget stopped a
call. Now they name the budget (#3671).
- A static test fails on any kubectl call that bypasses `dp_run` and
lists the lines. It matches by command position, so a call inside a
double-quoted `$(...)` is still checked (#3694).
- Per-pod and per-LoadBalancer capture files join namespace and name
with `_`, which neither can contain. A second run into the same
directory appends, with the run separator `capture-notes.txt` already
uses (#3679).
- `capture_node` and `capture_lb_node` no longer write their callers'
loop variables. A static test fails when a function assigns a name that
a `read -r` loop or another scope owns (#3675).
- The tests that shrank every timeout to one second now cut only the
read under test. A `timeout` shim decides by argv, so a slow machine
can't cut a read the test needs answered (#3685).
- `cilium-dbg bpf lb list` runs unfiltered in every affected node's
capture, not only for a failing LoadBalancer (#3689).

The call-site comment in `.chainsaw.yaml` and the collector paragraph in
`docs/agents/e2e-testing.md` now match the code. Every behaviour change
has a test in `hack/capture-dataplane.bats` that failed on the code
before it.

Two parts of the issues live elsewhere. The runner stopping a bats file
at its first failing test is #4620, and a conntrack read filtered by
service VIP is #4623.

From #4629 I fixed the items on lines this PR already changes: the node
capture now gets the run separator, a failed speaker log read writes a
note, the endpoint tcpdump says when it falls back to the geneve
interface, and the per-read timeout knobs are validated (a zero there
would otherwise look like a spent budget). The rest of #4629 stays open:
`ovn-lflows.txt` and `lb-speaker-logs.txt` are still replaced on each
run, and two comments still describe older behaviour.

- Closes #3694
- Closes #3671
- Closes #3679
- Closes #3675
- Closes #3685
- Closes #3689
- Refs #4629

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): bound the failure-path data-plane collector by a wall-clock budget, keep its per-pod and per-LoadBalancer captures from overwriting each other, and capture the cilium service map for ClusterIP failures.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Data-plane diagnostics now follow a configurable time budget,
reserving time for LoadBalancer captures and stopping operations when
the budget is spent.
* Capture reports distinguish failed, successful, unknown, and
budget-limited checks. LoadBalancers with successful probes are skipped
for heavy capture.
* Repeated captures are preserved together, and limits help control
LoadBalancer diagnostic collection.
* Diagnostics include additional network details from relevant nodes
during probe replay.
* **Documentation**
* Updated guidance covers capture time limits and how unanswered
EndpointSlice reads are reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:10:25 +03:00
IvanHunters f209c477e2 test(e2e): read the opensearch release history through the pinned name (#4689)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The opensearch suite now reads HelmRelease history through the shared
jsonpath, and a unit test fails if any Chainsaw suite goes back to a
copied literal. Before this, the suite carried its own copy of the
expression, and the upstream contract test in `internal/fluxcontract`
only checks `HELMRELEASE_HISTORY_JSONPATH`. I checked the copy against
helm-controller `api` v1.5.1, the version go.mod pins: the history field
is `history` and each snapshot has `status`. So the read works today, it
was just not pinned.

The suite passes `-o "jsonpath=${HELMRELEASE_HISTORY_JSONPATH}"` now,
which gives kubectl exactly the same argument as before. The new test
reads every suite under `hack/e2e-chainsaw/`. A suite that calls
`helmrelease_has_remediation_cycle` must fill the local it hands the
guard from the shared name, and no suite may assign that name itself. It
reuses the matchers that already pin `run-kubernetes.sh`. The test is
red on main and green with the fix.

One known limit. The second-assignment check counts over the whole file,
which fits `run-kubernetes.sh` because it runs as one shell. A Chainsaw
suite runs each `script` step in its own shell, so an unrelated
`statuses=` in another opensearch step would turn the test red with
nothing broken. That fails toward red, never toward a silent pass, so I
left it.

Closes #4550

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(e2e): the opensearch suite reads HelmRelease history through the shared jsonpath, which is now pinned against the upstream Flux type for every Chainsaw suite that runs the remediation guard.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Expanded validation across Chainsaw end-to-end suites to check
consistent HelmRelease history handling in remediation checks, including
the order and isolation of script steps.
* The OpenSearch suite now uses the shared history lookup. Empty-history
checks and remediation-cycle failure handling are unchanged, and there
is no change to end-user behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:08:51 +03:00
IvanHunters b4fbaf00a2 test(e2e): keep gateway and redis-tls op ceilings above their own waits (#4643)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Two Chainsaw script ops had a timeout smaller than the waits inside
them. When the op ceiling fires first, Chainsaw kills the shell: the
script never prints its own error, and a cleanup trap never runs.

The gateway listener-set step could poll for 660s under a 6m op. Its
cleanup trap also deleted two tenants with no limit on the wait. Each
delete now runs under an outer `timeout`, because kubectl's own
`--timeout` does not cover discovery or a stalled request. A tenant
delete gets 300s, the same 5m that `.chainsaw.yaml` gives a delete op.
The parent is deleted only after the child is gone, because deleting a
parent while its child is still uninstalling wedges the parent's
cleanup. A failed tenant delete fails the step with kubectl's own error,
so a leaked tenant shows up here and not in a later suite. The op is now
24m, which covers the polls plus those bounds.

The redis-tls CA-projection step has 30s read caps that add up to well
over its 5m op. Raising the op would not hold, because several reads run
in loops. So all reads now share a 240s step budget. No read starts
after the budget runs out, and a read that is still running gets cut off
when it does. A read that hits its bound, or finds the budget already
spent, says which read it was.

Two exec checks in the same step blamed the 15s redis-cli cap for any
failure, including a refused exec or a missing container. They now print
the exit status and what the exec printed.

The step also runs under bash now, for pipefail. The sandbox's `/bin/sh`
is dash, which rejects `set -o pipefail`. Chainsaw 0.2.15, the version
the sandbox image installs, supports a `shell:` field on script ops.
Without pipefail, a failed Secret read took base64's zero exit status
and was reported as a wrong source name.

The private-key check read the projection a second time, so a failed
second read made it pass. It now checks the value the step already
fetched.

I added `hack/chainsaw-op-budgets.bats` to guard all of this. It holds
the gateway op above its polls plus its delete bounds, keeps the
child-before-parent order, holds the redis budget under its op with
every kubectl call going through the clamped helper, and runs the redis
step's script against a stubbed kubectl to check each failure message
and the clamp. On main, 11 of its 12 tests fail. The one that passes is
the healthy-stub control.

Closes #4114

Closes #4250

Closes #4246

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches two Chainsaw suites and adds one unit test file under
`hack/`. It moves or renames nothing a downstream repository reads.

### Release note

```release-note
test(e2e): the gateway listener-set and redis-tls projection steps now fail on their own errors instead of being killed by the op timeout, and redis-tls reports the real cause of a failed read or exec.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added automated checks to verify gateway operations have sufficient
time to complete and cleanup deletions are bounded and correctly
ordered.
* Expanded gateway scenarios to cover cleanup failures: if child-tenant
deletion fails or times out, the parent is left in place; deletion
failures are reported.
* Updated Redis TLS checks with bounded read times and a shared
execution limit. Failure reports include command output and status, and
checks cover private-key material in the decoded projection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:07:37 +03:00
IvanHunters 1ca603a7b2 test(e2e): wait for the S3 gateway to accept a key before using it (#4694)
## What this PR does

The bucket e2e suite fails now and then because the S3 gateway rejects a
key that COSI already reports as granted. The error is `The access key
ID you provided does not exist in our records.` from `mc alias set`. In
job 109130131391 the BucketAccess assert finished at 21:23:43.57 and the
first `mc alias set` failed at 21:23:44.82. In job 109396174094 cap1 and
cap2 were accepted, and cap3 was rejected two seconds into the step.

Granted access is not the same as a usable key. In seaweedfs-cosi-driver
v0.3.1, `DriverGrantBucketAccess` writes the new key into
`/etc/iam/identity.json` on the filer and returns. The S3 gateway in
SeaweedFS 4.45 reloads that file later, from a filer metadata
subscription (`weed/s3api/auth_credentials_subscribe.go`). In our chart
the driver writes through `seaweedfs-filer` and the two gateways follow
`seaweedfs-filer-client`, so the change may have to pass through the
other filer replica first.

The suite now waits until the gateway accepts the key before it writes.
Every alias in both write steps goes through `await_key`, which uses the
same port-forward as the writes and so asks the same gateway pod. It
waits only while the answer is the InvalidAccessKeyId text, for at most
60s. Any other error fails at once, and so does an attempt cut off at
its 15s cap. When the window runs out, the step fails with a message
that names the BucketAccess.

One probe costs at most 77s, so both steps go from a 5m to an 8m op
timeout. The inner budgets now sum to 409s and 441s, and the budget
comments say so.

I checked the probe locally against a fake `mc`:

```text
case=ok rc=0 attempts=1 elapsed=0s
  Added `a` successfully.
case=lag rc=0 attempts=3 elapsed=5s
  Added `a` successfully.
case=other rc=1 attempts=1 elapsed=0s
  FAIL: mc alias set for BucketAccess bucket-test-admin failed (exit 1) with an error other than the key not being loaded yet
case=never rc=1 attempts=31 elapsed=60s
  FAIL: S3 gateway still rejects the key of BucketAccess bucket-test-admin after 60s of probing (the gateway never loaded it, or the driver never wrote it)
```

A hanging `mc` fails on its first attempt after 15s with exit 124.
`chainsaw lint test` from v0.2.15 accepts the file.

The read-only upload check gets fixed along the way. Before, an unloaded
`ro-user` key also made the upload fail, and that failure passed as the
expected denial.

One thing I left as it is: `await_key` is copied into both steps,
because chainsaw steps share no shell, and moving it to `_lib/` would
run every suite on each edit.

Closes #4580

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): the bucket suite waits for the S3 gateway to accept a new key before writing with it, instead of trusting BucketAccess readiness.
```
2026-10-05 14:07:16 +03:00
IvanHunters 858ca4d0a3 test(e2e): cover monitoring OIDC modes with a Chainsaw suite (#4678)
## What this PR does

Monitoring OIDC gets live-cluster e2e coverage again. It had none since
the two pre-Chainsaw BATS suites were removed without a port. This adds
a Chainsaw suite in `hack/e2e-chainsaw/monitoring/` that ports what they
asserted, plus the in-place switch from an inline config to a
`secretRef` that nothing covered.

The suite takes one Monitoring release in `tenant-test` through three
modes in a row: System, CustomConfig with an inline config, and
CustomConfig with a `secretRef`. It creates that release itself and
never reads the tenant-root monitoring instance, so it keeps working if
#4023 or #4479 turns root monitoring off.

What each phase checks:

- System mode: cozystack-api admits `spec.oidc` and both HelmReleases go
Ready. The Grafana CR points `auth.generic_oauth` at the cozy realm, has
no `role_attribute_path`, and has `skip_org_role_sync` and
`oauth_allow_insecure_email_lookup` forced. The KeycloakClient is
confidential and references the generated Secret, which holds 32
alphanumeric characters. The audience scope is bound to this release's
clientId. There is no chart-owned `KeycloakRealmGroup`. The users-Job
carries the users and the mode, and `grafana-deployment` reads the
client secret from the Secret.
- Switch to inline: the Grafana CR takes the tenant's map, keeps the
forced keys and drops every System-only key. The users-Job reruns with
the new list. The KeycloakClient, the audience scope and the client
Secret are pruned, and the Deployment stops referencing that Secret.
- Switch to `secretRef`: the inline `auth.generic_oauth` block is gone
from the Grafana CR. The operator Secret is mounted at
`/etc/grafana/oidc` on both the CR and the Deployment. This checks the
mount only and does not prove Grafana loads the file, see #4679.
- After the install and after each switch, neither release may show a
remediation cycle. That means no `failed` or `uninstalled` snapshot, and
the latest revision must be exactly 1, then 2, then 3. The revision
check is there because helm-controller truncates history after the
previous deployed snapshot. An upgrade that failed, rolled back and then
passed on retry leaves no failed entry, only extra revisions.

I did not assert `GF_PATHS_CUSTOM_INI`. I could not find anything in
Grafana that reads it (#4679), and pinning it would make possibly dead
wiring part of the contract.

The coverage split with the render tests is this.
`packages/system/monitoring/tests/oidc_test.yaml` keeps the
fresh-install render of each mode, including the CustomConfig negatives
(no Keycloak objects, no client Secret). This suite covers what a render
cannot see: admission, the Deployment the Grafana operator builds, and
what a switch between modes prunes or leaves behind. The Keycloak
objects are checked as written, so creating the client in Keycloak is
not asserted. Removing it is, because the operator's finalizer holds the
KeycloakClient until the client is gone from Keycloak.

Before the first apply the suite also waits until the tenant's
`cozystack-values` carries `oidc-enabled: "true"`. The outer release
copies `_cluster` into the inner release once, so a Monitoring applied
right after the tenant is created can render from stale values and fail
the System-mode guard for the full 15m. That is what the first CI run on
this PR hit.

The Keycloak CRDs are required, not probed for. The e2e install always
enables platform OIDC, so a missing CRD is a broken install. The removed
suites skipped in that case, and their probe could not tell an absent
API group from a present one.

The selector picks the suite up by name from
`cozystack.monitoring-application` and `cozystack.monitoring`. A change
to `system/monitoring`, `extra/monitoring`, `monitoring-rd` or
`grafana-operator` now runs only this suite instead of the full run. The
tenant-root Grafana bring-up stays covered by the install BATS while it
exists. One selector unit test relied on `cozystack.monitoring` reaching
no suite, so it now renames that source in its copy of the graph.

The second commit keeps Keycloak changes on every suite that drives a
Keycloak client. Before this PR a change to `system/keycloak` or
`system/keycloak-operator` escalated to the full run. With the
monitoring suite it would select monitoring alone, because
`monitoring-application` is the only source that depends on
`keycloak-operator`, and `kubernetes-latest` with its OIDC lifecycle
would drop out. `src_to_suites()` now maps both to `kubernetes-latest
monitoring`. The mapping has to be explicit because
`kubernetes-application` has no `dependsOn` on `keycloak-operator`
(#4680). A `postgres-operator` change now also reaches
`kubernetes-latest`, through Keycloak's database. The selector comment
and the e2e doc no longer use `system/postgres-operator` as the example
of a path with a source that reaches no suite.

Cost: one monitoring stack bring-up, two in-place upgrades and teardown.
I expect 10 to 20 minutes on every full run. The selector change makes
the CI run on this PR a full run, so the suite's own timing is in the
chainsaw report. A red run can spend up to 16m on each of four waits,
since each one sits a minute above the 15m timeout of the release it
waits on.

The suite cannot run locally, so I checked it this way:

- `chainsaw lint test` at v0.2.15, the sandbox version, passes.
- I ran the suite's own steps against a local envtest apiserver.
Chainsaw applies the real manifests, and a stand-in for helm and the
operators renders the chart from the stored Monitoring spec. With the
chart intact all nine steps pass. Each of 30 single breakages of the
chart or of the simulated controllers turns the expected step red.
- `hack/select-e2e_test.bats` and `hack/select-install_test.bats` pass.
The new Keycloak selection case fails without the mapping and when
`kubernetes-latest` is dropped from it.

A browser login through Keycloak is out of scope, same as in the removed
suites.

Left for later:

- KeycloakClient and scope `status` after the operator creates them in
Keycloak is not checked.
- `hack/e2e-chainsaw/README.md` lists suites by hand. I added
`monitoring`, but `backup`, `kafka-metadata` and `kubernetes-proxmox`
were already missing.
- The `operator diff selects all dependent app suites` selector test
could also check `monitoring`.
- The comment on the Keycloak mapping says both suites wait for the
finalizer to remove the client. The monitoring suite does wait.
`kubernetes-latest` only reads once after its upgrade.
- The ready checks compare `observedGeneration` with
`metadata.generation` through `contains()`. Both are fields of the same
object, not a number literal, so the v0.2.15 literal gotcha should not
apply, but no live run has passed this suite yet.
- The inline fixture carries no `enabled: "true"`, so that phase checks
a disabled OAuth block. The asserts do not depend on it.

Closes #4051

Closes #3785

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added end-to-end coverage for Monitoring installations using system
OIDC, inline custom configuration, and Secret-based configuration. The
tests verify readiness, rendered resources, configuration cleanup, and
upgrade history.
* Updated test selection so Keycloak-related changes also run the
Monitoring suite.
* **Documentation**
* Added Monitoring to the documented Chainsaw suite list and clarified
how changed paths determine test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:07:10 +03:00
IvanHunters 6e4c9c4b6a fix(ci): harden the PR labeler's title parsing and map recurring scopes (#4666)
## What this PR does

Fixes three problems in `.github/workflows/pr-labeler.yaml`. Each one
either killed the job or put a PR in the wrong area, and none of them
showed up anywhere.

- Both maps are now looked up with `Object.hasOwn`. A title typed or
scoped `constructor`, `toString`, `hasOwnProperty` or `__proto__` used
to resolve to an `Object.prototype` member and crash the step on
`l.startsWith`, which left the PR with no labels at all.
- Every leading `[Backport ...]` prefix is now stripped, not only the
first one. With two prefixes, the second was read as a bracket scope, so
the real scope and the `kind/*` label were both lost. The `area/release`
label hid the problem by keeping `area/uncategorized` from firing.
- Scopes that recur three or more times over the full PR history now map
to an area. So does each area's own name (`networking`, `storage`,
`database`, `virtualization`, `testing`, `dependencies`).

Scope-to-area decisions:

- `area/database`: `database`, `etcd-operator`, `opensearch`,
`opensearch-operator`
- `area/networking`: `networking`, `kubeovn`, `multus`, `kilo`,
`hetzner-robotlb`, `ouroboros` (it rewrites the CoreDNS Corefile to fix
hairpin NAT, so it is networking, not platform)
- `area/virtualization`: `virtualization`, `vm`, `kubevirt-operator`,
`kubevirt-cdi`, `kubevirt-instancetypes`
- `area/platform`: `operator`, `cozystack-operator`, `controller`,
`cozystack-controller`, `cozystack-basics`, `flux-shard-operator`,
`migrations`, `scheduler`, `cozystack-scheduler`, `reloader`, `bootbox`,
and `lineage`. I put `lineage` here on purpose: no existing label covers
it more precisely, and it is too narrow for an area of its own.
- `area/storage`: `storage`, `piraeus-operator`,
`backupstrategy-controller`
- `area/release`: `changelog`, `changelogs`
- `area/ci`: `workflows`, `labeler`
- `area/dependencies`: `dependencies`, `renovate`
- `area/api`: `registry` (the cozystack-api storage layer)
- `area/dashboard`: `dashboard-controller`
- `area/monitoring`: `monitoring-agents`
- `area/keycloak`: `keycloak-configure`
- `area/testing`: `testing`
- New `area/dx`, added to `.github/labels.yml`: `dx`, `hack`, `cozypkg`.
These are developer tooling (hack/ scripts, package Makefiles, cozypkg).
`area/build` covers image builds and `area/testing` covers tests, and
neither describes this work.

Left unmapped on purpose:

- `http-cache`: the application was removed from the tree.
- `openbao`: already mapped on main.
- `apps`, `system`, `docs`: catch-all scopes that don't point to one
area.
- `fix`, `bugfix`: malformed titles.
- `security`, `rbac`, `governance`, `codeowners`, `maintenance`,
`images`, `readme`, `telemetry`, `cluster-autoscaler`: no single area
fits any of them.

I did not add spelling normalisation. `kubeovn` got its own row, since
it is the only alternate spelling that recurs.

A new contract test, `hack/pr-labeler-contract.bats`, pulls the
workflow's own `script:` block out of the file and runs it under node
with stubbed `context`, `core` and `github`, so it tests the shipped
code, not a copy. On main it is red on the prototype titles (`THROW:
l.startsWith is not a function`), on the double backport (only
`area/release`), on the newly mapped scopes and on the area-name scopes.
It also checks that every map target exists in `labels.yml` and that no
key is written twice. The suite fails rather than skips when node is
missing. The older node-based suites skip, so this is the first one that
needs node on the unit-test runner.

Some things stay out of this PR. The scope table in
`docs/agents/contributing.md` is a summary. It has no row for `area/dx`
and does not mention that an area's own name works as a scope.
Separately, the singular `migration` still maps to `area/virtualization`
as it did before, while the new `migrations` maps to `area/platform`.

A backport PR whose scope is unmapped still never gets
`area/uncategorized`, even with a single prefix. The `area/release`
added by the strip satisfies the fallback check. This behaviour is the
same on main and is not changed here.

The test that checks map targets against `labels.yml` passes by itself
if the map literal cannot be extracted. The duplicate-key and area-name
tests go red in that case, so the suite as a whole still fails.

Closes #4125, closes #4124, closes #4122

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): the PR labeler no longer crashes on titles scoped like Object.prototype members, handles nested backport prefixes, and maps more recurring scopes, including a new area/dx.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `area/dx` label for issues and pull requests related to
developer tooling.
* **Improvements**
* Pull requests can now be categorized using a broader set of scope
aliases.
* Repeated leading backport prefixes are recognized and route the pull
request to the release area.
* Label mapping now avoids treating inherited property names as valid
mappings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:06:47 +03:00
IvanHunters 02fbe76418 fix(ci): close three blind spots in the e2e test selectors (#4658)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The e2e selectors miss Chainsaw suites that have no `chainsaw-test.yaml`
or `.yml` directly in their directory. Chainsaw v0.2.15 also runs a
directory that holds only numbered step files such as `01-install.yaml`,
and its discovery walks every subdirectory of the path the runner hands
it. A suite in either layout ran only when nothing was selected. No
narrowed selection named it, the full-suite list left it out, and
`select-install.sh --validate` never asked for its mapping. Now
`hack/select-e2e.sh` and `hack/select-install.sh` list a top-level
directory as a suite when Chainsaw would load a test from it or from
anywhere below it. No suite in the current tree uses either layout, so
today's suite list stays the same.

A file placed directly in `examples/backups/` (a shared `common.sh`, for
example) selected nothing, because it fell through to the inert
`examples/` rule. It now escalates to the full suite, same as
`examples/backups/_lib/`. `README.md` there stays inert through the
`*.md` rule.

The `test-chainsaw` comment in `packages/core/testing/Makefile` said
Chainsaw discovers every `chainsaw-test.yaml`, which was too narrow. I
fixed it, and the matching sentences in `docs/agents/e2e-testing.md`
too.

Every behaviour change has a bats test that fails against the scripts on
main.

Known gaps I left for a follow-up, none of them a wrong answer today:

- The `full_suite_list` helper in `hack/select-e2e_test.bats` repeats
the selector's `find`/`sed`, so a bug in that shared pattern would pass
every full-suite assertion. The step-file and nested tests compare
against literal lists, which covers the new layouts.
- The selectors are a little wider than Chainsaw: a broken
`chainsaw-test.y*ml` symlink, or one with no Test in it, still lists its
suite. That errs toward running more.
- A few test comments describe what used to happen rather than why the
rule exists.

Closes #4630

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ci): the e2e test selectors now see Chainsaw suites made of numbered step files or nested below a suite directory, and run the full suite for an edit to a file placed directly in examples/backups/.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* E2E suite discovery recognizes supported manifest and numbered-step
files at any depth and maps them to the top-level suite.
* Changes to files directly under `examples/backups/` trigger full-suite
selection.
* Suite selection reports an error if discovery fails or finds no
suites.
* **Documentation**
* Clarified supported suite layouts, disabled suite-file handling,
discovery and escalation rules, and per-path coverage guidance.
* **Tests**
* Added coverage for nested suites, numbered steps, unmapped suite
layouts, and backup-path selection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:06:18 +03:00
IvanHunters 9485f9118b test(tests): make negated BATS assertions able to fail (#3790)
## What this PR does

A hundred and twelve assertions across twelve BATS suites begin with
`!`. A hundred and seven of them pass no matter what they find. POSIX
exempts a command whose return value is inverted with `!` from `set -e`,
so the failing status is dropped where it is produced, and
`hack/cozytest.sh` closes each test function with `return 0`, so it
never resurfaces as the test's own status either. The other five are
followed by `|| { ... exit 1; }`, which reads the inverted status and
does work; they are rewritten too, because telling the two apart needs
the tail of every statement parsed, and a guard that guesses wrong there
is green on a real one.

Switching to the `bats` binary does not fix the hundred and seven. Its
ERR trap carries the identical exemption: `bash -c 'set -eE; trap "echo
TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap and
exits zero. The only negated assertion `bats` reports is one standing
last in the test body, where the inverted status happens to be what the
function returns. Two of the hundred and seven stood last; the rest did
not.

What was lost is specific rather than diffuse. `!` is how you spell
"this must not appear", and an absence is what no other assertion in a
test covers. The affected suites assert that a migration ran nothing
destructive, that no image was copied, that a flag was not passed, that
a backup cleanup never flushed the master, that an object does not
exist. On that half of their contract they held for any input.

Each site is now written as `if cmd; then echo "FAIL: ..."; false; fi`,
the form already used in the files where this had been noticed one at a
time, and each carries a message naming what must not have happened.
`hack/bats-no-negated-assert.bats` scans every `hack/**/*.bats` and
refuses the old form, so a new one cannot be added silently. Inverted
conditions (`if !`, `while !`, `until !`) are untouched, because `set
-e` is not what decides a condition. `[ ! -f x ]` is untouched for a
different reason and remains an ordinary failing assertion: the `!`
there is an operator of `test`, not negation of a command, so nothing
exempts its status.

The guard pins its own premise instead of describing it, on both sides.
One test asserts the runner still passes a bare negated assertion; a
second, that neither `set -e` nor the ERR trap sees an inverted status;
a third, that the prescribed `if cmd; then ...; false; fi` does fail,
under both runners, and *at* the form rather than somewhere before it,
since a run that died early would satisfy a bare status check and pin
nothing.

Every rewrite here rests on the third test. If `false` inside a `then`
body ever stopped propagating, the first test would still pass, because
a negated assertion would still be vacuous, while every converted
absence went silently green. A ban that prescribes a replacement has to
test the replacement too.

Every claim the guard makes is pinned by removing the thing it claims.
The recursion is exercised *through* the audit function against a
fixture in a subdirectory, not through a second copy of its `find`
expression: a copy keeps passing while the function loses its recursion,
and an audit that returns empty reads as a clean tree. Recursion matters
because `BATS_UNIT_FILES` is a non-recursive `wildcard`, so `make
unit-tests` never runs a suite placed in a subdirectory.
`hack/e2e-apps/` held such suites until they moved to Chainsaw and is
gone now, which is why the test pins against a fixture and not a real
directory.

The rule's surface was checked in the other direction too. The shape
appears in no `hack/**/*.sh` script as an assertion: the one hit,
`hack/testdata/install-wait/call-shapes.sh`, is fixture text that
`hack/run-kubernetes-install-wait_test.bats` parses, not a command
anything runs. It appears in no Chainsaw `script:` step either, so
`.bats` is the whole surface rather than the part that was convenient to
guard.

The scan is lexical, so its header names the statement boundaries it
owns rather than implying it understands shell: `;` `&` `&&` `||` `{`
`(`, the `)` closing a case pattern, and the words `then` `do` `else`,
each measured against the runner before being added. A keyword may
legally be preceded only by whitespace or by `;`, since every other
separator in front of `then`, `do` or `else` is a shell syntax error,
checked with `sh -n` rather than recalled. The scan therefore admits
exactly those two, and that part of the class is closed rather than
patched one spelling at a time. All three keywords are vacuous after
either spelling, so missing one is a silent miss, the single direction
this scan must not err in, given that its whole design trades noise for
safety. For the same reason it takes `!(cmd)` with no space before the
subshell, which is vacuous on dash and bash alike, and reads through a
`time` or `time -p` prefix, which is vacuous under `bats` and under
cozytest wherever `/bin/sh` is bash.

Five over-counts remain and are deliberate, each named in the header: `(
! cmd )` does propagate its inverted status and is reported anyway; so
would a `!` opening a statement after a redirection `&`; `x=$(true || !
false)` is reported although the `||` reads the inverted status; a
condition continued by a trailing `&&` instead of a backslash is
reported, because the strip only recognises a condition whose end it can
see; and `time ! cmd` is reported although on a dash `/bin/sh`, which
has no `time` keyword, it runs an external `time` that fails. Each costs
one rewritten line, where the opposite error costs a silent pass. They
are enumerated rather than engineered away on purpose, because a removed
error leaves no trace while a listed one tells the next reader what they
are looking at. The scan also errs *quiet* in at least two places, named
here rather than in the header. A file cut off midway through a line
continuation loses its last buffered statement, but that input is not a
runnable test, since `sh -n` rejects it with `unexpected end of file`.
And `time -- ! cmd`, which bash accepts, is vacuous under `bats` and not
peeled by the scan; nothing in the tree uses it, and I stopped adding
spellings there rather than chase every prefix bash allows. The test
naming the shapes claims only what its fixture holds. A lexical scan
over shell cannot promise it has enumerated the language, and a name
that promised it would be false whether or not a gap existed today.

The branch name is narrower than the subject and I am leaving it as it
is rather than churn it. This is not a defect in `hack/cozytest.sh` that
moving to the `bats` binary would fix. It is a POSIX rule both runners
inherit, and moving to `bats` would close two of the hundred and seven
sites.

One suite went red for real once its assertions could fail.
`seaweedfs-guard-parity` forbids four mutable discriminators by grepping
each chart, and both charts carry a template comment block arguing why
those signals were rejected, so the grep matched the argument and would
have reported a chart for documenting the rule it follows. The charts
are correct; the assertion was not. It now strips `{{/* ... */}}` before
matching. Two details there are load-bearing and neither is obvious. The
strip removes the comment *span*, not the line that opens it: dropping
the line is shorter and silently loses a discriminator sharing a line
with a comment, which is a chart classifying on a forbidden signal and a
test reporting that it does not. And the strip reports when the file
ended inside a comment, and the checks refuse to read its output when it
did, because every `{{/*` is read as an opener, so one inside a quoted
string (these charts carry long `fail (printf ...)` messages) would
swallow the rest of the file and leave the checks reading nothing.

One commit widens six comments that explained this trap as applying to a
"negated pipeline". It is not about pipelines, and the narrower wording
invites the conclusion that the other shapes are safe. Comment text
only; each of those files already uses the working form.

The last commit makes the one absence check in
`hack/e2e-install-cozystack.bats` that already used the `if` form tell a
missing object from a failed read. It asserted that the version
configmap was gone on `kubectl get`'s exit status alone, and that status
is 1 both for a NotFound and for an api-server the test cannot reach.

### Verification

Every unit suite this branch touches was run through `hack/cozytest.sh`
on the current base and is green. `migration-seaweedfs-db-adopt` passes
15 of 15 on macOS only when its temp directory sits under `$HOME`. Its
container cannot write into a bind mount under `/var/folders`, and
`/usr/bin/mktemp -d` ignores `TMPDIR`, so the run needs GNU `mktemp`
first on `PATH` as well. Measured on Linux before the rebase, it passed
15 of 15 as is.

The six sites that reached main after this branch was first written, in
`redis-backup-strategy` and `proxmox-e2e-fixture_test`, were each shown
to be able to fail: with the condition inverted, the suite goes red at
that assertion and prints its own `FAIL:` message. The nine sites in
`migration-seaweedfs-db-adopt` carry a two-sided check as well: with the
pattern swapped for one the same test proves is present, the rewritten
form fails the suite and the original `!` form passes it.

The guard itself is green under cozytest with bash and with dash as
`/bin/sh`, under the `bats` binary, and with mawk, gawk and BSD awk.
Each branch of its separator class and of the `time` and `!(cmd)`
handling was removed in turn, and each removal turns the shapes test
red.

`e2e-install-cozystack.bats` is an e2e suite and runs only in the E2E
workflow, so its one changed absence check is not exercised locally.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff file by file. The change touches
eighteen existing BATS files, adds one, and adds a convention to
`docs/agents/e2e-testing.md`. It does not touch
`hack/e2e-prepare-cluster.bats`, which is the node contract
`ansible-cozystack` and `talm` restate; it moves and renames nothing
under `hack/` and changes no make target's behaviour, which is what
`ccp` gates on; and it touches no package, schema, enum, default, CRD,
label, annotation or metric.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```

Closes #3565
2026-10-05 14:05:42 +03:00
IvanHunters 94abf6940b feat(kubernetes): let kubevirt-csi detach without the HotplugVolumes gate (#4451)
## What this PR does

This lets kubevirt-csi detach volumes without the deprecated KubeVirt
`HotplugVolumes` gate. The default stays as it is, so nothing changes at
runtime yet. Part of #4032.

The driver got a switch upstream for exactly this, and I bumped to it.
kubevirt/csi-driver#382 added `--enable-vmi-hotplug-fallback`, merged as
cb4770e4e475. With it set to false, the driver no longer calls the
VMI-level `removevolume` on a VMI owned by a VM. virt-api rejects that
call once `HotplugVolumes` is off.

Our wrapper now passes the flag to upstream and no longer has its own
hot-unplug path. `unpublishHotplugVolume` was a backport of
kubevirt/csi-driver#184, and the previous pin already contained that
change. Upstream detaches from the VMI when the VM is gone, then waits
for the VMI status to drop the volume before it returns. So the
wrapper's re-check never found the volume attached. Adding the ownership
check there would have been dead code.

Closes #4450

The chart leaves the flag at its default of true and does not render it.
The driver image in the chart is stamped only at release preparation, so
between releases main runs the previous release's binary, which does not
know the flag. Go's `flag` package exits on an unknown flag. Rendering
`=true` would crash-loop the CSI controller for no effect. The right
value depends on the KubeVirt CR, not on the tenant, so the comment next
to `HotplugVolumes` in `kubevirt-cr.yaml` points at the controller
Deployment. The follow-up PR drops the gate and renders the flag as
false.

Nothing changes on the node side. Main already pins f5de10d39556, and
cb4770e4e475 is that commit plus kubevirt/csi-driver#382.

A Go unit test runs an unpublish through the wrapper during a teardown,
with the flag on, off with a VM-owned VMI, and off with an orphaned VMI.
It checks that the flag reaches upstream and that only upstream makes
the VMI-level call. Upstream's own tests cover the ownership rule. A
second test holds that an NFS volume only releases the node from its
CiliumNetworkPolicy and never reaches the upstream hot-unplug. A
helm-unittest pins that the chart renders no such argument. I haven't
built the image or run e2e. Volume attach and detach on a tenant cluster
in e2e is the real check.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): kubevirt-csi supports --enable-vmi-hotplug-fallback, so it can detach volumes without the KubeVirt HotplugVolumes feature gate. The flag stays true for now and behaviour is unchanged.
```
2026-10-05 10:46:12 +03:00
Mattia Eleuteri cfa31d3a17 fix(postgres): record the live cluster's flavor on a backup
The chart refuses a flavor change on an existing Cluster, but the
Postgres app keeps the refused value. A BackupJob taken after such a
change recorded the app's flavor, not the one the data directory was
written by, so a RestoreJob would accept a target of the wrong image
family and refuse the right one.

The driver already reads the live Cluster to keep its barman
serverName; it now takes the flavor from that Cluster's imageName the
same way the chart's flavor guard compares images.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-05 08:52:32 +02:00
tym83 a3b802406c docs(agents): drop the note on hardcoded sidebar categories
The sidebars now follow every category but Administration, so the
note describes behaviour that no longer exists.

Assisted-by: LLM
Signed-off-by: tym83 <6355522@gmail.com>
2026-10-04 14:21:08 +05:00
tym83 b9ca07d021 feat(dashboard): show categories brought by tapped repositories in the sidebars
The marketplace and console sidebars rendered a fixed IaaS / PaaS / NaaS
list and dropped every other spec.dashboard.category. A repository tapped
with cozypkg that brings its own category was reachable only through the
full application list, and its installed applications had no console
section. Follow groupByCategory instead, the order the marketplace page
already uses, and keep Administration in the Admin portal.

Assisted-by: LLM
Signed-off-by: tym83 <6355522@gmail.com>
2026-10-04 14:20:35 +05:00
Yan Bondarenko 15b7092f3a fix(nats): enable JetStream for the generated account
With users configured and jetstream.enabled true, the chart enables
JetStream on the server but generates account A without it, so every
JetStream call from those users fails with error 10039, "JetStream not
enabled for account". The generated account now gets jetstream: true
unless it already sets JetStream itself. nats-server matches config keys
case-insensitively and walks the account map in no fixed order, so a
JetStream key in any casing counts as set. Otherwise the account would
carry both keys and the server would apply either one, changing from
restart to restart.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-04 10:51:16 +03:00
Yan Bondarenko 6248d458ad fix(platform): keep the legacy etcd-operator stopped after etcd adoption
Migration 50 scales the legacy etcd-operator to 0, adopts every legacy
etcd.aenix.io cluster with etcd-migrate and then scales the operator
back to 1. At that point it is still the legacy operator, and it has
nothing left to manage. Until a tenant's etcd HelmRelease reconciles the
new chart, Flux can upgrade it with the old one, which re-creates the
legacy EtcdCluster that etcd-migrate just orphan-deleted. The legacy
operator then reconciles it, re-creates the legacy StatefulSet and
rewrites the legacy etcd-headless Service, dropping the owner references
etcd-migrate pointed at the adopted EtcdMembers. When the new chart
prunes the legacy EtcdCluster, the garbage collector deletes the Service
with it, and the adopted members' etcd-<i>.etcd-headless names stop
resolving.

After a successful adoption the operator now stays at 0 until the
etcd-operator HelmRelease replaces the Deployment with the v1alpha2
operator later in the same upgrade. A failed or interrupted adoption
still restores it through the EXIT trap.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-03 18:25:49 +03:00
Aleksei Sviridkin 2df578bffd test(e2e): read the opensearch release history through the pinned name
The opensearch suite wrote the history jsonpath out as a literal, so
the contract test that runs HELMRELEASE_HISTORY_JSONPATH against the
upstream Flux type never saw the expression this suite hands kubectl.
A field rename upstream would have reached it only as an empty read.

The suite now takes the shared expression, and the contract test reads
every Chainsaw suite that calls the guard, so a suite copying the
literal, or assigning the shared name after sourcing the library, fails
in the unit lane instead of going unchecked. The read is matched inside
the script step that calls the guard, because every step runs in a
shell of its own and a read in another step never reaches the call.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 11:39:34 +03:00
Aleksei Sviridkin b42805bb4f docs(e2e): describe the HelmRelease history guard for the addons
The conventions still described a guard on the parent release alone.
They now state which footprint fails a run and on which releases,
why a footprint on a RetryOnFailure release is fatal even after a
retry recovered it and which recovery leaves no footprint at all,
why the default-strategy addons only get a note, how the strategy
read is kept from failing soft, and why the failure paths carry the
same classification without a verdict.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 06:59:32 +03:00
Aleksei Sviridkin 8cb1023788 fix(e2e): fail on a failed release of cilium and csi as on the parent
The history guard covered only the parent Kubernetes HelmRelease. A
failed action on one of the addon releases it installs left no mark
the suite inspected, so a run surfaced only the downstream symptom,
typically the node-join deadline, with nothing naming the release
behind it. The tenant CNI and CSI are the sharp cases, since they
carry a cluster-wide precondition.

The guard now reads every addon release of the parent, selected from
the cluster by name prefix so a release added later is covered too.
A "failed" or "uninstalled" Snapshot fails the run on a release whose
install strategy is RetryOnFailure: the parent, which the aggregated
apiserver stamps that way, and cilium and csi. The same footprint on
the other addons, which pair the default strategy with retries: -1,
is printed as a note and leaves the run green.

The footprint is not split into a teardown and a retried failure.
In helm-controller v1.5.0, the version the platform installs, and
unchanged in v1.5.1, a release with an active retry strategy never
remediates: atomic_release.go returns an upgrade for a failed
release before it reaches the remediation switch (lines 465-475), and
the two other paths that uninstall, a changed release target and
deletion, both call ClearHistory afterwards. On a RetryOnFailure
release an "uninstalled" Snapshot is therefore unreachable, and the
"failed" one is the only signal. A retry that later recovered still
fails the run, by design, with one blind spot the statuses cannot
close: a failed install that the very next attempt recovers is
marked superseded by Helm, because there is no deployed release to
supersede instead, and its history reads like an ordinary upgrade.

The guard runs after the assertions, while every deadline a failed
addon breaks ends the script earlier, so the same classification is
also printed without a verdict from the node-join collector and from
the exit handler. That handler is now armed at the tenant CR apply,
so the management-cluster waits before the tenant answers are
covered too. Every read is bounded and never folded into "no
history", and the parent must read back RetryOnFailure: an empty
value there means the jsonpath stopped resolving, and any other value
means the apiserver stamp is gone, either of which would turn its
footprint into a note.

The history read takes its jsonpath from HELMRELEASE_HISTORY_JSONPATH,
and the upstream-type pin accepts the bounded wrapper the read goes
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 06:59:31 +03:00
Aleksei Sviridkin 091b8549c2 build(images): run the tests of the packages the patches touch (#4326)
## What this PR does

Twelve first-party images that are built from an upstream tree with
patches on top now run `go test` on the packages those patches touch.
The step sits in the builder stage, right after the patches are applied.
Before this, ten of them ran `go build` only, and `go build` never
compiles a `_test.go`, so a test carried inside a patch was proven once
by hand and never again. The other two, opensearch-operator and
kube-ovn, ran a `go test` line written by hand, which a patch reaching a
new directory would slip past; the loop replaces both lines.

The package list is read from the patches themselves with `git apply
--numstat`, so a new patch is covered without an edit to the Dockerfile.
Each package is tested from its own directory, which finds nested
modules: in objectstorage, `internal/` is a module of its own and the
root module cannot name `./internal/runtime`. An empty list runs
nothing. The list is read into a variable before the loop, because a
`for` loop drops the exit status of a command substitution, and a patch
that git cannot read would otherwise give an empty list and a green
step.

A few images need more than the plain loop:

- opensearch-operator skips `controllers/`. A patch touches it, but its
suite starts an envtest control plane from etcd and kube-apiserver
binaries the builder does not have. The loop runs with `-race` and
replaces the hand-picked `go test -race` line, which also covered
`pkg/reconcilers/util`, a package no patch touches.
- kube-ovn drops its `go test -run '^TestVMIMigrationSchedulingRetry$'`
line. The test it named is copied into `pkg/controller`, which a patch
touches, so the loop runs it along with the rest of that package.
- redis-operator also runs `go vet -tags integration` in each touched
directory. Two of the files its patches edit are upstream integration
tests behind that tag, and `go test` compiles none of them.
- cluster-autoscaler and kubevirt-cloud-provider export `GOOS` and
`GOARCH` for the target in the whole builder stage, so their tests unset
both. A local `make image` builds amd64 and arm64 by default, and a test
binary for the other architecture runs on the build host only under
binfmt emulation. CI is not affected either way: the jobs that build
package images each pin one `PLATFORM` that matches the runner's
architecture.

The first run of the step on cluster-autoscaler found the carried
`fix-downscale.diff` out of step with the upstream tests it leaves in
place. The patch removes the `maxSize` refusal from `SetSize`, and one
case of `TestNodeGroupIncreaseSizeErrors` still expected that refusal.
The patch now drops that case and carries
`TestNodeGroupDeleteNodesAboveMaxSize` instead. That test covers the
downscale the patch is named for: `DeleteNodes` lowers a group through
`SetSize`, so with the refusal in place a group standing above its max
size could not shed a node. Against the unpatched `SetSize` the new test
fails. The image's behaviour does not change, and scale-up stays bounded
by the core scale-up orchestrator, which skips a node group already at
its max size.

CI builds all twelve images, and the step reports `ok` or `no test
files` for every package it tests. Locally, the redis-operator,
opensearch-operator and kube-ovn builder stages pass. The
cluster-autoscaler package was checked from the
`cluster-autoscaler-1.32.0` tag tarball, because the Dockerfile's full
clone of kubernetes/autoscaler does not survive my network. As negative
controls, an undefined call added to a redis integration test fails `go
vet -tags integration` while `go test` still passes, and the new
autoscaler test fails without the patch.

Left out on purpose, and the Dockerfile says so: `multus`. Its only
touched package, `pkg/multus`, creates network namespaces in its tests,
which a build container cannot do (`mount --make-rshared /var/run/netns:
operation not permitted`). `cilium` (C, no compile step) and
`piraeus-server` (Java) are out of scope.

Closes #4318

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: Dockerfiles under
`packages/*/images/` and one patch file. No values key, schema, make
target, `hack/` file or documented workflow changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```
2026-10-03 04:56:00 +03:00
Aleksei Sviridkin f32e0693c1 ci(dashboard): run the console's eslint in the UI test job
The console defines a lint script next to typecheck and test, but no
workflow ran it, so new findings landed unseen next to the old ones.
The backlog is cleared down to three react-hooks/exhaustive-deps
warnings. eslint exits zero on warnings alone, so the lint script caps
them at that count: the step gates from its first run, and a new
warning fails it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:09 +03:00
Aleksei Sviridkin 4d3796e41d refactor(dashboard): mark the deliberate setState calls inside effects
Each of these effects re-syncs local state to something that changes
outside the component. The rule's suggested rewrite derives the state
during render instead, which changes when the new value is committed
and what the first render shows, so each site carries a scoped disable
with its reason rather than a rewrite.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:08 +03:00
Aleksei Sviridkin be9b148e6c refactor(dashboard): move non-component exports out of component files
Vite's fast refresh can only hot-swap a module that exports nothing but
components; a file that also exports a hook, a helper or a registry
object forces a full page reload on every edit to it. Hooks, context
objects and the RJSF registry now live in their own modules, and the
components stay where they were.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:08 +03:00
Aleksei Sviridkin f1281e6176 refactor(dashboard): drop the unused ad binding in OverviewTab
The tab never reads it; the prop stays in the interface because the
detail page passes the same props to every tab.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:07 +03:00
Aleksei Sviridkin 8298c43e05 refactor(dashboard): type command palette instances as K8sResource
client.list was called without a type argument, so its items came back
as unknown and were cast to any to read metadata.name.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:07 +03:00