Commit Graph
7240 Commits
Author SHA1 Message Date
Aleksei Sviridkin ceb1afefc4 test(e2e): spell the find negation as -not in select-e2e_test
The negated-assertion audit reads a continued line that starts with
`!` as a bash statement, so the `! -type d` operand inside the find
expression on the continuation line turned the unit lane red on main.
`-not` is the same find predicate and is not a statement prefix.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 15:01:09 +03:00
IvanHunters ec4d26df54 fix(kubernetes): accept a golden worker image that exists only as a PVC (#4578)
## What this PR does

A worker pool on `osImage.builtin` refused a golden image that exists in
`cozy-public` as a PVC without its DataVolume, and the whole
KubernetesNodes release failed to render. CDI clones the PVC, not the
DataVolume, so that golden is a valid source, and vm-disk already
accepts it. This can happen once CDI garbage-collects DataVolumes
(`dataVolumeTTLSeconds`) or when someone restores a golden PVC on its
own.

Now the render fails only when neither the DataVolume nor the PVC
exists. The StorageClass and size checks already fall back to the PVC,
so a PVC-only golden is still checked against the pool, and the new test
suite pins that too.

Closes #4565

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): a worker pool on osImage.builtin now accepts a golden image that exists in cozy-public only as a PVC, matching vm-disk; it is refused only when neither the DataVolume nor the PVC exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Built-in golden images can now be used when available as either a
DataVolume or a PVC.
* **Bug Fixes**
* Missing golden images now produce an error that identifies the missing
image and suggests ways to resolve the issue. The error is suppressed
when the pool already has a successful clone.
* PVC-based golden images are checked against the pool’s storage class
and minimum disk size requirements.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:45:50 +03:00
IvanHunters f9f7562e56 fix(kamaji): resolve Gateway API availability once at startup (#4501)
## What this PR does

Carries the fix proposed upstream in clastix/kamaji#1327 (not yet merged
there; issue clastix/kamaji#1326) as a build patch on the 26.9.4-edge
tag that cozystack builds Kamaji from.

In Kamaji 26.9.4-edge, every TenantControlPlane reconcile checks whether
the Gateway API is available. The discovery call ignores the reconcile
context, so it is bounded only by the client's 32s default timeout,
which is longer than the 30s reconcile budget. The watches, however,
read availability only once, at startup.

With the patch, Kamaji resolves availability once when the manager
starts. The watches, the indexer, the reconciled resources and the
admission webhook's Gateway API check all use that one value. A
discovery error at startup now stops the manager, so it is no longer
mistaken for "Gateway API not installed". A Gateway API installed after
Kamaji starts needs a Kamaji restart. On this platform that case does
not come up: the kamaji package depends on cert-manager, and
cert-manager depends on gateway-api-crds.

The webhook's separate TLSRoute check still runs discovery on each
request, as before, and is listed as a follow-up in clastix/kamaji#1327.
Kamaji's chart sets no `timeoutSeconds` on the webhook, so the
apiserver's 10s default applies, and a discovery call that can take up
to the client's 32s can make that admission time out.

The patch is the upstream commit applied unchanged except for the
dropped docs hunk, which is not part of the image. It applies cleanly to
26.9.4-edge with no offsets. It modifies different files than the
existing `fix-kubelet-config-compat.diff`, so the two apply in either
order. The upstream tests stay in the patch so that it matches the
upstream commit; the image build does not run them. On the patched tree,
`go build ./...` passes, and `go test -race` passes for
`./controllers/...`, `./internal/webhook/...`,
`./internal/utilities/...`, `./internal/kubeadm/...` and `./cmd/...`. I
also checked that the new tests can fail: they go red when the
availability check is inverted and when the old per-reconcile lookup is
put back.

If clastix/kamaji#1327 changes before it merges, this patch should be
updated to match. It is dropped once Kamaji ships the change.

Refs #4489. This does not close #4489, because this change does not
explain the long stall reported there.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kamaji): Kamaji now checks for the Gateway API once at startup instead of on every TenantControlPlane reconcile, so a slow discovery endpoint can no longer use up the reconcile timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Gateway API availability is now checked at startup and consistently
applied to Gateway-related setup and configuration validation.
* Gateway configuration is rejected if the Gateway API was unavailable
when Kamaji started. If you install the required CRDs afterward, restart
Kamaji before configuring Gateway resources.
* Discovery errors now stop startup rather than allowing Kamaji to
continue with an uncertain Gateway API status.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:51 +03:00
IvanHunters b0e07cadbb fix(operator): rebuild each changed ArtifactGenerator generation once (#4594)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

After an upgrade, an ArtifactGenerator can report Ready on its new spec
while its artifacts were built from the old one, and source-watcher
never rebuilds them. That is how the bucket-system chart in #4593
shipped without cozy-lib.

The cause is upstream (fluxcd/source-watcher#384). source-watcher builds
from the spec it read when the reconcile started, but the final status
patch stamps Ready with the live generation. An upgrade changes the
packages source and the copy rules within seconds, so a spec change
often lands during a build.

The generator status has nothing that tells a stale build from a good
one. So the operator now rebuilds every changed generation once, after
source-watcher first reports it Ready, through the force-drift path the
operator already uses for fluxcd/pkg#934. An unchanged spec rebuilds to
the same digest, so ExternalArtifacts and HelmReleases do not move.
Generation 1 is skipped, since the race needs a spec change during a
build.

On the first rollout every generator past generation 1 gets one extra
rebuild, and its PackageSource shows
`Unknown/AwaitingSourceWatcherRecovery` until that rebuild lands.
Nothing waits on that condition today. The same rebuild repairs clusters
already stuck like #4593.

The recovery messages on the PackageSource no longer blame
fluxcd/pkg#934, because the same path now serves both cases.

### How it was tested

Unit tests in `internal/operator/packagesource_reconciler_test.go` cover
the forced rebuild on generation 2 and on a later generation, every case
where no rebuild must happen, the rebuild settling without a second one,
both failure orders (force fails, recording the generation fails), and a
stale recovery counter. Each condition of the new predicate was mutated
once to confirm a test fails.

I also ran this commit's operator image on an arm64 test cluster running
v1.7.0-alpha.3, the one where #4593 was found. Before that I had
repaired its bucket generator by hand: deleting the stale
ExternalArtifact and requesting a reconcile produced an artifact with
`charts/cozy-lib` and both bucket HelmReleases went Ready. With the
patched operator, the only generator past generation 1 got exactly one
forced rebuild and the verified-generation annotation. source-watcher
reported all its ExternalArtifacts up to date. The digests of all 214
ExternalArtifacts and the revisions of all 104 HelmReleases matched a
snapshot taken before the swap, and every PackageSource and HelmRelease
stayed Ready. The race itself was not reproduced on the cluster, since
that needs a spec change landing during a build.

Closes #4593

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(operator): rebuild an ArtifactGenerator once after each spec change, so an upgrade no longer leaves a component on artifacts built from the previous spec (for example a chart missing a library it now needs).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved PackageSource recovery when an artifact generator reports
ready: the system now verifies the generated output and triggers a
rebuild when needed.
* Failed rebuild or verification attempts can be retried during a later
reconciliation, and recovery tracking is cleared once the rebuilt
generation settles.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:44 +03:00
IvanHunters 50dec5d46a fix(platform): resume a retried migration Job from the stamped version (#4583)
## What this PR does

A retried pod of the platform migration Job no longer runs migrations
that already finished in the same upgrade. The Job gets
`CURRENT_VERSION` once, at render time, and a failed pod is replaced by
one with the same environment. By then every finished migration has
stamped the `cozystack-version` ConfigMap, but the new pod ignored that
and started from the rendered version again.

`run-migrations.sh` now starts from the version stamped in the
ConfigMap. If reading it fails, the pod fails and the Job retries it,
because falling back to the rendered version is exactly the re-run this
fixes. The rendered value is used only when the ConfigMap holds no
integer version, which is also when the chart itself could not get a
real version from it. Without that fallback a value like `v58` gave a
green Job that ran nothing.

The new `hack/run-migrations.bats` runs the script in the migrations
image's alpine base against a fake kubectl that keeps the stamp in a
file, so one pod's stamps are what the next pod reads. It needs docker,
like the other migration suites.

I also found that the ConfigMap existence check at the top of the script
treats any API error as a missing ConfigMap and skips every pending
migration. That is older than this change and is filed as #4579.

Closes #4555

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a retried platform migration Job now resumes from the version stamped in the cozystack-version ConfigMap instead of re-running migrations that already finished in the same upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Migration startup now resumes from a valid stamped version, avoiding
reruns of migrations that have already completed.
* Empty or non-numeric stamps fall back to the rendered version.
Out-of-range stamps or failed stamp reads stop migrations before they
run.
* If the migration ConfigMap is missing, migrations do not run and the
target version is stamped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:39 +03:00
IvanHunters bb2946990d fix(vm-instance)!: refuse guest memory below KubeVirt's 1M floor (#4569)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The vm-instance resource guard now refuses guest memory below 1M, the
same floor KubeVirt admission enforces. Before, a value such as `500m`
or `1Ki` rendered and then failed when the VirtualMachine was applied.

This is a breaking change. A memory above zero and below 1M rendered
before and is now refused at render time. KubeVirt v1.9.0 would not
admit such a VM either: `validateMemoryRequestsNegativeOrNull` refuses a
non-zero request below 1M
(`pkg/virt-api/webhooks/validating-webhook/admitters/vmi-create-admitter.go:1135`).
The VirtualMachine admitter runs the same check on its template.

The cpu, sockets and memory descriptions now list the values the guard
refuses, so the schema states the same rule as the error.

The error cannot quote an unquoted number as it was written, because
YAML has decoded it before the chart sees it. A large one also printed
in exponent form, so `4294967296` came out as `4.294967296e+09`. A
string is still quoted exactly as written. A number is now printed in
plain digits and marked "read as", which is the closest honest wording I
found.

Closes #4413

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(vm-instance)!: refuse `resources.memory` below 1M at render time, since KubeVirt does not admit a VM with less. A value above zero and below 1M rendered before and is now refused. The cpu, sockets and memory descriptions list the refused values, and the guard error marks an unquoted number as read rather than written.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* VM resource values are now checked when rendering: CPU cores and
sockets must be positive whole numbers no greater than 4294967295.
Unit-suffixed, fractional, zero, negative, and quoted values with
leading zeros are rejected.
* Memory values below `1M` are rejected, with clearer messages showing
the value as entered or interpreted.

* **Documentation**
* Updated resource guidance to describe these limits and validation
behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:15 +03:00
IvanHunters ed20c53449 fix(vm-disk): refuse to clone a golden image that does not exist (#4567)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Stacked on #4392, so the base is `fix/vm-default-images-health-check`.
Retarget to `main` once #4392 merges.

A new vm-disk with `source.image` now fails to render when the golden
image it names does not exist, and the error names the image and
`cozy-public`. Before this the chart cloned
`cozy-public/vm-default-images-<name>` without looking, and CDI waited
for that PVC forever while the VMDisk stayed Ready. That happens on a
typo, and on every cluster where the opt-in
`cozystack.vm-default-images` package is off, so the message also says
to add it to `bundles.enabledPackages`.

The golden counts as missing only when neither its DataVolume nor its
PVC exists. CDI clones the PVC, so a PVC with no DataVolume is still a
usable source, and a DataVolume whose PVC is not created yet is an
import about to start.

Under `helm template` every lookup is empty, so the check runs only when
the release namespace itself is visible. The same trick is already used
in the kubernetes and seaweedfs charts. Existing disks are exempt the
same way #4392 exempts them, because their DataVolume spec is immutable.
A disk refused while the catalog is off renders by itself once the
golden appears, since app HelmReleases retry on failure.

Tests are in `packages/apps/vm-disk/tests/dv_golden_image_test.yaml`:
refusal of a missing golden, a PVC without a DataVolume, a DataVolume
before its PVC, rendering with no cluster, and an existing disk whose
golden was removed.

Closes #4548

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(vm-disk): a new disk with source.image now fails with a clear error when the golden image does not exist in cozy-public, instead of waiting forever for a clone that never starts.
```
2026-10-05 14:42:52 +03:00
IvanHunters c7b8009964 fix(kubernetes): log kubevirt-csi temp NFS mount dir removal failure (#4562)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The kubevirt-csi node plugin now logs a warning when it cannot remove
the temp dir it uses for NFS publish. Before, the error was dropped, so
a dir left behind after a failed unmount was never reported.

The publish path temp-mounts the export root to create `/data`, then
removes the temp dir in a defer. The failure is logged, the same way the
unmount failure next to it already is, and not returned. A leftover temp
dir does not break the volume, and failing a publish that otherwise
worked only makes kubelet retry it.

A new unit test publishes an NFS volume through the fake mounter, where
the removal always fails, and checks that a warning naming the dir is
logged.

This is independent of #4451. That PR touches the same Go module but not
`node.go`.

A failed unmount still leaves the NFS root mounted. That is older than
this change and is tracked in #4561.

Closes #4466

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi node plugin now logs a warning when it cannot remove its temporary NFS mount dir instead of dropping the error
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Failed cleanup of a temporary mount directory now produces a warning
with the directory path, making the issue easier to identify. The volume
publish operation continues to succeed when cleanup fails.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:39:58 +03:00
Aleksei Sviridkin 2ddd55ce5b fix(build): honour PUSH and LOAD from the environment and in image-talos (#4654)
## What this PR does

A local build that asks to load images into docker no longer pushes
them. `hack/common-envs.mk` set `PUSH` and `LOAD` with plain `:=`, and a
makefile assignment beats the environment, so `LOAD=1 PUSH=0 make image`
quietly pushed a multi-arch build. Both now take the environment value,
and an empty one falls back to the default. I also made `PUSH` default
to 0 when `LOAD=1`, so `LOAD=1` alone only loads. Before this, `make
image LOAD=1` pushed and loaded at once. An explicit `PUSH` from the
environment or the command line still wins, and `OCI_EXPORT_DIR` still
forces both off. Every buildx target gets the values through
`BUILDX_ARGS`, so the fix lives in that one file.

The Talos installer was the one image that ignored `PUSH`. It is
published with skopeo, not buildx, so it never saw `--push=$(PUSH)`. Its
registry copies now run only when `PUSH=1`.

A build that pushes nothing no longer writes `PUSHED_TAGS_LOG`. That log
tells the multi-arch stitch which tags to move, so it must hold only
tags that were really pushed.

The cache helper lost an argument nobody used. No caller passes the
second argument of `cache-args`, and an explicit tag there would bypass
`CACHE_TAG`, so the amd64 and arm64 builds of that image would share one
cache ref.

CI sets neither `PUSH` nor `LOAD`, so CI builds do the same thing as
before. Each fix has a `make --dry-run` test in
`hack/common-envs_test.bats`.

Closes #4499

Closes #4504

Closes #4551

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The ccp skills pass `PUSH=1` explicitly, which still works. The website
docs do not mention `PUSH`, `LOAD` or `cache-args`.

### Release note

```release-note
fix(build): `LOAD` and `PUSH` set in the environment are now honoured, `LOAD=1` no longer pushes unless `PUSH=1` is also set, and the Talos installer is no longer published when `PUSH=0`.
```
2026-10-05 14:15:37 +03:00
Aleksei Sviridkin 2af8b0c229 ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.

Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.

Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.

The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.

`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.

Closes #4591

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:47 +03:00
Aleksei Sviridkin 9c025fa2b1 fix(installer): skip the packages artifact push and stamp with PUSH=0 (#4663)
## What this PR does

A local build with `PUSH=0` no longer publishes the packages artifact.
Before this change, `make image-packages` in `packages/core/installer`
ran `flux push artifact` no matter what `PUSH` was set to, and then
wrote the pushed digest into `values.yaml`. The `chart` target in the
same Makefile already respects `PUSH`.

With `PUSH` set to anything other than 1, the target now skips both the
push and the digest stamp. It prints one line saying so and leaves the
committed `platformSourceUrl` and `platformSourceRef` as they are. It
still writes `packages/core/platform/.build-revision`, which is
gitignored.

I chose to skip the stamp instead of computing a local digest, because
every reader of that value needs a digest that exists in the registry.
The chart passes `platformSourceRef` to the operator as
`--platform-source-ref`, and Flux then pulls
`cozystack-packages@<digest>` from the registry. The release scripts
`verify-promoted-packages.sh`, `promote-packages-artifact.sh`,
`stitch-multiarch.sh` and `nightly-mirror.sh` also treat it as a
registry digest. A digest of an artifact that was never pushed points at
nothing. On main an unpushed run would not even get that far, since the
recipe dies at `test -n "$DIGEST"`.

The gate is a make-level `ifeq ($(PUSH),1)`, so it reads whatever `PUSH`
ends up as after `hack/common-envs.mk`. On main that means `PUSH=0` on
the command line. I also applied the `common-envs.mk` change from #4654
locally, which takes `PUSH` from the environment and makes `LOAD=1`
default it to 0. With that applied, a dry run shows no push and no stamp
for `PUSH=0` in the environment, for `LOAD=1`, and for `PUSH=0` on the
command line, while `LOAD=1 PUSH=1` and an empty `PUSH` still push.

CI is not affected, because no workflow sets `PUSH` or `LOAD` and `PUSH`
stays at 1. I dry-ran the three CI invocations of this Makefile before
and after the change, on main and again with #4654 applied:

- `tags.yaml`, release stitch step: `make -C packages/core/installer
image-packages chart` with `REGISTRY`, a release tag as `IMAGE_TAG`, and
`PUSHED_TAGS_LOG`
- `pull-requests.yaml`, installer step of a same-repo PR: `make -C
packages/core/installer image` with a `pr-N-sha` tag,
`PLATFORM=linux/amd64`, `BUILDER`, and an empty `OCI_EXPORT_DIR`
- `build-main.yaml` and `build-release.yaml`: `make build`, which runs
the installer `image` target with defaults

```text
$ diff --recursive before/ after/     # tags.txt pr.txt build.txt
$ echo $?
0
```

There is one side effect. `OCI_EXPORT_DIR` forces `PUSH := 0`, so
running `image-packages` with it set no longer pushes, where before it
would have tried an anonymous push. No CI path does that. The fork PR
build already runs only `pre-checks image-operator` for the installer,
and the `make -n ... image OCI_EXPORT_DIR=/oci` scan in `e2e-fork.yaml`
greps only the `.oci.tar` names, which this change does not touch.

The new test dry-runs the target in both modes. With `PUSH=0` there must
be no `flux push artifact` and no `platformSourceRef` stamp; with `PUSH`
at its default, both must be present. The first test fails on main. I
mutated the guard three ways, and each time only the matching test went
red:

- condition forced true: the `PUSH=0` test fails
- compared against 2: the default-path test fails
- `endif` moved up so only the push is gated: the `PUSH=0` test fails on
the stamp check

One limitation is left as a follow-up. The test's `env -u PUSH -u LOAD`
only isolates it once #4654 lands, because on main a file assignment
wins over the environment anyway. It does not cover `PUSH=0` coming in
through `MAKEFLAGS`, so `make bats-unit-tests PUSH=0` would fail the
default-path test here and the ordering check in
`hack/stitch-multiarch_test.bats`. Nothing runs the suite that way
today. If that changes, passing an explicit `PUSH=1` on the default-path
`make` lines, or unsetting `MAKEFLAGS` the way
`hack/bats-unit-keep-going.bats` does, would fix it.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website's `development.md` tells developers to run `make
image-packages` with `REGISTRY` set to their own registry. That is the
default `PUSH=1` path, and it works as before. Neither the website nor
ccp mentions `PUSH=0` for this target.

Closes #4656

### Release note

```release-note
fix(installer): `make image-packages` no longer pushes the cozystack-packages artifact or stamps its digest into values.yaml when PUSH=0.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* You can run `image-packages` with `PUSH=0` to skip publishing the
artifact and keep the committed platform source reference unchanged.
* When pushing is enabled, the command publishes the artifact and
updates the platform source URL and digest-based reference.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:42 +03:00
Aleksei Sviridkin 33cf6972bb test(qdrant): exercise the tenant trust anchor end to end (#4701)
## What this PR does

The qdrant e2e suite now runs with TLS on and checks the tenant trust
anchor end to end. Before this, the fixture left `tls` unset with
`external: false`, so the chart rendered no CA and no
`TenantProjection`, and nothing in e2e reached the controller or
`tenantsecrets`. Setting `tls.enabled: true` turns TLS on without the
LoadBalancer that `external: true` would allocate.

The first new step checks that the anchor is published. The
`qdrant-test-ca` sentinel has to report `Ready=True/Projected`, and
`qdrant-test.tenant-ca` has to carry both the `tenant-ca` and
`tenantresource=true` labels. The projection must hold only `ca.crt`
with the same fingerprint as the chart's CA, and `tenantsecrets` must
serve it. The key-bearing `qdrant-test-ca` and `qdrant-test-tls` must
answer NotFound there, and any other error fails the step.

The second step checks that the anchor is what decides trust. It reads
the anchor through `tenantsecrets`, port-forwards the qdrant Service and
runs `openssl s_client` with that file as the only CA, hostname check
included. A CA with the same subject and another key must fail, and so
must the runner's default trust store. A handshake failure counts only
when the output shows a verification error, so a refused connection does
not pass as a rejection. Plaintext HTTP must get no response. Curl exits
7 and 28 fail the step, and a second verified handshake afterwards
proves the forward was still alive.

TLS does not break the existing steps, because none of them talk to
qdrant. `create-qdrant`, `wait-helmreleases-ready`,
`verify-statefulset-and-storage` and `verify-pvc-reclaim-on-delete` only
read Kubernetes objects: no `http://`, no port 6333 or 6334, no curl or
exec, and the suite has no backup round trip. The release now also
renders two Issuers, two Certificates and the sentinel. The chart
switches its probes to HTTPS with `enable_tls`, so `readyReplicas: 1`
now covers a pod serving TLS.

Local checks: `chainsaw lint test` (v0.2.15) says the file is valid.
`helm template` with the fixture's spec renders 2 Certificates, 2
Issuers and 1 TenantProjection, and none without TLS. I ran the
handshake logic against a local `openssl s_server` with a CA and leaf
shaped like the chart's. The anchor passes, the wrong CA and the default
store fail on verification, a wrong hostname fails, plaintext gets
`000`, and a closed port is not counted as a rejection.

A sentinel with a missing source (`SourceNotFound`) is not tested here,
the reconciler unit tests cover it.

Review notes I left as they are:

- Through `kubectl port-forward` the local socket is always accepted, so
curl exits 7 and 28 only catch a dead forward process. A false pass
would need qdrant to answer plaintext and a pod-side blip on that one
request at the same time.
- qdrant may reset the socket on the plaintext request, and some kubectl
versions drop the whole port-forward on a pod-side error. Then the
handshake after it turns the step red, not green. If CI shows `lost
connection to pod` there, the plaintext check should get its own
port-forward.
- The suite's `catch` has no `describe` of the `TenantProjection`. The
sentinel assert prints its conditions when it fails, but a red Secret
assert or script after it would not.
- The vendored chart's `servicemonitor.yaml` hardcodes `scheme: http`,
so metrics scraping fails for any qdrant with TLS on. That was already
the case before this PR, and the suite does not check metrics. Tracked
in #4703.

cert-manager does not set owner references here, so `qdrant-test-ca` and
`qdrant-test-tls` stay in the namespace with their private keys after
the app is deleted. That is tracked in #4704. The nats half of #3706 is
tracked in #4702.

Closes #3706

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/qdrant/`, which no entry in the
trigger map covers.

### Release note

```release-note
test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused.
```
2026-10-05 14:14:36 +03:00
Aleksei Sviridkin 7ea71042b7 ci(e2e): add a daily isp-hosted install lane on kind (#4712)
## What this PR does

This adds a daily CI lane that installs the `isp-hosted` platform
variant through the `hosted` installer variant. No lane ran either of
them before. The PR, nightly and tag lanes all install `isp-full` on
Talos, where Cozystack brings its own CNI and storage, and that is how
#3376 shipped unnoticed.

What `isp-hosted` changes is what it leaves to the host. The system
bundle swaps kube-ovn and Cilium for a `noop` networking Package and
drops multus, LINSTOR, MetalLB, cozy-proxy, snapshot-controller and the
rest of the shared data-plane set. The iaas bundle refuses to render.
Migrations are not a difference, `isp-full` enables them too.

So the lane runs on kind. kindnet, kube-proxy and local-path come from
the cluster, and there is no Cilium, no LINSTOR and no `replicated` or
`local` StorageClass. It installs the latest GHCR nightly instead of
building anything, at 05:00 UTC after the nightly mirror publishes, and
on `workflow_dispatch`. There is no pull request trigger, because a run
would install the nightly images and not the PR's.

A run passes when every HelmRelease is Ready, `cozystack-platform` and
`tenant-root` included. It also checks that the platform emitted the
`noop` networking Package and none of LINSTOR, MetalLB, multus or
KubeVirt, and that the cluster has no Cilium CRD and no `replicated` or
`local` class.

The lane reuses the e2e sandbox container for tooling and
`collect-report`, attached to the kind network.
`hack/isp-hosted-lane_test.bats` pins the wiring: the Package selects
`isp-hosted`, the install uses `variant=hosted`, the kind DNS domain
matches `networking.clusterDomain`, and kind is pinned by checksum and
digest. Six of its seven tests were red before the implementation
existed.

`hack/sandbox-runner-headroom.bats` gets a named exemption. That check
treats any workflow touching `packages/core/testing` as a lane that
boots the QEMU guests, and this one boots none. The exemption is checked
after the job match, and removing it turns the test red again.

The pins come from the kind v0.32.0 release: the binary sha256
`50030de2…` from its `kind-linux-amd64.sha256sum`, and
`kindest/node:v1.33.12@sha256:3f5c8443…` from its release notes.
v1.33.12 is the Kubernetes version the Talos lanes use, so the variant
is the only thing that differs. All three are plain `env` values in the
workflow, where a Renovate custom manager can match them.

### Real runs

The first real run on this branch,
[36961209755](https://github.com/cozystack/cozystack/actions/runs/36961209755),
went red on `tenant-root`:

```text
tenant-root  False  Helm install failed for release tenant-root/tenant-root with chart tenant@0.0.0+629b61300902: unable to build kubernetes objects from release manifest: resource mapping not found for name: "allow-external-communication" namespace: "tenant-root" from "": no matches for kind "CiliumNetworkPolicy" in version "cilium.io/v2"
```

This is #3387 reproduced live: `packages/apps/tenant` renders Cilium
policies on a cluster without Cilium. The other releases went Ready,
except `cozystack-basics`, which was still in an upgrade at the last
listing. The fix landed in #4710.

The run after it,
[37089699259](https://github.com/cozystack/cozystack/actions/runs/37089699259),
is green on the first attempt. It installed nightly
`0.0.0-nightly.20261003.37089458387.1`, built from 309875ab6, which
contains #4710. All four install tests passed, `tenant-root` included,
with no `cilium.io` CRD on the cluster. The install took about 9
minutes.

### Notes

The OpenAPI suite is not run. Its "Test kinds" case needs an Ingress
application, and kind has no LoadBalancer, so it would fail for a reason
unrelated to the variant.

The red run's job log is 347 MB. cozytest's live trace prints the whole
HelmRelease snapshot JSON from `cozy_wait_all_helmreleases_ready` on
every 2-second poll, and `gh run view --log` breaks at that size. The
isp-full lanes use the same helper, and it is tracked separately.

The red run spent about 31 minutes in install, most of it waiting out
the 30-minute gate; the green one took about 9 minutes on
`oracle-vm-24cpu-96gb-x86-64`. Moving to a smaller runner needs a
measured run, so it is a follow-up and not part of this PR.

Follow-ups from review, none blocking:

- No `vars.NIGHTLY_ENABLED` gate, so with the nightly switched off the
lane keeps testing a stale `nightly` tag. A `workflow_run` trigger on a
successful mirror would fix that too.
- The Package endpoint `cozy-hosted-control-plane` has to agree with
`KIND_CLUSTER: cozy-hosted`, and no test checks that yet.
- The operator registration check greps a Go literal in
`cmd/cozystack-operator/main.go`, so reformatting that table breaks it.
- The oras download has no checksum check, same as in `nightly.yaml`.
- cozytest stops at the first failing test, so on a red install the "no
Cilium" check never runs. Running it before the install would keep that
evidence in red runs.

Closes #3390

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds CI files, test files and one new make target. Nothing
under `hack/` is moved or renamed and no existing target changes, so the
ccp trigger does not apply.

### Release note

```release-note
ci(e2e): add a daily lane that installs the isp-hosted platform variant on kind
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added scheduled and manually triggered end-to-end checks for hosted
installations using the latest nightly packages.
* Validation covers platform readiness, package deployment, and expected
networking and storage behavior.
* Test runs collect diagnostic reports and attempt cleanup after
failures to support investigation.
* Added checks to verify hosted installation settings and ensure
prohibited network and storage components are absent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:05 +03:00
IvanHunters 14eab3cfe3 fix(backups): restrict the kafka-metadata CLI Pod and correct demo personas (#4695)
## What this PR does

This fixes two problems in the Kafka and NATS backup demos under
`examples/backups/`.

The Kafka metadata demo ran its CLI Pod without any securityContext, so
a namespace that enforces the restricted Pod Security level would reject
it. The Pod now gets the same restricted context as the CLI Pod in the
topic-data demo: `runAsNonRoot`, the `RuntimeDefault` seccomp profile,
no privilege escalation, all capabilities dropped. The new bats case
reads the overrides passed to `kubectl run` and fails if any of those
fields is missing or the patch is not sent as `--override-type=json`. It
is red on main, and it goes red on each of seven single-field mutations
of the patch while the other five cases stay green.

One known limit: `runAsNonRoot` needs an image that runs as a non-root
user. The default Strimzi image runs as UID 1001 and is fine. If
`KAFKA_IMAGE` is overridden with an image that runs as root, the kubelet
refuses to start the container, and the helper fails only when its 5
minute `kubectl wait` times out.

The kafka and nats walkthroughs told a tenant to run scripts that the
tenant roles forbid. Steps 03, 04, 06, 07 and `cleanup.sh` were in the
tenant column, and the scenario docs and several script comments had the
tenant create the backup Secret. Those rows now say admin and name the
part a tenant admin can do on its own, and the scenario docs say the
same. After this change the persona column in both tables matches the
tenant roles. kafka-metadata has no persona table, so I didn't change
anything there.

These are the tenant rules on pods and backup objects, plus one example
of the per-app secret rules. Each app chart's dashboard Role allows
`get`, `list` and `watch` on a few named Secrets and no other verb on
secrets. Nothing aggregated into the tenant roles grants `pods/exec`,
`pods/log`, `helmreleases`, `objectstorage.k8s.io`, `statefulsets` or
`kafkas.kafka.strimzi.io`.

```yaml
# packages/system/cozystack-basics/templates/clusterroles.yaml:100-112 (cozy:tenant:view:base, every tenant level)
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - persistentvolumes
  - endpoints
  - events
  - resourcequotas
  verbs:
  - get
  - list
  - watch
# packages/system/cozystack-basics/templates/clusterroles.yaml:238-246 (cozy:tenant:admin:base)
- apiGroups: [""]
  resources:
  - pods
  - services
  - persistentvolumes
  - endpoints
  - events
  verbs:
  - delete
# packages/system/backup-controller/templates/tenant-clusterroles.yaml:35-44 (cozy:backups:admin)
- apiGroups: ["backups.cozystack.io"]
  resources:
  - plans
  - backupjobs
  - restorejobs
  verbs:
  - create
  - update
  - patch
  - delete
# packages/apps/bucket/templates/dashboard-resourcemap.yaml:6-14 (bound at the tenant "use" level)
- apiGroups:
  - ""
  resources:
  - secrets
  resourceNames:
  {{- range $name, $user := .Values.users }}
  - {{ $.Release.Name }}-{{ $name }}-credentials
  {{- end }}
  verbs: ["get", "list", "watch"]
```

How each changed row runs into those rules:

| Row | What the script needs | Why a tenant can't |
| --- | --- | --- |
| 03 (kafka, nats) | waits on the `HelmRelease` and COSI objects, `get`
on the `bucket-<name>-backup` Secret, kafka also reads the CA Secret in
`tenant-root` | no `helmreleases` or `objectstorage.k8s.io` rule; the
bucket Role covers only `<release>-<user>-credentials` |
| 04 (kafka, nats) | waits on the `HelmRelease` and the Strimzi `Kafka`
or the `StatefulSet`, `create` on `secrets`, `create` on `pods`,
`pods/exec` | pods are `get`/`list`/`watch`/`delete` only, and nothing
grants `create` on secrets |
| 06 (kafka, nats) | `create` on `pods` and `pods/exec` for the CLI Pod,
kafka also `get` on `pods/log` for the refused restore | same pod rules,
and nothing grants `pods/log` |
| 07 (kafka, nats) | same as 04 | same as 04 |
| `cleanup.sh` (kafka, nats) | `delete` on `secrets`, on `backups`, and
on the cluster-scoped `BackupClass` and strategy | `cozy:backups:admin`
deletes only plans, backupjobs and restorejobs, and no tenant rule
covers secrets or cluster-scoped objects |

Step 05 stays a tenant step. Its `BackupJob` comes from
`cozy:backups:admin`, so in practice that means a tenant admin. The nats
demo's CLI Pod has the same missing securityContext as the
kafka-metadata one, and I left it for a separate issue.

Closes #4625

Closes #4636

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): the kafka-metadata backup demo now runs its CLI Pod with the restricted Pod Security context, and the kafka and nats backup demos now list steps 03, 04, 06, 07 and cleanup as admin steps, because they need Pod, exec, Secret and cluster-scoped permissions that tenant roles do not grant.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Security**
* Kafka metadata backup CLI Pods are recreated with restricted security
settings when they are not running or lack the required non-root
configuration.
* **Documentation**
* Kafka and NATS guides clarify which setup, restore, and cleanup steps
require administrator access and which job actions tenants can perform.
* Updated examples clarify that the demo provides required storage
Secrets and certificates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:54 +03:00
IvanHunters 1621c471f2 ci(dashboard): run eslint on the console and clear its errors (#4692)
## What this PR does

The console now passes its own eslint config, and CI runs it. On main
`pnpm lint` exits 1 with 54 problems, 51 of them errors, and
`ui-test.yaml` never ran it. On this branch it exits 0 with no errors,
and a Lint step sits between Typecheck and Test.

How each rule was handled:

- `no-explicit-any` (38 errors): replaced with real types. These commits
build a vite bundle byte-identical to main's.
- `react-refresh/only-export-components` (6 errors): the hooks, context
objects, `resolveSource` and the RJSF registry moved out of component
files into their own modules. `useTenantContext` stays in
`tenant-context.tsx`, so its importers and the tests that mock that
module are untouched.
- `no-unused-vars` (1 error): dropped the unused `ad` binding in
`OverviewTab`.
- `react-hooks/set-state-in-effect` (6 errors): relaxed per line with a
reason, not in the config.

The last rule is the only relaxation. Each of these effects re-syncs
local state to something that changes outside the component. The rule
wants the state derived during render, which changes when the value is
committed or what the first render shows, so it would be a behaviour
change. Per file:

- `components/ResourceQuotasField.tsx` re-syncs the inputs after the
form value is reset from outside.
- `components/SourceWidget.tsx` re-syncs the selected option when the
value loads asynchronously in edit forms.
- `components/command-palette/command-palette.tsx` resets query and
level whenever the palette closes.
- `components/command-palette/use-keyboard-nav.ts` moves the highlight
back to the top when the item list changes.
- `hooks/use-is-mac.ts` would return `true` on the first render on macOS
with a lazy initial state. Today it returns `false`.
- `lib/tenant-provider.tsx` falls back to an existing tenant once the
tenant list has loaded.

The eslint config itself is unchanged, and `no-explicit-any` stays on
everywhere.

`pnpm typecheck` and `pnpm test` (442 tests) pass at every commit. After
the file moves the bundle has the same chunk set, and only the entry
chunk differs, by reordered and renamed minified functions.

Three `react-hooks/exhaustive-deps` warnings from main are left as they
are, because fixing them changes when the effects run. One is in
`SourceWidget.tsx`, where the effect reads `options` but depends only on
`value`. The other two come from `ads = adList?.items || []` in
`use-command-items.tsx`, which gives two `useMemo` hooks a new
dependency on every render.

Things I did not do here:

- The step has no `--max-warnings`, so a new warning will not fail CI.
Once the three warnings are fixed, `--max-warnings 0` can go in.
- `SourceField.tsx` and `SourceWidget.tsx` each declare the same small
`SourceOptionSchema` interface. I kept them local since the two widgets
share nothing else.
- The job name stays `Typecheck and test`, so a required check with that
name keeps matching.
- `pnpm -r lint` only reaches `apps/console`.
`packages/{k8s-client,ui,types}` have no `lint` script, same as on main.

Closes #3911

Closes #4566

### Screenshots

No visual change: types, module layout and CI only.

### Downstream repositories

The diff touches only the console workspace and `ui-test.yaml`. The
console has no downstream copy since `cozystack-ui` was archived, and no
trigger map entry matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(dashboard): the console's eslint now runs in the UI test job, and the console passes it with no errors.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Tenant selection now uses available tenant namespaces, restores a
saved selection when possible, and falls back to the root tenant or the
first available tenant.
* Tenant options are sorted, and the selected tenant is kept in sync
with the available choices.
* **Chores**
* Automated checks now include linting alongside type checks and unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:31 +03:00
IvanHunters 9239555e6a ci(workflows): pin flux, yq and helm-unittest installs (#4645)
## What this PR does

Workflows no longer run the flux install script as root. Every step that
needed the flux CLI downloaded `https://fluxcd.io/install.sh` and ran it
with `sudo`, so whatever that host served at the moment ran on runners
that later get registry credentials. PR #4442 pinned the version, but
the script was still fetched and executed. All 12 of those steps now use
`fluxcd/flux2/action`, pinned to the v2.8.6 commit with `version:
"2.8.6"`. The action downloads the release tarball, checks it against
that release's checksums file, and adds the binary to `PATH`.

Two workflows still took yq from the floating latest release, nightly
and the fork e2e lane. They now use the same install the promotion steps
already have: yq v4.53.3 checked against a committed SHA-256. I verified
that hash against the release's `checksums` file and against the binary
itself.

The helm-unittest plugin was installed at whatever version was newest.
The plugin embeds its own Helm, so a plugin release could change chart
test results on a PR that touched no chart. It's pinned to v1.1.2 now,
the version CI already runs. A runner that carries a different version
gets it replaced.

A new contract test scans every workflow for these three tools, because
an unpinned install usually comes back by copying an older step. On main
it fails on the install script, the missing setup action, the
latest-release yq and the unpinned plugin. I ran one mutation per guard,
and each one failed only the test meant to catch it. The existing
contract tests that checked `FLUX_VERSION` inside the old steps now
check for the action step.

The e2e sandbox image still runs `install.sh` in its Dockerfile. That's
an image build, not a workflow step, so I left it for a separate change.

Closes #3619

Closes #3640

Closes #4510

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only workflow tool-install steps and `hack/*.bats`
contract tests. Nothing in the trigger map reaches them.

### Release note

```release-note
ci(workflows): install flux through the commit-pinned upstream setup action instead of running fluxcd.io/install.sh as root, pin yq with checksum verification in the nightly and fork e2e workflows, and pin the helm-unittest plugin version.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Reliability**
* CI workflows now use a consistent, pinned Flux version. The yq and
Helm plugin setup also verifies or enforces specific versions.
* **Tests**
* Added checks to ensure workflow tool versions are pinned and
consistent, and that downloaded yq releases are checksum-verified.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:02 +03:00
IvanHunters 40638429bd fix(release): collect refs stored as registry/image/tag/digest keys (#4697)
## What this PR does

The shared image-ref enumeration skipped refs stored the way upstream
ingress-nginx stores them: `registry`, `image`, `tag` and `digest` as
separate keys, with the repository in `image` and a digest that has no
`@`. Two refs in the tree use this layout, the ingress-nginx controller
and its default backend, both in
`packages/system/ingress-nginx/values.yaml`. Every tool that reads refs
through `hack/lib/image-refs.sh` missed both of them: the promote retag,
the nightly mirror, the multi-arch gate, the stitch step and the
promoted-packages verifier.

I added a sixth YAML shape that joins `<registry>/<image>@<digest>`. It
fires only on a complete `sha256:` digest, so upstream's unpinned
default (`digest: ""`) and image maps without a digest emit nothing. It
checks that `digest` is a string before `test()`, same as shape 3 does.
The library header and `docs/agents/image-refs.md` now say "the YAML
sub-shapes listed" instead of a number, so the next shape doesn't make
the count wrong again.

The real tree now gives 93 refs instead of 91, and the two new ones are
exactly the ingress-nginx pair. The retag and mirror output does not
change today, because both refs sit outside the owned registry and the
ownership filters drop them. Both digests are multi-arch indexes, so
`hack/verify-multiarch.sh --refs-file` with just those two passes and
the release gate stays green.

To show the consumers really pick the shape up, I ran both dry-runs on a
copy of the tree with the controller moved under the build registry, the
host being the whole `registry` value. In that copy the controller is
the only build-registry ref.

Before:

```text
$ REGISTRY=iad.ocir.io/idyksih5sir9/cozystack hack/promote-retag.sh 1.99.0 --dry-run
No cozystack-owned digest-pinned image refs found under iad.ocir.io/idyksih5sir9/cozystack/ — is this the rc's baked tree?
exit 1
$ hack/nightly-mirror.sh 1.99.0 <tree>/packages --dry-run
No cozystack-owned digest-pinned image refs found under iad.ocir.io/idyksih5sir9/cozystack/ in '<tree>/packages' — is this the baked main tree?
exit 1
```

After:

```text
$ REGISTRY=iad.ocir.io/idyksih5sir9/cozystack hack/promote-retag.sh 1.99.0 --dry-run
▸ iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller  sha256:a8ddcee59b63f6cbf93f4d3c9ffa1d1dda3c1d39497748d4484655356c68b3f6
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller@sha256:a8ddcee5… docker://iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller:1.99.0
exit 0
$ hack/nightly-mirror.sh 1.99.0 <tree>/packages --dry-run
▸ iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller  ->  ghcr.io/cozystack/cozystack/ingress-nginx-controller  sha256:a8ddcee5…
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/…/ingress-nginx-controller@sha256:a8ddcee5… docker://ghcr.io/cozystack/cozystack/ingress-nginx-controller:1.99.0
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/…/ingress-nginx-controller@sha256:a8ddcee5… docker://ghcr.io/cozystack/cozystack/ingress-nginx-controller:nightly
exit 0
```

The red test is a new case in `collect_refs_from_file understands every
YAML shape`. It has a fixture in the ingress-nginx layout, with a
non-string digest in front of it, and three near-misses that must emit
nothing: no digest, an empty digest, and a digest without `image`. On
main the positive case fails with `got: <nothing>`. Removing the rule,
the registry join, the full-digest check, or the type guard on `image`
or on `digest` turns that test red, and only that one. The bats suites
of all library consumers pass with both the old and the new library.

Two gaps remain in the tests and I left them out on purpose. No fixture
covers the new rule without a `registry` key, and none pins its type
guard on `registry`. Shapes 2 and 3 have the same unpinned `registry`
guard on main.

The overlay script has no such gap. `hack/overlay-main-images.sh` does
not use the library, and its line classifier already accepts `image:`,
`registry:`, `tag:` and `digest:` lines.

Closes #4518

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff. None of the changed files are
in it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(release): the image-ref enumeration used by promotion, the nightly mirror and the multi-arch gate now reads refs stored as separate registry/image/tag/digest keys, the layout of the ingress-nginx controller and default backend.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Image references can now be read from configurations that store the
image name and SHA-256 digest separately, with an optional registry.
* **Bug Fixes**
* Incomplete or invalid image and digest values are ignored rather than
producing malformed references.
* **Documentation**
  * Documented the additional image storage format.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:10:58 +03:00
IvanHunters 9196b9798e fix(e2e): bound the dataplane capture collector by wall clock (#4640)
## What this PR does

The e2e data-plane collector limited itself by counting pods and
LoadBalancers, but both of its callers stop it by the clock.
`hack/cozytest.sh` gives it 600s and the Chainsaw catch gives it 300s.
One affected pod could already run past the smaller limit, and the
LoadBalancer capture, which runs last, was the part that got lost. I
moved the collector to a wall-clock budget and fixed the other issues
filed against it.

- Every kubectl call now goes through `dp_run`. It cuts the call's
timeout to what is left of `COZY_DATAPLANE_BUDGET` and starts nothing
once the budget is spent. The default is 270s, and `cozytest.sh` passes
570s. The pod section stops 90s early, so the LoadBalancer section
always gets time to start. A call the deadline cut says so, a call cut
by its own bound names that bound, and a refused call says it was not
started and names its section; exec blocks carry that line inside the
block. The node capture and the LoadBalancer probe used to blame an
unanswered lookup or a missing cni-server when the budget stopped a
call. Now they name the budget (#3671).
- A static test fails on any kubectl call that bypasses `dp_run` and
lists the lines. It matches by command position, so a call inside a
double-quoted `$(...)` is still checked (#3694).
- Per-pod and per-LoadBalancer capture files join namespace and name
with `_`, which neither can contain. A second run into the same
directory appends, with the run separator `capture-notes.txt` already
uses (#3679).
- `capture_node` and `capture_lb_node` no longer write their callers'
loop variables. A static test fails when a function assigns a name that
a `read -r` loop or another scope owns (#3675).
- The tests that shrank every timeout to one second now cut only the
read under test. A `timeout` shim decides by argv, so a slow machine
can't cut a read the test needs answered (#3685).
- `cilium-dbg bpf lb list` runs unfiltered in every affected node's
capture, not only for a failing LoadBalancer (#3689).

The call-site comment in `.chainsaw.yaml` and the collector paragraph in
`docs/agents/e2e-testing.md` now match the code. Every behaviour change
has a test in `hack/capture-dataplane.bats` that failed on the code
before it.

Two parts of the issues live elsewhere. The runner stopping a bats file
at its first failing test is #4620, and a conntrack read filtered by
service VIP is #4623.

From #4629 I fixed the items on lines this PR already changes: the node
capture now gets the run separator, a failed speaker log read writes a
note, the endpoint tcpdump says when it falls back to the geneve
interface, and the per-read timeout knobs are validated (a zero there
would otherwise look like a spent budget). The rest of #4629 stays open:
`ovn-lflows.txt` and `lb-speaker-logs.txt` are still replaced on each
run, and two comments still describe older behaviour.

- Closes #3694
- Closes #3671
- Closes #3679
- Closes #3675
- Closes #3685
- Closes #3689
- Refs #4629

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): bound the failure-path data-plane collector by a wall-clock budget, keep its per-pod and per-LoadBalancer captures from overwriting each other, and capture the cilium service map for ClusterIP failures.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Data-plane diagnostics now follow a configurable time budget,
reserving time for LoadBalancer captures and stopping operations when
the budget is spent.
* Capture reports distinguish failed, successful, unknown, and
budget-limited checks. LoadBalancers with successful probes are skipped
for heavy capture.
* Repeated captures are preserved together, and limits help control
LoadBalancer diagnostic collection.
* Diagnostics include additional network details from relevant nodes
during probe replay.
* **Documentation**
* Updated guidance covers capture time limits and how unanswered
EndpointSlice reads are reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:10:25 +03:00
IvanHunters f209c477e2 test(e2e): read the opensearch release history through the pinned name (#4689)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The opensearch suite now reads HelmRelease history through the shared
jsonpath, and a unit test fails if any Chainsaw suite goes back to a
copied literal. Before this, the suite carried its own copy of the
expression, and the upstream contract test in `internal/fluxcontract`
only checks `HELMRELEASE_HISTORY_JSONPATH`. I checked the copy against
helm-controller `api` v1.5.1, the version go.mod pins: the history field
is `history` and each snapshot has `status`. So the read works today, it
was just not pinned.

The suite passes `-o "jsonpath=${HELMRELEASE_HISTORY_JSONPATH}"` now,
which gives kubectl exactly the same argument as before. The new test
reads every suite under `hack/e2e-chainsaw/`. A suite that calls
`helmrelease_has_remediation_cycle` must fill the local it hands the
guard from the shared name, and no suite may assign that name itself. It
reuses the matchers that already pin `run-kubernetes.sh`. The test is
red on main and green with the fix.

One known limit. The second-assignment check counts over the whole file,
which fits `run-kubernetes.sh` because it runs as one shell. A Chainsaw
suite runs each `script` step in its own shell, so an unrelated
`statuses=` in another opensearch step would turn the test red with
nothing broken. That fails toward red, never toward a silent pass, so I
left it.

Closes #4550

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(e2e): the opensearch suite reads HelmRelease history through the shared jsonpath, which is now pinned against the upstream Flux type for every Chainsaw suite that runs the remediation guard.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Expanded validation across Chainsaw end-to-end suites to check
consistent HelmRelease history handling in remediation checks, including
the order and isolation of script steps.
* The OpenSearch suite now uses the shared history lookup. Empty-history
checks and remediation-cycle failure handling are unchanged, and there
is no change to end-user behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:08:51 +03:00
IvanHunters b4fbaf00a2 test(e2e): keep gateway and redis-tls op ceilings above their own waits (#4643)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Two Chainsaw script ops had a timeout smaller than the waits inside
them. When the op ceiling fires first, Chainsaw kills the shell: the
script never prints its own error, and a cleanup trap never runs.

The gateway listener-set step could poll for 660s under a 6m op. Its
cleanup trap also deleted two tenants with no limit on the wait. Each
delete now runs under an outer `timeout`, because kubectl's own
`--timeout` does not cover discovery or a stalled request. A tenant
delete gets 300s, the same 5m that `.chainsaw.yaml` gives a delete op.
The parent is deleted only after the child is gone, because deleting a
parent while its child is still uninstalling wedges the parent's
cleanup. A failed tenant delete fails the step with kubectl's own error,
so a leaked tenant shows up here and not in a later suite. The op is now
24m, which covers the polls plus those bounds.

The redis-tls CA-projection step has 30s read caps that add up to well
over its 5m op. Raising the op would not hold, because several reads run
in loops. So all reads now share a 240s step budget. No read starts
after the budget runs out, and a read that is still running gets cut off
when it does. A read that hits its bound, or finds the budget already
spent, says which read it was.

Two exec checks in the same step blamed the 15s redis-cli cap for any
failure, including a refused exec or a missing container. They now print
the exit status and what the exec printed.

The step also runs under bash now, for pipefail. The sandbox's `/bin/sh`
is dash, which rejects `set -o pipefail`. Chainsaw 0.2.15, the version
the sandbox image installs, supports a `shell:` field on script ops.
Without pipefail, a failed Secret read took base64's zero exit status
and was reported as a wrong source name.

The private-key check read the projection a second time, so a failed
second read made it pass. It now checks the value the step already
fetched.

I added `hack/chainsaw-op-budgets.bats` to guard all of this. It holds
the gateway op above its polls plus its delete bounds, keeps the
child-before-parent order, holds the redis budget under its op with
every kubectl call going through the clamped helper, and runs the redis
step's script against a stubbed kubectl to check each failure message
and the clamp. On main, 11 of its 12 tests fail. The one that passes is
the healthy-stub control.

Closes #4114

Closes #4250

Closes #4246

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches two Chainsaw suites and adds one unit test file under
`hack/`. It moves or renames nothing a downstream repository reads.

### Release note

```release-note
test(e2e): the gateway listener-set and redis-tls projection steps now fail on their own errors instead of being killed by the op timeout, and redis-tls reports the real cause of a failed read or exec.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added automated checks to verify gateway operations have sufficient
time to complete and cleanup deletions are bounded and correctly
ordered.
* Expanded gateway scenarios to cover cleanup failures: if child-tenant
deletion fails or times out, the parent is left in place; deletion
failures are reported.
* Updated Redis TLS checks with bounded read times and a shared
execution limit. Failure reports include command output and status, and
checks cover private-key material in the decoded projection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:07:37 +03:00
IvanHunters 1ca603a7b2 test(e2e): wait for the S3 gateway to accept a key before using it (#4694)
## What this PR does

The bucket e2e suite fails now and then because the S3 gateway rejects a
key that COSI already reports as granted. The error is `The access key
ID you provided does not exist in our records.` from `mc alias set`. In
job 109130131391 the BucketAccess assert finished at 21:23:43.57 and the
first `mc alias set` failed at 21:23:44.82. In job 109396174094 cap1 and
cap2 were accepted, and cap3 was rejected two seconds into the step.

Granted access is not the same as a usable key. In seaweedfs-cosi-driver
v0.3.1, `DriverGrantBucketAccess` writes the new key into
`/etc/iam/identity.json` on the filer and returns. The S3 gateway in
SeaweedFS 4.45 reloads that file later, from a filer metadata
subscription (`weed/s3api/auth_credentials_subscribe.go`). In our chart
the driver writes through `seaweedfs-filer` and the two gateways follow
`seaweedfs-filer-client`, so the change may have to pass through the
other filer replica first.

The suite now waits until the gateway accepts the key before it writes.
Every alias in both write steps goes through `await_key`, which uses the
same port-forward as the writes and so asks the same gateway pod. It
waits only while the answer is the InvalidAccessKeyId text, for at most
60s. Any other error fails at once, and so does an attempt cut off at
its 15s cap. When the window runs out, the step fails with a message
that names the BucketAccess.

One probe costs at most 77s, so both steps go from a 5m to an 8m op
timeout. The inner budgets now sum to 409s and 441s, and the budget
comments say so.

I checked the probe locally against a fake `mc`:

```text
case=ok rc=0 attempts=1 elapsed=0s
  Added `a` successfully.
case=lag rc=0 attempts=3 elapsed=5s
  Added `a` successfully.
case=other rc=1 attempts=1 elapsed=0s
  FAIL: mc alias set for BucketAccess bucket-test-admin failed (exit 1) with an error other than the key not being loaded yet
case=never rc=1 attempts=31 elapsed=60s
  FAIL: S3 gateway still rejects the key of BucketAccess bucket-test-admin after 60s of probing (the gateway never loaded it, or the driver never wrote it)
```

A hanging `mc` fails on its first attempt after 15s with exit 124.
`chainsaw lint test` from v0.2.15 accepts the file.

The read-only upload check gets fixed along the way. Before, an unloaded
`ro-user` key also made the upload fail, and that failure passed as the
expected denial.

One thing I left as it is: `await_key` is copied into both steps,
because chainsaw steps share no shell, and moving it to `_lib/` would
run every suite on each edit.

Closes #4580

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): the bucket suite waits for the S3 gateway to accept a new key before writing with it, instead of trusting BucketAccess readiness.
```
2026-10-05 14:07:16 +03:00
IvanHunters 858ca4d0a3 test(e2e): cover monitoring OIDC modes with a Chainsaw suite (#4678)
## What this PR does

Monitoring OIDC gets live-cluster e2e coverage again. It had none since
the two pre-Chainsaw BATS suites were removed without a port. This adds
a Chainsaw suite in `hack/e2e-chainsaw/monitoring/` that ports what they
asserted, plus the in-place switch from an inline config to a
`secretRef` that nothing covered.

The suite takes one Monitoring release in `tenant-test` through three
modes in a row: System, CustomConfig with an inline config, and
CustomConfig with a `secretRef`. It creates that release itself and
never reads the tenant-root monitoring instance, so it keeps working if
#4023 or #4479 turns root monitoring off.

What each phase checks:

- System mode: cozystack-api admits `spec.oidc` and both HelmReleases go
Ready. The Grafana CR points `auth.generic_oauth` at the cozy realm, has
no `role_attribute_path`, and has `skip_org_role_sync` and
`oauth_allow_insecure_email_lookup` forced. The KeycloakClient is
confidential and references the generated Secret, which holds 32
alphanumeric characters. The audience scope is bound to this release's
clientId. There is no chart-owned `KeycloakRealmGroup`. The users-Job
carries the users and the mode, and `grafana-deployment` reads the
client secret from the Secret.
- Switch to inline: the Grafana CR takes the tenant's map, keeps the
forced keys and drops every System-only key. The users-Job reruns with
the new list. The KeycloakClient, the audience scope and the client
Secret are pruned, and the Deployment stops referencing that Secret.
- Switch to `secretRef`: the inline `auth.generic_oauth` block is gone
from the Grafana CR. The operator Secret is mounted at
`/etc/grafana/oidc` on both the CR and the Deployment. This checks the
mount only and does not prove Grafana loads the file, see #4679.
- After the install and after each switch, neither release may show a
remediation cycle. That means no `failed` or `uninstalled` snapshot, and
the latest revision must be exactly 1, then 2, then 3. The revision
check is there because helm-controller truncates history after the
previous deployed snapshot. An upgrade that failed, rolled back and then
passed on retry leaves no failed entry, only extra revisions.

I did not assert `GF_PATHS_CUSTOM_INI`. I could not find anything in
Grafana that reads it (#4679), and pinning it would make possibly dead
wiring part of the contract.

The coverage split with the render tests is this.
`packages/system/monitoring/tests/oidc_test.yaml` keeps the
fresh-install render of each mode, including the CustomConfig negatives
(no Keycloak objects, no client Secret). This suite covers what a render
cannot see: admission, the Deployment the Grafana operator builds, and
what a switch between modes prunes or leaves behind. The Keycloak
objects are checked as written, so creating the client in Keycloak is
not asserted. Removing it is, because the operator's finalizer holds the
KeycloakClient until the client is gone from Keycloak.

Before the first apply the suite also waits until the tenant's
`cozystack-values` carries `oidc-enabled: "true"`. The outer release
copies `_cluster` into the inner release once, so a Monitoring applied
right after the tenant is created can render from stale values and fail
the System-mode guard for the full 15m. That is what the first CI run on
this PR hit.

The Keycloak CRDs are required, not probed for. The e2e install always
enables platform OIDC, so a missing CRD is a broken install. The removed
suites skipped in that case, and their probe could not tell an absent
API group from a present one.

The selector picks the suite up by name from
`cozystack.monitoring-application` and `cozystack.monitoring`. A change
to `system/monitoring`, `extra/monitoring`, `monitoring-rd` or
`grafana-operator` now runs only this suite instead of the full run. The
tenant-root Grafana bring-up stays covered by the install BATS while it
exists. One selector unit test relied on `cozystack.monitoring` reaching
no suite, so it now renames that source in its copy of the graph.

The second commit keeps Keycloak changes on every suite that drives a
Keycloak client. Before this PR a change to `system/keycloak` or
`system/keycloak-operator` escalated to the full run. With the
monitoring suite it would select monitoring alone, because
`monitoring-application` is the only source that depends on
`keycloak-operator`, and `kubernetes-latest` with its OIDC lifecycle
would drop out. `src_to_suites()` now maps both to `kubernetes-latest
monitoring`. The mapping has to be explicit because
`kubernetes-application` has no `dependsOn` on `keycloak-operator`
(#4680). A `postgres-operator` change now also reaches
`kubernetes-latest`, through Keycloak's database. The selector comment
and the e2e doc no longer use `system/postgres-operator` as the example
of a path with a source that reaches no suite.

Cost: one monitoring stack bring-up, two in-place upgrades and teardown.
I expect 10 to 20 minutes on every full run. The selector change makes
the CI run on this PR a full run, so the suite's own timing is in the
chainsaw report. A red run can spend up to 16m on each of four waits,
since each one sits a minute above the 15m timeout of the release it
waits on.

The suite cannot run locally, so I checked it this way:

- `chainsaw lint test` at v0.2.15, the sandbox version, passes.
- I ran the suite's own steps against a local envtest apiserver.
Chainsaw applies the real manifests, and a stand-in for helm and the
operators renders the chart from the stored Monitoring spec. With the
chart intact all nine steps pass. Each of 30 single breakages of the
chart or of the simulated controllers turns the expected step red.
- `hack/select-e2e_test.bats` and `hack/select-install_test.bats` pass.
The new Keycloak selection case fails without the mapping and when
`kubernetes-latest` is dropped from it.

A browser login through Keycloak is out of scope, same as in the removed
suites.

Left for later:

- KeycloakClient and scope `status` after the operator creates them in
Keycloak is not checked.
- `hack/e2e-chainsaw/README.md` lists suites by hand. I added
`monitoring`, but `backup`, `kafka-metadata` and `kubernetes-proxmox`
were already missing.
- The `operator diff selects all dependent app suites` selector test
could also check `monitoring`.
- The comment on the Keycloak mapping says both suites wait for the
finalizer to remove the client. The monitoring suite does wait.
`kubernetes-latest` only reads once after its upgrade.
- The ready checks compare `observedGeneration` with
`metadata.generation` through `contains()`. Both are fields of the same
object, not a number literal, so the v0.2.15 literal gotcha should not
apply, but no live run has passed this suite yet.
- The inline fixture carries no `enabled: "true"`, so that phase checks
a disabled OAuth block. The asserts do not depend on it.

Closes #4051

Closes #3785

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added end-to-end coverage for Monitoring installations using system
OIDC, inline custom configuration, and Secret-based configuration. The
tests verify readiness, rendered resources, configuration cleanup, and
upgrade history.
* Updated test selection so Keycloak-related changes also run the
Monitoring suite.
* **Documentation**
* Added Monitoring to the documented Chainsaw suite list and clarified
how changed paths determine test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:07:10 +03:00
IvanHunters 6e4c9c4b6a fix(ci): harden the PR labeler's title parsing and map recurring scopes (#4666)
## What this PR does

Fixes three problems in `.github/workflows/pr-labeler.yaml`. Each one
either killed the job or put a PR in the wrong area, and none of them
showed up anywhere.

- Both maps are now looked up with `Object.hasOwn`. A title typed or
scoped `constructor`, `toString`, `hasOwnProperty` or `__proto__` used
to resolve to an `Object.prototype` member and crash the step on
`l.startsWith`, which left the PR with no labels at all.
- Every leading `[Backport ...]` prefix is now stripped, not only the
first one. With two prefixes, the second was read as a bracket scope, so
the real scope and the `kind/*` label were both lost. The `area/release`
label hid the problem by keeping `area/uncategorized` from firing.
- Scopes that recur three or more times over the full PR history now map
to an area. So does each area's own name (`networking`, `storage`,
`database`, `virtualization`, `testing`, `dependencies`).

Scope-to-area decisions:

- `area/database`: `database`, `etcd-operator`, `opensearch`,
`opensearch-operator`
- `area/networking`: `networking`, `kubeovn`, `multus`, `kilo`,
`hetzner-robotlb`, `ouroboros` (it rewrites the CoreDNS Corefile to fix
hairpin NAT, so it is networking, not platform)
- `area/virtualization`: `virtualization`, `vm`, `kubevirt-operator`,
`kubevirt-cdi`, `kubevirt-instancetypes`
- `area/platform`: `operator`, `cozystack-operator`, `controller`,
`cozystack-controller`, `cozystack-basics`, `flux-shard-operator`,
`migrations`, `scheduler`, `cozystack-scheduler`, `reloader`, `bootbox`,
and `lineage`. I put `lineage` here on purpose: no existing label covers
it more precisely, and it is too narrow for an area of its own.
- `area/storage`: `storage`, `piraeus-operator`,
`backupstrategy-controller`
- `area/release`: `changelog`, `changelogs`
- `area/ci`: `workflows`, `labeler`
- `area/dependencies`: `dependencies`, `renovate`
- `area/api`: `registry` (the cozystack-api storage layer)
- `area/dashboard`: `dashboard-controller`
- `area/monitoring`: `monitoring-agents`
- `area/keycloak`: `keycloak-configure`
- `area/testing`: `testing`
- New `area/dx`, added to `.github/labels.yml`: `dx`, `hack`, `cozypkg`.
These are developer tooling (hack/ scripts, package Makefiles, cozypkg).
`area/build` covers image builds and `area/testing` covers tests, and
neither describes this work.

Left unmapped on purpose:

- `http-cache`: the application was removed from the tree.
- `openbao`: already mapped on main.
- `apps`, `system`, `docs`: catch-all scopes that don't point to one
area.
- `fix`, `bugfix`: malformed titles.
- `security`, `rbac`, `governance`, `codeowners`, `maintenance`,
`images`, `readme`, `telemetry`, `cluster-autoscaler`: no single area
fits any of them.

I did not add spelling normalisation. `kubeovn` got its own row, since
it is the only alternate spelling that recurs.

A new contract test, `hack/pr-labeler-contract.bats`, pulls the
workflow's own `script:` block out of the file and runs it under node
with stubbed `context`, `core` and `github`, so it tests the shipped
code, not a copy. On main it is red on the prototype titles (`THROW:
l.startsWith is not a function`), on the double backport (only
`area/release`), on the newly mapped scopes and on the area-name scopes.
It also checks that every map target exists in `labels.yml` and that no
key is written twice. The suite fails rather than skips when node is
missing. The older node-based suites skip, so this is the first one that
needs node on the unit-test runner.

Some things stay out of this PR. The scope table in
`docs/agents/contributing.md` is a summary. It has no row for `area/dx`
and does not mention that an area's own name works as a scope.
Separately, the singular `migration` still maps to `area/virtualization`
as it did before, while the new `migrations` maps to `area/platform`.

A backport PR whose scope is unmapped still never gets
`area/uncategorized`, even with a single prefix. The `area/release`
added by the strip satisfies the fallback check. This behaviour is the
same on main and is not changed here.

The test that checks map targets against `labels.yml` passes by itself
if the map literal cannot be extracted. The duplicate-key and area-name
tests go red in that case, so the suite as a whole still fails.

Closes #4125, closes #4124, closes #4122

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): the PR labeler no longer crashes on titles scoped like Object.prototype members, handles nested backport prefixes, and maps more recurring scopes, including a new area/dx.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `area/dx` label for issues and pull requests related to
developer tooling.
* **Improvements**
* Pull requests can now be categorized using a broader set of scope
aliases.
* Repeated leading backport prefixes are recognized and route the pull
request to the release area.
* Label mapping now avoids treating inherited property names as valid
mappings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:06:47 +03:00
IvanHunters 02fbe76418 fix(ci): close three blind spots in the e2e test selectors (#4658)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The e2e selectors miss Chainsaw suites that have no `chainsaw-test.yaml`
or `.yml` directly in their directory. Chainsaw v0.2.15 also runs a
directory that holds only numbered step files such as `01-install.yaml`,
and its discovery walks every subdirectory of the path the runner hands
it. A suite in either layout ran only when nothing was selected. No
narrowed selection named it, the full-suite list left it out, and
`select-install.sh --validate` never asked for its mapping. Now
`hack/select-e2e.sh` and `hack/select-install.sh` list a top-level
directory as a suite when Chainsaw would load a test from it or from
anywhere below it. No suite in the current tree uses either layout, so
today's suite list stays the same.

A file placed directly in `examples/backups/` (a shared `common.sh`, for
example) selected nothing, because it fell through to the inert
`examples/` rule. It now escalates to the full suite, same as
`examples/backups/_lib/`. `README.md` there stays inert through the
`*.md` rule.

The `test-chainsaw` comment in `packages/core/testing/Makefile` said
Chainsaw discovers every `chainsaw-test.yaml`, which was too narrow. I
fixed it, and the matching sentences in `docs/agents/e2e-testing.md`
too.

Every behaviour change has a bats test that fails against the scripts on
main.

Known gaps I left for a follow-up, none of them a wrong answer today:

- The `full_suite_list` helper in `hack/select-e2e_test.bats` repeats
the selector's `find`/`sed`, so a bug in that shared pattern would pass
every full-suite assertion. The step-file and nested tests compare
against literal lists, which covers the new layouts.
- The selectors are a little wider than Chainsaw: a broken
`chainsaw-test.y*ml` symlink, or one with no Test in it, still lists its
suite. That errs toward running more.
- A few test comments describe what used to happen rather than why the
rule exists.

Closes #4630

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ci): the e2e test selectors now see Chainsaw suites made of numbered step files or nested below a suite directory, and run the full suite for an edit to a file placed directly in examples/backups/.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* E2E suite discovery recognizes supported manifest and numbered-step
files at any depth and maps them to the top-level suite.
* Changes to files directly under `examples/backups/` trigger full-suite
selection.
* Suite selection reports an error if discovery fails or finds no
suites.
* **Documentation**
* Clarified supported suite layouts, disabled suite-file handling,
discovery and escalation rules, and per-path coverage guidance.
* **Tests**
* Added coverage for nested suites, numbered steps, unmapped suite
layouts, and backup-path selection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:06:18 +03:00
IvanHunters 9485f9118b test(tests): make negated BATS assertions able to fail (#3790)
## What this PR does

A hundred and twelve assertions across twelve BATS suites begin with
`!`. A hundred and seven of them pass no matter what they find. POSIX
exempts a command whose return value is inverted with `!` from `set -e`,
so the failing status is dropped where it is produced, and
`hack/cozytest.sh` closes each test function with `return 0`, so it
never resurfaces as the test's own status either. The other five are
followed by `|| { ... exit 1; }`, which reads the inverted status and
does work; they are rewritten too, because telling the two apart needs
the tail of every statement parsed, and a guard that guesses wrong there
is green on a real one.

Switching to the `bats` binary does not fix the hundred and seven. Its
ERR trap carries the identical exemption: `bash -c 'set -eE; trap "echo
TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap and
exits zero. The only negated assertion `bats` reports is one standing
last in the test body, where the inverted status happens to be what the
function returns. Two of the hundred and seven stood last; the rest did
not.

What was lost is specific rather than diffuse. `!` is how you spell
"this must not appear", and an absence is what no other assertion in a
test covers. The affected suites assert that a migration ran nothing
destructive, that no image was copied, that a flag was not passed, that
a backup cleanup never flushed the master, that an object does not
exist. On that half of their contract they held for any input.

Each site is now written as `if cmd; then echo "FAIL: ..."; false; fi`,
the form already used in the files where this had been noticed one at a
time, and each carries a message naming what must not have happened.
`hack/bats-no-negated-assert.bats` scans every `hack/**/*.bats` and
refuses the old form, so a new one cannot be added silently. Inverted
conditions (`if !`, `while !`, `until !`) are untouched, because `set
-e` is not what decides a condition. `[ ! -f x ]` is untouched for a
different reason and remains an ordinary failing assertion: the `!`
there is an operator of `test`, not negation of a command, so nothing
exempts its status.

The guard pins its own premise instead of describing it, on both sides.
One test asserts the runner still passes a bare negated assertion; a
second, that neither `set -e` nor the ERR trap sees an inverted status;
a third, that the prescribed `if cmd; then ...; false; fi` does fail,
under both runners, and *at* the form rather than somewhere before it,
since a run that died early would satisfy a bare status check and pin
nothing.

Every rewrite here rests on the third test. If `false` inside a `then`
body ever stopped propagating, the first test would still pass, because
a negated assertion would still be vacuous, while every converted
absence went silently green. A ban that prescribes a replacement has to
test the replacement too.

Every claim the guard makes is pinned by removing the thing it claims.
The recursion is exercised *through* the audit function against a
fixture in a subdirectory, not through a second copy of its `find`
expression: a copy keeps passing while the function loses its recursion,
and an audit that returns empty reads as a clean tree. Recursion matters
because `BATS_UNIT_FILES` is a non-recursive `wildcard`, so `make
unit-tests` never runs a suite placed in a subdirectory.
`hack/e2e-apps/` held such suites until they moved to Chainsaw and is
gone now, which is why the test pins against a fixture and not a real
directory.

The rule's surface was checked in the other direction too. The shape
appears in no `hack/**/*.sh` script as an assertion: the one hit,
`hack/testdata/install-wait/call-shapes.sh`, is fixture text that
`hack/run-kubernetes-install-wait_test.bats` parses, not a command
anything runs. It appears in no Chainsaw `script:` step either, so
`.bats` is the whole surface rather than the part that was convenient to
guard.

The scan is lexical, so its header names the statement boundaries it
owns rather than implying it understands shell: `;` `&` `&&` `||` `{`
`(`, the `)` closing a case pattern, and the words `then` `do` `else`,
each measured against the runner before being added. A keyword may
legally be preceded only by whitespace or by `;`, since every other
separator in front of `then`, `do` or `else` is a shell syntax error,
checked with `sh -n` rather than recalled. The scan therefore admits
exactly those two, and that part of the class is closed rather than
patched one spelling at a time. All three keywords are vacuous after
either spelling, so missing one is a silent miss, the single direction
this scan must not err in, given that its whole design trades noise for
safety. For the same reason it takes `!(cmd)` with no space before the
subshell, which is vacuous on dash and bash alike, and reads through a
`time` or `time -p` prefix, which is vacuous under `bats` and under
cozytest wherever `/bin/sh` is bash.

Five over-counts remain and are deliberate, each named in the header: `(
! cmd )` does propagate its inverted status and is reported anyway; so
would a `!` opening a statement after a redirection `&`; `x=$(true || !
false)` is reported although the `||` reads the inverted status; a
condition continued by a trailing `&&` instead of a backslash is
reported, because the strip only recognises a condition whose end it can
see; and `time ! cmd` is reported although on a dash `/bin/sh`, which
has no `time` keyword, it runs an external `time` that fails. Each costs
one rewritten line, where the opposite error costs a silent pass. They
are enumerated rather than engineered away on purpose, because a removed
error leaves no trace while a listed one tells the next reader what they
are looking at. The scan also errs *quiet* in at least two places, named
here rather than in the header. A file cut off midway through a line
continuation loses its last buffered statement, but that input is not a
runnable test, since `sh -n` rejects it with `unexpected end of file`.
And `time -- ! cmd`, which bash accepts, is vacuous under `bats` and not
peeled by the scan; nothing in the tree uses it, and I stopped adding
spellings there rather than chase every prefix bash allows. The test
naming the shapes claims only what its fixture holds. A lexical scan
over shell cannot promise it has enumerated the language, and a name
that promised it would be false whether or not a gap existed today.

The branch name is narrower than the subject and I am leaving it as it
is rather than churn it. This is not a defect in `hack/cozytest.sh` that
moving to the `bats` binary would fix. It is a POSIX rule both runners
inherit, and moving to `bats` would close two of the hundred and seven
sites.

One suite went red for real once its assertions could fail.
`seaweedfs-guard-parity` forbids four mutable discriminators by grepping
each chart, and both charts carry a template comment block arguing why
those signals were rejected, so the grep matched the argument and would
have reported a chart for documenting the rule it follows. The charts
are correct; the assertion was not. It now strips `{{/* ... */}}` before
matching. Two details there are load-bearing and neither is obvious. The
strip removes the comment *span*, not the line that opens it: dropping
the line is shorter and silently loses a discriminator sharing a line
with a comment, which is a chart classifying on a forbidden signal and a
test reporting that it does not. And the strip reports when the file
ended inside a comment, and the checks refuse to read its output when it
did, because every `{{/*` is read as an opener, so one inside a quoted
string (these charts carry long `fail (printf ...)` messages) would
swallow the rest of the file and leave the checks reading nothing.

One commit widens six comments that explained this trap as applying to a
"negated pipeline". It is not about pipelines, and the narrower wording
invites the conclusion that the other shapes are safe. Comment text
only; each of those files already uses the working form.

The last commit makes the one absence check in
`hack/e2e-install-cozystack.bats` that already used the `if` form tell a
missing object from a failed read. It asserted that the version
configmap was gone on `kubectl get`'s exit status alone, and that status
is 1 both for a NotFound and for an api-server the test cannot reach.

### Verification

Every unit suite this branch touches was run through `hack/cozytest.sh`
on the current base and is green. `migration-seaweedfs-db-adopt` passes
15 of 15 on macOS only when its temp directory sits under `$HOME`. Its
container cannot write into a bind mount under `/var/folders`, and
`/usr/bin/mktemp -d` ignores `TMPDIR`, so the run needs GNU `mktemp`
first on `PATH` as well. Measured on Linux before the rebase, it passed
15 of 15 as is.

The six sites that reached main after this branch was first written, in
`redis-backup-strategy` and `proxmox-e2e-fixture_test`, were each shown
to be able to fail: with the condition inverted, the suite goes red at
that assertion and prints its own `FAIL:` message. The nine sites in
`migration-seaweedfs-db-adopt` carry a two-sided check as well: with the
pattern swapped for one the same test proves is present, the rewritten
form fails the suite and the original `!` form passes it.

The guard itself is green under cozytest with bash and with dash as
`/bin/sh`, under the `bats` binary, and with mawk, gawk and BSD awk.
Each branch of its separator class and of the `time` and `!(cmd)`
handling was removed in turn, and each removal turns the shapes test
red.

`e2e-install-cozystack.bats` is an e2e suite and runs only in the E2E
workflow, so its one changed absence check is not exercised locally.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff file by file. The change touches
eighteen existing BATS files, adds one, and adds a convention to
`docs/agents/e2e-testing.md`. It does not touch
`hack/e2e-prepare-cluster.bats`, which is the node contract
`ansible-cozystack` and `talm` restate; it moves and renames nothing
under `hack/` and changes no make target's behaviour, which is what
`ccp` gates on; and it touches no package, schema, enum, default, CRD,
label, annotation or metric.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```

Closes #3565
2026-10-05 14:05:42 +03:00
IvanHunters 94abf6940b feat(kubernetes): let kubevirt-csi detach without the HotplugVolumes gate (#4451)
## What this PR does

This lets kubevirt-csi detach volumes without the deprecated KubeVirt
`HotplugVolumes` gate. The default stays as it is, so nothing changes at
runtime yet. Part of #4032.

The driver got a switch upstream for exactly this, and I bumped to it.
kubevirt/csi-driver#382 added `--enable-vmi-hotplug-fallback`, merged as
cb4770e4e475. With it set to false, the driver no longer calls the
VMI-level `removevolume` on a VMI owned by a VM. virt-api rejects that
call once `HotplugVolumes` is off.

Our wrapper now passes the flag to upstream and no longer has its own
hot-unplug path. `unpublishHotplugVolume` was a backport of
kubevirt/csi-driver#184, and the previous pin already contained that
change. Upstream detaches from the VMI when the VM is gone, then waits
for the VMI status to drop the volume before it returns. So the
wrapper's re-check never found the volume attached. Adding the ownership
check there would have been dead code.

Closes #4450

The chart leaves the flag at its default of true and does not render it.
The driver image in the chart is stamped only at release preparation, so
between releases main runs the previous release's binary, which does not
know the flag. Go's `flag` package exits on an unknown flag. Rendering
`=true` would crash-loop the CSI controller for no effect. The right
value depends on the KubeVirt CR, not on the tenant, so the comment next
to `HotplugVolumes` in `kubevirt-cr.yaml` points at the controller
Deployment. The follow-up PR drops the gate and renders the flag as
false.

Nothing changes on the node side. Main already pins f5de10d39556, and
cb4770e4e475 is that commit plus kubevirt/csi-driver#382.

A Go unit test runs an unpublish through the wrapper during a teardown,
with the flag on, off with a VM-owned VMI, and off with an orphaned VMI.
It checks that the flag reaches upstream and that only upstream makes
the VMI-level call. Upstream's own tests cover the ownership rule. A
second test holds that an NFS volume only releases the node from its
CiliumNetworkPolicy and never reaches the upstream hot-unplug. A
helm-unittest pins that the chart renders no such argument. I haven't
built the image or run e2e. Volume attach and detach on a tenant cluster
in e2e is the real check.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): kubevirt-csi supports --enable-vmi-hotplug-fallback, so it can detach volumes without the KubeVirt HotplugVolumes feature gate. The flag stays true for now and behaviour is unchanged.
```
2026-10-05 10:46:12 +03:00
Aleksei Sviridkin 2df578bffd test(e2e): read the opensearch release history through the pinned name
The opensearch suite wrote the history jsonpath out as a literal, so
the contract test that runs HELMRELEASE_HISTORY_JSONPATH against the
upstream Flux type never saw the expression this suite hands kubectl.
A field rename upstream would have reached it only as an empty read.

The suite now takes the shared expression, and the contract test reads
every Chainsaw suite that calls the guard, so a suite copying the
literal, or assigning the shared name after sourcing the library, fails
in the unit lane instead of going unchecked. The read is matched inside
the script step that calls the guard, because every step runs in a
shell of its own and a read in another step never reaches the call.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 11:39:34 +03:00
Aleksei Sviridkin 091b8549c2 build(images): run the tests of the packages the patches touch (#4326)
## What this PR does

Twelve first-party images that are built from an upstream tree with
patches on top now run `go test` on the packages those patches touch.
The step sits in the builder stage, right after the patches are applied.
Before this, ten of them ran `go build` only, and `go build` never
compiles a `_test.go`, so a test carried inside a patch was proven once
by hand and never again. The other two, opensearch-operator and
kube-ovn, ran a `go test` line written by hand, which a patch reaching a
new directory would slip past; the loop replaces both lines.

The package list is read from the patches themselves with `git apply
--numstat`, so a new patch is covered without an edit to the Dockerfile.
Each package is tested from its own directory, which finds nested
modules: in objectstorage, `internal/` is a module of its own and the
root module cannot name `./internal/runtime`. An empty list runs
nothing. The list is read into a variable before the loop, because a
`for` loop drops the exit status of a command substitution, and a patch
that git cannot read would otherwise give an empty list and a green
step.

A few images need more than the plain loop:

- opensearch-operator skips `controllers/`. A patch touches it, but its
suite starts an envtest control plane from etcd and kube-apiserver
binaries the builder does not have. The loop runs with `-race` and
replaces the hand-picked `go test -race` line, which also covered
`pkg/reconcilers/util`, a package no patch touches.
- kube-ovn drops its `go test -run '^TestVMIMigrationSchedulingRetry$'`
line. The test it named is copied into `pkg/controller`, which a patch
touches, so the loop runs it along with the rest of that package.
- redis-operator also runs `go vet -tags integration` in each touched
directory. Two of the files its patches edit are upstream integration
tests behind that tag, and `go test` compiles none of them.
- cluster-autoscaler and kubevirt-cloud-provider export `GOOS` and
`GOARCH` for the target in the whole builder stage, so their tests unset
both. A local `make image` builds amd64 and arm64 by default, and a test
binary for the other architecture runs on the build host only under
binfmt emulation. CI is not affected either way: the jobs that build
package images each pin one `PLATFORM` that matches the runner's
architecture.

The first run of the step on cluster-autoscaler found the carried
`fix-downscale.diff` out of step with the upstream tests it leaves in
place. The patch removes the `maxSize` refusal from `SetSize`, and one
case of `TestNodeGroupIncreaseSizeErrors` still expected that refusal.
The patch now drops that case and carries
`TestNodeGroupDeleteNodesAboveMaxSize` instead. That test covers the
downscale the patch is named for: `DeleteNodes` lowers a group through
`SetSize`, so with the refusal in place a group standing above its max
size could not shed a node. Against the unpatched `SetSize` the new test
fails. The image's behaviour does not change, and scale-up stays bounded
by the core scale-up orchestrator, which skips a node group already at
its max size.

CI builds all twelve images, and the step reports `ok` or `no test
files` for every package it tests. Locally, the redis-operator,
opensearch-operator and kube-ovn builder stages pass. The
cluster-autoscaler package was checked from the
`cluster-autoscaler-1.32.0` tag tarball, because the Dockerfile's full
clone of kubernetes/autoscaler does not survive my network. As negative
controls, an undefined call added to a redis integration test fails `go
vet -tags integration` while `go test` still passes, and the new
autoscaler test fails without the patch.

Left out on purpose, and the Dockerfile says so: `multus`. Its only
touched package, `pkg/multus`, creates network namespaces in its tests,
which a build container cannot do (`mount --make-rshared /var/run/netns:
operation not permitted`). `cilium` (C, no compile step) and
`piraeus-server` (Java) are out of scope.

Closes #4318

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: Dockerfiles under
`packages/*/images/` and one patch file. No values key, schema, make
target, `hack/` file or documented workflow changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```
2026-10-03 04:56:00 +03:00
Aleksei Sviridkin f32e0693c1 ci(dashboard): run the console's eslint in the UI test job
The console defines a lint script next to typecheck and test, but no
workflow ran it, so new findings landed unseen next to the old ones.
The backlog is cleared down to three react-hooks/exhaustive-deps
warnings. eslint exits zero on warnings alone, so the lint script caps
them at that count: the step gates from its first run, and a new
warning fails it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:09 +03:00
Aleksei Sviridkin 4d3796e41d refactor(dashboard): mark the deliberate setState calls inside effects
Each of these effects re-syncs local state to something that changes
outside the component. The rule's suggested rewrite derives the state
during render instead, which changes when the new value is committed
and what the first render shows, so each site carries a scoped disable
with its reason rather than a rewrite.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:08 +03:00
Aleksei Sviridkin be9b148e6c refactor(dashboard): move non-component exports out of component files
Vite's fast refresh can only hot-swap a module that exports nothing but
components; a file that also exports a hook, a helper or a registry
object forces a full page reload on every edit to it. Hooks, context
objects and the RJSF registry now live in their own modules, and the
components stay where they were.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:08 +03:00
Aleksei Sviridkin f1281e6176 refactor(dashboard): drop the unused ad binding in OverviewTab
The tab never reads it; the prop stays in the interface because the
detail page passes the same props to every tab.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:07 +03:00
Aleksei Sviridkin 8298c43e05 refactor(dashboard): type command palette instances as K8sResource
client.list was called without a type argument, so its items came back
as unknown and were cast to any to read metadata.name.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:07 +03:00
Aleksei Sviridkin 05ff14dbc1 refactor(dashboard): type the RJSF templates and source fields instead of any
RJSF's generic defaults are any, and the templates restated them, so
every schema read went untyped. The templates now default to unknown
and FormContextType, and the source widgets read option schemas
through a small local interface.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:07 +03:00
Aleksei Sviridkin def917a5d5 refactor(dashboard): type the backup form pages instead of any
The console's own rules forbid any, and eslint flags every one of these.
The create pages only read a handful of reference fields, so a local
interface per page covers them; the rest of the spec still passes
through from RJSF untouched.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:06 +03:00
Aleksei Sviridkin 534f8f1db9 docs(backups): stop presenting admin-only demo steps as tenant work
The kafka and nats walkthroughs put steps 03, 04, 06, 07 and cleanup
under the tenant persona, and the scenario docs and helper comments
had the tenant create the backup Secret. The scripts read, create and
delete Secrets, wait on HelmReleases, COSI objects and operator
resources, exec into a CLI Pod, read Pod logs, and delete the Backup
and the cluster-scoped BackupClass and strategy. The tenant roles grant
none of that, so a tenant following the docs hit Forbidden at the
first read. The tables now say admin and name the part a tenant admin can
do on its own, and the scenario docs say the same.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:02 +03:00
Aleksei Sviridkin 874e1e44b7 fix(backups): run the kafka-metadata demo CLI Pod as restricted
The kafka-metadata walkthrough started its long-lived CLI Pod with a
bare kubectl run, so the Pod carried no securityContext and a namespace
enforcing the restricted Pod Security level would reject it. The kafka
topic-data walkthrough already layers the restricted context onto its
CLI Pod; the metadata demo now applies the same JSON patch. A running
CLI Pod left by an earlier run without that context is replaced rather
than reused, since a Pod's securityContext cannot change in place.

The new bats cases read the overrides kubectl run receives and fail
unless both the Pod-level and the container-level fields are present
and the patch is sent as --override-type=json, and check that an owned
Pod without the restricted context is recreated.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:43:02 +03:00
Aleksei Sviridkin cc731e3105 test(e2e): keep redis-tls failures legible under the op ceiling
The CA-projection step caps each kubectl at 30s, but more capped reads
run than its 5m op can cover, so a degraded apiserver could spend the
ceiling and leave nothing but "signal: killed". The reads now share a
step budget held under the op, and a read that hits its bound or finds
the budget spent says which read it was.

Two exec checks reported the 15s redis-cli cap for any failure, sending
a reader after a slow redis when the exec was refused or the container
was missing. They now report the exit status and what the exec
printed. The step also runs under bash with pipefail: under dash a
failed Secret read took base64's zero status and surfaced as a wrong
source name. The private-key check reads the projection already
fetched, since a second read that failed made it pass.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:42:53 +03:00
Aleksei Sviridkin a057d738d1 test(e2e): give the gateway listener-set op room for its own waits
The step's polls can wait 660s under a 6m op, and its cleanup trap
deletes two tenants with no bound on the wait. When the op ceiling
fires first, Chainsaw kills the shell: the poll that was running never
reports, the trap never runs, and both tenants stay behind for every
later suite.

The deletes keep waiting, child before parent, since a parent deleted
while its child is still uninstalling wedges its own cleanup. Each
delete runs under an outer timeout with a kill grace, because kubectl's
--timeout bounds only the wait after the request and not discovery or
a stalled request. The parent is deleted only once the child is gone.
A failed tenant delete fails the step with kubectl's error, its exit
status and the tenant left behind, rather than passing with the leak.
The op is sized above the polls plus those bounds, and a unit guard
holds it to that sum and the cleanup to that order.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:42:53 +03:00
Aleksei Sviridkin b1085782c7 test(e2e): fail the isp-hosted no-Cilium check on any cilium.io CRD
The check looked for one CRD by name, so a cluster carrying only part
of Cilium's CRDs passed as a cluster without Cilium, and a failed
lookup read as an absent CRD and passed too. It now lists the CRD
groups once, fails if that list cannot be read, and fails on any CRD
in cilium.io.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:42:38 +03:00
Aleksei Sviridkin bd504e0b4e ci(e2e): run the isp-hosted install daily against the nightly
The lane installs the latest GHCR nightly on a kind cluster once a day,
after the nightly has published. It takes no pull request trigger: a
run would install the nightly's images, not the pull request's, and
report on code the pull request did not change.

The unit test holds the lane's wiring in place, since a lane that
quietly installed another variant would test something other than
its name says. The sandbox headroom check reads any workflow touching
packages/core/testing as one that boots the QEMU guests; this lane
starts only the sandbox container, so it is exempted by name, after
the job match, and is sized like the others if it ever starts calling
the guest targets.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:42:38 +03:00
Aleksei Sviridkin 3e36b35795 test(e2e): add an install suite for the isp-hosted variant on kind
isp-hosted shipped unable to converge and nothing noticed, because no
lane installs it: every e2e lane runs isp-full on Talos, where
Cozystack provides the CNI and storage itself. This suite installs it
the way it is meant to run, through the hosted installer variant on a
cluster that brings its own CNI, kube-proxy and default StorageClass,
and passes only when every HelmRelease is Ready and the variant really
left out the data plane it delegates to the host.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:42:38 +03:00
IvanHunters 309875ab62 test(controller): seed the DataVolume informer for controller-runtime 0.25 (#4729)
## What this PR does

`main` does not build `internal/controller` tests, so `Unit & controller
tests` is red on every open PR. #4380 added DataVolume watch tests that
set `FakeInformer.Synced` and lock `SyncedLock`. #4473 then moved to
controller-runtime v0.25.1, where `FakeInformer` keeps its sync state in
a private channel and exposes only `Synced()` and `HasSynced()`. Each PR
was green on its own base.

The tests now seed the DataVolume informer into
`FakeInformers.InformersByGVK` and call `Synced()` when they want it
synced. Seeding is needed because `FakeInformers` builds every informer
it creates itself already synced, so there is no way to get an unsynced
one from it otherwise. The unsynced cases still assert that no reader is
installed, so they would fail if the informer came up synced.

`go vet ./...` is clean and `go test ./internal/controller/` passes.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Improved automated checks for workload monitoring across different
data synchronization states. These changes strengthen validation of
watch behavior without changing functionality visible to users.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-03 01:36:33 +03:00
Aleksei Sviridkin 2eeaa59055 test(controller): seed the DataVolume informer for controller-runtime 0.25
controller-runtime 0.25 made FakeInformer's sync state a private
channel, so the DataVolume watch tests no longer compile: they toggled
the exported Synced field and its lock. FakeInformers also builds every
informer it creates already synced, so an unsynced informer has to be
seeded into InformersByGVK and marked synced with Synced() later.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:29:03 +03:00
Aleksei Sviridkin 89a1fe52d9 build(multus): say why the image build runs no tests of the patched package
The other patched Go images now test the packages their patches touch
in the builder stage. The one package the multus patch touches,
pkg/multus, creates network namespaces in its tests, which a build
container is not allowed to do, so that image keeps building without a
test step. The Dockerfile records the reason, so the absence reads as a
decision and not as an omission.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:02 +03:00
Aleksei Sviridkin f51be95430 build(objectstorage-controller): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Each package is tested from its own directory rather than as a ./...
path of the root module, because internal/ is a module of its own and
the root module cannot name ./internal/runtime.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:02 +03:00
Aleksei Sviridkin b2a5f12e73 build(kubernetes): run the tests of the packages the patches touch
The cluster-autoscaler and kubevirt-cloud-provider image builds ran go
build only, and go build never compiles a _test.go, so nothing ran the
tests of the code the patches change. Run go test in the builder stage
on the packages the patches touch. The list is read from the patches
themselves, so a new patch is covered without an edit to the Dockerfile.

Both builder stages export GOOS and GOARCH for the target, so the tests
unset them: a local make image builds amd64 and arm64 by default, and a
test binary for the other architecture runs on the build host only
under binfmt emulation.

The first run of that step on cluster-autoscaler found the carried
downscale patch out of step with the upstream tests it leaves in place:
SetSize no longer refuses a size above maxSize, while one case of
TestNodeGroupIncreaseSizeErrors still expected that refusal. The patch
now drops that case, and carries a test for the downscale it is named
for instead. DeleteNodes lowers a group one node at a time through
SetSize, so with the refusal in place a group standing above its max
size could not shed a node at all. The new test deletes a node from
such a group and expects the replica count to drop; against the
unpatched SetSize it fails with the refusal the patch removes. Scale-up
past maxSize stays bounded by the core scale-up orchestrator, which
skips a node group already at MaxSize.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:01 +03:00
Aleksei Sviridkin 070d235d1c build(kube-ovn): run the tests of the packages the patches touch
The image build ran one test by name, the one shipped next to the
migration patch, and go build never compiles a _test.go, so the rest of
the tests in the packages the patches change never ran.

Run go test in the builder stage on the packages the patches touch. The
list is read from the patches themselves, so a new patch is covered
without an edit to the Dockerfile. It replaces the hand-picked go test
line: the shipped test lives in pkg/controller, which a patch touches,
so the loop runs it with the rest of that package.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:01 +03:00
Aleksei Sviridkin 59c3174b44 build(kamaji): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:00 +03:00
Aleksei Sviridkin 0b00b69267 build(capi-providers-cpprovider): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:00 +03:00