Commit Graph
7281 Commits
Author SHA1 Message Date
Mattia Eleuteri 5956b459fb fix(backups): give each Postgres instance its own default WAL prefix
The default CNPG strategy archived a Postgres app under
<namespace>-<name>, inside a destinationPath that is also keyed by
namespace and name, and the driver fell back to the Cluster name when a
strategy set no serverName. Deleting the app does not empty the bucket,
so an app recreated under the same name archived its WALs and base
backups into its predecessor's stream: both start on timeline 1, and a
later restore of the old archive could pick the new cluster's base
backup or WAL segments.

barman-cloud refuses a non-empty archive only while
.check-empty-wal-archive exists in PGDATA. CNPG removes that file as
soon as ContinuousArchiving turns True, and with no plugin attached
archive_command is a successful no-op, so the file is gone minutes
after initdb, before the first BackupJob attaches the plugin.

Suffix the default serverName with the application UID, in the
strategy template and in the driver's fallback. The template guards
the UID with `with`, so an object without one renders the previous name
rather than "<no value>". A Cluster that already has the plugin keeps
its serverName, so only clusters never backed up take the new prefix,
and restores keep reading the source prefix from driverMetadata.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-10-06 16:35:42 +03:00
Aleksei Sviridkin 9734a16692 feat(postgres): add a postgis image flavor (#4587)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds a `flavor: postgresql | postgis` enum to the `postgres` app.
`postgis` runs the cluster on the CloudNativePG PostGIS operand image,
so tenants can enable `postgis`, `postgis_raster`, `postgis_topology`,
`postgis_sfcgal`, `postgis_tiger_geocoder` and `address_standardizer`
through the existing `databases.<name>.extensions`. The default stays
`postgresql` and renders exactly the image it rendered before this
change.

This picks up #2672 by @matthieu-robin, which the stale bot closed after
a rebase request, with these changes:

- The postgis tags in #2672 (`17.7-3.5`, `16.11-3.5`, ...) do not exist
in `ghcr.io/cloudnative-pg/postgis`; the registry answers 404, so the
flavor would have ended in `ImagePullBackOff`. The map now uses tags
that exist, `<minor>-<postgis>-standard-trixie`, on the same PostgreSQL
minor as `files/versions.yaml`.
- v18 is supported: PostGIS images for 18 now exist upstream.
- `standard` rather than `system`: the chart archives through the
barman-cloud plugin since the backup rework, so the operand no longer
needs the barman binaries of the deprecated `system` variant.
- `hack/update-versions.sh` derives the postgis map from the minors it
picks, and `hack/check-postgres-postgis-versions.bats` fails if the two
maps drift to different minors or away from `standard-trixie`.
- The chart refuses a flavor change on a live `Cluster`. The default
images are built on Debian bullseye (glibc 2.31) and the postgis ones on
trixie (glibc 2.41): CNPG would roll the swap out as a plain image
update, under an initialised data directory, which silently invalidates
every index on collatable text, and the postgis-to-postgresql direction
also drops the libraries behind every PostGIS object. Only the image
name is compared, so minor bumps and registry mirrors keep passing.
- `pgrouting` is not claimed: it is absent from the `18.1-3.6.1` build
the chart pins and only appears in later postgis builds.

A general mechanism (CNPG ImageVolume extensions) was considered and
left out: it needs PostgreSQL 18 and Kubernetes 1.35 or the
`ImageVolume` gate, so it cannot serve v13-v17 nor most management
clusters today, and PostGIS is the only extension CloudNativePG ships as
a full operand image. No other extension request exists in this
repository.

Testing:

- `helm unittest packages/apps/postgres`: 145/145, including 6 new cases
in `tests/flavor_test.yaml` (default image, postgis on the Cluster and
on the init Job, the guard in both directions with a mocked live
Cluster, and a mirrored digest-pinned image taking a new minor).
Removing the guard makes exactly the two guard cases fail.
- `hack/cozytest.sh hack/check-postgres-postgis-versions.bats` passes,
and fails on a shifted minor or a `system` tag.
- `make -C packages/apps/postgres generate` leaves no diff.
- The rendered
`ghcr.io/cloudnative-pg/postgis:18.1-3.6.1-standard-trixie` was started
locally: `CREATE EXTENSION` for citext, postgis, postgis_raster and
postgis_topology succeeds and `ST_Buffer` answers. A three-instance
cluster on the `system` sibling of that image has been running PostGIS
in production since 2026-09-29.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#63 (issue: the
provider models every field by hand, and the default handling needs a
decision there)
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(postgres): add a `flavor` field accepting `postgresql` (default) or `postgis`. `postgis` bootstraps the cluster from the CloudNativePG PostGIS image, so PostGIS extensions can be enabled per database through `databases.<name>.extensions`. The flavor is fixed at creation: changing it on an existing cluster is refused, and so is restoring a backup into a cluster of the other flavor, whether through a RestoreJob or `bootstrap.enabled`; move data across flavors with a logical dump.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a `flavor` option for PostgreSQL clusters. The default
`postgresql` flavor remains available, while `postgis` selects an image
with PostGIS extensions.
  - PostGIS images are available for PostgreSQL versions 13–18.
- Cluster flavor is fixed at creation. To switch flavors, create a new
cluster and migrate data with a logical dump.
- **Bug Fixes**
- Restores are refused when the backup and target cluster use different
flavors.
- **Documentation**
- Updated PostgreSQL configuration guidance to describe flavor options
and limitations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 16:17:22 +03:00
Aleksei Sviridkin 0e99ec012a docs(observability): add a distributed tracing guide (#3772)
## What this PR does

Adds `docs/observability/distributed-tracing.md`, the operator and
tenant guide for distributed tracing (epic #3761, design
cozystack/community `design-proposals/distributed-tracing`).

The guide covers the per-tenant backend and collector (#3762, #3763):
the in-namespace OTLP endpoints, the standard `OTEL_*` variables to
point an instrumented workload at them, sampling, redaction and tenancy,
and a per-engine table of what each managed engine can emit itself.
Verification against the engines (#3766) showed that none of them pushes
OTLP spans server-side, so the guide's answer for every engine is
client-side instrumentation, with the ClickHouse span table as the one
partial exception.

It also covers shared-central tracing (#3775, with the per-tenant write
rate in #4648): hosting the store in tenant-root, opting a tenant in,
the shared disk and the write rate with its limits (counted before
sampling, a 3 MiB request cap, the allowance a pod start lets through,
at most 10 collector replicas), the trust boundary with the collector's
write-only and Grafana's read-only logins, the lifecycle windows on
enabling and disabling, and the order to stop hosting in. Those details
used to live in the `tracingCentral` field description, which #3775 cut
to a few sentences because it ships as field help in the dashboard.

Both #3775 and #4648 are merged, and every figure and refusal the guide
quotes was checked against the charts on `main`.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The guide is a file under `docs/` in this repository and changes no
package, values key or API, so no trigger in the downstream map applies.

### Release note

```release-note
docs(observability): add a distributed tracing guide covering the per-tenant collector, client-side instrumentation per managed engine, and shared-central tracing in tenant-root.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added a distributed-tracing guide covering tenant-local VictoriaTraces
storage, OTLP collectors, workload configuration, engine tracing
limitations, sampling, and optional attribute redaction.
* Documented shared-central tracing prerequisites, including tenant-root
hosting and tenant opt-in, plus write-rate, request-size, and replica
limits; credentials and network rules; tenant isolation and lifecycle
behavior; and conditions that stop hosting.
  * Clarified that cross-signal correlation is not configured.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 14:39:00 +03:00
Alexey Artamonov b3103af693 docs(observability): add a distributed tracing guide
Tracing reaches a workload only if it is instrumented and pointed at
the collector, and the managed engines push no OTLP spans of their
own, so operators and tenants need one place that says what the
platform provides and what is left to the application: the in-namespace
OTLP endpoints and the standard OTEL_* variables, sampling and
redaction, and what each engine can and cannot emit.

Shared-central tracing has more to it than the Tenant field description
can carry, since that text is shown as field help in the dashboard:
hosting the store in tenant-root, the shared disk and the per-tenant
write rate with its limits, the write-only and read-only logins and the
trust boundary, the windows on turning it on and off, and the order to
stop hosting in. The guide holds that detail.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 14:34:55 +03:00
Aleksei Sviridkin 11499c30e1 feat(monitoring): hold each shared-central tenant to a write rate (#4648)
## What this PR does

Closes the part of the distributed tracing design that #3775 left open:
the per-tenant collector rate-limits its tenant, so that a noisy or
hostile tenant cannot exhaust a shared backend (cozystack/community
`design-proposals/distributed-tracing`, Security and §2).

**Why the disk cap is not enough.** #3775 bounds every traces store by
disk usage, but VictoriaTraces always keeps the last two days, so a
tenant writing faster than the cap allows over two days fills the disk
every tenant on the shared store writes to. Only a per-tenant write rate
closes that.

**The collector is built here.** No upstream OpenTelemetry distribution
or processor limits throughput, and neither the open-source vmauth nor
VictoriaTraces limits a tenant's rate. Elastic's EDOT Collector
registers Elastic's `ratelimitprocessor` among its default components,
but it ships under the Elastic License 2.0 and carries far more
components than the chart uses. The collector is therefore built with
the OpenTelemetry Collector Builder from the upstream components the
chart already uses, at the same v0.136.0, plus that `ratelimitprocessor`
(Apache-2.0). Its v0.8.2 is the release built against collector
v0.135.0, the last one before v0.137.0, and it builds against v0.136.0
unchanged. The image is versioned by the collector release with a
cozystack respin counter (`v0.136.0-cozystack.0`), runs as the upstream
image's user, and is built with the other package images.

**How the rate is set and enforced.** The platform sets the rate with
`monitoring.tracingCentralTenantBytesPerSecond`, and it reaches every
tenant in `_cluster` through the `cozystack-values` Secret. A change
there is a change in the values every monitoring HelmRelease reads, so
Flux upgrades each of them; a rate read with `lookup` at render time
would never reach a tenant whose release is already in sync. The
platform writes the key only for a positive rate, because every release
in every tenant reads that Secret, and refuses to render on a value that
is not a whole number of bytes per second in plain decimal digits, up to
1 TiB/s ("1Mi", 1.5, "0100"), rather than reading it as the default.
Without it, or on a value that is not a positive number, the collector
holds the tenant to the default of about 97 KiB/s, at which one tenant
writes at most the disk cap of a default 10Gi store over two days,
counted uncompressed; the store keeps spans compressed, so they take
less than that. A larger store needs the rate raised with it. Each
central tenant's collector splits the rate across its replicas and
refuses data above it before the batch processor, so the application
sees the refusal, and right after the memory limiter, ahead of the
sampler, which adds a tracestate to every span it keeps, and of the
tenant attribute added to every resource, so it counts a request as the
receiver admitted it. The rate is therefore counted before sampling, and
the store keeps only the sampled share of it. Over OTLP/HTTP it gets a
429, which OTLP exporters retry. Over OTLP/gRPC the limiter answers
RESOURCE_EXHAUSTED without RetryInfo, which the OTLP specification makes
final, so a gRPC exporter drops those spans; the Tenant's
`tracingCentral`, the Monitoring `tracingCollector` and the platform key
tell central tenants to export over HTTP. The limiter sees the requests
the receiver admits, so in central mode the receiver caps a request at 3
MiB over both OTLP/HTTP and gRPC, decompressed for HTTP. The limiter
counts a request as the collector re-encodes it, which can be larger
than it arrived (13% for a span carrying only its IDs, the smallest
valid one, and up to 1.4x for spans with empty-valued attributes), so
burst is at least 5 MiB, above 1.4 times the cap, since a request larger
than burst never passes. Each collector pod starts with a full bucket,
so a pod start lets through above the rate up to 5 MiB or one second of
the pod's share of the rate, whichever is larger, and the chart refuses
to render a central tenant's collector with more than 10 replicas.

**Only the collector can write.** The rate holds only if the collector's
login is the only way the tenant writes. Each central tenant now has two
VMUsers on its one account: the collector's, which may only write, and
its Grafana datasource's, which may only read, each with its own Secret.
With one login for both, any user the tenant admits to its Grafana, a
Viewer included, could POST through the datasource proxy to `../insert`,
which vmauth cleans before it picks a route, and write into the shared
store past the limiter. The tenant's Grafana pod may reach the vmauth
too, so its image is the platform's, like the collector's, and only the
`auth.ini` key is mounted from the Secret a tenant names in
`oidc.customConfig.secretRef`: an image of the tenant's choosing with
the collector's Secret mounted would otherwise write past the limiter as
well.

**The collector image is the platform's.** The apps API defaults a
Monitoring spec from its schema but does not prune keys the schema
leaves out, and the Monitoring chart hands its values to the inner chart
as they are, so a tenant could set `tracingCollector.image` and run any
collector it liked, one without the limiter included. That key is no
longer passed through.

**Trade-offs.** The limiter is a vendor component, not an OpenTelemetry
one, and Elastic marks it as in development; it moves with Elastic's
releases, so bumping the collector means picking the matching release of
it (v0.74.0 builds against collector v1.67.0), and collector v0.136.0 is
a year old. It logs each refused request at error level, so a tenant
held at its limit produces steady log lines in its own collector.
Several tenants writing at their limits together can still exceed the
cap within two days; the rate bounds each tenant, not their sum. The
limiter keeps its state in each pod's memory, so a tenant admin who
restarts its collector pods over and over gains, per pod start, up to 5
MiB or one second of the pod's share of the rate, whichever is larger;
closing that needs a limiter with shared state (the processor's
gubernator mode), which is out of scope here. A sender whose connection
stays on one replica gets that replica's share of the rate. In central
mode the receiver refuses a request larger than 3 MiB: over OTLP/HTTP
with a 400, which senders do not retry, where the receiver default is 20
MiB, and over gRPC, where the default is 4 MiB. Changing the rate
upgrades every release in every tenant once, since it travels in the
Secret they all read. Until CI builds the image,
`tracingCollector.image` in `values.yaml` still names the upstream
distribution, which has no rate limiter, so a central-mode collector
from an unbuilt tree does not start; CI and releases stamp the built
image, and the image repository in the build registry has to be public
before a cluster can pull it.

**Upgrade.** Collectors in both modes move to the built image, which
carries the same components at the same version; central collectors also
roll for the new processor. The replica cap fails the render of a
central tenant's collector with more than 10 replicas, while a collector
writing to the tenant's own store keeps any replica count; the collector
has shipped only in the v1.7.0 alphas. Each central tenant's existing
VMUser loses its read route, and a second VMUser and Secret appear for
Grafana, whose datasource moves to the new login in the same release, so
the tenant's trace reads may fail until vmauth loads the new user.
Nothing else changes for clusters that do not set the rate: the platform
writes no new key, so no other release upgrades.

**Testing.** helm-unittest covers the platform writing the rate into
`_cluster` only for a positive whole number and refusing a malformed
one, numbers Helm reads as float64 included, the collector's limiter
(the configured rate, the split across replicas, burst, the receiver's
request cap in central mode and not in local mode, the default when
unset and on a word, zero or a negative value, its place right after the
memory limiter, with and without redaction, its absence in local mode),
the replica cap for a central tenant's collector and its absence for a
local one, the image key no longer passing through, the two logins (the
collector's with only the write route, Grafana's with only the read
route, separate Secrets and passwords), the Grafana image not passing
through, and only `auth.ini` being mounted from a `secretRef` Secret for
a central tenant; every term was checked by mutation. On a test cluster,
with the chart's rendered VMAuth, VMUsers, VTCluster, Grafana datasource
and collector: with one login for both, a Grafana Viewer's POST to
`../insert` through the datasource proxy landed a span in the shared
store; with the two logins vmauth answered `400 missing route` for every
encoding of the traversal tried, reads went through, and the collector's
login could write but not read. With the built collector at 100%
sampling and the receiver capped at 4 MiB, a fresh pod accepted a
request just under that cap made of 53-byte spans and one made of
ID-only spans, refused a larger one with a 400, and answered 429 once
the bucket was drained; the 53-byte spans it accepted were read back
through the read-only login; the 3 MiB cap the chart renders has not
been run on a cluster. A pod mounting a Secret through the chart's
`secretRef` volume as rendered for a central tenant got only `auth.ini`,
and one naming the collector's Secret, which has no such key, did not
start. The design's Testing section also asks for a sustained span flood
against a disk-capped store to show that the oldest spans are evicted
and ingest is not blocked; there is no end-to-end lane for
shared-central tracing yet, so that flood test is a follow-up, like the
cross-tenant isolation test #3775 names. On a test cluster, under the
network policies the tenant chart renders, the built image validated the
rendered config in both modes and wrote and read through the shared
store as before; a tenant given a small limit got 200 for its first
OTLP/HTTP requests and 429 for the rest of a burst, with the processor
logging each refusal, while another tenant under the default rate got
200 for all of the same requests.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#729
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The new platform key `monitoring.tracingCentralTenantBytesPerSecond`
belongs in the hand-written `platform-package.md` reference on the
website, tracked in cozystack/website#729. The Monitoring schema and the
Tenant's `tracingCentral` change only descriptions; the provider models
neither, so it needs nothing. The central-mode section #3772 adds to the
tracing guide covers the rate.

### Release note

```release-note
feat(monitoring): hold each shared-central tracing tenant to a write rate. The platform sets the rate with monitoring.tracingCentralTenantBytesPerSecond, and each central tenant's collector refuses spans above it, which bounds what one tenant writes into the store every tenant shares. The traces collector is now built by cozystack from the upstream components plus Elastic's rate-limit processor, and its image can no longer be set from a Monitoring application. A central tenant's Grafana reads the shared store with a login that cannot write, and its image can no longer be set from a Monitoring application.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Central tracing now applies a configurable per-tenant write-rate limit
shared across collector replicas. The default is approximately 97 KiB
per second, and collector replica counts are limited to 10.
* Each starting replica can temporarily exceed its rate share by up to 5
MiB, or by one second of that share when it is larger. Tenants share
central storage, so combined writes can fill the store and cause older
traces to be evicted.

* **Documentation**
* OTLP/HTTP is recommended for central tracing: excess traffic is
retried over HTTP, while OTLP/gRPC traffic may be dropped.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 13:29:16 +03:00
Alexey Artamonov e63b9e439c fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and
burst was the receiver's 4 MiB cap. A request grows on its way to the
limiter: the sampler adds a tracestate to every span it keeps, and the
limiter counts a request as the collector re-encodes it, which writes
the empty value of every attribute the wire may omit, so a valid
request of spans with empty-valued attributes is counted 1.4x. A
request the receiver admitted near its cap was therefore refused on
every retry.

The limiter now runs right after memory_limiter, so it counts the
request before sampling. The receiver caps a request at 3 MiB and
burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would
also be what each pod start lets through above the rate. The rate is
counted before sampling, so the store keeps only the sampled share of
it. The texts say that a request above the cap gets a 400, which
senders do not retry, and that each collector rollout a tenant causes
lets that burst through again on every pod.

The replica bound that keeps that allowance small now applies only to
a central tenant's collector, whose render fails above ten replicas: a
collector writing to the tenant's own store has no rate, and the
replica counts it shipped with stay valid.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 65f0caae2b fix(monitoring): keep a central tenant's write login out of its Grafana
A tenant's Grafana pod may reach the shared traces store's vmauth, and
the Secret a tenant names in oidc.customConfig.secretRef was mounted
into it whole. The apps API does not prune keys the Monitoring schema
leaves out, so a tenant could also set grafana.image: an image of its
choosing, with the collector's Secret mounted, writes past the rate
limit the collector enforces.

Grafana's image is the platform's, like the collector's, and for a
central tenant only the auth.ini key Grafana reads is mounted from the
named Secret. Other tenants have no collector login to keep out and
keep the whole Secret, which an auth.ini reading a sibling key with
$__file{} relies on.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov cc51cac63d fix(monitoring): give a central tenant's Grafana a login that cannot write
The collector holds a central tenant to its write rate, but the one
login it writes with was also the login of the tenant's Grafana
datasource, and it could both read and write. vmauth cleans the request
path before it picks a route, so any user the tenant admits to its
Grafana, a Viewer included, could POST through the datasource proxy to
../insert and write into the shared store past the limiter.

Each central tenant now has two VMUsers on its one account: the
collector's, which may only write, and Grafana's, which may only read,
each with its own chart-owned Secret and password. A namespace name
holds no dot, so the reader's username never matches another tenant's.
On upgrade the datasource moves to the new login in the same release
that creates it, so reads may fail until vmauth loads the new user.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 3497855e15 fix(platform): read the tracing rate only as plain decimal digits
Sprig's int64 reads a string with a leading zero as octal and turns a
parse error into 0, so "0100" would reach the collectors as 64, and
"08" or digits beyond int64 would leave the default in place without
a word. A float beyond int64 converts differently on each CPU
architecture, and the collector adds its replica count to the rate
before splitting it.

A string rate must now be plain decimal digits, and any rate above
1 TiB/s fails the render; the collector also falls back to the default
on digits it cannot hold rather than start with a rate of zero, which
the processor refuses.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov a0aea139fc fix(platform): refuse a malformed shared-central tracing rate
A rate written as a quantity or a fraction ("1Mi", "100KiB", 1.5) was
read as zero, so the platform wrote no key and every collector kept
the default without a word. The platform render now fails on anything
but a whole number of bytes per second. Helm reads a number from
values as a float64, which prints a million as 1e+06, so a whole
number is checked as a number and written as int64 digits.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov fc0db49187 fix(monitoring): bound what a central tenant writes above its rate
Each collector pod starts with a full token bucket, and burst was at
least 64 MiB, the largest request vtinsert accepts. A tenant controls
when its collector pods start, through any Monitoring change or by
deleting them, and how many there are, so it could write tens of
mebibytes above its rate per pod start and gigabytes per rollout of
many replicas, which the two-day bound does not survive.

The limiter only sees requests the receiver admits, so in central mode
the receiver now caps them at 4 MiB, decompressed for HTTP, and burst
drops to that floor; a pod start lets at most 4 MiB through above the
rate. The replica count is capped at 10. The texts now say what the
default rate guarantees: one tenant writes at most a default store's
disk cap over two days counted uncompressed, rather than that it
cannot fill it, and they name the per-pod-start allowance and that a
sender pinned to one replica gets that replica's share.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov f8deb8e171 fix(platform): write the tracing rate only when it is positive
apps.yaml wrote the rate for any non-zero value, so a negative one
reached every tenant's cozystack-values Secret and upgraded every
release in every tenant, while the collectors ignored it and kept the
default.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 177b207a27 feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.

The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.

Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 32a6c71972 fix(monitoring): keep the collector image out of a tenant's reach
The apps API defaults a Monitoring spec from its schema but does not
prune keys the schema leaves out, and the Monitoring chart hands its
values to the inner chart as they are. A tenant could therefore set
tracingCollector.image and run any collector it liked in its
namespace; with a per-tenant rate limit in that collector, the same
key would let a tenant drop the limit. The image is the platform's,
so it is no longer passed through.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 474377e8ca build(monitoring): build the traces collector with a rate limiter
A shared-central traces store needs each tenant held to a write rate,
or one tenant can fill the disk every tenant shares. No upstream
OpenTelemetry distribution or processor limits throughput. Elastic's
EDOT Collector registers Elastic's ratelimitprocessor among its default
components, but it ships under the Elastic License 2.0 and carries far
more components than the chart uses.

The collector is therefore built here with the OpenTelemetry Collector
Builder from the upstream components the chart already uses, at the
same v0.136.0, plus that ratelimitprocessor (Apache-2.0). Its v0.8.2
is the release built against v0.135.0, the last before v0.137.0. The
image is versioned by the collector release with a cozystack respin
counter, runs as the upstream image's user, and is built with the
other package images.

The builder stage is alpine, which has no git, so a proxy error during
its go install would have had neither the direct fallback nor a retry;
the stage installs git. The module-fetch contract saw only go build, so
it now checks go install too, and takes only git as cover for a
versioned go install, which resolves outside the stage's module.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
myasnikovdaniil a5d12e78d4 fix(ci): reconcile three unit-test contracts broken by adjacent merges (#4751)
## What this PR does

Unit tests on main are red, and that blocks E2E on every open PR. Three
contracts broke the same way. In each case a check from one PR and a
line from another merged a few minutes apart on 2026-10-05, with no
textual conflict. Any one of them alone keeps the job red, so they are
fixed together.

The first is the pin on how a suite reads HelmRelease history. #4689
added it. It accepts `$(kubectl ...` and, since #3594, the bounded
`$(_cozy_guard_kubectl ...` wrapper. Nothing in front of kubectl is
allowed. The monitoring suite from #4678 bounds that read with
`$(timeout 30 kubectl ...`, and the bound is wanted. So the pin now
accepts an optional `timeout [-k N] N` in front of kubectl. The prefix
is spelled out on purpose. Any text would also let through `$(echo
kubectl ...)`, which names the shared expansion without running it, and
that form is now a refuse case. I checked it with mutations: the new
accept cases fail against the old pattern, and a pin that takes any
prefix fails a refuse case.

The second is the flux install in workflows. #4645 forbids running
fluxcd.io/install.sh and moved every workflow to the SHA-pinned
fluxcd/flux2 action. The isp-hosted lane from #4712 still ran the
script. It now uses the same action and version.

The third is the negated-assertion audit. It reads the find operand in
`\( ! -type d` in select-e2e_test.bats as a shell negation and fails.
The line now says `-not -type d`. This is the same change as #4748, so
this PR supersedes it. The audit bug itself is #4752.

After the Go fix, the full log of the unit job had only these two bats
failures left.

Closes #4750
Closes #4755

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): fix three unit-test contracts that went red on main after adjacent merges.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* History checks now recognize direct `kubectl` reads and reads through
the supported wrapper, including timeout-bounded commands with an
optional kill-grace setting. Commands that only include `kubectl` as an
argument, use similarly named commands, or place a timeout before the
wrapper are not treated as reads.
* **Chores**
* The hosted end-to-end workflow now uses a pinned Flux version, and
suite selection handles directory exclusions consistently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 23:17:08 +05:00
Aleksei Sviridkin f4a6eb809c fix(nats): enable JetStream for the generated account (#4675)
## What this PR does

With `users` configured and `jetstream.enabled: true`, the application
chart enables JetStream on the server but generates account `A` without
it. Authentication and ordinary NATS messaging work, while
`$JS.API.INFO` returns error 10039, "JetStream not enabled for account".

The generated account now gets `jetstream: true` when persistent
messaging is enabled and the account does not set JetStream itself.
Tenant-supplied limits, explicit disablement, other accounts and the
existing generated-user merge are kept. nats-server matches config keys
case-insensitively and walks the account map in no fixed order, so a
`JetStream` key in any casing counts as set. Otherwise the account would
carry both keys and the server would apply either one, changing from
restart to restart.

Existing instances with `users` pick this up on upgrade. The rendered
NATS config changes, so the StatefulSet rolls its pods once, and account
`A` gets JetStream without account limits unless
`config.merge.accounts.A.jetstream` sets them.

### Validation

- `helm unittest` passes all 75 application-chart tests. Eight cases
cover this change. Against the original template the generated-account
and null-override cases fail. The other six pin what the change must
keep, and the mixed-case one fails as soon as the check is
case-sensitive.
- Dropping the `jetstream.enabled` check, the case folding or the
explicit-setting check turns at least one case red each time.
- The application chart and the vendored system chart were rendered, and
the resulting account configuration was run on NATS Server 2.11.10 in an
isolated standalone setup. The original template reproduces error 10039.
With this change a file-backed stream can be created, published to and
read, and explicit disablement and a one-stream limit stay enforced.
- `make generate` with cozyvalues-gen v1.7.0 leaves the generated files
unchanged.

### Screenshots

Not a UI change.

### Downstream repositories

The diff changes an application template and chart tests only. Public
values, schemas, output resource names and tooling contracts are
unchanged. The website NATS documentation contains no account-level
workaround to remove. No downstream trigger applies.


- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note



```release-note
fix(nats): enable JetStream on the account generated for users when jetstream.enabled is true, keeping any explicit account-level JetStream setting. Instances with users roll their NATS pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Generated NATS accounts now enable JetStream when configured, without
overriding an existing JetStream setting. Configured passwords are added
while preserving existing account settings.

* **Tests**
* Added coverage for enabled and disabled JetStream, explicit limits and
false or empty values, case-insensitive setting names, null overrides,
and manually configured accounts when generated users are absent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 18:00:37 +03:00
Aleksei Sviridkin 0cae0e87ad test(e2e): spell the find negation as -not in the suite-list helper
The negated-assertion audit splits a line on `(` and reads the `!`
after `\(` as a shell negation that errexit would exempt, so the
helper's find predicate fails the audit and the unit tests on main.
`-not` is the same predicate in GNU, BSD and busybox find and does not
look like one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 17:57:05 +03:00
Aleksei Sviridkin c854e054b9 ci(workflows): install flux in the isp-hosted lane through the pinned action
The isp-hosted lane fetched fluxcd.io/install.sh and ran it as root,
which the CI tool-install contract forbids for every workflow: it runs
whatever the site serves at that moment. The lane landed next to the
contract and kept the old step, leaving the unit tests red on main.

It now uses the same SHA-pinned fluxcd/flux2 action and version as the
other workflows.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 17:57:05 +03:00
Aleksei Sviridkin f527ef17ee test(fluxcontract): accept a timeout-bounded history read
The monitoring suite bounds its HelmRelease history read with
`timeout 30 kubectl`, which keeps a hung API call from eating the
step's whole budget and the diagnostic with it. The shared-read pin
admitted kubectl only bare or through the bounded wrapper, so the two
landed side by side and left the unit tests red on main.

The pin now takes an optional `timeout [-k N] N` in front of kubectl,
the forms the suites already use for bounded reads. The bound is
spelled out instead of allowing any prefix, since
`$(echo kubectl ...)` would name the expansion without running it;
that form, bare and behind a timeout, is now a refuse case. So is a
timeout in front of _cozy_guard_kubectl: timeout runs a program, the
wrapper is a shell function, and the read would fail every time. The
command name has to end at a blank, so `kubectlx` or
`_cozy_guard_kubectl_v2` is not taken for either one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-05 17:57:04 +03:00
Aleksei Sviridkin 7bf6906a80 fix(platform): keep the legacy etcd-operator stopped after etcd adoption (#4738)
## What this PR does

Migration 50 scales the legacy etcd-operator to 0, adopts every legacy
`etcd.aenix.io` cluster with `etcd-migrate`, and then scales the
operator back to 1. At that point the Deployment still runs the legacy
operator, and after a successful adoption it has nothing left to manage.
Bringing it back undoes part of the adoption.

Observed on a production cluster upgraded from 1.4.3 to 1.6.4. The API
audit log and the Helm history of a tenant's `etcd` release show this
sequence, in UTC on the same day.

- 13:21:12 migration 50 scales
`cozy-etcd-operator/etcd-operator-controller-manager` to 0.
- 13:22:59 `etcd-migrate` updates `etcd-headless`, pointing its owner
references at the adopted EtcdMembers, and orphan-deletes the legacy
EtcdCluster and StatefulSet.
- 13:23:04 migration 50 scales the operator back to 1. It is still the
legacy operator, the v1alpha2 one replaced it only at 13:40.
- 13:23:42 Flux upgrades the tenant's `etcd` release with the old chart
(Helm revision 87), because the new chart artifact appeared only at
13:24:40. The old chart re-creates the legacy EtcdCluster at 13:23:43.
- 13:23:43 to 13:23:45 the legacy operator reconciles it, updates
`etcd-headless` twice and re-creates the legacy StatefulSet `etcd`.
- 13:36:49 Flux upgrades the release to the new chart (revision 88),
which prunes the legacy EtcdCluster.
- 13:36:50 the garbage collector deletes `etcd-headless` and the
re-created StatefulSet.

The same happened in all five tenants that run an etcd. The pods
survived because EtcdMembers own them, but they keep `spec.subdomain:
etcd-headless`, so their `etcd-<i>.etcd-headless.<ns>.svc` names stopped
resolving. From then on every member logged `no such host` for its
peers, the v1alpha2 EtcdCluster had no status conditions, and the
`etcd-defrag` CronJob failed every hour with `context deadline
exceeded`. The transitional `etcd-headless` Service the etcd chart ships
for this window (#3270) does not cover it. Helm patched the existing
Service 202 ms before the garbage collector removed it, and did not
create it again because it was already part of the release.

With this change the operator stays at 0 after a successful adoption.
The etcd-operator HelmRelease replaces the Deployment with the v1alpha2
operator later in the same upgrade, and no later migration waits for the
operator. A failed or interrupted adoption still restores it to 1
through the existing EXIT trap, so a failed run does not leave the
tenant etcds without an operator. If that HelmRelease cannot reconcile,
the adopted clusters wait for it with or without this change, because
the legacy operator does not reconcile v1alpha2 EtcdClusters.

Clusters already upgraded with this bug can be repaired without
restarting anything. Creating the Service from the tenant's own `etcd`
release is enough.

```
helm -n <ns> get manifest etcd | awk 'BEGIN{RS="\n---\n"} /\nkind: Service\n/ && /\n  name: etcd-headless\n/' | kubectl -n <ns> create -f -
```

On the cluster above the v1alpha2 EtcdCluster reported `Available` with
reason `QuorumHealthy` within 11 seconds of the Service being created,
in every tenant, and the members stopped logging `no such host`.

### Validation

- `hack/cozytest.sh hack/migration-50-etcd-adopt.bats` passes all 29
tests. A new test asserts that a successful adoption scales the operator
to 0 and never back to 1. Against the current script it fails with
`FAIL: the legacy operator was scaled back up after a successful
adoption`.
- The existing test for a failed adoption still passes, so the EXIT trap
still restores the operator to 1.
- The new test checks the absence with `if grep ...; then echo "FAIL:
..."; false; fi`, the form the other suites use. A bare `! grep`
assertion cannot fail a test under `hack/cozytest.sh`, so it could not
have caught this. The other negated assertions in this file had the same
weakness, and #3790 has rewritten them on main. With all of them made
able to fail in a scratch copy, the suite still passes against this
change.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. It changes a migration script and its bats test. No package,
values, schema, CRD, make target or `hack/` layout changes, so no entry
matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): keep the legacy etcd-operator stopped after migration 50 adopts the legacy etcd clusters. Scaling it back up let it reconcile the legacy EtcdCluster that Flux re-created from a not yet upgraded etcd chart, re-create the legacy StatefulSet and take the legacy etcd-headless Service away from the adopted members, which the garbage collector then deleted. The adopted members lost their DNS names, the EtcdCluster never reported a status and etcd defrag failed every hour. Clusters already upgraded can recreate the Service from the etcd release manifest.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Migration Updates**
* After successful etcd adoption, the legacy operator remains scaled to
zero replicas for replacement by the etcd operator; it is no longer
scaled back up before the migration is stamped.
* If adoption fails or exits before completion, the legacy operator is
restored to one replica.
* The dry-run message now describes the zero-replica outcome of
successful adoption.
  * Successful adoption continues to record the migration version stamp.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 17:53:30 +03:00
Aleksei Sviridkin 415430bcc7 fix(e2e): fail on a failed release of cilium and csi as on the parent, and report addon remediation (#3594)
## What this PR does

The history guard in the tenant Kubernetes e2e suite checked only the
parent `kubernetes-<test>` HelmRelease. A failed action on one of the
addon releases it installs left nothing the suite looked at, so a run
showed only the downstream symptom, usually the node-join deadline, and
never named the release behind it.

The guard now reads every addon release of the parent. It selects them
from the cluster by name prefix, so a release added to the chart later
is covered too. The rule:

- A `failed` or `uninstalled` Snapshot fails the run on any release
whose install strategy is `RetryOnFailure`. That is the parent (the
aggregated apiserver stamps it on every Application HelmRelease) and the
tenant `cilium` and `csi`.
- The same footprint on the other addons, which run the default strategy
with `retries: -1`, is printed as a note and the run stays green. For
the CRD addons that configuration is tracked in #3555.

A transient `cilium` or `csi` failure that later recovers on a retry
still turns the run red, by design, as long as its Snapshot still reads
`failed`. One recovery leaves no such Snapshot: when an install fails
and the very next attempt succeeds, Helm marks the failed install
`superseded`, because there is no deployed release to supersede, and the
history reads like an ordinary upgrade. The guard cannot see that case
from the statuses alone. The code and the docs now say so, a test pins
it, and detection is left to a follow-up.

An earlier revision of this PR tried to tell a teardown apart from a
retried failure and made only `uninstalled` fatal. That split does not
hold. In helm-controller v1.5.0, the version the platform installs, a
release with an active retry strategy never remediates.
`atomic_release.go` returns an upgrade for a failed release before it
reaches the remediation switch, and the two other paths that uninstall
(a changed release target, and deletion) both clear history afterwards.
So on these releases `uninstalled` never shows up, and `failed` is the
only signal. The parent keeps the rule it already had on main, and
`cilium` and `csi` now follow it too.

The failure paths get the same reading. The guard runs after the
assertions, while the deadlines a failed addon breaks end the script
earlier, so the same classification is also printed, without a verdict,
from the node-join collector and from the exit handler. That handler is
now armed at the tenant CR apply, so the management-cluster waits before
the tenant answers are covered too. Smaller changes that came with this:

- Every guard read is bounded, and a failed read is never folded into
"no history".
- The parent must read back a non-empty strategy. An empty value there
means the jsonpath stopped resolving, and without this check every
footprint would quietly turn into a note.
- The history read uses `HELMRELEASE_HISTORY_JSONPATH`, and the
upstream-type pin in `internal/fluxcontract` now accepts the bounded
wrapper the read goes through.

`docs/agents/e2e-testing.md` §6 and checklist item 8 describe the new
rule.

On related issues: this adds the `local` that #3774 reports missing in
`helmrelease_has_remediation_cycle`, but not the sweep over the other
`_lib/` helpers that the issue also asks for. For #3773 it rewrites the
guard comment the issue quotes. The waits without a backstop are not
touched.

Closes #3553.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff. It touches
`hack/e2e-chainsaw/_lib/`, `hack/*.bats`, one Go test under
`internal/fluxcontract`, two spend-order comments in the kubernetes
suites, and `docs/agents/e2e-testing.md`. No package, schema, CRD, make
target or user-facing document changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): the HelmRelease history guard now also covers the tenant addon releases. A failed or uninstalled Snapshot fails the run on the parent, cilium and csi, which retry in place; other addons get a note. The same classification is printed on the failure paths where the guard itself cannot run.
```
2026-10-05 17:12:38 +03:00
IvanHunters 0639ce39aa fix(kamaji): drop any subchart nested under the vendored kamaji chart (#4459)
## What this PR does

A subchart nested under the vendored kamaji chart would deploy no matter
what the values say, so this PR removes that directory in `make update`
and adds a test that fails if anything renders from it. The vendored
chart declares a `kamaji-etcd` dependency gated on `kamaji-etcd.deploy`,
and the subchart itself is not vendored. The issue proposed pinning
`kamaji-etcd.deploy: false` in the package values. I tested that, and it
would not have helped: through this package the condition is never
evaluated at all.

Helm skips the condition because the umbrella chart declares no
dependencies. `processDependencyEnabled` returns immediately when the
chart it starts from has no `dependencies` in its `Chart.yaml`, and
`cozy-kamaji` has none. The vendored `kamaji` chart is loaded from
`charts/` without being declared, so Helm never recurses into it and
never reads its `kamaji-etcd.deploy` condition. The same early return is
in Helm v3 and v4. I put `kamaji-etcd-0.15.0.tgz` into
`charts/kamaji/charts/` and rendered the package three ways:

- default values: StatefulSet, Job, DataStore and four Certificates from
`kamaji-etcd`
- `kamaji.kamaji-etcd.deploy: false` in the package values: the same
etcd
- `deploy: false` edited into the vendored `charts/kamaji/values.yaml`:
the same etcd

Rendering `charts/kamaji` on its own with `kamaji-etcd.deploy=false`
renders no etcd, which confirms the umbrella is what skips the
condition. So any chart that ends up under `charts/kamaji/charts/`
deploys whatever the values say, and no values change can close the
hazard.

This PR changes two things:

- `make update` removes `charts/kamaji/charts` after extracting the
upstream chart. Upstream does not ship that directory today, so the
recipe still reproduces the current tree exactly. I checked that by
running it on a copy of the package. I also ran it against a fake
upstream tarball that carried the subchart: the directory was extracted
and then removed.
- The package gets its first helm-unittest suite. It asserts that no
template of a subchart nested under `charts/kamaji/charts/` renders a
document, and a `test` target lets `hack/helm-unit-tests.sh` pick it up
in `make unit-tests`. With the subchart tgz dropped in, the suite fails.
With the `rm` line removed from the recipe, the fake-upstream update
leaves the tgz behind and the suite fails too. `/tests` is anchored in
`.helmignore`, so the packaged chart's file list is unchanged.

I left the `kamaji-etcd` block in the vendored values alone. It is not
quite dead: `certmanager_issuer.yaml` reads it to decide whether to keep
the self-signed issuer. That only matters when
`certManager.issuerRef.name` is set, and this package does not set it.
`helm template` output is byte-identical to main.

Closes #4397

### Screenshots

Not a UI change.

### Downstream repositories

The change touches only the kamaji package's update recipe, its test
suite and its `.helmignore`. The closest trigger is package `Makefile`s
→ the website's `development.md`, but that page describes `make update`
and `make test` in general terms that still hold, and the rendered chart
is unchanged.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kamaji): `make update` now drops any subchart nested under the vendored kamaji chart, and a unit test fails if one appears, because such a subchart (for example the bundled kamaji-etcd) would deploy regardless of `kamaji-etcd.deploy`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Helm chart updates now avoid including nested charts that could deploy
unexpectedly.
* Helm chart packages exclude test files, keeping them out of
distributed chart artifacts.

* **Tests**
  * Added a chart check to verify nested charts render no resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:48 +03:00
IvanHunters 2188f16461 test(vminstance): cover VMDisk clones from an image and from a disk (#4586)
## What this PR does

Closes #3482.

The `vminstance` suite now covers both VMDisk clone paths, which had no
e2e test at all. One disk is cloned from a golden image with
`source.image`, and a second disk is cloned from the first with
`source.disk`. For each clone the test checks that the DataVolume points
at the expected source PVC and becomes `Ready`, that its PVC is `Bound`,
and that the VMDisk reports `Ready` and `WorkloadsReady`, the same way
the standalone VMDisk test does. The `Bound` check is separate because
CDI derives `Ready` from the DataVolume phase alone, and can reach
`Succeeded` from the claim's annotations before the claim binds; a
`phase: Succeeded` assert would only repeat `Ready`.

The golden image is a fixture the suite creates itself.
`vm-default-images` is an opt-in package and the e2e install does not
enable it, so `alpine-3.21` from the default collection is not there to
clone. The suite imports the same Alpine 3.21 image into `cozy-public`
under the name `e2e-alpine-3.21`, so it can never collide with an image
the package owns. This is the only download in the new test, and the
`source.disk` case reuses the first clone as its base disk.

The long wait sits on the DataVolume, not on the HelmRelease. Since
#4380 a vm-disk release goes Ready once the DataVolume exists and the
clone's progress is reported through the VMDisk `WorkloadsReady`
condition, so the test does not assert the HelmRelease at all. Chainsaw
deletes the clones before the golden image at cleanup.

The clone itself is only proven by this PR's E2E run, on the `local`
class of the PR lane.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/vminstance/`, and nothing in
the trigger map covers it.

### Release note

```release-note
test(vminstance): e2e now covers VMDisk clones from a golden image (source.image) and from another disk (source.disk).
```
2026-10-05 15:14:38 +03:00
IvanHunters 4b1a75a332 test(vminstance): dry-run evict the VM launcher pod (#4585)
## What this PR does

Part of #4422.

The e2e suite now asks KubeVirt whether it would let a VM launcher pod
be evicted, without evicting it. Until now nothing in e2e evicted a
launcher pod, so the drain behaviour set up in #4403 was known only from
reading KubeVirt. Once the `vminstance` VM is running, the suite sends a
dry-run Eviction (`dryRun=All`) for its virt-launcher pod and checks the
answer. Nothing gets deleted.

A dry run still goes through the KubeVirt eviction webhook, which is
registered with `sideEffects: NoneOnDryRun` in v1.9.0. With
`LiveMigrateIfPossible` its answer follows the VMI's `LiveMigratable`
condition. That condition depends on storage, so the test reads it
instead of assuming it:

- `LiveMigratable=True`: the eviction must be refused with the webhook's
`Eviction triggered evacuation of VMI "<ns>/<name>"` message (HTTP 429).
A drain retries on that while the VM migrates away.
- `LiveMigratable=False`: the eviction must be allowed. The old
`LiveMigrate` value refused this case with "is not live-migratable".
KubeVirt v1.9.0 creates no PodDisruptionBudget for a VMI, so no PDB can
refuse it either.
- Anything else, including a missing condition, fails the test.

The allowed answer alone proves little, because the webhook also lets an
eviction through under `None`. So the test also asserts
`evictionStrategy: LiveMigrateIfPossible` in the VMI spec, where
virt-api copies the cluster value at creation. After the dry run it
checks that the launcher pod is the same object and not terminating, and
that the VMI has no `evacuationNodeName`.

On the PR lane only the `False` branch runs. Post-install prep pins
`StorageProfile/local` to RWO because DRBD is off there, and an RWO disk
makes the VMI non-migratable. Nightly and tag runs use `replicated`,
where I expect the `True` branch, but I have not seen this suite run on
that lane yet. A real drain and a workload update after a KubeVirt bump
are still open in #4422.

I ran the script against a stub `kubectl`. Both expected answers pass,
and it fails on every wrong answer I tried, such as a refusal for a
non-migratable VMI, a PDB-style 429, a missing condition, two launcher
pods, or a replaced pod after the dry run. `chainsaw lint` (v0.2.15) and
shellcheck pass.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/vminstance/` and the suite
README, and nothing in the trigger map covers either.

### Release note

```release-note
test(vminstance): the e2e suite now dry-run evicts the VM launcher pod and checks the KubeVirt eviction decision against the VMI's LiveMigratable condition.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added end-to-end coverage for dry-run eviction of a running virtual
machine’s launcher pod. The test checks the VM’s live-migration policy
and `LiveMigratable` condition, expecting eviction to be refused when
the VM is live-migratable and accepted otherwise.
* Verified that the dry run leaves the launcher pod unchanged and does
not record an evacuation node on the VM.
* **Documentation**
* Documented the test’s behavior and the local `jq` requirement for
running it.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:32 +03:00
IvanHunters 6a98acb61d fix(api): leave failed golden images out of the image options (#4568)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The VM image picker no longer offers golden images whose import failed.
A golden keeps its PVC in `cozy-public` after the import fails, so a
picker built on the PVC list offered an image whose clone never
completes.

The `image` option source in cozystack-api now reads the
`vm-default-images-*` DataVolumes next to the PVCs and drops names whose
DataVolume is in phase `Failed`. I filter rather than mark them, because
such an option can never be used and `OptionItem` has no disabled state.
A golden that is still importing stays in the list, since CDI holds a
clone until the source is populated, and so does a PVC with no
DataVolume. When the DataVolumes cannot be read, the PVCs go out
unfiltered. The Option list drops a source whose provider errors, and an
empty picker would be worse than the old behaviour. cozystack-api gets a
read grant on `cdi.kubevirt.io` DataVolumes for this.

The console did not use that source at all. `SourceField` listed the
PVCs itself, as the tenant, so the fix above would not reach the picker
users see. It now reads the same Option list the other dynamic dropdowns
use. With that, nothing in the console lists `cozy-public` PVCs, so I
removed the read grant on them from the `cozy:tenant:dashboard` Role.
That is a visible change: a tenant loses `kubectl get pvc --namespace
cozy-public`, which was only ever granted for this picker in #2843.

One limitation: in CDI v1.66.1, which cozystack ships, a golden import
reaches `Failed` only on a checksum mismatch, and a broken URL keeps it
in `ImportInProgress`. So this filter covers less than the issue
assumed. I filed #4564 for that.

Closes #4553

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

I have no screenshot. The picker looks the same; only the list behind it
changes. `apps/console/src/components/SourceField.test.tsx` covers it:
the picker shows the images the option source serves and not the raw PVC
names.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(api): the VM image picker no longer offers golden images whose import failed. The console now reads images from the image option source, and tenants no longer get read access to PersistentVolumeClaims in cozy-public.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* The image picker now displays available images from the tenant’s image
options.
* **Bug Fixes**
* Images whose imports have failed are excluded from the picker. Images
remain available when import status is still in progress, has succeeded,
or cannot be retrieved.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:25 +03:00
IvanHunters 86bf2ba8ec fix(kubernetes): let kubevirt-csi hot-unplug from an orphaned VMI (#4563)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This lets the kubevirt-csi controller detach a volume from a VM instance
that no VM owns. Right now that call is always denied. Closes #4452.

The driver has a fallback for a volume that is still attached to the VMI
but is no longer in the VM spec. It calls the VMI-level `removevolume`
subresource. The `-kcsi` Role grants only the two VM-level subresources,
so the call ends in 403. Nothing else reaches this service account with
that grant. Only its own Role and ClusterRole name it as a subject, and
the group bindings that cover every service account grant nothing on
this subresource. At v1.9.0, `kubevirt.io:default` grants only
`kubevirts`, `version` and `guestfs`. `kubevirt.io:edit` and
`kubevirt.io:admin` do include the VMI subresource, but they aggregate
only into the built-in `edit` and `admin` roles, and no tenant binding
gives those to this account. The cozystack tenant roles aggregate on
their own labels and don't list it.

The fallback is still needed after the flag in #4453 goes to false. The
flag only skips VMIs that a VM owns. When a VM is deleted with
`propagationPolicy=Orphan`, the KubeVirt VM controller drops its
finalizer and leaves the VMI running (`vm.go` at v1.9.0). The tenant
node stays up with its volumes attached. When a pod moves off that node,
the driver has to detach through the VMI, and on a 403 the hot-plug pod
keeps holding the infra volume. Until #4453 lands, the fallback also
runs for VM-owned VMIs and gets the same 403.

The grant is `update` on `virtualmachineinstances/removevolume` and
nothing more. The account can already remove a volume from any VM in the
namespace, so it gets no new reach. virt-api also rejects the VMI-level
call on a VM-owned VMI once `HotplugVolumes` is off.
`virtualmachineinstances/addvolume` stays out because the driver only
attaches through the VM.

The new helm-unittest pins the Role's `subresources.kubevirt.io` rule to
these three subresources with `update`. I haven't reproduced the 403 on
a cluster. This shows it on a tenant namespace, before and after:

```bash
kubectl auth can-i update virtualmachineinstances.subresources.kubevirt.io --subresource=removevolume \
  --namespace <tenant-ns> --as system:serviceaccount:<tenant-ns>:<release>-kcsi
```

Upstream kubevirt/csi-driver has the same gap in
`deploy/infra-cluster-service-account.yaml`.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi controller can now hot-unplug a volume from a VMI that no VM owns, instead of failing with 403 Forbidden.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Kubernetes CSI volume removal for virtual machine instances,
including cases where a volume is not listed in the virtual machine
specification. This helps the driver complete volume removal in
scenarios where the Kubernetes virtualization API accepts the
instance-level request. Existing permissions for adding and removing
volumes from virtual machines remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 15:14:19 +03:00
IvanHunters ec4d26df54 fix(kubernetes): accept a golden worker image that exists only as a PVC (#4578)
## What this PR does

A worker pool on `osImage.builtin` refused a golden image that exists in
`cozy-public` as a PVC without its DataVolume, and the whole
KubernetesNodes release failed to render. CDI clones the PVC, not the
DataVolume, so that golden is a valid source, and vm-disk already
accepts it. This can happen once CDI garbage-collects DataVolumes
(`dataVolumeTTLSeconds`) or when someone restores a golden PVC on its
own.

Now the render fails only when neither the DataVolume nor the PVC
exists. The StorageClass and size checks already fall back to the PVC,
so a PVC-only golden is still checked against the pool, and the new test
suite pins that too.

Closes #4565

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): a worker pool on osImage.builtin now accepts a golden image that exists in cozy-public only as a PVC, matching vm-disk; it is refused only when neither the DataVolume nor the PVC exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Built-in golden images can now be used when available as either a
DataVolume or a PVC.
* **Bug Fixes**
* Missing golden images now produce an error that identifies the missing
image and suggests ways to resolve the issue. The error is suppressed
when the pool already has a successful clone.
* PVC-based golden images are checked against the pool’s storage class
and minimum disk size requirements.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:45:50 +03:00
IvanHunters f9f7562e56 fix(kamaji): resolve Gateway API availability once at startup (#4501)
## What this PR does

Carries the fix proposed upstream in clastix/kamaji#1327 (not yet merged
there; issue clastix/kamaji#1326) as a build patch on the 26.9.4-edge
tag that cozystack builds Kamaji from.

In Kamaji 26.9.4-edge, every TenantControlPlane reconcile checks whether
the Gateway API is available. The discovery call ignores the reconcile
context, so it is bounded only by the client's 32s default timeout,
which is longer than the 30s reconcile budget. The watches, however,
read availability only once, at startup.

With the patch, Kamaji resolves availability once when the manager
starts. The watches, the indexer, the reconciled resources and the
admission webhook's Gateway API check all use that one value. A
discovery error at startup now stops the manager, so it is no longer
mistaken for "Gateway API not installed". A Gateway API installed after
Kamaji starts needs a Kamaji restart. On this platform that case does
not come up: the kamaji package depends on cert-manager, and
cert-manager depends on gateway-api-crds.

The webhook's separate TLSRoute check still runs discovery on each
request, as before, and is listed as a follow-up in clastix/kamaji#1327.
Kamaji's chart sets no `timeoutSeconds` on the webhook, so the
apiserver's 10s default applies, and a discovery call that can take up
to the client's 32s can make that admission time out.

The patch is the upstream commit applied unchanged except for the
dropped docs hunk, which is not part of the image. It applies cleanly to
26.9.4-edge with no offsets. It modifies different files than the
existing `fix-kubelet-config-compat.diff`, so the two apply in either
order. The upstream tests stay in the patch so that it matches the
upstream commit; the image build does not run them. On the patched tree,
`go build ./...` passes, and `go test -race` passes for
`./controllers/...`, `./internal/webhook/...`,
`./internal/utilities/...`, `./internal/kubeadm/...` and `./cmd/...`. I
also checked that the new tests can fail: they go red when the
availability check is inverted and when the old per-reconcile lookup is
put back.

If clastix/kamaji#1327 changes before it merges, this patch should be
updated to match. It is dropped once Kamaji ships the change.

Refs #4489. This does not close #4489, because this change does not
explain the long stall reported there.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kamaji): Kamaji now checks for the Gateway API once at startup instead of on every TenantControlPlane reconcile, so a slow discovery endpoint can no longer use up the reconcile timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Gateway API availability is now checked at startup and consistently
applied to Gateway-related setup and configuration validation.
* Gateway configuration is rejected if the Gateway API was unavailable
when Kamaji started. If you install the required CRDs afterward, restart
Kamaji before configuring Gateway resources.
* Discovery errors now stop startup rather than allowing Kamaji to
continue with an uncertain Gateway API status.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:51 +03:00
IvanHunters b0e07cadbb fix(operator): rebuild each changed ArtifactGenerator generation once (#4594)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

After an upgrade, an ArtifactGenerator can report Ready on its new spec
while its artifacts were built from the old one, and source-watcher
never rebuilds them. That is how the bucket-system chart in #4593
shipped without cozy-lib.

The cause is upstream (fluxcd/source-watcher#384). source-watcher builds
from the spec it read when the reconcile started, but the final status
patch stamps Ready with the live generation. An upgrade changes the
packages source and the copy rules within seconds, so a spec change
often lands during a build.

The generator status has nothing that tells a stale build from a good
one. So the operator now rebuilds every changed generation once, after
source-watcher first reports it Ready, through the force-drift path the
operator already uses for fluxcd/pkg#934. An unchanged spec rebuilds to
the same digest, so ExternalArtifacts and HelmReleases do not move.
Generation 1 is skipped, since the race needs a spec change during a
build.

On the first rollout every generator past generation 1 gets one extra
rebuild, and its PackageSource shows
`Unknown/AwaitingSourceWatcherRecovery` until that rebuild lands.
Nothing waits on that condition today. The same rebuild repairs clusters
already stuck like #4593.

The recovery messages on the PackageSource no longer blame
fluxcd/pkg#934, because the same path now serves both cases.

### How it was tested

Unit tests in `internal/operator/packagesource_reconciler_test.go` cover
the forced rebuild on generation 2 and on a later generation, every case
where no rebuild must happen, the rebuild settling without a second one,
both failure orders (force fails, recording the generation fails), and a
stale recovery counter. Each condition of the new predicate was mutated
once to confirm a test fails.

I also ran this commit's operator image on an arm64 test cluster running
v1.7.0-alpha.3, the one where #4593 was found. Before that I had
repaired its bucket generator by hand: deleting the stale
ExternalArtifact and requesting a reconcile produced an artifact with
`charts/cozy-lib` and both bucket HelmReleases went Ready. With the
patched operator, the only generator past generation 1 got exactly one
forced rebuild and the verified-generation annotation. source-watcher
reported all its ExternalArtifacts up to date. The digests of all 214
ExternalArtifacts and the revisions of all 104 HelmReleases matched a
snapshot taken before the swap, and every PackageSource and HelmRelease
stayed Ready. The race itself was not reproduced on the cluster, since
that needs a spec change landing during a build.

Closes #4593

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(operator): rebuild an ArtifactGenerator once after each spec change, so an upgrade no longer leaves a component on artifacts built from the previous spec (for example a chart missing a library it now needs).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved PackageSource recovery when an artifact generator reports
ready: the system now verifies the generated output and triggers a
rebuild when needed.
* Failed rebuild or verification attempts can be retried during a later
reconciliation, and recovery tracking is cleared once the rebuilt
generation settles.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:44 +03:00
IvanHunters 50dec5d46a fix(platform): resume a retried migration Job from the stamped version (#4583)
## What this PR does

A retried pod of the platform migration Job no longer runs migrations
that already finished in the same upgrade. The Job gets
`CURRENT_VERSION` once, at render time, and a failed pod is replaced by
one with the same environment. By then every finished migration has
stamped the `cozystack-version` ConfigMap, but the new pod ignored that
and started from the rendered version again.

`run-migrations.sh` now starts from the version stamped in the
ConfigMap. If reading it fails, the pod fails and the Job retries it,
because falling back to the rendered version is exactly the re-run this
fixes. The rendered value is used only when the ConfigMap holds no
integer version, which is also when the chart itself could not get a
real version from it. Without that fallback a value like `v58` gave a
green Job that ran nothing.

The new `hack/run-migrations.bats` runs the script in the migrations
image's alpine base against a fake kubectl that keeps the stamp in a
file, so one pod's stamps are what the next pod reads. It needs docker,
like the other migration suites.

I also found that the ConfigMap existence check at the top of the script
treats any API error as a missing ConfigMap and skips every pending
migration. That is older than this change and is filed as #4579.

Closes #4555

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a retried platform migration Job now resumes from the version stamped in the cozystack-version ConfigMap instead of re-running migrations that already finished in the same upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Migration startup now resumes from a valid stamped version, avoiding
reruns of migrations that have already completed.
* Empty or non-numeric stamps fall back to the rendered version.
Out-of-range stamps or failed stamp reads stop migrations before they
run.
* If the migration ConfigMap is missing, migrations do not run and the
target version is stamped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:39 +03:00
IvanHunters bb2946990d fix(vm-instance)!: refuse guest memory below KubeVirt's 1M floor (#4569)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The vm-instance resource guard now refuses guest memory below 1M, the
same floor KubeVirt admission enforces. Before, a value such as `500m`
or `1Ki` rendered and then failed when the VirtualMachine was applied.

This is a breaking change. A memory above zero and below 1M rendered
before and is now refused at render time. KubeVirt v1.9.0 would not
admit such a VM either: `validateMemoryRequestsNegativeOrNull` refuses a
non-zero request below 1M
(`pkg/virt-api/webhooks/validating-webhook/admitters/vmi-create-admitter.go:1135`).
The VirtualMachine admitter runs the same check on its template.

The cpu, sockets and memory descriptions now list the values the guard
refuses, so the schema states the same rule as the error.

The error cannot quote an unquoted number as it was written, because
YAML has decoded it before the chart sees it. A large one also printed
in exponent form, so `4294967296` came out as `4.294967296e+09`. A
string is still quoted exactly as written. A number is now printed in
plain digits and marked "read as", which is the closest honest wording I
found.

Closes #4413

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(vm-instance)!: refuse `resources.memory` below 1M at render time, since KubeVirt does not admit a VM with less. A value above zero and below 1M rendered before and is now refused. The cpu, sockets and memory descriptions list the refused values, and the guard error marks an unquoted number as read rather than written.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* VM resource values are now checked when rendering: CPU cores and
sockets must be positive whole numbers no greater than 4294967295.
Unit-suffixed, fractional, zero, negative, and quoted values with
leading zeros are rejected.
* Memory values below `1M` are rejected, with clearer messages showing
the value as entered or interpreted.

* **Documentation**
* Updated resource guidance to describe these limits and validation
behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:43:15 +03:00
IvanHunters ed20c53449 fix(vm-disk): refuse to clone a golden image that does not exist (#4567)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Stacked on #4392, so the base is `fix/vm-default-images-health-check`.
Retarget to `main` once #4392 merges.

A new vm-disk with `source.image` now fails to render when the golden
image it names does not exist, and the error names the image and
`cozy-public`. Before this the chart cloned
`cozy-public/vm-default-images-<name>` without looking, and CDI waited
for that PVC forever while the VMDisk stayed Ready. That happens on a
typo, and on every cluster where the opt-in
`cozystack.vm-default-images` package is off, so the message also says
to add it to `bundles.enabledPackages`.

The golden counts as missing only when neither its DataVolume nor its
PVC exists. CDI clones the PVC, so a PVC with no DataVolume is still a
usable source, and a DataVolume whose PVC is not created yet is an
import about to start.

Under `helm template` every lookup is empty, so the check runs only when
the release namespace itself is visible. The same trick is already used
in the kubernetes and seaweedfs charts. Existing disks are exempt the
same way #4392 exempts them, because their DataVolume spec is immutable.
A disk refused while the catalog is off renders by itself once the
golden appears, since app HelmReleases retry on failure.

Tests are in `packages/apps/vm-disk/tests/dv_golden_image_test.yaml`:
refusal of a missing golden, a PVC without a DataVolume, a DataVolume
before its PVC, rendering with no cluster, and an existing disk whose
golden was removed.

Closes #4548

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(vm-disk): a new disk with source.image now fails with a clear error when the golden image does not exist in cozy-public, instead of waiting forever for a clone that never starts.
```
2026-10-05 14:42:52 +03:00
IvanHunters c7b8009964 fix(kubernetes): log kubevirt-csi temp NFS mount dir removal failure (#4562)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The kubevirt-csi node plugin now logs a warning when it cannot remove
the temp dir it uses for NFS publish. Before, the error was dropped, so
a dir left behind after a failed unmount was never reported.

The publish path temp-mounts the export root to create `/data`, then
removes the temp dir in a defer. The failure is logged, the same way the
unmount failure next to it already is, and not returned. A leftover temp
dir does not break the volume, and failing a publish that otherwise
worked only makes kubelet retry it.

A new unit test publishes an NFS volume through the fake mounter, where
the removal always fails, and checks that a warning naming the dir is
logged.

This is independent of #4451. That PR touches the same Go module but not
`node.go`.

A failed unmount still leaves the NFS root mounted. That is older than
this change and is tracked in #4561.

Closes #4466

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): the kubevirt-csi node plugin now logs a warning when it cannot remove its temporary NFS mount dir instead of dropping the error
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Failed cleanup of a temporary mount directory now produces a warning
with the directory path, making the issue easier to identify. The volume
publish operation continues to succeed when cleanup fails.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:39:58 +03:00
Aleksei Sviridkin 2ddd55ce5b fix(build): honour PUSH and LOAD from the environment and in image-talos (#4654)
## What this PR does

A local build that asks to load images into docker no longer pushes
them. `hack/common-envs.mk` set `PUSH` and `LOAD` with plain `:=`, and a
makefile assignment beats the environment, so `LOAD=1 PUSH=0 make image`
quietly pushed a multi-arch build. Both now take the environment value,
and an empty one falls back to the default. I also made `PUSH` default
to 0 when `LOAD=1`, so `LOAD=1` alone only loads. Before this, `make
image LOAD=1` pushed and loaded at once. An explicit `PUSH` from the
environment or the command line still wins, and `OCI_EXPORT_DIR` still
forces both off. Every buildx target gets the values through
`BUILDX_ARGS`, so the fix lives in that one file.

The Talos installer was the one image that ignored `PUSH`. It is
published with skopeo, not buildx, so it never saw `--push=$(PUSH)`. Its
registry copies now run only when `PUSH=1`.

A build that pushes nothing no longer writes `PUSHED_TAGS_LOG`. That log
tells the multi-arch stitch which tags to move, so it must hold only
tags that were really pushed.

The cache helper lost an argument nobody used. No caller passes the
second argument of `cache-args`, and an explicit tag there would bypass
`CACHE_TAG`, so the amd64 and arm64 builds of that image would share one
cache ref.

CI sets neither `PUSH` nor `LOAD`, so CI builds do the same thing as
before. Each fix has a `make --dry-run` test in
`hack/common-envs_test.bats`.

Closes #4499

Closes #4504

Closes #4551

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The ccp skills pass `PUSH=1` explicitly, which still works. The website
docs do not mention `PUSH`, `LOAD` or `cache-args`.

### Release note

```release-note
fix(build): `LOAD` and `PUSH` set in the environment are now honoured, `LOAD=1` no longer pushes unless `PUSH=1` is also set, and the Talos installer is no longer published when `PUSH=0`.
```
2026-10-05 14:15:37 +03:00
Aleksei Sviridkin 2af8b0c229 ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

One transient error from proxy.golang.org is enough to fail an image
build or the `Unit & controller tests` job today. The error is an HTTP/2
stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or
falls back after it. This PR adds three things: a direct fallback, a
bounded retry and a module cache.

Image builds now fall back to fetching modules from their origin when
the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a
build arg from `hack/common-envs.mk`, and every Go builder stage
declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error.
With Go's default comma it does that only on a 404 or 410. The value
comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because
buildkit can't reach a mirror on localhost or a `file://` proxy. Every
workflow job that runs Go on the runner sets the same value as env.

Explicit module downloads in Dockerfiles are retried three times with a
growing pause. The loop is inline, in one form that the contract test
pins. I tried a shared script from a named build context first, but
buildx refuses any `--build-context` for the four Dockerfiles still
pinned to the `dockerfile:1.2` frontend. The images also build from
different contexts, so no single file can be copied in. Raising those
pins changes how those images build and needs a real build to check, so
I left it for a follow-up. Two alpine stages, token-proxy and
shadowbox's `ss-server`, left the download to `go build`. The alpine
golang images have no git, so the direct fallback can't fetch for them,
and they now get the same download step. The runner jobs use
`hack/go-mod-download.sh` with the same bound.

The unit test job now starts from a module cache instead of downloading
everything. It restores `GOMODCACHE` with `actions/cache`, keyed on
every `go.sum` in the tree, and then prefetches all tracked modules
through the retry, so a cache miss still works. A cache saved by a PR
run is visible only to that PR, but one saved on the default branch can
be restored by any PR
([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)).
That is why the build cache warmer on main writes the same key before
its image build. This way the first run of a PR, which is the one that
flakes, starts warm. The warmer restores only an exact key, so old
module versions don't pile up, and its Go steps can't fail the image
build. Both runner labels are Linux, so the key matches. Both also check
out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the
same cache path, but I have not seen it on the warmer's runner yet. The
release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env
only, no cache.

`hack/go-module-fetch-contract.bats` covers all three. Each test is red
on main, and I checked each guard with a mutation.

Closes #4591

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Reliability**
- Go module downloads now fall back to direct access when the public
proxy encounters errors.
- Builds and automated checks retry failed module downloads, helping
them recover from temporary network issues.
- **Performance**
- CI caches Go modules to reduce repeated downloads and speed up builds
and tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:47 +03:00
Aleksei Sviridkin 9c025fa2b1 fix(installer): skip the packages artifact push and stamp with PUSH=0 (#4663)
## What this PR does

A local build with `PUSH=0` no longer publishes the packages artifact.
Before this change, `make image-packages` in `packages/core/installer`
ran `flux push artifact` no matter what `PUSH` was set to, and then
wrote the pushed digest into `values.yaml`. The `chart` target in the
same Makefile already respects `PUSH`.

With `PUSH` set to anything other than 1, the target now skips both the
push and the digest stamp. It prints one line saying so and leaves the
committed `platformSourceUrl` and `platformSourceRef` as they are. It
still writes `packages/core/platform/.build-revision`, which is
gitignored.

I chose to skip the stamp instead of computing a local digest, because
every reader of that value needs a digest that exists in the registry.
The chart passes `platformSourceRef` to the operator as
`--platform-source-ref`, and Flux then pulls
`cozystack-packages@<digest>` from the registry. The release scripts
`verify-promoted-packages.sh`, `promote-packages-artifact.sh`,
`stitch-multiarch.sh` and `nightly-mirror.sh` also treat it as a
registry digest. A digest of an artifact that was never pushed points at
nothing. On main an unpushed run would not even get that far, since the
recipe dies at `test -n "$DIGEST"`.

The gate is a make-level `ifeq ($(PUSH),1)`, so it reads whatever `PUSH`
ends up as after `hack/common-envs.mk`. On main that means `PUSH=0` on
the command line. I also applied the `common-envs.mk` change from #4654
locally, which takes `PUSH` from the environment and makes `LOAD=1`
default it to 0. With that applied, a dry run shows no push and no stamp
for `PUSH=0` in the environment, for `LOAD=1`, and for `PUSH=0` on the
command line, while `LOAD=1 PUSH=1` and an empty `PUSH` still push.

CI is not affected, because no workflow sets `PUSH` or `LOAD` and `PUSH`
stays at 1. I dry-ran the three CI invocations of this Makefile before
and after the change, on main and again with #4654 applied:

- `tags.yaml`, release stitch step: `make -C packages/core/installer
image-packages chart` with `REGISTRY`, a release tag as `IMAGE_TAG`, and
`PUSHED_TAGS_LOG`
- `pull-requests.yaml`, installer step of a same-repo PR: `make -C
packages/core/installer image` with a `pr-N-sha` tag,
`PLATFORM=linux/amd64`, `BUILDER`, and an empty `OCI_EXPORT_DIR`
- `build-main.yaml` and `build-release.yaml`: `make build`, which runs
the installer `image` target with defaults

```text
$ diff --recursive before/ after/     # tags.txt pr.txt build.txt
$ echo $?
0
```

There is one side effect. `OCI_EXPORT_DIR` forces `PUSH := 0`, so
running `image-packages` with it set no longer pushes, where before it
would have tried an anonymous push. No CI path does that. The fork PR
build already runs only `pre-checks image-operator` for the installer,
and the `make -n ... image OCI_EXPORT_DIR=/oci` scan in `e2e-fork.yaml`
greps only the `.oci.tar` names, which this change does not touch.

The new test dry-runs the target in both modes. With `PUSH=0` there must
be no `flux push artifact` and no `platformSourceRef` stamp; with `PUSH`
at its default, both must be present. The first test fails on main. I
mutated the guard three ways, and each time only the matching test went
red:

- condition forced true: the `PUSH=0` test fails
- compared against 2: the default-path test fails
- `endif` moved up so only the push is gated: the `PUSH=0` test fails on
the stamp check

One limitation is left as a follow-up. The test's `env -u PUSH -u LOAD`
only isolates it once #4654 lands, because on main a file assignment
wins over the environment anyway. It does not cover `PUSH=0` coming in
through `MAKEFLAGS`, so `make bats-unit-tests PUSH=0` would fail the
default-path test here and the ordering check in
`hack/stitch-multiarch_test.bats`. Nothing runs the suite that way
today. If that changes, passing an explicit `PUSH=1` on the default-path
`make` lines, or unsetting `MAKEFLAGS` the way
`hack/bats-unit-keep-going.bats` does, would fix it.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website's `development.md` tells developers to run `make
image-packages` with `REGISTRY` set to their own registry. That is the
default `PUSH=1` path, and it works as before. Neither the website nor
ccp mentions `PUSH=0` for this target.

Closes #4656

### Release note

```release-note
fix(installer): `make image-packages` no longer pushes the cozystack-packages artifact or stamps its digest into values.yaml when PUSH=0.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* You can run `image-packages` with `PUSH=0` to skip publishing the
artifact and keep the committed platform source reference unchanged.
* When pushing is enabled, the command publishes the artifact and
updates the platform source URL and digest-based reference.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:42 +03:00
Aleksei Sviridkin 33cf6972bb test(qdrant): exercise the tenant trust anchor end to end (#4701)
## What this PR does

The qdrant e2e suite now runs with TLS on and checks the tenant trust
anchor end to end. Before this, the fixture left `tls` unset with
`external: false`, so the chart rendered no CA and no
`TenantProjection`, and nothing in e2e reached the controller or
`tenantsecrets`. Setting `tls.enabled: true` turns TLS on without the
LoadBalancer that `external: true` would allocate.

The first new step checks that the anchor is published. The
`qdrant-test-ca` sentinel has to report `Ready=True/Projected`, and
`qdrant-test.tenant-ca` has to carry both the `tenant-ca` and
`tenantresource=true` labels. The projection must hold only `ca.crt`
with the same fingerprint as the chart's CA, and `tenantsecrets` must
serve it. The key-bearing `qdrant-test-ca` and `qdrant-test-tls` must
answer NotFound there, and any other error fails the step.

The second step checks that the anchor is what decides trust. It reads
the anchor through `tenantsecrets`, port-forwards the qdrant Service and
runs `openssl s_client` with that file as the only CA, hostname check
included. A CA with the same subject and another key must fail, and so
must the runner's default trust store. A handshake failure counts only
when the output shows a verification error, so a refused connection does
not pass as a rejection. Plaintext HTTP must get no response. Curl exits
7 and 28 fail the step, and a second verified handshake afterwards
proves the forward was still alive.

TLS does not break the existing steps, because none of them talk to
qdrant. `create-qdrant`, `wait-helmreleases-ready`,
`verify-statefulset-and-storage` and `verify-pvc-reclaim-on-delete` only
read Kubernetes objects: no `http://`, no port 6333 or 6334, no curl or
exec, and the suite has no backup round trip. The release now also
renders two Issuers, two Certificates and the sentinel. The chart
switches its probes to HTTPS with `enable_tls`, so `readyReplicas: 1`
now covers a pod serving TLS.

Local checks: `chainsaw lint test` (v0.2.15) says the file is valid.
`helm template` with the fixture's spec renders 2 Certificates, 2
Issuers and 1 TenantProjection, and none without TLS. I ran the
handshake logic against a local `openssl s_server` with a CA and leaf
shaped like the chart's. The anchor passes, the wrong CA and the default
store fail on verification, a wrong hostname fails, plaintext gets
`000`, and a closed port is not counted as a rejection.

A sentinel with a missing source (`SourceNotFound`) is not tested here,
the reconciler unit tests cover it.

Review notes I left as they are:

- Through `kubectl port-forward` the local socket is always accepted, so
curl exits 7 and 28 only catch a dead forward process. A false pass
would need qdrant to answer plaintext and a pod-side blip on that one
request at the same time.
- qdrant may reset the socket on the plaintext request, and some kubectl
versions drop the whole port-forward on a pod-side error. Then the
handshake after it turns the step red, not green. If CI shows `lost
connection to pod` there, the plaintext check should get its own
port-forward.
- The suite's `catch` has no `describe` of the `TenantProjection`. The
sentinel assert prints its conditions when it fails, but a red Secret
assert or script after it would not.
- The vendored chart's `servicemonitor.yaml` hardcodes `scheme: http`,
so metrics scraping fails for any qdrant with TLS on. That was already
the case before this PR, and the suite does not check metrics. Tracked
in #4703.

cert-manager does not set owner references here, so `qdrant-test-ca` and
`qdrant-test-tls` stay in the namespace with their private keys after
the app is deleted. That is tracked in #4704. The nats half of #3706 is
tracked in #4702.

Closes #3706

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `hack/e2e-chainsaw/qdrant/`, which no entry in the
trigger map covers.

### Release note

```release-note
test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused.
```
2026-10-05 14:14:36 +03:00
Aleksei Sviridkin 7ea71042b7 ci(e2e): add a daily isp-hosted install lane on kind (#4712)
## What this PR does

This adds a daily CI lane that installs the `isp-hosted` platform
variant through the `hosted` installer variant. No lane ran either of
them before. The PR, nightly and tag lanes all install `isp-full` on
Talos, where Cozystack brings its own CNI and storage, and that is how
#3376 shipped unnoticed.

What `isp-hosted` changes is what it leaves to the host. The system
bundle swaps kube-ovn and Cilium for a `noop` networking Package and
drops multus, LINSTOR, MetalLB, cozy-proxy, snapshot-controller and the
rest of the shared data-plane set. The iaas bundle refuses to render.
Migrations are not a difference, `isp-full` enables them too.

So the lane runs on kind. kindnet, kube-proxy and local-path come from
the cluster, and there is no Cilium, no LINSTOR and no `replicated` or
`local` StorageClass. It installs the latest GHCR nightly instead of
building anything, at 05:00 UTC after the nightly mirror publishes, and
on `workflow_dispatch`. There is no pull request trigger, because a run
would install the nightly images and not the PR's.

A run passes when every HelmRelease is Ready, `cozystack-platform` and
`tenant-root` included. It also checks that the platform emitted the
`noop` networking Package and none of LINSTOR, MetalLB, multus or
KubeVirt, and that the cluster has no Cilium CRD and no `replicated` or
`local` class.

The lane reuses the e2e sandbox container for tooling and
`collect-report`, attached to the kind network.
`hack/isp-hosted-lane_test.bats` pins the wiring: the Package selects
`isp-hosted`, the install uses `variant=hosted`, the kind DNS domain
matches `networking.clusterDomain`, and kind is pinned by checksum and
digest. Six of its seven tests were red before the implementation
existed.

`hack/sandbox-runner-headroom.bats` gets a named exemption. That check
treats any workflow touching `packages/core/testing` as a lane that
boots the QEMU guests, and this one boots none. The exemption is checked
after the job match, and removing it turns the test red again.

The pins come from the kind v0.32.0 release: the binary sha256
`50030de2…` from its `kind-linux-amd64.sha256sum`, and
`kindest/node:v1.33.12@sha256:3f5c8443…` from its release notes.
v1.33.12 is the Kubernetes version the Talos lanes use, so the variant
is the only thing that differs. All three are plain `env` values in the
workflow, where a Renovate custom manager can match them.

### Real runs

The first real run on this branch,
[36961209755](https://github.com/cozystack/cozystack/actions/runs/36961209755),
went red on `tenant-root`:

```text
tenant-root  False  Helm install failed for release tenant-root/tenant-root with chart tenant@0.0.0+629b61300902: unable to build kubernetes objects from release manifest: resource mapping not found for name: "allow-external-communication" namespace: "tenant-root" from "": no matches for kind "CiliumNetworkPolicy" in version "cilium.io/v2"
```

This is #3387 reproduced live: `packages/apps/tenant` renders Cilium
policies on a cluster without Cilium. The other releases went Ready,
except `cozystack-basics`, which was still in an upgrade at the last
listing. The fix landed in #4710.

The run after it,
[37089699259](https://github.com/cozystack/cozystack/actions/runs/37089699259),
is green on the first attempt. It installed nightly
`0.0.0-nightly.20261003.37089458387.1`, built from 309875ab6, which
contains #4710. All four install tests passed, `tenant-root` included,
with no `cilium.io` CRD on the cluster. The install took about 9
minutes.

### Notes

The OpenAPI suite is not run. Its "Test kinds" case needs an Ingress
application, and kind has no LoadBalancer, so it would fail for a reason
unrelated to the variant.

The red run's job log is 347 MB. cozytest's live trace prints the whole
HelmRelease snapshot JSON from `cozy_wait_all_helmreleases_ready` on
every 2-second poll, and `gh run view --log` breaks at that size. The
isp-full lanes use the same helper, and it is tracked separately.

The red run spent about 31 minutes in install, most of it waiting out
the 30-minute gate; the green one took about 9 minutes on
`oracle-vm-24cpu-96gb-x86-64`. Moving to a smaller runner needs a
measured run, so it is a follow-up and not part of this PR.

Follow-ups from review, none blocking:

- No `vars.NIGHTLY_ENABLED` gate, so with the nightly switched off the
lane keeps testing a stale `nightly` tag. A `workflow_run` trigger on a
successful mirror would fix that too.
- The Package endpoint `cozy-hosted-control-plane` has to agree with
`KIND_CLUSTER: cozy-hosted`, and no test checks that yet.
- The operator registration check greps a Go literal in
`cmd/cozystack-operator/main.go`, so reformatting that table breaks it.
- The oras download has no checksum check, same as in `nightly.yaml`.
- cozytest stops at the first failing test, so on a red install the "no
Cilium" check never runs. Running it before the install would keep that
evidence in red runs.

Closes #3390

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds CI files, test files and one new make target. Nothing
under `hack/` is moved or renamed and no existing target changes, so the
ccp trigger does not apply.

### Release note

```release-note
ci(e2e): add a daily lane that installs the isp-hosted platform variant on kind
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added scheduled and manually triggered end-to-end checks for hosted
installations using the latest nightly packages.
* Validation covers platform readiness, package deployment, and expected
networking and storage behavior.
* Test runs collect diagnostic reports and attempt cleanup after
failures to support investigation.
* Added checks to verify hosted installation settings and ensure
prohibited network and storage components are absent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:14:05 +03:00
IvanHunters 14eab3cfe3 fix(backups): restrict the kafka-metadata CLI Pod and correct demo personas (#4695)
## What this PR does

This fixes two problems in the Kafka and NATS backup demos under
`examples/backups/`.

The Kafka metadata demo ran its CLI Pod without any securityContext, so
a namespace that enforces the restricted Pod Security level would reject
it. The Pod now gets the same restricted context as the CLI Pod in the
topic-data demo: `runAsNonRoot`, the `RuntimeDefault` seccomp profile,
no privilege escalation, all capabilities dropped. The new bats case
reads the overrides passed to `kubectl run` and fails if any of those
fields is missing or the patch is not sent as `--override-type=json`. It
is red on main, and it goes red on each of seven single-field mutations
of the patch while the other five cases stay green.

One known limit: `runAsNonRoot` needs an image that runs as a non-root
user. The default Strimzi image runs as UID 1001 and is fine. If
`KAFKA_IMAGE` is overridden with an image that runs as root, the kubelet
refuses to start the container, and the helper fails only when its 5
minute `kubectl wait` times out.

The kafka and nats walkthroughs told a tenant to run scripts that the
tenant roles forbid. Steps 03, 04, 06, 07 and `cleanup.sh` were in the
tenant column, and the scenario docs and several script comments had the
tenant create the backup Secret. Those rows now say admin and name the
part a tenant admin can do on its own, and the scenario docs say the
same. After this change the persona column in both tables matches the
tenant roles. kafka-metadata has no persona table, so I didn't change
anything there.

These are the tenant rules on pods and backup objects, plus one example
of the per-app secret rules. Each app chart's dashboard Role allows
`get`, `list` and `watch` on a few named Secrets and no other verb on
secrets. Nothing aggregated into the tenant roles grants `pods/exec`,
`pods/log`, `helmreleases`, `objectstorage.k8s.io`, `statefulsets` or
`kafkas.kafka.strimzi.io`.

```yaml
# packages/system/cozystack-basics/templates/clusterroles.yaml:100-112 (cozy:tenant:view:base, every tenant level)
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - persistentvolumes
  - endpoints
  - events
  - resourcequotas
  verbs:
  - get
  - list
  - watch
# packages/system/cozystack-basics/templates/clusterroles.yaml:238-246 (cozy:tenant:admin:base)
- apiGroups: [""]
  resources:
  - pods
  - services
  - persistentvolumes
  - endpoints
  - events
  verbs:
  - delete
# packages/system/backup-controller/templates/tenant-clusterroles.yaml:35-44 (cozy:backups:admin)
- apiGroups: ["backups.cozystack.io"]
  resources:
  - plans
  - backupjobs
  - restorejobs
  verbs:
  - create
  - update
  - patch
  - delete
# packages/apps/bucket/templates/dashboard-resourcemap.yaml:6-14 (bound at the tenant "use" level)
- apiGroups:
  - ""
  resources:
  - secrets
  resourceNames:
  {{- range $name, $user := .Values.users }}
  - {{ $.Release.Name }}-{{ $name }}-credentials
  {{- end }}
  verbs: ["get", "list", "watch"]
```

How each changed row runs into those rules:

| Row | What the script needs | Why a tenant can't |
| --- | --- | --- |
| 03 (kafka, nats) | waits on the `HelmRelease` and COSI objects, `get`
on the `bucket-<name>-backup` Secret, kafka also reads the CA Secret in
`tenant-root` | no `helmreleases` or `objectstorage.k8s.io` rule; the
bucket Role covers only `<release>-<user>-credentials` |
| 04 (kafka, nats) | waits on the `HelmRelease` and the Strimzi `Kafka`
or the `StatefulSet`, `create` on `secrets`, `create` on `pods`,
`pods/exec` | pods are `get`/`list`/`watch`/`delete` only, and nothing
grants `create` on secrets |
| 06 (kafka, nats) | `create` on `pods` and `pods/exec` for the CLI Pod,
kafka also `get` on `pods/log` for the refused restore | same pod rules,
and nothing grants `pods/log` |
| 07 (kafka, nats) | same as 04 | same as 04 |
| `cleanup.sh` (kafka, nats) | `delete` on `secrets`, on `backups`, and
on the cluster-scoped `BackupClass` and strategy | `cozy:backups:admin`
deletes only plans, backupjobs and restorejobs, and no tenant rule
covers secrets or cluster-scoped objects |

Step 05 stays a tenant step. Its `BackupJob` comes from
`cozy:backups:admin`, so in practice that means a tenant admin. The nats
demo's CLI Pod has the same missing securityContext as the
kafka-metadata one, and I left it for a separate issue.

Closes #4625

Closes #4636

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): the kafka-metadata backup demo now runs its CLI Pod with the restricted Pod Security context, and the kafka and nats backup demos now list steps 03, 04, 06, 07 and cleanup as admin steps, because they need Pod, exec, Secret and cluster-scoped permissions that tenant roles do not grant.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Security**
* Kafka metadata backup CLI Pods are recreated with restricted security
settings when they are not running or lack the required non-root
configuration.
* **Documentation**
* Kafka and NATS guides clarify which setup, restore, and cleanup steps
require administrator access and which job actions tenants can perform.
* Updated examples clarify that the demo provides required storage
Secrets and certificates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:54 +03:00
IvanHunters 1621c471f2 ci(dashboard): run eslint on the console and clear its errors (#4692)
## What this PR does

The console now passes its own eslint config, and CI runs it. On main
`pnpm lint` exits 1 with 54 problems, 51 of them errors, and
`ui-test.yaml` never ran it. On this branch it exits 0 with no errors,
and a Lint step sits between Typecheck and Test.

How each rule was handled:

- `no-explicit-any` (38 errors): replaced with real types. These commits
build a vite bundle byte-identical to main's.
- `react-refresh/only-export-components` (6 errors): the hooks, context
objects, `resolveSource` and the RJSF registry moved out of component
files into their own modules. `useTenantContext` stays in
`tenant-context.tsx`, so its importers and the tests that mock that
module are untouched.
- `no-unused-vars` (1 error): dropped the unused `ad` binding in
`OverviewTab`.
- `react-hooks/set-state-in-effect` (6 errors): relaxed per line with a
reason, not in the config.

The last rule is the only relaxation. Each of these effects re-syncs
local state to something that changes outside the component. The rule
wants the state derived during render, which changes when the value is
committed or what the first render shows, so it would be a behaviour
change. Per file:

- `components/ResourceQuotasField.tsx` re-syncs the inputs after the
form value is reset from outside.
- `components/SourceWidget.tsx` re-syncs the selected option when the
value loads asynchronously in edit forms.
- `components/command-palette/command-palette.tsx` resets query and
level whenever the palette closes.
- `components/command-palette/use-keyboard-nav.ts` moves the highlight
back to the top when the item list changes.
- `hooks/use-is-mac.ts` would return `true` on the first render on macOS
with a lazy initial state. Today it returns `false`.
- `lib/tenant-provider.tsx` falls back to an existing tenant once the
tenant list has loaded.

The eslint config itself is unchanged, and `no-explicit-any` stays on
everywhere.

`pnpm typecheck` and `pnpm test` (442 tests) pass at every commit. After
the file moves the bundle has the same chunk set, and only the entry
chunk differs, by reordered and renamed minified functions.

Three `react-hooks/exhaustive-deps` warnings from main are left as they
are, because fixing them changes when the effects run. One is in
`SourceWidget.tsx`, where the effect reads `options` but depends only on
`value`. The other two come from `ads = adList?.items || []` in
`use-command-items.tsx`, which gives two `useMemo` hooks a new
dependency on every render.

Things I did not do here:

- The step has no `--max-warnings`, so a new warning will not fail CI.
Once the three warnings are fixed, `--max-warnings 0` can go in.
- `SourceField.tsx` and `SourceWidget.tsx` each declare the same small
`SourceOptionSchema` interface. I kept them local since the two widgets
share nothing else.
- The job name stays `Typecheck and test`, so a required check with that
name keeps matching.
- `pnpm -r lint` only reaches `apps/console`.
`packages/{k8s-client,ui,types}` have no `lint` script, same as on main.

Closes #3911

Closes #4566

### Screenshots

No visual change: types, module layout and CI only.

### Downstream repositories

The diff touches only the console workspace and `ui-test.yaml`. The
console has no downstream copy since `cozystack-ui` was archived, and no
trigger map entry matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(dashboard): the console's eslint now runs in the UI test job, and the console passes it with no errors.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Tenant selection now uses available tenant namespaces, restores a
saved selection when possible, and falls back to the root tenant or the
first available tenant.
* Tenant options are sorted, and the selected tenant is kept in sync
with the available choices.
* **Chores**
* Automated checks now include linting alongside type checks and unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:31 +03:00
IvanHunters 9239555e6a ci(workflows): pin flux, yq and helm-unittest installs (#4645)
## What this PR does

Workflows no longer run the flux install script as root. Every step that
needed the flux CLI downloaded `https://fluxcd.io/install.sh` and ran it
with `sudo`, so whatever that host served at the moment ran on runners
that later get registry credentials. PR #4442 pinned the version, but
the script was still fetched and executed. All 12 of those steps now use
`fluxcd/flux2/action`, pinned to the v2.8.6 commit with `version:
"2.8.6"`. The action downloads the release tarball, checks it against
that release's checksums file, and adds the binary to `PATH`.

Two workflows still took yq from the floating latest release, nightly
and the fork e2e lane. They now use the same install the promotion steps
already have: yq v4.53.3 checked against a committed SHA-256. I verified
that hash against the release's `checksums` file and against the binary
itself.

The helm-unittest plugin was installed at whatever version was newest.
The plugin embeds its own Helm, so a plugin release could change chart
test results on a PR that touched no chart. It's pinned to v1.1.2 now,
the version CI already runs. A runner that carries a different version
gets it replaced.

A new contract test scans every workflow for these three tools, because
an unpinned install usually comes back by copying an older step. On main
it fails on the install script, the missing setup action, the
latest-release yq and the unpinned plugin. I ran one mutation per guard,
and each one failed only the test meant to catch it. The existing
contract tests that checked `FLUX_VERSION` inside the old steps now
check for the action step.

The e2e sandbox image still runs `install.sh` in its Dockerfile. That's
an image build, not a workflow step, so I left it for a separate change.

Closes #3619

Closes #3640

Closes #4510

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only workflow tool-install steps and `hack/*.bats`
contract tests. Nothing in the trigger map reaches them.

### Release note

```release-note
ci(workflows): install flux through the commit-pinned upstream setup action instead of running fluxcd.io/install.sh as root, pin yq with checksum verification in the nightly and fork e2e workflows, and pin the helm-unittest plugin version.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Reliability**
* CI workflows now use a consistent, pinned Flux version. The yq and
Helm plugin setup also verifies or enforces specific versions.
* **Tests**
* Added checks to ensure workflow tool versions are pinned and
consistent, and that downloaded yq releases are checksum-verified.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:11:02 +03:00
IvanHunters 40638429bd fix(release): collect refs stored as registry/image/tag/digest keys (#4697)
## What this PR does

The shared image-ref enumeration skipped refs stored the way upstream
ingress-nginx stores them: `registry`, `image`, `tag` and `digest` as
separate keys, with the repository in `image` and a digest that has no
`@`. Two refs in the tree use this layout, the ingress-nginx controller
and its default backend, both in
`packages/system/ingress-nginx/values.yaml`. Every tool that reads refs
through `hack/lib/image-refs.sh` missed both of them: the promote retag,
the nightly mirror, the multi-arch gate, the stitch step and the
promoted-packages verifier.

I added a sixth YAML shape that joins `<registry>/<image>@<digest>`. It
fires only on a complete `sha256:` digest, so upstream's unpinned
default (`digest: ""`) and image maps without a digest emit nothing. It
checks that `digest` is a string before `test()`, same as shape 3 does.
The library header and `docs/agents/image-refs.md` now say "the YAML
sub-shapes listed" instead of a number, so the next shape doesn't make
the count wrong again.

The real tree now gives 93 refs instead of 91, and the two new ones are
exactly the ingress-nginx pair. The retag and mirror output does not
change today, because both refs sit outside the owned registry and the
ownership filters drop them. Both digests are multi-arch indexes, so
`hack/verify-multiarch.sh --refs-file` with just those two passes and
the release gate stays green.

To show the consumers really pick the shape up, I ran both dry-runs on a
copy of the tree with the controller moved under the build registry, the
host being the whole `registry` value. In that copy the controller is
the only build-registry ref.

Before:

```text
$ REGISTRY=iad.ocir.io/idyksih5sir9/cozystack hack/promote-retag.sh 1.99.0 --dry-run
No cozystack-owned digest-pinned image refs found under iad.ocir.io/idyksih5sir9/cozystack/ — is this the rc's baked tree?
exit 1
$ hack/nightly-mirror.sh 1.99.0 <tree>/packages --dry-run
No cozystack-owned digest-pinned image refs found under iad.ocir.io/idyksih5sir9/cozystack/ in '<tree>/packages' — is this the baked main tree?
exit 1
```

After:

```text
$ REGISTRY=iad.ocir.io/idyksih5sir9/cozystack hack/promote-retag.sh 1.99.0 --dry-run
▸ iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller  sha256:a8ddcee59b63f6cbf93f4d3c9ffa1d1dda3c1d39497748d4484655356c68b3f6
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller@sha256:a8ddcee5… docker://iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller:1.99.0
exit 0
$ hack/nightly-mirror.sh 1.99.0 <tree>/packages --dry-run
▸ iad.ocir.io/idyksih5sir9/cozystack/ingress-nginx-controller  ->  ghcr.io/cozystack/cozystack/ingress-nginx-controller  sha256:a8ddcee5…
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/…/ingress-nginx-controller@sha256:a8ddcee5… docker://ghcr.io/cozystack/cozystack/ingress-nginx-controller:1.99.0
DRY-RUN skopeo copy --multi-arch all docker://iad.ocir.io/…/ingress-nginx-controller@sha256:a8ddcee5… docker://ghcr.io/cozystack/cozystack/ingress-nginx-controller:nightly
exit 0
```

The red test is a new case in `collect_refs_from_file understands every
YAML shape`. It has a fixture in the ingress-nginx layout, with a
non-string digest in front of it, and three near-misses that must emit
nothing: no digest, an empty digest, and a digest without `image`. On
main the positive case fails with `got: <nothing>`. Removing the rule,
the registry join, the full-digest check, or the type guard on `image`
or on `digest` turns that test red, and only that one. The bats suites
of all library consumers pass with both the old and the new library.

Two gaps remain in the tests and I left them out on purpose. No fixture
covers the new rule without a `registry` key, and none pins its type
guard on `registry`. Shapes 2 and 3 have the same unpinned `registry`
guard on main.

The overlay script has no such gap. `hack/overlay-main-images.sh` does
not use the library, and its line classifier already accepts `image:`,
`registry:`, `tag:` and `digest:` lines.

Closes #4518

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff. None of the changed files are
in it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(release): the image-ref enumeration used by promotion, the nightly mirror and the multi-arch gate now reads refs stored as separate registry/image/tag/digest keys, the layout of the ingress-nginx controller and default backend.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Image references can now be read from configurations that store the
image name and SHA-256 digest separately, with an optional registry.
* **Bug Fixes**
* Incomplete or invalid image and digest values are ignored rather than
producing malformed references.
* **Documentation**
  * Documented the additional image storage format.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:10:58 +03:00
IvanHunters 9196b9798e fix(e2e): bound the dataplane capture collector by wall clock (#4640)
## What this PR does

The e2e data-plane collector limited itself by counting pods and
LoadBalancers, but both of its callers stop it by the clock.
`hack/cozytest.sh` gives it 600s and the Chainsaw catch gives it 300s.
One affected pod could already run past the smaller limit, and the
LoadBalancer capture, which runs last, was the part that got lost. I
moved the collector to a wall-clock budget and fixed the other issues
filed against it.

- Every kubectl call now goes through `dp_run`. It cuts the call's
timeout to what is left of `COZY_DATAPLANE_BUDGET` and starts nothing
once the budget is spent. The default is 270s, and `cozytest.sh` passes
570s. The pod section stops 90s early, so the LoadBalancer section
always gets time to start. A call the deadline cut says so, a call cut
by its own bound names that bound, and a refused call says it was not
started and names its section; exec blocks carry that line inside the
block. The node capture and the LoadBalancer probe used to blame an
unanswered lookup or a missing cni-server when the budget stopped a
call. Now they name the budget (#3671).
- A static test fails on any kubectl call that bypasses `dp_run` and
lists the lines. It matches by command position, so a call inside a
double-quoted `$(...)` is still checked (#3694).
- Per-pod and per-LoadBalancer capture files join namespace and name
with `_`, which neither can contain. A second run into the same
directory appends, with the run separator `capture-notes.txt` already
uses (#3679).
- `capture_node` and `capture_lb_node` no longer write their callers'
loop variables. A static test fails when a function assigns a name that
a `read -r` loop or another scope owns (#3675).
- The tests that shrank every timeout to one second now cut only the
read under test. A `timeout` shim decides by argv, so a slow machine
can't cut a read the test needs answered (#3685).
- `cilium-dbg bpf lb list` runs unfiltered in every affected node's
capture, not only for a failing LoadBalancer (#3689).

The call-site comment in `.chainsaw.yaml` and the collector paragraph in
`docs/agents/e2e-testing.md` now match the code. Every behaviour change
has a test in `hack/capture-dataplane.bats` that failed on the code
before it.

Two parts of the issues live elsewhere. The runner stopping a bats file
at its first failing test is #4620, and a conntrack read filtered by
service VIP is #4623.

From #4629 I fixed the items on lines this PR already changes: the node
capture now gets the run separator, a failed speaker log read writes a
note, the endpoint tcpdump says when it falls back to the geneve
interface, and the per-read timeout knobs are validated (a zero there
would otherwise look like a spent budget). The rest of #4629 stays open:
`ovn-lflows.txt` and `lb-speaker-logs.txt` are still replaced on each
run, and two comments still describe older behaviour.

- Closes #3694
- Closes #3671
- Closes #3679
- Closes #3675
- Closes #3685
- Closes #3689
- Refs #4629

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): bound the failure-path data-plane collector by a wall-clock budget, keep its per-pod and per-LoadBalancer captures from overwriting each other, and capture the cilium service map for ClusterIP failures.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Data-plane diagnostics now follow a configurable time budget,
reserving time for LoadBalancer captures and stopping operations when
the budget is spent.
* Capture reports distinguish failed, successful, unknown, and
budget-limited checks. LoadBalancers with successful probes are skipped
for heavy capture.
* Repeated captures are preserved together, and limits help control
LoadBalancer diagnostic collection.
* Diagnostics include additional network details from relevant nodes
during probe replay.
* **Documentation**
* Updated guidance covers capture time limits and how unanswered
EndpointSlice reads are reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:10:25 +03:00
IvanHunters f209c477e2 test(e2e): read the opensearch release history through the pinned name (#4689)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The opensearch suite now reads HelmRelease history through the shared
jsonpath, and a unit test fails if any Chainsaw suite goes back to a
copied literal. Before this, the suite carried its own copy of the
expression, and the upstream contract test in `internal/fluxcontract`
only checks `HELMRELEASE_HISTORY_JSONPATH`. I checked the copy against
helm-controller `api` v1.5.1, the version go.mod pins: the history field
is `history` and each snapshot has `status`. So the read works today, it
was just not pinned.

The suite passes `-o "jsonpath=${HELMRELEASE_HISTORY_JSONPATH}"` now,
which gives kubectl exactly the same argument as before. The new test
reads every suite under `hack/e2e-chainsaw/`. A suite that calls
`helmrelease_has_remediation_cycle` must fill the local it hands the
guard from the shared name, and no suite may assign that name itself. It
reuses the matchers that already pin `run-kubernetes.sh`. The test is
red on main and green with the fix.

One known limit. The second-assignment check counts over the whole file,
which fits `run-kubernetes.sh` because it runs as one shell. A Chainsaw
suite runs each `script` step in its own shell, so an unrelated
`statuses=` in another opensearch step would turn the test red with
nothing broken. That fails toward red, never toward a silent pass, so I
left it.

Closes #4550

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(e2e): the opensearch suite reads HelmRelease history through the shared jsonpath, which is now pinned against the upstream Flux type for every Chainsaw suite that runs the remediation guard.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Expanded validation across Chainsaw end-to-end suites to check
consistent HelmRelease history handling in remediation checks, including
the order and isolation of script steps.
* The OpenSearch suite now uses the shared history lookup. Empty-history
checks and remediation-cycle failure handling are unchanged, and there
is no change to end-user behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:08:51 +03:00
IvanHunters b4fbaf00a2 test(e2e): keep gateway and redis-tls op ceilings above their own waits (#4643)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Two Chainsaw script ops had a timeout smaller than the waits inside
them. When the op ceiling fires first, Chainsaw kills the shell: the
script never prints its own error, and a cleanup trap never runs.

The gateway listener-set step could poll for 660s under a 6m op. Its
cleanup trap also deleted two tenants with no limit on the wait. Each
delete now runs under an outer `timeout`, because kubectl's own
`--timeout` does not cover discovery or a stalled request. A tenant
delete gets 300s, the same 5m that `.chainsaw.yaml` gives a delete op.
The parent is deleted only after the child is gone, because deleting a
parent while its child is still uninstalling wedges the parent's
cleanup. A failed tenant delete fails the step with kubectl's own error,
so a leaked tenant shows up here and not in a later suite. The op is now
24m, which covers the polls plus those bounds.

The redis-tls CA-projection step has 30s read caps that add up to well
over its 5m op. Raising the op would not hold, because several reads run
in loops. So all reads now share a 240s step budget. No read starts
after the budget runs out, and a read that is still running gets cut off
when it does. A read that hits its bound, or finds the budget already
spent, says which read it was.

Two exec checks in the same step blamed the 15s redis-cli cap for any
failure, including a refused exec or a missing container. They now print
the exit status and what the exec printed.

The step also runs under bash now, for pipefail. The sandbox's `/bin/sh`
is dash, which rejects `set -o pipefail`. Chainsaw 0.2.15, the version
the sandbox image installs, supports a `shell:` field on script ops.
Without pipefail, a failed Secret read took base64's zero exit status
and was reported as a wrong source name.

The private-key check read the projection a second time, so a failed
second read made it pass. It now checks the value the step already
fetched.

I added `hack/chainsaw-op-budgets.bats` to guard all of this. It holds
the gateway op above its polls plus its delete bounds, keeps the
child-before-parent order, holds the redis budget under its op with
every kubectl call going through the clamped helper, and runs the redis
step's script against a stubbed kubectl to check each failure message
and the clamp. On main, 11 of its 12 tests fail. The one that passes is
the healthy-stub control.

Closes #4114

Closes #4250

Closes #4246

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches two Chainsaw suites and adds one unit test file under
`hack/`. It moves or renames nothing a downstream repository reads.

### Release note

```release-note
test(e2e): the gateway listener-set and redis-tls projection steps now fail on their own errors instead of being killed by the op timeout, and redis-tls reports the real cause of a failed read or exec.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added automated checks to verify gateway operations have sufficient
time to complete and cleanup deletions are bounded and correctly
ordered.
* Expanded gateway scenarios to cover cleanup failures: if child-tenant
deletion fails or times out, the parent is left in place; deletion
failures are reported.
* Updated Redis TLS checks with bounded read times and a shared
execution limit. Failure reports include command output and status, and
checks cover private-key material in the decoded projection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:07:37 +03:00
IvanHunters 1ca603a7b2 test(e2e): wait for the S3 gateway to accept a key before using it (#4694)
## What this PR does

The bucket e2e suite fails now and then because the S3 gateway rejects a
key that COSI already reports as granted. The error is `The access key
ID you provided does not exist in our records.` from `mc alias set`. In
job 109130131391 the BucketAccess assert finished at 21:23:43.57 and the
first `mc alias set` failed at 21:23:44.82. In job 109396174094 cap1 and
cap2 were accepted, and cap3 was rejected two seconds into the step.

Granted access is not the same as a usable key. In seaweedfs-cosi-driver
v0.3.1, `DriverGrantBucketAccess` writes the new key into
`/etc/iam/identity.json` on the filer and returns. The S3 gateway in
SeaweedFS 4.45 reloads that file later, from a filer metadata
subscription (`weed/s3api/auth_credentials_subscribe.go`). In our chart
the driver writes through `seaweedfs-filer` and the two gateways follow
`seaweedfs-filer-client`, so the change may have to pass through the
other filer replica first.

The suite now waits until the gateway accepts the key before it writes.
Every alias in both write steps goes through `await_key`, which uses the
same port-forward as the writes and so asks the same gateway pod. It
waits only while the answer is the InvalidAccessKeyId text, for at most
60s. Any other error fails at once, and so does an attempt cut off at
its 15s cap. When the window runs out, the step fails with a message
that names the BucketAccess.

One probe costs at most 77s, so both steps go from a 5m to an 8m op
timeout. The inner budgets now sum to 409s and 441s, and the budget
comments say so.

I checked the probe locally against a fake `mc`:

```text
case=ok rc=0 attempts=1 elapsed=0s
  Added `a` successfully.
case=lag rc=0 attempts=3 elapsed=5s
  Added `a` successfully.
case=other rc=1 attempts=1 elapsed=0s
  FAIL: mc alias set for BucketAccess bucket-test-admin failed (exit 1) with an error other than the key not being loaded yet
case=never rc=1 attempts=31 elapsed=60s
  FAIL: S3 gateway still rejects the key of BucketAccess bucket-test-admin after 60s of probing (the gateway never loaded it, or the driver never wrote it)
```

A hanging `mc` fails on its first attempt after 15s with exit 124.
`chainsaw lint test` from v0.2.15 accepts the file.

The read-only upload check gets fixed along the way. Before, an unloaded
`ro-user` key also made the upload fail, and that failure passed as the
expected denial.

One thing I left as it is: `await_key` is copied into both steps,
because chainsaw steps share no shell, and moving it to `_lib/` would
run every suite on each edit.

Closes #4580

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): the bucket suite waits for the S3 gateway to accept a new key before writing with it, instead of trusting BucketAccess readiness.
```
2026-10-05 14:07:16 +03:00
IvanHunters 858ca4d0a3 test(e2e): cover monitoring OIDC modes with a Chainsaw suite (#4678)
## What this PR does

Monitoring OIDC gets live-cluster e2e coverage again. It had none since
the two pre-Chainsaw BATS suites were removed without a port. This adds
a Chainsaw suite in `hack/e2e-chainsaw/monitoring/` that ports what they
asserted, plus the in-place switch from an inline config to a
`secretRef` that nothing covered.

The suite takes one Monitoring release in `tenant-test` through three
modes in a row: System, CustomConfig with an inline config, and
CustomConfig with a `secretRef`. It creates that release itself and
never reads the tenant-root monitoring instance, so it keeps working if
#4023 or #4479 turns root monitoring off.

What each phase checks:

- System mode: cozystack-api admits `spec.oidc` and both HelmReleases go
Ready. The Grafana CR points `auth.generic_oauth` at the cozy realm, has
no `role_attribute_path`, and has `skip_org_role_sync` and
`oauth_allow_insecure_email_lookup` forced. The KeycloakClient is
confidential and references the generated Secret, which holds 32
alphanumeric characters. The audience scope is bound to this release's
clientId. There is no chart-owned `KeycloakRealmGroup`. The users-Job
carries the users and the mode, and `grafana-deployment` reads the
client secret from the Secret.
- Switch to inline: the Grafana CR takes the tenant's map, keeps the
forced keys and drops every System-only key. The users-Job reruns with
the new list. The KeycloakClient, the audience scope and the client
Secret are pruned, and the Deployment stops referencing that Secret.
- Switch to `secretRef`: the inline `auth.generic_oauth` block is gone
from the Grafana CR. The operator Secret is mounted at
`/etc/grafana/oidc` on both the CR and the Deployment. This checks the
mount only and does not prove Grafana loads the file, see #4679.
- After the install and after each switch, neither release may show a
remediation cycle. That means no `failed` or `uninstalled` snapshot, and
the latest revision must be exactly 1, then 2, then 3. The revision
check is there because helm-controller truncates history after the
previous deployed snapshot. An upgrade that failed, rolled back and then
passed on retry leaves no failed entry, only extra revisions.

I did not assert `GF_PATHS_CUSTOM_INI`. I could not find anything in
Grafana that reads it (#4679), and pinning it would make possibly dead
wiring part of the contract.

The coverage split with the render tests is this.
`packages/system/monitoring/tests/oidc_test.yaml` keeps the
fresh-install render of each mode, including the CustomConfig negatives
(no Keycloak objects, no client Secret). This suite covers what a render
cannot see: admission, the Deployment the Grafana operator builds, and
what a switch between modes prunes or leaves behind. The Keycloak
objects are checked as written, so creating the client in Keycloak is
not asserted. Removing it is, because the operator's finalizer holds the
KeycloakClient until the client is gone from Keycloak.

Before the first apply the suite also waits until the tenant's
`cozystack-values` carries `oidc-enabled: "true"`. The outer release
copies `_cluster` into the inner release once, so a Monitoring applied
right after the tenant is created can render from stale values and fail
the System-mode guard for the full 15m. That is what the first CI run on
this PR hit.

The Keycloak CRDs are required, not probed for. The e2e install always
enables platform OIDC, so a missing CRD is a broken install. The removed
suites skipped in that case, and their probe could not tell an absent
API group from a present one.

The selector picks the suite up by name from
`cozystack.monitoring-application` and `cozystack.monitoring`. A change
to `system/monitoring`, `extra/monitoring`, `monitoring-rd` or
`grafana-operator` now runs only this suite instead of the full run. The
tenant-root Grafana bring-up stays covered by the install BATS while it
exists. One selector unit test relied on `cozystack.monitoring` reaching
no suite, so it now renames that source in its copy of the graph.

The second commit keeps Keycloak changes on every suite that drives a
Keycloak client. Before this PR a change to `system/keycloak` or
`system/keycloak-operator` escalated to the full run. With the
monitoring suite it would select monitoring alone, because
`monitoring-application` is the only source that depends on
`keycloak-operator`, and `kubernetes-latest` with its OIDC lifecycle
would drop out. `src_to_suites()` now maps both to `kubernetes-latest
monitoring`. The mapping has to be explicit because
`kubernetes-application` has no `dependsOn` on `keycloak-operator`
(#4680). A `postgres-operator` change now also reaches
`kubernetes-latest`, through Keycloak's database. The selector comment
and the e2e doc no longer use `system/postgres-operator` as the example
of a path with a source that reaches no suite.

Cost: one monitoring stack bring-up, two in-place upgrades and teardown.
I expect 10 to 20 minutes on every full run. The selector change makes
the CI run on this PR a full run, so the suite's own timing is in the
chainsaw report. A red run can spend up to 16m on each of four waits,
since each one sits a minute above the 15m timeout of the release it
waits on.

The suite cannot run locally, so I checked it this way:

- `chainsaw lint test` at v0.2.15, the sandbox version, passes.
- I ran the suite's own steps against a local envtest apiserver.
Chainsaw applies the real manifests, and a stand-in for helm and the
operators renders the chart from the stored Monitoring spec. With the
chart intact all nine steps pass. Each of 30 single breakages of the
chart or of the simulated controllers turns the expected step red.
- `hack/select-e2e_test.bats` and `hack/select-install_test.bats` pass.
The new Keycloak selection case fails without the mapping and when
`kubernetes-latest` is dropped from it.

A browser login through Keycloak is out of scope, same as in the removed
suites.

Left for later:

- KeycloakClient and scope `status` after the operator creates them in
Keycloak is not checked.
- `hack/e2e-chainsaw/README.md` lists suites by hand. I added
`monitoring`, but `backup`, `kafka-metadata` and `kubernetes-proxmox`
were already missing.
- The `operator diff selects all dependent app suites` selector test
could also check `monitoring`.
- The comment on the Keycloak mapping says both suites wait for the
finalizer to remove the client. The monitoring suite does wait.
`kubernetes-latest` only reads once after its upgrade.
- The ready checks compare `observedGeneration` with
`metadata.generation` through `contains()`. Both are fields of the same
object, not a number literal, so the v0.2.15 literal gotcha should not
apply, but no live run has passed this suite yet.
- The inline fixture carries no `enabled: "true"`, so that phase checks
a disabled OAuth block. The asserts do not depend on it.

Closes #4051

Closes #3785

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added end-to-end coverage for Monitoring installations using system
OIDC, inline custom configuration, and Secret-based configuration. The
tests verify readiness, rendered resources, configuration cleanup, and
upgrade history.
* Updated test selection so Keycloak-related changes also run the
Monitoring suite.
* **Documentation**
* Added Monitoring to the documented Chainsaw suite list and clarified
how changed paths determine test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:07:10 +03:00
IvanHunters 6e4c9c4b6a fix(ci): harden the PR labeler's title parsing and map recurring scopes (#4666)
## What this PR does

Fixes three problems in `.github/workflows/pr-labeler.yaml`. Each one
either killed the job or put a PR in the wrong area, and none of them
showed up anywhere.

- Both maps are now looked up with `Object.hasOwn`. A title typed or
scoped `constructor`, `toString`, `hasOwnProperty` or `__proto__` used
to resolve to an `Object.prototype` member and crash the step on
`l.startsWith`, which left the PR with no labels at all.
- Every leading `[Backport ...]` prefix is now stripped, not only the
first one. With two prefixes, the second was read as a bracket scope, so
the real scope and the `kind/*` label were both lost. The `area/release`
label hid the problem by keeping `area/uncategorized` from firing.
- Scopes that recur three or more times over the full PR history now map
to an area. So does each area's own name (`networking`, `storage`,
`database`, `virtualization`, `testing`, `dependencies`).

Scope-to-area decisions:

- `area/database`: `database`, `etcd-operator`, `opensearch`,
`opensearch-operator`
- `area/networking`: `networking`, `kubeovn`, `multus`, `kilo`,
`hetzner-robotlb`, `ouroboros` (it rewrites the CoreDNS Corefile to fix
hairpin NAT, so it is networking, not platform)
- `area/virtualization`: `virtualization`, `vm`, `kubevirt-operator`,
`kubevirt-cdi`, `kubevirt-instancetypes`
- `area/platform`: `operator`, `cozystack-operator`, `controller`,
`cozystack-controller`, `cozystack-basics`, `flux-shard-operator`,
`migrations`, `scheduler`, `cozystack-scheduler`, `reloader`, `bootbox`,
and `lineage`. I put `lineage` here on purpose: no existing label covers
it more precisely, and it is too narrow for an area of its own.
- `area/storage`: `storage`, `piraeus-operator`,
`backupstrategy-controller`
- `area/release`: `changelog`, `changelogs`
- `area/ci`: `workflows`, `labeler`
- `area/dependencies`: `dependencies`, `renovate`
- `area/api`: `registry` (the cozystack-api storage layer)
- `area/dashboard`: `dashboard-controller`
- `area/monitoring`: `monitoring-agents`
- `area/keycloak`: `keycloak-configure`
- `area/testing`: `testing`
- New `area/dx`, added to `.github/labels.yml`: `dx`, `hack`, `cozypkg`.
These are developer tooling (hack/ scripts, package Makefiles, cozypkg).
`area/build` covers image builds and `area/testing` covers tests, and
neither describes this work.

Left unmapped on purpose:

- `http-cache`: the application was removed from the tree.
- `openbao`: already mapped on main.
- `apps`, `system`, `docs`: catch-all scopes that don't point to one
area.
- `fix`, `bugfix`: malformed titles.
- `security`, `rbac`, `governance`, `codeowners`, `maintenance`,
`images`, `readme`, `telemetry`, `cluster-autoscaler`: no single area
fits any of them.

I did not add spelling normalisation. `kubeovn` got its own row, since
it is the only alternate spelling that recurs.

A new contract test, `hack/pr-labeler-contract.bats`, pulls the
workflow's own `script:` block out of the file and runs it under node
with stubbed `context`, `core` and `github`, so it tests the shipped
code, not a copy. On main it is red on the prototype titles (`THROW:
l.startsWith is not a function`), on the double backport (only
`area/release`), on the newly mapped scopes and on the area-name scopes.
It also checks that every map target exists in `labels.yml` and that no
key is written twice. The suite fails rather than skips when node is
missing. The older node-based suites skip, so this is the first one that
needs node on the unit-test runner.

Some things stay out of this PR. The scope table in
`docs/agents/contributing.md` is a summary. It has no row for `area/dx`
and does not mention that an area's own name works as a scope.
Separately, the singular `migration` still maps to `area/virtualization`
as it did before, while the new `migrations` maps to `area/platform`.

A backport PR whose scope is unmapped still never gets
`area/uncategorized`, even with a single prefix. The `area/release`
added by the strip satisfies the fallback check. This behaviour is the
same on main and is not changed here.

The test that checks map targets against `labels.yml` passes by itself
if the map literal cannot be extracted. The duplicate-key and area-name
tests go red in that case, so the suite as a whole still fails.

Closes #4125, closes #4124, closes #4122

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): the PR labeler no longer crashes on titles scoped like Object.prototype members, handles nested backport prefixes, and maps more recurring scopes, including a new area/dx.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `area/dx` label for issues and pull requests related to
developer tooling.
* **Improvements**
* Pull requests can now be categorized using a broader set of scope
aliases.
* Repeated leading backport prefixes are recognized and route the pull
request to the release area.
* Label mapping now avoids treating inherited property names as valid
mappings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:06:47 +03:00
IvanHunters 02fbe76418 fix(ci): close three blind spots in the e2e test selectors (#4658)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The e2e selectors miss Chainsaw suites that have no `chainsaw-test.yaml`
or `.yml` directly in their directory. Chainsaw v0.2.15 also runs a
directory that holds only numbered step files such as `01-install.yaml`,
and its discovery walks every subdirectory of the path the runner hands
it. A suite in either layout ran only when nothing was selected. No
narrowed selection named it, the full-suite list left it out, and
`select-install.sh --validate` never asked for its mapping. Now
`hack/select-e2e.sh` and `hack/select-install.sh` list a top-level
directory as a suite when Chainsaw would load a test from it or from
anywhere below it. No suite in the current tree uses either layout, so
today's suite list stays the same.

A file placed directly in `examples/backups/` (a shared `common.sh`, for
example) selected nothing, because it fell through to the inert
`examples/` rule. It now escalates to the full suite, same as
`examples/backups/_lib/`. `README.md` there stays inert through the
`*.md` rule.

The `test-chainsaw` comment in `packages/core/testing/Makefile` said
Chainsaw discovers every `chainsaw-test.yaml`, which was too narrow. I
fixed it, and the matching sentences in `docs/agents/e2e-testing.md`
too.

Every behaviour change has a bats test that fails against the scripts on
main.

Known gaps I left for a follow-up, none of them a wrong answer today:

- The `full_suite_list` helper in `hack/select-e2e_test.bats` repeats
the selector's `find`/`sed`, so a bug in that shared pattern would pass
every full-suite assertion. The step-file and nested tests compare
against literal lists, which covers the new layouts.
- The selectors are a little wider than Chainsaw: a broken
`chainsaw-test.y*ml` symlink, or one with no Test in it, still lists its
suite. That errs toward running more.
- A few test comments describe what used to happen rather than why the
rule exists.

Closes #4630

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ci): the e2e test selectors now see Chainsaw suites made of numbered step files or nested below a suite directory, and run the full suite for an edit to a file placed directly in examples/backups/.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* E2E suite discovery recognizes supported manifest and numbered-step
files at any depth and maps them to the top-level suite.
* Changes to files directly under `examples/backups/` trigger full-suite
selection.
* Suite selection reports an error if discovery fails or finds no
suites.
* **Documentation**
* Clarified supported suite layouts, disabled suite-file handling,
discovery and escalation rules, and per-path coverage guidance.
* **Tests**
* Added coverage for nested suites, numbered steps, unmapped suite
layouts, and backup-path selection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 14:06:18 +03:00