Commit Graph
6731 Commits
Author SHA1 Message Date
scooby87 cffe509dcf feat(monitoring): per-tenant OTLP collector (tracing phase 2) (#3763)
## What this PR does

Phase 2 of distributed tracing (design cozystack/community#38, epic
#3761), built on the phase-1 backend (#3762, merged): a per-tenant
**OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It
lives in the tenant namespace (like a per-tenant vmagent), so
app→collector traffic is intra-namespace and never crosses the tenant
boundary. No OTel operator exists in the platform, so this is a plain
Deployment + ConfigMap + Service (the fluent-bit pattern).

## Changes

- **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`,
HTTP `:4318`) → `memory_limiter` + head sampling
(`probabilistic_sampler`, default 10%) + optional `attributes/redact`
(PII/secret control point) + `batch` → `otlphttp` exporter to the
`tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428`
single, endpoints per upstream docs). PSS-restricted securityContext,
digest-pinned image via `cozy-lib.image`, config-checksum roll.
- **app-facing Service `otel-traces`** (4317/4318).
- **`tracingCollector` values** — `enabled` / `replicas` /
`samplingPercentage` (schema-bounded 0-100) / `redactAttributes` /
`resources`. A **fail-guard** rejects an enabled collector fronting more
than one `tracingStorages` backend (it exports to a single backend by
design).
- **`otel-traces` WorkloadMonitor** and dashboard resource-map entry,
rendered under the collector's own gate. The same resource-map Role also
gets the pre-existing `alermanager` typo corrected to `alertmanager`, so
the alertmanager WorkloadMonitor is readable from the dashboard.
- helm-unittest coverage for the collector and its WorkloadMonitor;
regenerated `values.schema.json` / `README.md` / `monitoring-rd`
ApplicationDefinition.

## Verification

- `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all
pass.
- `helm template` renders cleanly; `make generate` idempotent.
- Live check on a test cluster (single-mode backend): the collector
starts with `redactAttributes` containing a numeric, a colon-bearing and
an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted
and is queryable from the backend through the Jaeger API, with the
redacted attributes removed and the others kept.
- Local `/branch-review` — **LGTM** (one blocker + four recommendations
fixed, each with a test: multi-backend guard, memory_limiter cgroup
limit, configurable replicas, sampling bounds, trust-boundary wording).

## Scope / deferred (epic #3761)

`AccountID`/`ProjectID` header injection for the shared-central topology
(phase 4) and engine integrations (phase 3) are follow-up PRs.

### Screenshots

N/A, no UI change.

### Downstream repositories

The diff touches only `packages/extra/monitoring`,
`packages/system/monitoring` and `packages/system/monitoring-rd`. The
website's monitoring reference page is regenerated from the package
README by the release bot, and the Terraform provider models only the
tenant's `monitoring` flag, not these values, so no follow-up is needed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage in cluster or single-instance modes,
with retention settings and Grafana datasources.
* Added an enabled-by-default OpenTelemetry collector with configurable
replicas, sampling, attribute redaction, and resource limits.
* Added workload monitoring and automatic resource recommendations for
configured trace components.
  * Added readiness health checks for both tracing backend modes.

* **Bug Fixes**
* Corrected dashboard permissions for alert monitoring and trace
collector resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 16:27:45 +03:00
Aleksei Sviridkin 368d00b5d4 test(e2e): remove unused monitoring suites and audit BATS runners (#4019)
## What this PR does

I removed the two monitoring OIDC BATS suites that had no runner after
the Chainsaw migration, and brought the runner audit and wrapper OIDC
test from [#3778](https://github.com/cozystack/cozystack/pull/3778).

Also added checks for audience mapper and `OIDC_MODE=System`. The named
`KeycloakRealmGroup` test documents coverage the document-count and kind
assertions already provided.

Runner audit reports unused BATS suites, missing files in runner paths,
and unresolved Make variables (including `$(E2E_SUITE)`).

Live Monitoring API admission, Grafana operator reconciliation and mode
transitions are not tested here.

### Validation

- `make -j4 unit-tests SHELL=/usr/bin/dash` passed.
- Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart
tests passed.
- Runner audit: 12 cases passed under dash and Bats 1.14.0. The full
shell syntax and BATS sweep passed under dash too.
- Three chart mutations and seven runner mutations failed as expected.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```
2026-09-25 16:55:05 +04:00
scooby87 03e2da5805 feat(postgres,mariadb)!: drop plaintext user passwords (#4078)
## What this PR does

Removes the plaintext `users[].password` field from the `apps/postgres`
and `apps/mariadb` charts (and the mariadb `root` account), so
managed-database passwords can no longer be pinned in tenant values or
committed to git. Passwords are always chart-generated into the
`<release>-credentials` Secret and preserved across reconciles via
`lookup`, matching the model `redis`/`valkey` already use. This closes
the "User passwords" item of the portal upstream epic
(aenix-org/cozyportal#1060).

The generated password never lands in the init-script Secret or the SQL
it runs: the postgres init-job reads each user's password from the
mounted `<release>-credentials` Secret at run time and applies it
through a psql variable (`ALTER ROLE … WITH PASSWORD :'pw'`), so it does
not linger in the init-script manifest a tenant can read. It does live
in cleartext `stringData` in the `<release>-credentials` Secret itself,
which is a chart manifest and therefore kept in Helm release history for
`MaxHistory` revisions like any chart-generated secret — that residue is
exactly why rotation cannot be built on the chart (see below). A
`password` or a `passwordRotation` key left over in a tenant's values is
accepted-but-inert and draws an admission warning, so an operator
editing it is told it has no effect rather than getting a silent 200.

Backup/restore into a copy keeps the target's chart-managed credentials
working: the CNPG driver clears `bootstrap.enabled` once recovery is
healthy so the init-job re-applies the generated passwords onto the
recovered roles, and the MariaDB driver excludes `mysql.global_priv`
from the dump (`ignoreGlobalPriv`) for backups captured from this
release onward, so restoring one of those into a copy does not clobber
the target's chart-managed users or `root`. Excluding the grant table
has a flip side, and it applies to every MariaDB restore, not only a
to-copy one: only accounts declared through `User` CRs (the chart's
`users` map) are captured, so an account created out of band directly in
MySQL is in no backup and a restore brings back its data and `mysql.db`
grants but not the account itself. This is also a backup-side fix only:
a restore from a backup taken **before** this change still replays the
source's `mysql.global_priv` and overwrites the target's users and
`root`, and `root` then needs an out-of-band reset (the operator applies
`rootPasswordSecretKeyRef` only at datadir bootstrap, so there is no
in-band path). See `docs/operations/backup-classes.md` for the
convergence window, the out-of-band-account limitation, and this
old-backup caveat.

Password **rotation** is deliberately **not** in this PR. A
chart-rendered credentials Secret keeps its cleartext in Helm release
history for `MaxHistory` revisions, so a rotation built on it cannot
revoke a leaked password; rotation is being reworked as a controller
that owns the Secret — design proposal in cozystack/community#72.

Migration is seamless: existing releases keep their current passwords on
upgrade (the value already lives in `<release>-credentials`, which
`lookup` re-reads and preserves); a `password` still present in a
tenant's values is ignored, so no existing HelmRelease fails validation.

Verification: `helm unittest` passes for both charts (postgres 87,
mariadb 126, including render-time username guards that reject any name
k8s would later reject as an invalid Secret key or User resource name);
`go build ./...`, `go vet`, the backup-controller tests and the
apps-apiserver tests are green; `make generate` was run with the
CI-pinned `cozyvalues-gen` v1.6.0 so `values.schema.json`, the READMEs,
the API types, and the `cozyrds` ApplicationDefinitions are all in sync
(the branch was rebased onto `main`, which reconciled cleanly with the
read-replica autoscaling feature that landed there).

### Screenshots

N/A — no UI change.

### Downstream repositories

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: affected — `content/en/docs/next/cozystack-api/go-types.md`
is hand-written and still shows `Password:` in the Go examples, and the
managed-app reference pages regenerate from these chart READMEs, so both
document `users[name].password` until refreshed (not yet filed).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: affected — the provider hand-maintains the postgres/mariadb
schemas and per-field defaults; this PR removes `users[].password`. A
follow-up issue/PR is needed there (not yet filed).

`terraform-provider-cozystack` and `website` are reached by this diff;
both boxes are left unticked because the follow-ups are not yet filed.
Same-repo future work this change suggests: the controller-owned
password rotation (design in cozystack/community#72), and extending the
"passwords only in the Secret" model to the other engines that still
accept plaintext (`clickhouse`, `mongodb`, `nats`, `opensearch`,
`rabbitmq`, `vpn`).

### Release note

```release-note
feat(postgres,mariadb)!: passwords for managed PostgreSQL and MariaDB users can no longer be set via `users[].password` — they are always auto-generated and stored only in the `<release>-credentials` Secret, never rendered into the init-script or the SQL it runs. Existing releases keep their current passwords on upgrade; a `password` (or a `passwordRotation`) left in values is ignored with an admission warning. Password rotation is deferred to a follow-up controller design (cozystack/community#72). MariaDB backups now exclude the `mysql.global_priv` grant table (`ignoreGlobalPriv`), so a restore recreates only accounts declared through the chart's `users` map — an account created out of band directly in MySQL is not captured and must be recreated after a restore.
```
2026-09-25 15:49:53 +03:00
myasnikovdaniil 0cb37dba9e docs(release): add changelog for v1.6.3 (#4077)
This PR adds the changelog for release `v1.6.3`.

✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.3.md`.
2026-09-25 15:41:55 +05:00
Alexey Artamonov 9d6fea7017 fix(monitoring): correct the alertmanager name in the dashboard Role
The Role listed the WorkloadMonitor as 'alermanager', so the real
'alertmanager' monitor was never granted: a tenant dashboard is
forbidden from reading its status and the alertmanager health tile
breaks. The test pins the correct name and the absence of the typo.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 13:35:34 +03:00
Alexey Artamonov 60b1b05434 feat(monitoring): add per-tenant OTLP collector for distributed tracing
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.

The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.

redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.

The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 13:35:34 +03:00
Aleksei Sviridkin 2d2186d1f0 feat(monitoring): VictoriaTraces backend + Grafana datasource (tracing phase 1) (#3762)
## What this PR does

Phase 1 of distributed tracing (design: cozystack/community#38; epic:
#3761): a **VictoriaTraces backend and Grafana traces datasource** in
the monitoring stack, mirroring the logs (`VLCluster`) implementation.
No new operator: `VTCluster`/`VTSingle` already ship in
victoria-metrics-operator.

- **`tracingStorages`** values in `extra/monitoring` (annotated,
schema-generating) and `system/monitoring` (defaults), per-entry `mode:
cluster|single`. **Empty by default, so tracing is off out of the box**:
nothing produces spans until the collector ships, so no backend is
provisioned until an operator adds an entry.
- **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or
`VTSingle` (edge/dev/small); `retentionPeriod` plus optional
`retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup
label**, reused by the existing post-delete hook, so no hook change is
needed.
- **Render-time validation** (`_helpers.tpl`): the installed
VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise
reach the operator and fail after admission. Each `tracingStorages`
entry is checked before anything renders: `name` must be an RFC 1123
label of at most 42 characters (the `vtstorage-<name>-<hash>` pod
revision label is capped at 63) and unique across the list, `mode` must
be `cluster` or `single`, `storage` must be a positive quantity (a zero
size makes the operator mount an EmptyDir in single mode, and a negative
one fails when the PVC is created), and `retentionDiskUsageBytes` must
be a VictoriaMetrics `BytesString` (the operator rejects the
Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage`
field is also typed as a quantity in the schema, and the rendered name
is quoted so a numeric name stays a string.
- **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces
datasource per entry (built-in Grafana core datasource, no plugin).
- **VPA**: a `VerticalPodAutoscaler` for each trace component
(`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle`
Deployment), matching the metrics/logs sizing, with `updateMode:
Initial`.
- **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on
`status.updateStatus`, alongside the existing `VLCluster` gate.
- **VictoriaLogs datasource fix**
(`templates/vlogs/grafana-datasource.yaml`): the range over
`logsStorages` emitted no `---` between documents, so two or more
entries collapsed into one document and only the last datasource
survived. Each entry is now its own document.
- **Tests and schema**: helm-unittest suites for off-by-default, cluster
vs single, datasource type and both query URLs, retention, PVC label,
VPA targets, every validation guard with its boundaries, and per-entry
vlogs datasource documents, plus regenerated `values.schema.json`,
`README.md` and the `monitoring-rd` ApplicationDefinition.

#### Upgrade impact

With default values the only rendered object that changes is the
`monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle`
`healthCheckExprs` and `tracingStorages: []`. Flux evaluates those
expressions only against objects in the release inventory, so they are
inert while tracing is off. No VictoriaTraces object is created on
upgrade. Installs with two or more `logsStorages` entries additionally
gain the `GrafanaDatasource` objects that were previously dropped (see
the VictoriaLogs fix above); nothing else under VictoriaLogs or
VictoriaMetrics changes.

#### Verification

- `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring`
(11 tests) pass. Each validation guard and each branch of the
zero-storage pattern was mutation-checked: reverting it turns its own
test red.
- `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly.
`tracingStorages: []` (the default) renders nothing. An invalid,
duplicate or over-long `name`, a zero or negative `storage`, or a bad
`retentionDiskUsageBytes` (`8Gi`) is refused at render with an
actionable message; a numeric `name` renders as a string. `make
generate` is idempotent.
- **Datasource URLs verified against upstream docs** ([VictoriaTraces
and
Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)):
cluster `vtselect:10471/select/jaeger`, single
`vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana
datasource; the Tempo Query API (`/select/tempo`, `type: tempo`,
VictoriaTraces v0.9.4+) has a richer TraceQL UX but is
upstream-experimental, so Jaeger is the default. Both URLs are pinned by
unit tests.
- The `retentionDiskUsageBytes` pattern, the zero-storage check and the
name limit were checked against victoria-metrics-operator v0.68.4
(`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>`
StatefulSet name) and the Kubernetes `ControllerRevisionName` naming.

#### Scope / deferred (tracked in #3761)

Per-tenant OTLP Collector, engine integrations, shared-central topology,
and docs are follow-up PRs. This PR intentionally does **not** enable
the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and
the gRPC listener lands with the collector.

Builds on the now-merged design proposal cozystack/community#38.

### Screenshots

Not a UI change.

### Downstream repositories

Walked against the diff, no box ticked yet:

- cozystack/website: `monitoring` is in the website `Makefile`
`SERVICES` list, so its reference page is regenerated from
`packages/extra/monitoring/README.md` automatically on the next stable
tag. A user guide for tracing belongs to the docs follow-up in #3761; no
website issue or PR exists for it yet.
- cozystack/terraform-provider-cozystack: the provider has no
`Monitoring` resource (only the tenant's `monitoring` flag), so the new
schema field does not reach it.
- The remaining repositories in the trigger map are not reached by this
diff: no platform values, variants, installer values, `hack/` tooling,
node prerequisites, ApplicationDefinition semantics or telemetry names
change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage with cluster or single-node
deployment modes.
* Supports retention periods, persistent storage sizing, storage
classes, and optional disk-usage limits.
* Automatically configures Grafana tracing data sources for each enabled
trace backend.
* Monitoring readiness reflects the health of configured trace storage
services.
  * Invalid trace storage settings are rejected during configuration.
* **Bug Fixes**
* Every VictoriaLogs storage now gets its own Grafana datasource; with
several log storages, all but the last were previously dropped.
* **Tests**
* Added coverage for storage rendering, retention settings, validation
guards, Grafana endpoints, health checks, and cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 14:21:26 +04:00
Aleksei Sviridkin 47df27de57 fix(api): let one ApplicationDefinition own each application kind (#4424)
## What this PR does

Nothing enforced that `spec.application.kind` is unique across
ApplicationDefinitions, and every consumer that maps a kind back to a
definition resolved a duplicate its own way. The chartRef reconciler
selected HelmReleases by the kind label alone and let every definition
rewrite every release of its kind; the lineage webhook and the CA
projection reconciler took whichever definition came first in an
unordered list; cozystack-api registered one resource per definition. A
second definition declaring a kind already in use could therefore
repoint that kind's tenant releases at its own artifact, whatever the
definition was named. Fixes #4361.

This adds one ownership rule, in `internal/shared/appdefowner`: the
definition created first owns the kind, with ties broken by name. Every
Go consumer goes through it:

- the chartRef reconciler rewrites releases only for the owner of their
kind, emits a `KindClaimed` Warning event on a definition that is not
the owner, and re-reconciles the next owner when the current one is
deleted or changes kind;
- cozystack-api registers the owner only and logs the definitions it
leaves out; its restart hash now moves when ownership moves, while a
cluster without duplicate kinds hashes exactly as before, so the upgrade
itself does not restart the API;
- the lineage webhook and the CA projection reconciler resolve the
owner, and the CA projection watch now also reacts to a kind change;
- the lineage webhook gains a validating endpoint that refuses a new
ApplicationDefinition whose kind another definition already declares. An
update that keeps its kind is allowed, so a duplicate that predates the
check keeps upgrading, and so is a second definition from the same Helm
release, because a release that renames its definition object creates
the new one before it deletes the old one. It is registered with
`failurePolicy: Ignore`, and a failure to read the existing definitions
lets the request through with a warning instead of an error answer,
which Ignore would not cover: ApplicationDefinitions ship with the
platform, so the check must never block an install or upgrade, and the
ownership rule holds whatever it lets through.

Two decisions differ from the wording of the issue, deliberately:

- a duplicate kind is not a fail-stop at cozystack-api start-up. One
malformed or duplicate definition taking the API down for every tenant
is the problem #4359's design comment calls out, so the API serves the
owner and logs the rest instead;
- "created first owns" rather than "the platform's definition owns". It
needs no notion of provenance, and a definition that arrives later can
never take a kind over. The consequence worth knowing: deleting the
owner hands its releases to the next definition declaring that kind.

Known gaps left for follow-ups, both pre-existing:

- the dashboard console (`packages/system/dashboard/images/console`)
still picks the first definition matching a kind or plural in the list
it receives. It switches no release, but with a duplicate present it can
show the duplicate's form;
- `plural` collisions are not checked: a definition with a new kind but
an existing plural still takes the plural's endpoint at registration.
Existing releases are not reachable through it, since List and Get
filter by the kind label.

Tests cover the rule itself, each consumer, the webhook (create, update
keeping and changing kind) and the chart; every guard was
mutation-checked term by term.

### Screenshots

Not a UI change.

### Downstream repositories

The change alters `ApplicationDefinition` semantics (kind uniqueness and
ownership), which the trigger map routes to `cozystack/website`
(`content/en/docs/next/cozystack-api/application-definitions.md`). The
follow-up there is not opened yet; it needs the maintainers to settle
the behaviour first. No other repository in the map is reached: no CRD,
kind, plural, prefix or values change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(api): an application kind is now owned by one ApplicationDefinition, the one created first. A second definition declaring a kind already in use is refused at admission, and one that predates the check no longer manages that kind's releases or its API registration.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application kinds now have a consistent owner: the earliest-created
definition takes precedence, with name used to break timestamp ties.
* New definitions that claim a kind already owned by another definition
are rejected. Existing duplicates can keep their current kind, and
definitions from the same Helm release can be renamed.
* **Bug Fixes**
* Runtime configuration and Helm releases now use the owning definition,
preventing duplicates from replacing its configuration.
  * Ownership changes trigger affected definitions to be reconciled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 13:31:24 +04:00
IvanHunters 0bae19fccc feat(lineage-webhook): refuse a second ApplicationDefinition for a kind
A definition declaring an application kind that another definition already
declares now fails when it is applied, instead of sitting next to the first
one where it has no effect. An update that keeps its kind is allowed, so a
duplicate that predates the check keeps upgrading, and so is a second
definition from the same Helm release, since a release that renames its
definition object creates the new one before it deletes the old one.

The webhook is registered with failurePolicy Ignore: ApplicationDefinitions
ship with the platform itself, so an outage of the webhook must never block
a bootstrap or an upgrade, and a failure to read the existing definitions
lets the request through with a warning rather than answering with an error,
which Ignore would not cover. It is the feedback layer; which definition
manages a kind is decided by the ownership rule whatever it lets through.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:39:35 +03:00
IvanHunters 0b7dd23b99 fix(appdef): let one ApplicationDefinition own each application kind
Nothing enforced that spec.application.kind is unique, and each consumer that
maps a kind back to a definition resolved a duplicate its own way: the
chartRef reconciler let every definition rewrite every release of its kind,
the lineage webhook and the CA projection reconciler took whichever came
first in an unordered list, and cozystack-api registered one resource per
definition. A second definition declaring a kind already in use could
therefore repoint that kind's tenant releases at its own artifact.

Add one ownership rule, the definition created first owns the kind with ties
broken by name, and route every consumer through it. A definition that
arrives later with a kind already in use no longer manages that kind's
releases whatever it is named: the chartRef reconciler skips it with a
Warning event and re-reconciles the next owner when the current one goes,
cozystack-api registers the owner only and logs the rest, its restart hash
moves when ownership moves, and the lineage and CA projection lookups
resolve the owner.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:39:35 +03:00
Aleksei Sviridkin 6748014165 fix(ci): overlay the right line's refs for a PR on a feature branch (#4460)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A PR stacked on a feature branch now overlays the image refs of the line
it grew from, so its E2E no longer falls back to the refs committed in
the tree.

The Finalize job pulls `cozystack-packages:<base branch>` and overlays
its image refs onto the packages the PR did not rebuild. For a stacked
PR the base is something like `fix/foo`. The slash is not valid in an
OCI tag, and no workflow publishes an artifact for that branch anyway.
So the pull failed, the step only printed a warning, and every unbuilt
package ran its committed release ref. That is how stacked PRs got the
Grafana plugin 404 described in #4391.

Now `main` and `release-X.Y` bases read their own artifact. Any other
base reads the artifact of whichever of main and the release lines it
has the fewest commits on top of, and a notice says which one. A stack
on a release line keeps that line's images, for example #3895, which
sits on `release-1.5`. On a tie main wins, so a stack on a line cut from
main reads main's artifact until the stack contains a commit of the
line's own. Always taking main there would put main's images under the
line's charts, and that is the install failure from #3437. The new bats
case builds a small git fixture and checks the tag pulled for six bases,
one of them a deleted branch.

The other half of #4391, the stale `grafana-dashboards.tag`, is a
release-managed digest pin and is not touched here. It gets re-stamped
at release prep.

Part of #4391

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ci): E2E for a PR stacked on a feature branch overlays the image refs of the line the stack grew from, instead of falling back to the committed release refs.
```
2026-09-25 03:53:37 +04:00
Aleksei Sviridkin b981ea7f26 fix(backups): stop the nats, kafka-metadata and rabbitmq backup demos from reading empty values (#4458)
## What this PR does

The nats, kafka-metadata and rabbitmq backup demos carried the two
empty-read defects the redis demo already fixed.

**Empty read from an attach race (nats, kafka-metadata).** Both
seed/verify helpers ran each CLI call in a throwaway `kubectl run -i`
pod. That pod's stdout returns over an attach that only carries what the
container writes after the attach registers, so a CLI that finishes
first comes back empty with exit 0. Neither NATS nor Kafka ships a
long-lived pod that carries its CLI (unlike rabbitmq, which already
execs `rabbitmqctl` in its `-server-0` pod), so each demo now runs one
long-lived CLI pod and `kubectl exec` into it: an exec'd process owns
its pipes, so its output cannot be lost that way, and stderr stays
attached so a failed read says why. `cleanup.sh` removes the pod.

**Single read of the platform strategy (rabbitmq, kafka-metadata).**
Both runners read the platform's `cozy-default-<engine>` strategy once,
right after install, where the Helm upgrade that renders it is gated on
`dependsOn` and can be absent for minutes. Both now `kubectl wait
--for=create` for it, so it polls through NotFound and fails only on a
real error.

Unit tests for the two rewritten helpers are added under `hack/`, in the
`cozytest.sh` dialect, and assert the reply comes back through exec (a
throwaway run pod makes the stub fail).

Validation: kafka-metadata and rabbitmq are exercised by their in-tree
chainsaw E2E suites. The nats demo has no chainsaw suite in this
repository, so its change is covered by the unit test only.

### Screenshots

No UI changes.

### Downstream repositories

<!-- Walked the trigger map in docs/agents/contributing.md against the
diff. The change is confined to examples/backups/* demo scripts, one
demo README, and two hack/*.bats files. No repository's trigger matches:
it touches no packages/apps, no values.schema.json, no
ApplicationDefinition, no platform values, no hack/package.mk or make
target, and no commit convention. -->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): stop the nats, kafka-metadata and rabbitmq backup demos from reading empty CLI output
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Kafka and NATS backup demos reuse their CLI pods and limit cleanup to
pods belonging to the respective demo.
* NATS commands no longer include the password in the server URL, and
command errors remain visible.
* The RabbitMQ demo waits for its backup strategy to appear and reports
missing backup images more clearly.
* **Tests**
* Added coverage for reusing or creating demo pods, ownership checks,
password handling, and command failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 02:05:56 +04:00
Aleksei Sviridkin 8b6537f062 fix(e2e): collect the CDI controller log and CDI pods in cozyreport (#4414)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The e2e report now keeps the `cdi-deployment` log and the YAML of every
pod with CDI's `app=containerized-data-importer` label, in any namespace
and any phase.

In #4374 a Completed importer pod held the worker disk PVC in
`Terminating`, and the report had nothing to show why CDI did not delete
it. CDI deletes the importer pod once after a successful import and does
not retry. The only record of a failed delete is the controller log,
which was never collected. The pod itself was out of reach too, because
the broken-pod walk skips Completed pods, so there was no way to see if
it ever got a `deletionTimestamp`.

The controller log gets a 20000-line tail instead of the usual 2000. CDI
runs at `-v=1` and logs every reconcile, and the delete can happen an
hour before the report is taken.

It does not solve the underlying problem from #4374. It makes the next
occurrence diagnosable.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(e2e): the e2e diagnostics report now includes the CDI controller log and CDI pods in every phase.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Diagnostic reports now include up to 20,000 lines each from the
current and previous Containerized Data Importer deployment logs.
* Reports include YAML snapshots of all importer-associated pods across
namespaces, regardless of pod phase or deployment probe results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 01:52:49 +04:00
IvanHunters c355de63ed fix(backups): exec into a CLI Pod in the nats demo
nats_cli seeded and verified JetStream streams through a throwaway `kubectl run -i` Pod per call, with the same attach-race empty-read the redis demo fixed. NATS ships no long-lived Pod carrying the `nats` CLI, so run one nats-box Pod and `kubectl exec` into it; stream_message_count keeps the exec's stderr so a failed read says why. The password travels in NATS_PASSWORD on stdin rather than the --server URL the exec request records in the apiserver audit log.

The Pod carries cozystack.io/backup-demo=nats: nats_cli_pod reuses only a Ready Pod the demo owns, replaces a leftover in a terminal phase, and refuses a same-named Pod it does not own; cleanup.sh reaps by that label with --grace-period=1. Unit tests under hack/ pin the exec path, the preserved stderr, and the password staying out of argv.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 00:07:30 +03:00
IvanHunters 9cea7dcd90 fix(backups): exec into a CLI Pod in the kafka-metadata demo
kafka_run seeded and verified topics through a throwaway `kubectl run -i` Pod per call. That Pod's stdout returns over an attach that only carries what the container writes after the attach registers, so a CLI that finishes first comes back empty with exit 0 -- the empty-read defect fixed for the redis demo. Run one long-lived CLI Pod and `kubectl exec` into it: an exec'd process owns its pipes, so its output cannot be lost that way, and stderr stays attached (the topic_meta / topic_partitions reads keep it) so a failed read says why.

The Pod carries cozystack.io/backup-demo=kafka-metadata: kafka_cli_pod reuses only a Ready Pod the demo owns, replaces a leftover in a terminal phase, and refuses a same-named Pod it does not own; cleanup.sh reaps by that label with --grace-period=1. Unit tests under hack/ pin the exec path -- a throwaway run Pod or a re-added stderr redirect reddens them.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 00:07:20 +03:00
IvanHunters 7b94287970 fix(backups): wait for the platform strategy in rabbitmq and kafka demos
The rabbitmq and kafka-metadata backup demos read the platform's cozy-default-<engine> strategy once, right after install, where the Helm upgrade that renders it is gated on dependsOn and can leave it absent for minutes; the demo then failed immediately instead of waiting. Replace the single read with kubectl wait --for=create, so it polls through NotFound and fails only on a real error.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 00:07:08 +03:00
Alexey Artamonov 237f2e2bbd fix(postgres): say how to get past a refused extension clause
Before the charset guard, an extension value was spliced into CREATE
EXTENSION whole, so a release can carry a clause after the name, such
as earthdistance CASCADE, and that is the usual way to pull in a
dependency. The guard refuses such a value on upgrade too, and keeping
it that way is right, but the tenant needs to know what to write
instead. The reject message now says: drop the clause, which changes
nothing for an extension that is already installed because the script
never drops one and CREATE EXTENSION IF NOT EXISTS leaves it alone, and
list the dependencies first, since extensions are created in list
order.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 00:03:22 +03:00
Aleksei Sviridkin 9c5c5f9d4e fix(ci): overlay the right line's refs for a PR on a feature branch
The base-branch overlay pulled cozystack-packages:<base_ref>. A stacked
PR's base is a feature branch such as fix/foo: the slash is not valid
in an OCI tag and no artifact is published for it, so the pull failed,
the step only warned, and every package the PR did not rebuild ran its
committed release ref. With a committed ref older than its chart, that
broke E2E (Grafana plugin 404).

main and release-X.Y still read their own artifact. Any other base
reads the artifact of whichever of main and the release lines it has
the fewest commits on top of. Always picking main would put main's
images under a release line's charts for a stack rooted on that line,
the cross-generation mix that fails install.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 23:54:28 +03:00
Aleksei Sviridkin ab0c9f6e29 docs(e2e): document jq requirement for bucket suite (#4457)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Documents `jq` as a host prerequisite for the `bucket` Chainsaw suite,
whose command steps use it to parse S3 responses.

Fixes #4295

### Screenshots

N/A — documentation-only change.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
docs(e2e): document jq as a host prerequisite for the bucket Chainsaw suite
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated the local-run prerequisites for the `bucket` suite to include
`jq`, alongside `mc` and `nc`.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 00:45:01 +04:00
Aleksei Sviridkin 564f540f46 fix(kubernetes): render the OIDC UID header only from v1.33 (#4407)
## What this PR does

Stacked on #4398; review the diff against `fix/kamaji-pin-hash-reset`.

The tenant Kubernetes chart now renders
`--requestheader-uid-headers=X-Remote-Uid` only from an effective
Kubernetes version of v1.33, where `RemoteRequestHeaderUID` is beta and
on by default. It no longer renders a `--feature-gates` term of its own
on any version. On v1.32 it used to add a term enabling that gate, which
is alpha there, whenever it could read the operator's entries and they
left the gate unset; no release tag contains that path, so dropping it
now changes nothing a user has run.

The chart still reads `--emulated-version`, because emulation moves the
effective version: `--emulated-version=1.32` on a v1.35 tenant puts the
gate back to alpha-and-off, and the header flag is not rendered there.
It still reads the operator's `--feature-gates` entries for an opt-out
(`RemoteRequestHeaderUID=false`, or `AllBeta=false` with no explicit
`=true`), so the header flag never lands next to a disabled gate.
`AllAlpha` is no longer read, since it says nothing about a beta gate.

An unquoted `--emulated-version=value` entry whose kube version has a
non-zero patch component now fails the render and names the entry. The
rule mirrors upstream rather than the regex in #4396: `toVersionMap`
parses each value with `version.Parse` and refuses only `ver.Patch() !=
0` (`staging/src/k8s.io/component-base/compatibility/registry.go` lines
320-326 at v1.35.0, the same code in
`staging/src/k8s.io/apiserver/pkg/util/version/registry.go` at v1.31.0).
`version.Parse` takes `^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$`
(`versionMatchRE` in `k8s.io/apimachinery/pkg/util/version/version.go`,
identical at v0.31.1 and v0.35.0). So `1.32`, `v1.32`, `1.32-rc`,
`1.32.0` and `1.32.0-rc` all start an apiserver emulating 1.32, and
`1.32.4` and `1.32.4-rc` do not. A `^[0-9]+\.[0-9]+$` check would have
refused the first four. The chart extracts the numeric run with the same
pattern, reads the minor from it, and fails only on a non-zero third
component. A quoted value is still left unread.

Two deliberate narrowings:

- The chart no longer refuses a mix of bare and `kube:`-prefixed
`--feature-gates` terms. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it renders.
The apiserver still refuses the mix
(`component-base/compatibility/registry.go:405` at v1.35.0), and the
chart now passes it through like the other fatal gate values it already
leaves to the apiserver: unknown names, unparseable values, quoted
terms.
- On v1.32 the chart now neither renders nor judges the UID flags, as it
already did on v1.31. An operator's own `--requestheader-uid-headers`
entry without the gate reaches an apiserver that refuses it, which is
how every released chart version behaved. A cluster built from `main`
that already runs a v1.32 tenant with OIDC on loses the chart's
`--requestheader-uid-headers` and gate term on the next render; no
release carried them. That holds only while no release is cut from
`main` before this merges; a release that carries the v1.32 gate term
first would turn this into a breaking change for v1.32 tenants, whose
migration is to write both flags into `extraArgs` themselves.

An element such as `kube=1.32=x` is now split at its first `=` only, the
way `toVersionMap` splits it with `SplitN(compVer, "=", 2)`. The chart
used to skip that element, which the apiserver reads as 1.32, and so
rendered the header flag against an emulated 1.32. That gap predates
this branch.

Rebased onto #4398 at `fb2e57c3b`, which reads a two-element
`--feature-gates` and `--emulated-version` separately. That behaviour
carries over unchanged under the v1.33 floor: behind a hidden
`--emulated-version` only an explicit `RemoteRequestHeaderUID=false` is
refused beside a header entry, while a blanket `AllBeta=false` waits for
a reading, because a lower emulated version can make the gate alpha
again; nothing is refused as an opt-out behind a hidden
`--feature-gates` value; the empty-element refusal applies behind both;
and behind either one `X-Remote-Uid` is added only beside a uid-headers
list of the operator's. The README breaking-changes note from #4398 now
says the chart refuses neither the mixed gate spelling nor two kube
emulated versions. In `docs/oidc-tenant.md` the paragraphs that list
refusals now name the managed-flag move and refusal from #4398 and no
longer carry a count, and the two-element paragraph now says what the
chart renders behind a hidden value.

Tests: `helm unittest` goes from 308 on #4398 to 298. 26 cases are
deleted. 25 of them pinned the v1.32 gate term, `AllAlpha` handling or
the mixed-spelling refusal. One emulated a version below 1.32, which the
below-floor emulation case now covers. v1.32 cases that tested parsing
rather than the gate are retargeted to v1.34. Sixteen cases are added:
the patch-level refusal, the version parsing below the floor and above
it (on v1.35 emulating `1.33-rc`, `v1.33`, `1.33.0` and `kube=1.33=x`,
where only a correct minor renders the header), the first-`=` split, a
pin that a mixed gate spelling now passes through, emulation landing
exactly on 1.33, and a v1.32 opt-out beside a header entry passing
through untouched.

Coverage notes, not applied: no case pins that the patch-level refusal
stays off with `spec.oidc.mode: None`, where the block that holds it
does not run. The case "splits a component element at its first equals
sign only" pins that the kube element of `kube=1.32=x` is read. It does
not pin the split itself, because the version pattern ignores what
follows the numbers either way.

The `extraArgs` field description still calls the field the escape hatch
for apiserver flags unrelated to OIDC. It is unchanged here, and it
stays accurate for the v1.32 advice in the guide, because the UID flags
belong to the aggregation layer rather than to OIDC.

Optional, not applied: on v1.32 the chart could refuse an operator's
`--requestheader-uid-headers` entry that comes without the gate, since
the gate is off by default there and the apiserver will not start. The
same skip also drops two refusals the branch history had at 1.32, a
header entry beside `RemoteRequestHeaderUID=false` and a header list
ending in a comma; both now reach an apiserver that refuses them. It
stays out so that v1.32 matches v1.31 and the released charts, where the
chart judges none of these flags.

Not applied: for a value with a zero-prefixed major such as `01.32.4`,
the patch refusal suggests `01.32`. The apiserver refuses both values
over the leading zero, so the configuration fails either way and only
the suggested fix is off.

Why the patch refusal stays while the mixed-spelling refusal goes: the
patch refusal is about `--emulated-version`, which the chart reads
itself to decide what to render. The mixed spelling concerns
`--feature-gates`, which the chart no longer writes to.

Not split out: the `--emulated-version` parsing and the patch-level
refusal share the fix commit with the floor move, because both change
how the same value is read. The change also removes a comment from
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` that became false, so
select-e2e runs the full suite for this PR.

Default path: renders of the parent and this branch through
`tests/values/common.yaml` and `tests/values/oidc-system.yaml` at v1.31
to v1.35 are byte-identical, once the randomly generated Kamaji token
fields are masked, except for OIDC System at v1.32, which loses exactly
the two UID lines.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only field that changes is the description of `version`, which
returns to its earlier wording. The website's kubernetes reference page
is regenerated from the README on the next tag, and the Terraform
provider keeps its own copy of that description, which already reads
"Kubernetes major.minor version to deploy.".

Closes #4373

Closes #4396

### Release note

```release-note
fix(kubernetes): the tenant chart renders the aggregation-layer UID header flag with OIDC on only from an effective Kubernetes v1.33, and never renders a --feature-gates term of its own. v1.32 tenants render no UID flags, which matches every released version, so nothing regresses. This supersedes the v1.32 clause of the #3539 release note. An unquoted --emulated-version entry with a patch-level kube version such as 1.32.4, which the kube-apiserver refuses, now fails the render and names the entry.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* OIDC-enabled clusters on Kubernetes 1.33 and later now receive the UID
request-header setting without an added feature-gate flag.
* Chart rendering now checks Kubernetes and Talos version compatibility
and prevents rendering when worker deployments do not meet ownership or
retention requirements.
* **Bug Fixes**
* Emulated Kubernetes versions with non-zero patch components now fail
with a clear error; supported version formats are recognized more
consistently.
* **Documentation**
  * Updated version and OIDC guidance to reflect current chart behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 00:01:56 +04:00
IvanHunters 9e3ed1f2cb fix(backups): stop the redis backup demo from reading empty values (#4399)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The redis backup round-trip suite went red on PRs that do not touch
redis, each time on an empty read (#4395). The report snapshots of those
runs point to two separate causes, and I fix both here.

The empty marker reads came from the demo helper that ran redis-cli in a
throwaway `kubectl run -i --rm` Pod. kubectl gets that Pod's stdout over
an attach, and containerd's attach only carries output written after the
attach is registered. kubectl falls back to the Pod log only when the
attach fails. A redis-cli that finishes in that gap comes back as an
empty string with exit 0. In the failed runs, the master's log shows the
key present, and sentinel logged no failover, yet the GET read ''.
`redis_cmd` now runs redis-cli by `kubectl exec` in a sentinel Pod,
which already has it. An exec'd process gets its own pipes, so its
output can't be lost that way, and its exit code comes back. The
password now goes over stdin rather than a Pod env var.

The missing-strategy failure was real: `cozy-default-redis` did not
exist yet. It gets rendered behind a lookup of the platform bucket, so
it appears only after a forced Helm upgrade. helm-controller holds that
upgrade while `cozystack-basics` is not Ready, and the e2e install's
late platform patches keep it not Ready for several minutes. The demo
read the object once. It now waits for the object with `kubectl wait
--for=create` and a 15 minute bound. That wait fails right away on
anything other than NotFound.

The demo's reads also stop sending stderr to `/dev/null`, so the next
failure names its cause. `hack/redis-backup-demo-helpers.bats` covers
`redis_cmd` with stubbed kubectl and redis-cli: it reads through exec,
keeps the password out of argv, sends no AUTH when there is no Secret,
and fails loudly when the exec fails or sentinel has no master.

Closes #4395

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI changes.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `examples/backups/redis/` and a new unit test
under `hack/`. No entry in the trigger map covers either.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): the Redis backup demo reads its keys through `kubectl exec`, so a short-lived Pod can no longer drop its output, and it waits for the platform's `cozy-default-redis` strategy to appear instead of failing on its first read.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Redis backup and restore commands now report Kubernetes and Secret
read errors instead of hiding them, while still treating a missing
password Secret as empty.
* Redis commands pass authentication securely and fail clearly when
Sentinel cannot identify a master.
* Backup demos wait for the Redis backup strategy to appear and provide
clearer error messages.
* **Tests**
* Added coverage for Redis authentication handling, command failures,
and missing Sentinel masters.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 22:22:25 +03:00
nightcityblade fdcbc5c83c docs(e2e): document jq requirement for bucket suite
Assisted-by: LLM
Signed-off-by: nightcityblade <nightcityblade@gmail.com>
2026-09-25 02:35:29 +08:00
Aleksei Sviridkin 339cffe70f feat(kamaji)!: update to 26.9.4-edge and read every apiServer extraArgs entry (#4398)
## What this PR does

BREAKING CHANGE: a Kubernetes application whose
`controlPlane.apiServer.extraArgs` names a flag the tenant control plane
now owns fails to render until that entry is removed, except that an
`--enable-admission-plugins=` entry is moved into the new
`controlPlane.apiServer.admissionControllers` and a
`--kubelet-preferred-address-types=` entry into the new
`controlPlane.kubelet.preferredAddressTypes`.

Closes #3541.

Updates the Kamaji build this repo ships and rewrites the part of the
tenant Kubernetes chart whose correctness rested on the old one. The new
pin is `26.9.4-edge`, which is what `make update` picks.

Two upstream changes reach us. The tenant control plane no longer folds
the kube-apiserver argument list into a map keyed by flag name, so
entries supplied through the control plane spec arrive at the apiserver
verbatim, duplicates included. And the reset that rewrites those
arguments from scratch keys on a hash of the list rather than on its
length, so swapping one flag for another no longer strands the old one
on the running Deployment.

The OIDC block in the tenant chart read only the last entry of each
flag. That was right while earlier entries were dropped before the
apiserver saw them, and it is wrong now. An operator whose `extraArgs`
carried `--feature-gates=RemoteRequestHeaderUID=false` followed by an
unrelated `--feature-gates` entry got
`--requestheader-uid-headers=X-Remote-Uid` rendered next to a gate the
apiserver has switched off, and the control plane would not start. A
blanket `AllBeta=false` has the same shape, and so does an
`--emulated-version` entry naming kube followed by one naming another
component.

The chart now reads every entry with the accumulation each flag
performs. Occurrences of `--requestheader-uid-headers` append into one
list, so where a list of the operator's own omits `X-Remote-Uid` the
chart adds its own element beside it instead of refusing to render: the
union is what the apiserver validates, and their header keeps working.
Feature-gate terms collect per component and are applied in one pass, so
the chart carries only its own term, placed last where it wins. Naming
the kube component both bare and `kube:`-prefixed now fails the render,
because the apiserver refuses that configuration and nothing the chart
renders repairs it. `--emulated-version` accumulates too, and two
entries naming kube stop the apiserver outright, so the chart renders
nothing rather than pick a version that will not exist.

The same upstream change narrows what `extraArgs` can do, and the chart
now refuses the part that stopped working. Kamaji 26.9.4-edge sets a
list of kube-apiserver flags itself and drops any `extraArgs` entry that
names one before the apiserver sees the list, where `26.3.6-edge` merged
`extraArgs` last and let the entry win. A tenant with
`--enable-admission-plugins=NodeRestriction,AlwaysPullImages` loses both
plugins on the first reconcile after the upgrade, with nothing to say
so. The drop happens on the running control plane whatever the chart
renders, so the chart cannot prevent it; what it can do is refuse to
render such an entry, in every `oidc.mode`, and name it. The list
mirrors `buildKubeAPIServerCommand` at the pinned tag for the control
plane this chart renders (an etcd DataStore, konnectivity on, no
`dataStoreOverrides`), and the `controlPlane.apiServer.extraArgs`
description in the chart README carries it. A flag the control plane
only defaults, such as `--authorization-mode` or
`--requestheader-group-headers`, is still yours to set.
`--enable-admission-plugins` gets a replacement: the new
`controlPlane.apiServer.admissionControllers` passes a list through to
the KamajiControlPlane `spec.admissionControllers`, which the control
plane renders into that flag, and the application API checks each name
against the enum the KamajiControlPlane CRD declares. The list is
rendered only when non-empty, so a tenant that sets nothing renders
exactly as before, and a non-empty list replaces the control plane's
default set, as the `extraArgs` entry did under the previous build.
`--kubelet-preferred-address-types` gets the same treatment: the list
the chart used to hardcode on the KamajiControlPlane is now
`controlPlane.kubelet.preferredAddressTypes`, which renders that list
when left empty. Neither entry is refused where it can be moved: an
`--enable-admission-plugins=` or `--kubelet-preferred-address-types=`
entry is moved into its field while the field is empty, on every render,
and left out of the rendered `extraArgs`, so a tenant carrying one keeps
its setting across the Kamaji update with no edit and no ordering to get
right. The entry is refused beside a non-empty field, when it names
nothing, and in the two-entry spelling.

A separate defect in the same package is fixed here. The
TenantControlPlane schema reaches existing clusters through the
`kamaji-crds` subchart's templated CRDs, which every release upgrade
applies; `crds/` holds the copies a fresh install lays down first.
Directory-based renders lost that schema entirely, `helm template`,
`lint` and `package` alike, because the parent `.helmignore` matched the
subchart's `hack/` directory at any depth and that is where the
templates keep the schema they read with `.Files.Get`. Helm honours
`.helmignore` for a directory and skips it for an archive, so the
cluster never saw the difference and no local check could catch a schema
defect. The patterns are anchored to the chart root and a unit test
asserts the rendered CRDs carry their versions.

The bump is not cleanly reversible, and that is the one thing to know
before rolling it out. The args annotation holds a checksum once the new
build has written it, and the old build reads that annotation with
`strconv.Atoi`. The parse fails and `resetKubeAPIServerFlags` returns
early, ahead of both the args reset and the annotation write that would
put a number back. So after a rollback to `26.3.6-edge` the count-keyed
reset is disabled on every TenantControlPlane: a later removal of an
`extraArgs` entry no longer clears the flag from the running Deployment,
and it stays that way until someone writes a number into
`kube-apiserver.kamaji.clastix.io/args` by hand.

Findings from a live upgrade on a dev cluster, none of them caused by
the diff. The DNS-service-IP check moved out of a CEL rule on the CRD
into the admission webhook (`internal/webhook/handlers/tcp_dns.go`), so
it now depends on that webhook being reachable where the apiserver used
to enforce it by itself. The manager's readiness probe targets port 8081
while the webhook serves on 9443, so a Ready pod does not prove 9443
accepts connections; with `failurePolicy: Fail` on the
TenantControlPlane webhook, that window rejects writes rather than
letting them through unvalidated. The readiness probe this build adds to
kube-scheduler and kube-controller-manager survives a rollback, since
the old build neither sets nor clears those two; that one is benign.

Seven limits worth naming rather than leaving to be found. The
mixed-spelling refusal quotes the entry carrying each spelling, so one
entry holding both, `--feature-gates=Foo=true,kube:Bar=true`, is quoted
twice in its own message; the refusal is right and the message reads
oddly. The unit test that pins the rendered CRD schemas asserts on a
count, so a CRD missing from the render altogether fails on an empty
comparison rather than naming the CRD that went missing. There is one
window during the upgrade itself: the chart used to fold an operator's
own `--feature-gates` terms into the entry it rendered, because the old
build would otherwise drop them, and it now renders only its own term.
Nothing orders a tenant release after the Kamaji operator has rolled, so
a `v1.32` tenant that sets gate terms of its own with OIDC on can lose
them until the new operator reconciles it, which it then does on its
own. The same window takes a `--requestheader-uid-headers` list of the
operator's own that lacks `X-Remote-Uid`: the chart now renders its own
entry beside it instead of refusing, and the old operator keeps one
value per flag name, the chart's, so the operator's header is dropped
until the new operator reconciles. Which entry an error message quotes
is pinned only where one entry is eligible. Three variables choose
between equally eligible entries, the gate dictionary taking the last
setting for a name and the uid-headers ones the first, and no test
distinguishes first from last when two entries qualify, so a future edit
could make a message name the wrong entry of the two without a test
noticing. What the chart renders and what it refuses is pinned
exhaustively; it is the provenance in three message strings that is not.
The block is deliberately inconsistent about two shapes that stop the
apiserver equally hard: two `--emulated-version` entries naming kube
make it withhold silently, while a kube-prefixed and a bare gate
spelling in the same configuration make it refuse the render. The
refusal is right where the apiserver rejects the configuration whatever
the chart does; the withholding is right where the chart cannot read a
version it must render against. They are two rules, not one, and a third
that covers both is fair to ask for. Moving the refusal out from under
those conditions so it fires everywhere is the obvious third rule and it
is not free: the apiserver refuses the mix on `v1.31` too, but from
`k8s.io/apiserver/pkg/util/version` rather than from the component-base
`componentGlobalsRegistry.Set()` the message names, so hoisting it means
rewriting that sentence as well. Reading a flag and its value written as
two entries would be better than withholding for it, and is a change of
behaviour rather than a correction, so it is not here. And two sentences
about `--emulated-version` in `docs/oidc-tenant.md` sit thirty lines
apart and read as a retraction in sequence, though each is true at its
own layer: `toVersionMap` accepts an element naming a foreign component,
and the registry rejects it later.

What this leaves open, by name. The guard for a valueless flag at the
end of `extraArgs` covers `--requestheader-uid-headers`,
`--feature-gates` and `--emulated-version` only; any other flag that
takes a value swallows the appended `--authentication-config` the same
way, which is an input mistake the chart does not look for. The
managed-flag refusal matches names as written, as Kamaji does: an
underscore spelling such as `--secure_port` passes both, and the
apiserver normalises it to `--secure-port`, where the operator's value
wins as the later one, the same as before this change. Rendering
`--authentication-config` ahead of the operator's entries would close
that class for OIDC, since only the UID flags would then follow them; it
changes the rendered order, and so rolls every OIDC tenant that has
`extraArgs` of its own, and belongs in a separate change after the OIDC
floor move. One pre-existing comment needs checking: the note above
`extraArgs` in `templates/cluster.yaml` says that omitting the field
would leave the live array untouched under JSON merge patch, which reads
against Helm's `createPatch`, where a key the previous manifest carried
and the new one does not is patched to null; what that comment was
guarding against is still to be verified. The KamajiControlPlane CRD
accepts only the plugin names it lists, so a plugin outside that list
that the apiserver leaves off by default can no longer be enabled; that
limit is Kamaji's, since it owns the flag and restricts the field. A
name moved out of an `extraArgs` entry is not checked against that list
by the chart, so one outside it fails when helm-controller applies the
KamajiControlPlane, with an error that names it, rather than at render;
the chart keeps no copy of the list to check it with. A repeated address
type in a moved entry is dropped, keeping the first place it appears,
because the field is a set. With `oidc.mode: None` the chart does not
read `extraArgs` for the OIDC flags at all, and two shapes regress with
the bump there: a kube feature-gate term written both bare and
`kube:`-prefixed across entries, and two `--emulated-version` entries
naming kube. The old build kept one entry per flag, the new one passes
both, and the apiserver refuses both at startup. Next to a flag and
value written as two entries the chart keeps the refusals that hold
whatever the hidden value is, but it does not read positions: a hidden
`--feature-gates` term keeps the opt-out refusal off even when it sits
before the explicit opt-out and so cannot undo it.

What the live run could not exercise: the konnectivity datapath, because
the stand had no tenant workers, and `--requestheader-uid-headers`,
which comes from the tenant Kubernetes chart and no image swap reaches.
Both are covered by render and unit tests only.

One shape the old collapse was quietly repairing, with OIDC on, needs a
guard now that it is gone. A bare `--requestheader-uid-headers` in
`extraArgs` used to be overwritten by the entry of the same name the
chart renders, because the collapse kept one value per flag name; the
same held for a bare `--feature-gates` on `v1.32`, where the chart
renders a gate entry too. A bare `--emulated-version`, or a bare
`--feature-gates` on any other version, had no entry of the chart's to
be overwritten by and was emitted bare in sorted position, where it took
the next flag as its value, so it never worked. Without it those entries
reach the apiserver as written, and the apiserver gives a valueless long
flag the next argument whatever that argument looks like. What follows
the operator's last entry is the `--authentication-config` the chart
appends, so the flag swallows it and the AuthenticationConfiguration is
never applied. On `--requestheader-uid-headers` the path is accepted as
a header name and the control plane comes up with OIDC silently off,
failing every login for that tenant with nothing in the cluster to say
why; the other two reject it and the apiserver exits. The chart refuses
to render that shape and says which entry to fix.

Two things in the image build fall out of the bump. The datastore
deletion fix carried here as a patch landed upstream verbatim, so its
patch no longer applies, and one failing patch fails the whole apply of
the directory. The Go module overlay is obsolete for the opposite
reason: this tag already carries `golang.org/x/net`,
`golang.org/x/crypto` and `go.opentelemetry.io/otel/sdk` above the
versions the overlay forced, so keeping it would downgrade all three.

What I checked: `helm unittest` passes at 307 cases for the package,
with the OIDC suite carrying every shape above, and the image builder
stage builds end to end at the new tag with the one remaining patch.
Kamaji's own support for Kubernetes 1.36 and 1.37 does not move the
tenant version enum here; that is a separate update of the `kubernetes`
package. No image reference in `packages/system/kamaji/values.yaml` is
touched, since CI stamps that digest.

### Screenshots

No UI changes.

### Downstream repositories

I walked the trigger map file by file against the diff. The chart
templates, the vendored Kamaji chart, the image build and the docs reach
nothing downstream. The one generated file that reaches downstream is
`packages/apps/kubernetes/values.schema.json`, which gains
`controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` and a changed `extraArgs`
description; both new fields are optional with defaults, so a stored
Kubernetes object needs no change, no field is removed or renamed and no
version enum moves. The new fields are the one trigger the map fires on:
the Terraform provider keeps a hand-written schema.
`packages/apps/kubernetes/README.md` feeds the website's managed-apps
reference pages, which a bot regenerates on a stable tag.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: model `controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` on the kubernetes resource
(schema, model, expand/flatten).
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kamaji): update Kamaji to 26.9.4-edge. Tenant kube-apiserver arguments are now derived from a hash of the control plane's extraArgs instead of from its length, so on the first reconcile after the upgrade every tenant control plane rewrites those arguments from scratch and rolls once. The rebuilt list also orders them differently, Kamaji's own flags sorted first and the operator's kept verbatim after them, so the roll happens whether or not the tenant sets any extraArgs. A tenant with OIDC enabled takes a second roll when the chart's own flags land. Rolling the Kamaji image back to an earlier build does not restore the old behaviour: the older code reads that annotation as a number, gives up when it finds a checksum, and from then on never clears a flag you remove from extraArgs, on any TenantControlPlane, until the annotation is set back to a number by hand. The tenant Kubernetes chart reads every extraArgs entry the way the apiserver accumulates them: a --requestheader-uid-headers list that omits X-Remote-Uid now gets the chart's own element beside it instead of failing the render, a feature gate set in an earlier entry is honoured, and naming the kube feature-gates component both bare and kube:-prefixed fails the render, because the apiserver refuses that configuration outright. Writing --feature-gates or --emulated-version as a flag and a value in two separate entries now keeps the chart from rendering UID flags on its own, because an entry the chart cannot read may hide an opt-out; a --requestheader-uid-headers list of your own still gets X-Remote-Uid beside it, and write --flag=value to get the rest back. Action required for tenants that pass their own apiserver flags: Kamaji now sets a list of flags itself and drops a controlPlane.apiServer.extraArgs entry naming one, where the previous build let that entry win, and the Kubernetes application now refuses to render such an entry, except for the two it moves into a field. A tenant carrying one of the others stops reconciling after the upgrade, and no later change to that tenant applies, until the entry is removed; the entry no longer has an effect either way. The list is in the description of controlPlane.apiServer.extraArgs. An --enable-admission-plugins=value entry is moved into the new controlPlane.apiServer.admissionControllers, a list that replaces the control plane's default set as the entry did, and a --kubelet-preferred-address-types=value entry into the new controlPlane.kubelet.preferredAddressTypes, while that field is empty; nothing has to be edited for them. Before upgrading, list the affected entries with: kubectl get kuberneteses.apps.cozystack.io --all-namespaces --output json | jq --raw-output '["--advertise-address","--client-ca-file","--enable-admission-plugins","--service-cluster-ip-range","--kubelet-client-certificate","--kubelet-client-key","--kubelet-preferred-address-types","--proxy-client-cert-file","--proxy-client-key-file","--requestheader-allowed-names","--requestheader-client-ca-file","--secure-port","--service-account-key-file","--service-account-signing-key-file","--tls-cert-file","--tls-private-key-file","--etcd-servers","--etcd-cafile","--etcd-certfile","--etcd-keyfile","--etcd-prefix","--etcd-compaction-interval","--egress-selector-config-file"] as $owned | .items[] | .metadata as $m | (.spec.controlPlane.apiServer.extraArgs // [])[] | select((split("=")[0]) as $f | $owned | index($f)) | "\($m.namespace)/\($m.name): \(.)"' Entries it lists for --enable-admission-plugins or --kubelet-preferred-address-types need a change only when written as two entries, when they name nothing, or next to a non-empty field of their own.
```






<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Configure tenant control-plane admission controllers and kubelet
preferred address types.
- Choose a cert-manager issuer for webhook certificates; self-signed
certificates remain the default.
- **Bug Fixes**
- Improved OIDC argument handling for repeated, split, quoted, and
conflicting settings, with clearer render-time errors.
- DataStores are recognized as unused when no tenant control planes
reference them.
- **Validation**
- DataStore endpoints are checked for length, count, and IPv6 address
formatting.
- Chart-managed tenant control-plane arguments are rejected when
supplied through `extraArgs`.
- **Documentation**
  - Updated Kubernetes OIDC and certificate issuer guidance.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 22:26:41 +04:00
Aleksei Sviridkin d3b7e759bd chore: clean up stray console file and test fixtures (#4455)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Removes a kubeconfig that got committed by accident next to the console
sources, and ignores the filename so it doesn't come back. The vSphere
migration test fixtures now use synthetic values instead of
environment-specific ones. The tests only check the shape of the
inventory record, so they pass unchanged.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
chore: remove a stray kubeconfig from the console sources and use synthetic values in vSphere migration test fixtures.
```
2026-09-24 21:39:40 +04:00
Aleksei Sviridkin 047c715775 chore: clean up stray console file and test fixtures
Drop a local kubeconfig that was committed by accident alongside the
console sources and ignore the filename so it cannot come back. Replace
environment-specific values in the vSphere migration fixtures with
synthetic ones: the tests only care about the shape of the inventory
record, not where it was captured.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 20:37:49 +03:00
Aleksei Sviridkin 31711685e0 fix(workloadmonitor): populate VM instance-profile label from the KubeVirt preference annotation (#4448)
## What this PR does

The WorkloadMonitor reconciler copies a VM's instance profile onto the
`workloads.cozystack.io/kubevirt-vmi-instance-profile` label of the
`Workload` object, which the billing metric pipeline consumes.
`getWorkloadMetadata` read the annotation
`kubevirt.io/cluster-instanceprofile-name`, but KubeVirt never stamps
that key: it records the cluster preference (which carries the guest OS
profile) under `kubevirt.io/cluster-preference-name`. So the profile
label was never populated, and a VM's instance profile — a Windows
guest, for example — never reached billing. The sibling instance-type
label worked only because its key,
`kubevirt.io/cluster-instancetype-name`, is the real one.

This reads the correct annotation. The emitted label name is unchanged,
so the rest of the pipeline is untouched. The wrong key appears exactly
once in the tree, at the read site, and nothing writes it;
`kubevirt.io/cluster-preference-name` is the constant
`ClusterPreferenceAnnotation` in `kubevirt.io/api` (`core/v1/types.go`),
while no `cluster-instanceprofile-name` constant exists there.

Verified on a dev cluster: with the fix, a Windows-profile VM's
virt-launcher pod (which carries `kubevirt.io/cluster-preference-name =
windows.2k22.virtio`) now yields the
`workloads.cozystack.io/kubevirt-vmi-instance-profile` label on its
`Workload`, and the profile propagates through the billing metric
pipeline into the usage report's lifetime record. A table test pins the
exact annotation key, keeps the instance-type case as a control, and
asserts the nonexistent old key yields no label.

### Downstream repositories

Walked the trigger map against the diff: this is an internal controller
label-key fix, with no change to any CRD, values.schema.json, chart
values, hack/*, namespace, variant, release asset, or telemetry metric.
No listed downstream repository is affected.

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(workloadmonitor): populate the VM instance-profile billing label from the KubeVirt cluster-preference annotation so VM instance profiles (for example Windows) reach billing
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Workload metadata now uses KubeVirt’s cluster preference annotation
for the instance-profile label. The unsupported cluster instance-profile
annotation is ignored.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 21:20:48 +04:00
Alexey Artamonov ce74fdeea9 docs(backups): describe restored credentials and pre-upgrade residue
Say that a restored copy converges its application passwords shortly
after recovery, that MariaDB backups leave out the grant table, and that
dropping the password field does not rewrite copies made before the
upgrade: old Backup snapshots and Helm revisions still hold the password
that stays live. The credentials projector comment counts only the
engines that still take a password from values.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Alexey Artamonov f02ca8cbc2 test(backups): log in as an application user after each restore
The backup examples and e2e suites no longer set passwords. They read
the generated ones from <release>-credentials with jq, since a username
may contain a dot that jsonpath treats as a path step, and log in as an
application user after each restore, so a restore that leaves the
credentials unusable fails the run. The MariaDB run also checks that an
account created out of band on the source is absent from the copy. The
unreachable-target restore step asserts a terminal Failed attributed to
the recovery window, since the reason itself is best effort.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Alexey Artamonov 9fd590a5a7 fix(backups): keep a MariaDB restore target's accounts
The operator ships mysql.global_priv with every logical dump, so a
restore into a copy replaced the target's grant table with the source's
password hashes: application logins failed, and root, which the operator
sets only at datadir bootstrap, stayed wrong. Set ignoreGlobalPriv on
every Backup the driver creates. An account created out of band on the
source is therefore not carried into the copy.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Alexey Artamonov 70dcd9daed fix(backups): let a restored Postgres app converge its credentials
A restore sets bootstrap.enabled on the target app, which skips the
init-job, so the recovered roles keep the source's password hashes while
the target's credentials Secret holds generated ones and every
application login fails. Once recovery converges, the driver records
RecoveryConverged and then clears bootstrap.enabled, so the next
HelmRelease reconcile runs the init-job and applies the passwords. Ready
carries RestoreCompletedCredentialsPending and an Event says the
convergence is still pending. A clear that keeps failing is retried
under BootstrapDisablePending within a fixed grace window after
convergence and then fails the restore. bootstrap.enabled is serialized
without omitempty so the patch sends false instead of dropping the key.

Running the init-job on a restored cluster exposes its delete pass to a
catalog the spec does not describe: the driver sets spec.databases to
the Backup's snapshot, but recovery replays WAL past it, so a database
created after the Backup comes back still marked as managed and would
be dropped with its data. The first run after a restore, told apart by
the per-RestoreJob bootstrap.newServerName that the cluster has not yet
recorded, releases such a database and its roles from management
instead, and the run records the value once every pass has succeeded.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Alexey Artamonov 9afdc49894 feat(apps): warn when a removed field is still set on a database app
The apps API hands the stored spec to the HelmRelease unchanged, so a
users[].password left on a Postgres or MariaDB application is silently
ignored by the chart. Return a warning on create and update for each
such user, and for the superuser key each chart refuses to manage, so
the operator editing it learns it has no effect. Also warn when an
update introduces a "_"/"-" collision between usernames, database names
or Grant names, which the charts reject only on install.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Alexey Artamonov 9f857f64e1 feat(mariadb)!: generate every user password, root included
users[].password is gone from the chart, its schema and the API type.
Every password, root included, is generated into <release>-credentials
and kept across reconciles by lookup, and a password left in values is
ignored. An upgrade that finds no credentials Secret fails instead of
minting new passwords, because mariadb-operator applies root only at
datadir bootstrap and would advertise a root it never set. The Secret
carries k8s.mariadb.com/watch so the operator applies a changed password
without waiting for its requeue interval.

A username or database name becomes a Secret key and a resource name, so
render-time guards reject a name outside the DNS-1123 rule or past its
limit: 80 for a username, as the User CRD allows, and 64 for a database,
as the server allows. Names that differ only in "_" and "-" collide on
one resource; that is rejected on install only, since an existing
release already applied the overwrite. A user in both admin and readonly
of a database keeps a single ALL grant under the unchanged Grant name,
because renaming a Grant makes the operator revoke the live privilege. A
role member past 128 characters, the server's account-name limit, and a
database name of 65-80 characters, which the Database CRD accepts but
the server does not, are rejected on install and on upgrade only when
their Grant or Database does not exist yet, so a release that already
applied one keeps upgrading. User CR names and keys are quoted so a
username that reads as a YAML scalar stays a string.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:10:57 +03:00
Alexey Artamonov 2b5877eff9 feat(postgres)!: generate user passwords and read them at runtime
users[].password is gone from the chart, its schema and the API type.
Every application-user password is generated into <release>-credentials
and kept across reconciles by lookup, so an upgraded release keeps the
password it already has and a password left in values is ignored. The
init-job mounts that Secret and applies each password with ALTER ROLE
and a psql variable, so no password is rendered into the init-script
Secret that Helm keeps in release history. While bootstrap is enabled
the Secret is marked credentials-pending, and NOTES say the generated
passwords apply once the init-job runs.

Names reach superuser SQL and the init-script shell, so render-time
guards reject a username, database or extension name outside
[-._a-zA-Z0-9] or past the length PostgreSQL silently truncates: 63
bytes for a role or extension, 54 for a database, whose <db>_readonly
role would otherwise truncate and fail the next run. A username over 63
bytes that the credentials Secret already carries still upgrades.
users.postgres, the CNPG superuser, is rejected in every bootstrap
state. Changing a database's key in values drops the database, so the
reject message and the README give the rename that keeps the data: a
platform administrator renames it and its roles as the CNPG superuser
before the key changes.

The init-job cannot hang its Helm hook: the readiness wait ends on
elapsed time below activeDeadlineSeconds and the Job never retries, so a
purged restore target fails with a diagnostic. The delete-users pass
reads managed users from pg_shdescription instead of parsing the
version-dependent \du+ layout, which on PostgreSQL 16 and later matched
nothing and left a removed user able to log in; it compares names as
CREATE ROLE stores them and reassigns a removed user's objects in every
database before dropping it. A database without roles renders, and
extension names are quoted for hyphenated names and folded to lowercase
so they resolve as they did unquoted. A value that carries its own
surrounding quotes, the only way a hyphenated extension rendered before,
has that pair removed and keeps its case, so such a release still
upgrades. The README reads a password with jq, since jsonpath splits a
dotted key.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:10:57 +03:00
Myasnikov Daniil 9faa18a239 test(hack): detect BATS suites without a runner
Nested monitoring suites remained unexecuted after their runner was
removed. Audit both runner inventories so missing invocations and stale
suite paths fail in the unit lane. Make-variable arguments must be
reported even when their values hide the file suffix.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-24 20:00:27 +05:00
Timofei Larkin ce3eb30e03 fix(api): refuse only what a name declaration actually breaks (#4429)
## What this PR does

Follow-up to #4367, taking the non-blocking findings from its approving
review. Each one is a case where a name declaration (`x-cozystack-name`)
was handled more harshly, or more quietly, than the declaration itself
warrants.

- **An unparseable schema is no longer blamed on a declaration.** A
schema that is not a JSON object has no declaration in it, and the spec
schema loader already reports it. The API server used to refuse every
create of the kind with "declares an invalid x-cozystack-name". It now
treats the schema as declaring nothing: creates keep working, and
`Kubernetes` and `Kafka` keep their legacy caps.
- **Annotation keywords no longer take a kind offline.** Any unknown key
inside a declaration used to refuse creates, so a generator adding a
`title` or an example would have done the same. JSON Schema annotation
keywords (`title`, `default`, `examples`, `deprecated`, `readOnly`,
`writeOnly`, `$comment`) and `x-` extensions are now ignored. A
constraint keyword the server does not implement, such as `format` or a
misspelled `maxLenght`, is still refused, because accepting it would
drop it silently.
- **The docs no longer overclaim.** They said a declaration no name can
satisfy is refused. That is true only for lengths: a pattern is checked
to compile, not to match anything, so `$^` rejects every name and names
the pattern as the reason. The docs now say that.
- **The API review gate sees the declarations the server refuses.** The
gate only read a declaration that was an object, and only diffed the
keywords it knew. A declaration turned into a string or null, or a
misspelled keyword, looked like no change. A changed declaration now
goes through the same parser the server uses, and the gate reports it if
it cannot be enforced. The parser moves to `pkg/apis/apps/validation` so
there is one copy of the rules.
- **The declaration no longer leaks into the published OpenAPI for
`spec`.** The whole `openAPISchema` is published as the schema of
`spec`, so `x-cozystack-name` appeared there as a vendor extension of
`spec`. It is now stripped from the published copy. `metadata.name`
still carries no limit in the published OpenAPI: it is the shared
`ObjectMeta` reference, and a per-kind copy is a bigger change than this
one.

Not changed: a pattern that can never match is still accepted. Deciding
whether a regular expression matches any string within the declared
lengths is a much larger piece of code than the mistake it would catch.

### How it is tested

- `TestUnparseableSchemaDeclaresNothing`: three non-object schemas;
creates still pass, and the Kubernetes legacy cap still holds.
- `TestNameDeclarationIgnoresAnnotations`: a declaration carrying
`title`, `examples`, `$comment` and an `x-` extension still enforces its
`maxLength`.
- `TestInvalidNameSchemaFailsClosed` gains `format` as an unsupported
constraint keyword.
- `TestClassifyNameDeclaration` gains a declaration turned into a
string, a misspelled keyword and a null declaration, all reported, and
an added annotation, which is not.
- `TestPatchSpecLeavesNameDeclarationOffSpec`: the published `spec`
drops `x-cozystack-name` and keeps an unrelated vendor extension.

### Downstream repositories

No repository is affected beyond the follow-ups already noted on #4367
(the `cozystack/website` pages on annotations and
ApplicationDefinitions). Those are still unfiled, so no box is ticked.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(api): An ApplicationDefinition whose openAPISchema cannot be parsed no longer refuses every create of its kind as an invalid name declaration, and annotation keywords (`title`, `examples`, `x-` extensions and the like) inside `x-cozystack-name` are now ignored instead of taking the kind offline. The API review gate now reports a changed name declaration that the API server would refuse, and `x-cozystack-name` no longer appears on the published OpenAPI schema of `spec`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Application name declarations now support minimum and maximum lengths
and regular-expression patterns. Matching patterns are checked for
compilation; a pattern that matches no names rejects all names.
* **Bug Fixes**
* Invalid or unsupported name constraints now produce configuration
errors and block every create of the affected kind.
* Descriptive annotations and `x-` extensions are ignored when
validating name declarations.
* The custom name declaration is omitted from the published API
specification.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 18:46:35 +04:00
IvanHunters 1f9bad0643 fix(workloadmonitor): populate VM instance-profile label from the KubeVirt preference annotation
getWorkloadMetadata filled the workloads.cozystack.io/kubevirt-vmi-instance-profile
label from the annotation kubevirt.io/cluster-instanceprofile-name, a key KubeVirt
never stamps. KubeVirt records the cluster preference (which carries the guest OS
profile) under kubevirt.io/cluster-preference-name, so the profile label was never
emitted and a VM's instance profile never reached the billing metric. The sibling
instance-type label worked only because its key is the real one.

Read the correct annotation. The emitted label name is unchanged, so the downstream
metric pipeline is untouched.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-24 17:18:06 +03:00
Aleksei Sviridkin 4b6933c569 fix(ingress): trust CF-Connecting-IP only from Cloudflare ranges (#4444)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

With `cloudflareProxy: true` on a tenant Ingress app, any client that
reaches the load balancer directly can pick its own source address by
sending a `CF-Connecting-IP` header, and that address is what
`whitelist-source-range` checks. So the tenant `whitelist`, and any
per-Ingress whitelist behind the same controller, can be bypassed with
one header.

The chart passes the Cloudflare ranges to the controller as
`set_real_ip_from`. That is an nginx directive, not an ingress-nginx
ConfigMap key, so the controller drops it without a warning.
`use-forwarded-headers: "true"` turns the realip module on, and it then
trusts every source from the default `proxy-real-ip-cidr`, which is
`0.0.0.0/0` (ingress-nginx v1.11.5,
`internal/ingress/controller/config/config.go` and the
`set_real_ip_from` loop in `nginx.tmpl`). This has been the case since
the option was added.

The fix renames the key to `proxy-real-ip-cidr`. Traffic that comes
through Cloudflare keeps the visitor address as before. A direct
connection keeps its own address, whatever it puts in the header. If
something between Cloudflare and the controller rewrites the source
address, the header stops being trusted and the whitelist sees that hop
instead; that is the correct result for a header that can only be
trusted from known senders.

The new test checks that the trusted ranges land under the key the
controller reads, that they are not a catch-all, and that
`set_real_ip_from` is gone.

### Screenshots

No UI change.

### Downstream repositories

Template-only change in `packages/extra/ingress`: no values key, schema,
package, platform, installer or `hack/` change, so nothing in the
trigger map is reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ingress): with `cloudflareProxy` enabled, `CF-Connecting-IP` is now trusted only from Cloudflare's address ranges. Before, the Cloudflare ranges were passed under a key ingress-nginx ignores, so any client reaching the load balancer directly could set the header and get past the ingress `whitelist`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Updated Cloudflare proxy IP configuration to use ingress-nginx’s
supported setting, preserving forwarded-header and client IP handling.
* Ensured Cloudflare proxy IP ranges remain configured when a whitelist
is set.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 17:10:03 +04:00
Aleksei Sviridkin 3800f63513 fix(ingress): trust CF-Connecting-IP only from Cloudflare ranges
cloudflareProxy passed the Cloudflare ranges to the controller under
set_real_ip_from, which is an nginx directive and not an ingress-nginx
ConfigMap key, so the controller dropped it without a warning. With
use-forwarded-headers on, the realip module then trusted every source
from the default proxy-real-ip-cidr of 0.0.0.0/0, and any client that
reached the load balancer directly could set CF-Connecting-IP to an
address of its choice. That address is what whitelist-source-range
checks, so the tenant whitelist, and any per-Ingress one, could be
walked past with one header.

proxy-real-ip-cidr is the key the controller reads for the trusted
ranges.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:04:27 +03:00
Aleksei Sviridkin a92e89e969 docs(kubernetes): describe the OIDC UID header from v1.33
The tenant OIDC guide, the version field description and the e2e
helper comment still described a --feature-gates term rendered on
v1.32. Left as they were, they would tell an operator that the chart
switches that alpha gate on, which it no longer does. The guide now
says what a v1.32 tenant has to write itself to get the UID, and which
--emulated-version values fail the render, so an operator who meets
that refusal finds it explained. The breaking-changes note in the
README said the chart refuses the mixed gate spelling while OIDC is
on; with that refusal gone it would promise a render error that no
longer comes.

The version description returns to its earlier wording in values.yaml
and in the update-versions.sh template, because that template rewrites
the line on every version bump and would otherwise bring the old text
back. make generate carries it into the schema, README, API types and
ApplicationDefinition.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:10:53 +03:00
Aleksei Sviridkin 0078727620 fix(kubernetes): render the OIDC UID header only from v1.33
RemoteRequestHeaderUID is alpha and off on v1.32, so the chart had to
enable an alpha gate on v1.32 tenants with OIDC on to make
--requestheader-uid-headers start. From v1.33 the gate is beta and on
by default. No release has shipped the v1.32 path, so moving the floor
to an effective v1.33 now removes a behaviour no released chart had:
the chart renders no --feature-gates term of its own, and below the
floor it renders no UID flag and runs none of the UID checks, as it
already did on v1.31. The refusal of a list that ends in a bare flag
still applies there, because it guards the --authentication-config the
chart appends on every version.

The refusal of a kube gate spelling mixed between bare and
kube:-prefixed terms goes too. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it
renders, and the chart already passes other gate values the apiserver
refuses, unknown names and unparseable values, through to it.

--emulated-version is still read, because emulation moves the
effective version below the floor. Its kube version is now read from
the numbers at the start of the value, with the element split at its
first equals sign as toVersionMap splits it, so v1.32, 1.32.0 and
1.32-rc all read as 1.32. With OIDC on, an --emulated-version entry
without quotes whose kube version has a non-zero patch component fails
the render and names the entry: the apiserver refuses such a value and
would otherwise only crash-loop.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:10:53 +03:00
Myasnikov Daniil 16d23c8a53 test(monitoring): preserve OIDC coverage without live BATS suites
Rendering the inner chart alone cannot catch a wrapper dropping tenant
OIDC settings. The audience mapper and System-mode reconcile flag also
need assertions before the unused live suites are removed.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-24 17:08:47 +05:00
Timur Tukaev 5f5d8e9b7d docs(governance): nominate @themoriarti as Reviewer for the Proxmox integration (#4439)
## What this PR does

Nominates @themoriarti (Marian Koreniuk) as a **Reviewer** for the
Proxmox integration, following the process in
[CONTRIBUTOR_LADDER.md](https://github.com/cozystack/cozystack/blob/main/CONTRIBUTOR_LADDER.md#reviewer):

- `.github/CODEOWNERS`: adds @themoriarti as a code owner of
`packages/system/capi-providers-infraprovider-proxmox/` and
`packages/system/proxmox-*/` (Proxmox CCM and CSI). Each rule keeps the
full `/packages/system/` owner set, so no existing owner loses coverage,
and the catch-all invariant (`hack/codeowners-invariant.bats`) holds.
- `MAINTAINERS.md`: adds him to the Reviewers table.

The scope is deliberately narrow: shared code such as
`packages/apps/kubernetes*`, `api/apps` and the generic in-cluster IPAM
provider stays with the existing owners.

### Why

Marian has maintained the Proxmox integration since the first Proxmox
bundle and is now landing the Cluster API infrastructure provider, CCM
and CSI (#4087, #4088, #4089). Against the Reviewer requirements:

- well over 10 accepted PRs, contributing since 2024;
- has reviewed more than 20 PRs;
- in-depth knowledge of, and commitment to, the Proxmox area.

**Sponsors:** @kvaps (Ænix) and @kingdonb (Urmanac), which satisfies the
"at least one sponsor from a different employer" requirement.

### Notes for merging

- The Proxmox package directories arrive with #4087 and #4089. Merging
this PR after them is cleanest, though a rule for a path that does not
exist yet is harmless.
- GitHub silently ignores CODEOWNERS handles without write access. After
merge, an org admin needs to add @themoriarti as a repository
collaborator with the **Write** role (currently read).

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
docs(governance): add @themoriarti as Reviewer for the Proxmox integration.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Expanded code ownership and review coverage for Proxmox integrations,
including infrastructure provider, cloud controller manager, and storage
components. No end-user functionality changed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 17:07:39 +05:00
Aleksei Sviridkin 7d7f363cf2 docs(kubernetes): list the Kamaji update's extraArgs breaks
The Kamaji update changes what an existing tenant's extraArgs does, and
the package README is where an operator reads upgrade breaks. Record
there that an entry naming a flag the control plane owns stops taking
effect and, except for the two flags the chart moves into a field,
stops the release from rendering; which plugin names the admission
field accepts; and that repeated feature-gate spellings and kube
emulated versions, which used to collapse to the last entry, now reach
the apiserver and stop it from starting.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:13 +03:00
Aleksei Sviridkin 4e9f83ac0d fix(kubernetes)!: move or refuse extraArgs flags the control plane owns
From 26.9.4-edge the tenant control plane sets a list of kube-apiserver
flags itself and drops any extraArgs entry naming one before the
apiserver sees the list; 26.3.6-edge let such an entry win. An existing
tenant that set, for example, its own admission plugins through
extraArgs would lose them on the first reconcile after the upgrade with
nothing to say so, and a flag written with its value in a separate
entry leaves that value behind as a positional argument the apiserver
refuses.

Two of those flags have a KamajiControlPlane field the control plane
renders them from. An --enable-admission-plugins=value or
--kubelet-preferred-address-types=value entry is moved into that field
while the field is empty, and left out of the rendered extraArgs, so
the tenant keeps its setting across the upgrade without an edit. The
move is computed from the values on every render; the last entry wins,
as it did while the control plane kept one entry per flag name, and a
repeated address type is dropped because that field is a set. The
post-upgrade Job that patches extraArgs onto the KamajiControlPlane
reads the same list.

Every other entry naming an owned flag is refused at render time in
every OIDC mode, since the chart cannot stop the drop and the refusal
is where the operator learns of it. So are the two moved flags beside a
non-empty field, with a value that names nothing or is quoted, and in
the two-entry spelling. The list mirrors the Kamaji builder for the
shape of control plane this chart renders, and the extraArgs
description names the same flags.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:12 +03:00
Aleksei Sviridkin 0f9b8dc16c feat(kubernetes): expose admission controllers and kubelet address types
The KamajiControlPlane carries two lists the tenant control plane
renders into kube-apiserver flags: spec.admissionControllers into
--enable-admission-plugins and spec.kubelet.preferredAddressTypes into
--kubelet-preferred-address-types. The chart rendered neither from the
tenant values; the address types were a fixed list. Pass both through.

Admission controllers are rendered only when the list is non-empty, so
a tenant that sets none keeps the control plane's own default set, and
a list replaces that default rather than adding to it. An empty or
absent list of address types renders the list the chart rendered
before, so a tenant that sets nothing renders exactly as it did.

The names are declared as enums copied from the KamajiControlPlane
CRD at the provider pin, so the application API refuses a misspelled
name where the tenant edits it instead of at apply time. Both fields
are optional.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:11 +03:00
Aleksei Sviridkin 43cbcf9f46 fix(kamaji): anchor the ignore patterns to the chart root
Helm skips .helmignore when it reads a chart out of an archive and
honours it when it reads one out of a directory. A pattern with no
slash is matched against the base name at any depth, so `hack` reached
charts/kamaji-crds/hack, where the CRD templates keep the schema they
pull in with .Files.Get. The cluster, which gets the archive, has always
had the full CRDs; every local render, lint and package produced three
CustomResourceDefinitions with an empty spec, and no check that runs
before a cluster sees the chart could have caught a schema defect.

There is no hack directory at the top level of this package, so the
pattern excluded that subchart and nothing else. `images` is anchored
alongside it against the same trap; it matches only the intended
directory today and the packaged file list is unchanged by it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:01 +03:00
Aleksei Sviridkin 88264216b7 docs(kubernetes): restate the OIDC UID flags for the merged reading
The migration note told operators to move the extraArgs entry count in
the same edit so the running control plane would rewrite its apiserver
arguments; the control plane compares a hash of the list now, so any
change already forces the rebuild and no second edit is owed in either
direction. The paragraphs describing a value that survives only as the
last entry of its flag, and an empty value reaching the apiserver as a
bare flag, describe a build that is no longer shipped.

What replaces them is the accumulation each flag actually performs: a
uid-headers list the apiserver validates as the union of every entry,
feature gates settled across entries with the last term winning and one
spelling of the kube component throughout, and an emulated version that
two entries cannot both name.

One contract the earlier text sold is narrowed to what the chart
keeps. The render checks run only where the chart renders something, so
the three readings it refuses to guess at switch them off as well, and a
shape that would otherwise be refused passes through to a control plane
that does not start. A flag and value written as two entries is a
narrower case, and the page says which checks hold next to one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:01 +03:00
Aleksei Sviridkin 5b370d533f fix(kubernetes): read every extraArgs entry for the OIDC UID flags
The tenant control plane hands the kube-apiserver every entry of a flag,
so reading only the last one misses an opinion the apiserver acts on. An
earlier --feature-gates entry turning RemoteRequestHeaderUID off, or an
earlier --emulated-version naming kube, was previously dropped before the
apiserver saw it; now it decides, and rendering the uid header flag
against the later entry alone stops the control plane from starting.

Repeats of --requestheader-uid-headers append, so the chart adds its own
element beside a list that omits X-Remote-Uid instead of refusing to
render: the union satisfies the check the refusal existed for.
--feature-gates terms accumulate into one component bucket, so the chart
carries only its own term rather than a copy of the operator's, and
naming the kube component both bare and kube:-prefixed is refused
outright, which nothing the chart renders can repair.

A flag and its value written as two entries is read the same way the
apiserver reads it. pflag pairs them, so the value belongs to an entry
this code reads separately: the uid header flag counts as present, which
is what the gate-disabled check needs, while --feature-gates and
--emulated-version decide what is rendered and are not guessed at. What
holds whatever such a value hides still applies next to one: an empty
header element is refused, an explicit opt-out beside a header entry is
refused unless the hidden value is a gate term that could undo it, and
X-Remote-Uid is added beside an operator's own header list, which it can
only help. The chart's other additions wait for a reading.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:58:12 +03:00
Aleksei Sviridkin b31aa03a53 feat(kamaji)!: update to 26.9.4-edge
The tenant kube-apiserver argument list is no longer collapsed into a
map keyed by flag name: entries supplied through the control plane spec
reach the apiserver verbatim, duplicates included, and the reset that
rewrites them from scratch keys on a hash of the list rather than on its
length, so a same-count flag swap no longer strands the old flag on the
running Deployment.

The datastore deletion fix landed upstream verbatim, so its patch no
longer applies, and one failing patch fails the whole apply of the
directory. The module overlay is obsolete for the opposite reason: this
tag already carries golang.org/x/net, golang.org/x/crypto and
go.opentelemetry.io/otel/sdk above the versions the overlay forced, so
keeping it would downgrade all three.

The etcd.deploy override goes with it. The chart's dependency condition
is kamaji-etcd.deploy and the subchart is not vendored here, so the key
has never selected anything.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:58:12 +03:00
Aleksei Sviridkin 76b3652297 fix(kubevirt)!: evict non-migratable VMs instead of blocking the drain (#4403)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A node drain now completes when a VM on that node cannot live-migrate.
The VM is shut down and started again instead of holding the drain
forever. VMs that can migrate still migrate. The change is the
cluster-wide `evictionStrategy` in the KubeVirt CR, from `LiveMigrate`
to `LiveMigrateIfPossible`.

The drain is held by virt-api's eviction webhook, not by a
PodDisruptionBudget. With `LiveMigrate` and a VMI that is not
live-migratable, the webhook denies every eviction with HTTP 429
([pod-eviction-admitter.go#L147-L150](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L147-L150)).
`kubectl drain` retries a 429 until `--timeout`, which is infinite by
default. KubeVirt v1.9.0 does not create PDBs for VMIs at all; its
disruption-budget controller only deletes old ones
([disruptionbudget.go#L353-L406](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go#L353-L406)).
With `LiveMigrateIfPossible` the same webhook lets the eviction through
for such a VMI
([L152-L155](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L152-L155)).
The VM then comes back according to its `runStrategy`.

RWO disks are not the only case. A VM with a passthrough GPU is not
migratable either, unless the `VGPULiveMigration` gate is on and the GPU
is a mediated device
([vm.go#L1310-L1319](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-handler/vm.go#L1310-L1319)).
The gate is off in Cozystack, so any `vm-instance` with `gpus` set
blocked the drain of its node until now.

In KubeVirt a drain and a launcher update go through the same eviction.
After a KubeVirt upgrade the workload updater live-migrates outdated
VMIs that can migrate and evicts the rest, because
`workloadUpdateMethods` lists `Evict` as the fallback
([workload-updater.go#L421-L424](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L421-L424)).
The eviction call
([L614-L620](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L614-L620))
hits the same webhook as a drain. So a `vm-instance` VM that cannot
migrate either blocks both, which is what happens today, or is restarted
by both, which is what happens after this PR once the VM runs under the
new strategy.

This has a cost. A non-migratable VM now restarts on a drain. That
covers VMs on RWO storage and VMs with GPU or host device passthrough.
The webhook does not look at who asked for the eviction, so this holds
for any eviction through the API, not only a drain. A VM whose disk is
bound to the drained node, for example on a LINSTOR class without DRBD,
stays down until the node is back.

The upgrade itself restarts nothing, and existing VMs switch only after
their next restart. KubeVirt writes the cluster strategy into the VMI
spec when the VMI is created
([defaults.go#L208-L211](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/defaults/defaults.go#L208-L211)),
and the webhook reads that field first
([migrations.go#L116-L122](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/util/migrations/migrations.go#L116-L122)).
A VMI that was running before the upgrade keeps `LiveMigrate`, so a
drain still waits on it and the updater's eviction still gets 429. After
one restart the VMI carries `LiveMigrateIfPossible`. From then on every
KubeVirt bump shuts it down through the updater, in batches of 10 every
minute, the `batchEvictionSize` and `batchEvictionInterval` set in the
CR
([kubevirt-cr.yaml#L139-L140](https://github.com/cozystack/cozystack/blob/25d4a8c951b9fa7249b56ae79328d2cfd4f3a9c0/packages/system/kubevirt/templates/kubevirt-cr.yaml#L139-L140)).
Reading the source, a CPU socket increase on such a VM under
`vmRolloutStrategy: LiveUpdate` goes the same way, since the
`VCPUChange` condition sends it to the updater
([L320-L325](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L320-L325)).
I have not run that one.

I kept `Evict` even so, because `kubernetes-nodes` workers depend on it.
They set `evictionStrategy: External`, and for that strategy the webhook
marks the VMI for evacuation instead of evicting it
([pod-eviction-admitter.go#L156-L157](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L156-L157)).
CAPK sees the mark on a VMI that cannot migrate, drains the tenant node
and deletes the VMI
([machine.go#L414-L454](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/blob/v0.1.10/pkg/kubevirt/machine.go#L414-L454)).
The worker VM has `runStrategy: Always`, so KubeVirt starts it again on
the new launcher. That is how a GPU worker or any other non-migratable
worker gets the new launcher. Without `Evict` those workers would stay
on the old launcher for good. Worker behaviour does not change with this
PR, and the new test pins the method list so the fallback is not dropped
later.

`vm-instance` renders no `evictionStrategy` and has no value for it, so
a tenant has no way to opt out of either restart. KubeVirt would honour
a per-VM value over the cluster one. I filed #4419 to expose it.

I picked this over the options in the issue because KubeVirt already
decides migratability per VMI at eviction time. The platform CR does not
need to know anything about storage, and a chart shipping a VM does not
have to remember an override. Deriving the strategy from the disk access
mode in a chart would also miss the GPU case.

To get the new drain behaviour for a non-migratable VM that is already
running, restart it once after the upgrade. Until then a drain of its
node still blocks, as it does today.

The comment in the CR is on the same line as the key on purpose. `make
update` inserts blocks right before the `evictionStrategy:` line, so a
comment above the key would get separated from it.

A VM with `runStrategy: RerunOnFailure`, `Manual` or `Once` may stay off
after an eviction. If the guest shuts down cleanly on the ACPI request
within its grace period, the VMI ends `Succeeded`. `RerunOnFailure`
restarts only a failed VMI, and `Manual` and `Once` restart nothing on
their own
([vm.go#L1039-L1149](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go#L1039-L1149)).
`Once` stays off even after an unclean stop. Only the default `Always`
brings the VM back in every case.

What I did not verify. I have not drained a node with such a VM on a
live cluster, and I have not watched the updater restart them during an
upgrade. Everything above comes from the KubeVirt v1.9.0 and CAPK
v0.1.10 source. I did not check how `talosctl upgrade` handles a drain
that keeps getting 429. I also did not check how the cluster autoscaler
treats a node with a non-migratable VM now that its launcher pod can be
evicted. The pod has no `safe-to-evict` annotation, so for a VM on
node-local storage that likely comes down to the autoscaler's
`skip-nodes-with-local-storage` setting. Finally, I did not check
whether the upstream `VMCannotBeEvicted` alert is deployed in Cozystack.
For non-migratable VMs it stops firing after this change, since they are
evictable now.

Closes #4360

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubevirt)!: VMs that cannot live-migrate no longer block a node drain. The cluster-wide eviction strategy is now LiveMigrateIfPossible instead of LiveMigrate. VMs that can migrate still migrate. VMs that cannot, for example with a ReadWriteOnce disk or GPU or host device passthrough, are shut down and started again by their runStrategy. This applies to any pod eviction, not only a drain. A VM whose disk is bound to the drained node stays down until the node returns. A VM that is already running keeps the old strategy until its next restart, so restart non-migratable VMs once after upgrading if their nodes need to drain. The KubeVirt workload update uses the same eviction: once a non-migratable vm-instance VM runs under the new strategy, every later KubeVirt upgrade shuts it down, 10 VMs per minute. A VM with runStrategy RerunOnFailure, Manual or Once that shuts down cleanly on eviction stays off until it is started again. kubernetes-nodes workers are not affected.
```
2026-09-24 15:57:44 +04:00