Commit Graph
4106 Commits
Author SHA1 Message Date
Aleksei Sviridkin 2d2186d1f0 feat(monitoring): VictoriaTraces backend + Grafana datasource (tracing phase 1) (#3762)
## What this PR does

Phase 1 of distributed tracing (design: cozystack/community#38; epic:
#3761): a **VictoriaTraces backend and Grafana traces datasource** in
the monitoring stack, mirroring the logs (`VLCluster`) implementation.
No new operator: `VTCluster`/`VTSingle` already ship in
victoria-metrics-operator.

- **`tracingStorages`** values in `extra/monitoring` (annotated,
schema-generating) and `system/monitoring` (defaults), per-entry `mode:
cluster|single`. **Empty by default, so tracing is off out of the box**:
nothing produces spans until the collector ships, so no backend is
provisioned until an operator adds an entry.
- **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or
`VTSingle` (edge/dev/small); `retentionPeriod` plus optional
`retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup
label**, reused by the existing post-delete hook, so no hook change is
needed.
- **Render-time validation** (`_helpers.tpl`): the installed
VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise
reach the operator and fail after admission. Each `tracingStorages`
entry is checked before anything renders: `name` must be an RFC 1123
label of at most 42 characters (the `vtstorage-<name>-<hash>` pod
revision label is capped at 63) and unique across the list, `mode` must
be `cluster` or `single`, `storage` must be a positive quantity (a zero
size makes the operator mount an EmptyDir in single mode, and a negative
one fails when the PVC is created), and `retentionDiskUsageBytes` must
be a VictoriaMetrics `BytesString` (the operator rejects the
Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage`
field is also typed as a quantity in the schema, and the rendered name
is quoted so a numeric name stays a string.
- **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces
datasource per entry (built-in Grafana core datasource, no plugin).
- **VPA**: a `VerticalPodAutoscaler` for each trace component
(`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle`
Deployment), matching the metrics/logs sizing, with `updateMode:
Initial`.
- **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on
`status.updateStatus`, alongside the existing `VLCluster` gate.
- **VictoriaLogs datasource fix**
(`templates/vlogs/grafana-datasource.yaml`): the range over
`logsStorages` emitted no `---` between documents, so two or more
entries collapsed into one document and only the last datasource
survived. Each entry is now its own document.
- **Tests and schema**: helm-unittest suites for off-by-default, cluster
vs single, datasource type and both query URLs, retention, PVC label,
VPA targets, every validation guard with its boundaries, and per-entry
vlogs datasource documents, plus regenerated `values.schema.json`,
`README.md` and the `monitoring-rd` ApplicationDefinition.

#### Upgrade impact

With default values the only rendered object that changes is the
`monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle`
`healthCheckExprs` and `tracingStorages: []`. Flux evaluates those
expressions only against objects in the release inventory, so they are
inert while tracing is off. No VictoriaTraces object is created on
upgrade. Installs with two or more `logsStorages` entries additionally
gain the `GrafanaDatasource` objects that were previously dropped (see
the VictoriaLogs fix above); nothing else under VictoriaLogs or
VictoriaMetrics changes.

#### Verification

- `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring`
(11 tests) pass. Each validation guard and each branch of the
zero-storage pattern was mutation-checked: reverting it turns its own
test red.
- `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly.
`tracingStorages: []` (the default) renders nothing. An invalid,
duplicate or over-long `name`, a zero or negative `storage`, or a bad
`retentionDiskUsageBytes` (`8Gi`) is refused at render with an
actionable message; a numeric `name` renders as a string. `make
generate` is idempotent.
- **Datasource URLs verified against upstream docs** ([VictoriaTraces
and
Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)):
cluster `vtselect:10471/select/jaeger`, single
`vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana
datasource; the Tempo Query API (`/select/tempo`, `type: tempo`,
VictoriaTraces v0.9.4+) has a richer TraceQL UX but is
upstream-experimental, so Jaeger is the default. Both URLs are pinned by
unit tests.
- The `retentionDiskUsageBytes` pattern, the zero-storage check and the
name limit were checked against victoria-metrics-operator v0.68.4
(`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>`
StatefulSet name) and the Kubernetes `ControllerRevisionName` naming.

#### Scope / deferred (tracked in #3761)

Per-tenant OTLP Collector, engine integrations, shared-central topology,
and docs are follow-up PRs. This PR intentionally does **not** enable
the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and
the gRPC listener lands with the collector.

Builds on the now-merged design proposal cozystack/community#38.

### Screenshots

Not a UI change.

### Downstream repositories

Walked against the diff, no box ticked yet:

- cozystack/website: `monitoring` is in the website `Makefile`
`SERVICES` list, so its reference page is regenerated from
`packages/extra/monitoring/README.md` automatically on the next stable
tag. A user guide for tracing belongs to the docs follow-up in #3761; no
website issue or PR exists for it yet.
- cozystack/terraform-provider-cozystack: the provider has no
`Monitoring` resource (only the tenant's `monitoring` flag), so the new
schema field does not reach it.
- The remaining repositories in the trigger map are not reached by this
diff: no platform values, variants, installer values, `hack/` tooling,
node prerequisites, ApplicationDefinition semantics or telemetry names
change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage with cluster or single-node
deployment modes.
* Supports retention periods, persistent storage sizing, storage
classes, and optional disk-usage limits.
* Automatically configures Grafana tracing data sources for each enabled
trace backend.
* Monitoring readiness reflects the health of configured trace storage
services.
  * Invalid trace storage settings are rejected during configuration.
* **Bug Fixes**
* Every VictoriaLogs storage now gets its own Grafana datasource; with
several log storages, all but the last were previously dropped.
* **Tests**
* Added coverage for storage rendering, retention settings, validation
guards, Grafana endpoints, health checks, and cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 14:21:26 +04:00
IvanHunters 0bae19fccc feat(lineage-webhook): refuse a second ApplicationDefinition for a kind
A definition declaring an application kind that another definition already
declares now fails when it is applied, instead of sitting next to the first
one where it has no effect. An update that keeps its kind is allowed, so a
duplicate that predates the check keeps upgrading, and so is a second
definition from the same Helm release, since a release that renames its
definition object creates the new one before it deletes the old one.

The webhook is registered with failurePolicy Ignore: ApplicationDefinitions
ship with the platform itself, so an outage of the webhook must never block
a bootstrap or an upgrade, and a failure to read the existing definitions
lets the request through with a warning rather than answering with an error,
which Ignore would not cover. It is the feedback layer; which definition
manages a kind is decided by the ownership rule whatever it lets through.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:39:35 +03:00
Aleksei Sviridkin 564f540f46 fix(kubernetes): render the OIDC UID header only from v1.33 (#4407)
## What this PR does

Stacked on #4398; review the diff against `fix/kamaji-pin-hash-reset`.

The tenant Kubernetes chart now renders
`--requestheader-uid-headers=X-Remote-Uid` only from an effective
Kubernetes version of v1.33, where `RemoteRequestHeaderUID` is beta and
on by default. It no longer renders a `--feature-gates` term of its own
on any version. On v1.32 it used to add a term enabling that gate, which
is alpha there, whenever it could read the operator's entries and they
left the gate unset; no release tag contains that path, so dropping it
now changes nothing a user has run.

The chart still reads `--emulated-version`, because emulation moves the
effective version: `--emulated-version=1.32` on a v1.35 tenant puts the
gate back to alpha-and-off, and the header flag is not rendered there.
It still reads the operator's `--feature-gates` entries for an opt-out
(`RemoteRequestHeaderUID=false`, or `AllBeta=false` with no explicit
`=true`), so the header flag never lands next to a disabled gate.
`AllAlpha` is no longer read, since it says nothing about a beta gate.

An unquoted `--emulated-version=value` entry whose kube version has a
non-zero patch component now fails the render and names the entry. The
rule mirrors upstream rather than the regex in #4396: `toVersionMap`
parses each value with `version.Parse` and refuses only `ver.Patch() !=
0` (`staging/src/k8s.io/component-base/compatibility/registry.go` lines
320-326 at v1.35.0, the same code in
`staging/src/k8s.io/apiserver/pkg/util/version/registry.go` at v1.31.0).
`version.Parse` takes `^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$`
(`versionMatchRE` in `k8s.io/apimachinery/pkg/util/version/version.go`,
identical at v0.31.1 and v0.35.0). So `1.32`, `v1.32`, `1.32-rc`,
`1.32.0` and `1.32.0-rc` all start an apiserver emulating 1.32, and
`1.32.4` and `1.32.4-rc` do not. A `^[0-9]+\.[0-9]+$` check would have
refused the first four. The chart extracts the numeric run with the same
pattern, reads the minor from it, and fails only on a non-zero third
component. A quoted value is still left unread.

Two deliberate narrowings:

- The chart no longer refuses a mix of bare and `kube:`-prefixed
`--feature-gates` terms. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it renders.
The apiserver still refuses the mix
(`component-base/compatibility/registry.go:405` at v1.35.0), and the
chart now passes it through like the other fatal gate values it already
leaves to the apiserver: unknown names, unparseable values, quoted
terms.
- On v1.32 the chart now neither renders nor judges the UID flags, as it
already did on v1.31. An operator's own `--requestheader-uid-headers`
entry without the gate reaches an apiserver that refuses it, which is
how every released chart version behaved. A cluster built from `main`
that already runs a v1.32 tenant with OIDC on loses the chart's
`--requestheader-uid-headers` and gate term on the next render; no
release carried them. That holds only while no release is cut from
`main` before this merges; a release that carries the v1.32 gate term
first would turn this into a breaking change for v1.32 tenants, whose
migration is to write both flags into `extraArgs` themselves.

An element such as `kube=1.32=x` is now split at its first `=` only, the
way `toVersionMap` splits it with `SplitN(compVer, "=", 2)`. The chart
used to skip that element, which the apiserver reads as 1.32, and so
rendered the header flag against an emulated 1.32. That gap predates
this branch.

Rebased onto #4398 at `fb2e57c3b`, which reads a two-element
`--feature-gates` and `--emulated-version` separately. That behaviour
carries over unchanged under the v1.33 floor: behind a hidden
`--emulated-version` only an explicit `RemoteRequestHeaderUID=false` is
refused beside a header entry, while a blanket `AllBeta=false` waits for
a reading, because a lower emulated version can make the gate alpha
again; nothing is refused as an opt-out behind a hidden
`--feature-gates` value; the empty-element refusal applies behind both;
and behind either one `X-Remote-Uid` is added only beside a uid-headers
list of the operator's. The README breaking-changes note from #4398 now
says the chart refuses neither the mixed gate spelling nor two kube
emulated versions. In `docs/oidc-tenant.md` the paragraphs that list
refusals now name the managed-flag move and refusal from #4398 and no
longer carry a count, and the two-element paragraph now says what the
chart renders behind a hidden value.

Tests: `helm unittest` goes from 308 on #4398 to 298. 26 cases are
deleted. 25 of them pinned the v1.32 gate term, `AllAlpha` handling or
the mixed-spelling refusal. One emulated a version below 1.32, which the
below-floor emulation case now covers. v1.32 cases that tested parsing
rather than the gate are retargeted to v1.34. Sixteen cases are added:
the patch-level refusal, the version parsing below the floor and above
it (on v1.35 emulating `1.33-rc`, `v1.33`, `1.33.0` and `kube=1.33=x`,
where only a correct minor renders the header), the first-`=` split, a
pin that a mixed gate spelling now passes through, emulation landing
exactly on 1.33, and a v1.32 opt-out beside a header entry passing
through untouched.

Coverage notes, not applied: no case pins that the patch-level refusal
stays off with `spec.oidc.mode: None`, where the block that holds it
does not run. The case "splits a component element at its first equals
sign only" pins that the kube element of `kube=1.32=x` is read. It does
not pin the split itself, because the version pattern ignores what
follows the numbers either way.

The `extraArgs` field description still calls the field the escape hatch
for apiserver flags unrelated to OIDC. It is unchanged here, and it
stays accurate for the v1.32 advice in the guide, because the UID flags
belong to the aggregation layer rather than to OIDC.

Optional, not applied: on v1.32 the chart could refuse an operator's
`--requestheader-uid-headers` entry that comes without the gate, since
the gate is off by default there and the apiserver will not start. The
same skip also drops two refusals the branch history had at 1.32, a
header entry beside `RemoteRequestHeaderUID=false` and a header list
ending in a comma; both now reach an apiserver that refuses them. It
stays out so that v1.32 matches v1.31 and the released charts, where the
chart judges none of these flags.

Not applied: for a value with a zero-prefixed major such as `01.32.4`,
the patch refusal suggests `01.32`. The apiserver refuses both values
over the leading zero, so the configuration fails either way and only
the suggested fix is off.

Why the patch refusal stays while the mixed-spelling refusal goes: the
patch refusal is about `--emulated-version`, which the chart reads
itself to decide what to render. The mixed spelling concerns
`--feature-gates`, which the chart no longer writes to.

Not split out: the `--emulated-version` parsing and the patch-level
refusal share the fix commit with the floor move, because both change
how the same value is read. The change also removes a comment from
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` that became false, so
select-e2e runs the full suite for this PR.

Default path: renders of the parent and this branch through
`tests/values/common.yaml` and `tests/values/oidc-system.yaml` at v1.31
to v1.35 are byte-identical, once the randomly generated Kamaji token
fields are masked, except for OIDC System at v1.32, which loses exactly
the two UID lines.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only field that changes is the description of `version`, which
returns to its earlier wording. The website's kubernetes reference page
is regenerated from the README on the next tag, and the Terraform
provider keeps its own copy of that description, which already reads
"Kubernetes major.minor version to deploy.".

Closes #4373

Closes #4396

### Release note

```release-note
fix(kubernetes): the tenant chart renders the aggregation-layer UID header flag with OIDC on only from an effective Kubernetes v1.33, and never renders a --feature-gates term of its own. v1.32 tenants render no UID flags, which matches every released version, so nothing regresses. This supersedes the v1.32 clause of the #3539 release note. An unquoted --emulated-version entry with a patch-level kube version such as 1.32.4, which the kube-apiserver refuses, now fails the render and names the entry.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* OIDC-enabled clusters on Kubernetes 1.33 and later now receive the UID
request-header setting without an added feature-gate flag.
* Chart rendering now checks Kubernetes and Talos version compatibility
and prevents rendering when worker deployments do not meet ownership or
retention requirements.
* **Bug Fixes**
* Emulated Kubernetes versions with non-zero patch components now fail
with a clear error; supported version formats are recognized more
consistently.
* **Documentation**
  * Updated version and OIDC guidance to reflect current chart behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 00:01:56 +04:00
Aleksei Sviridkin 339cffe70f feat(kamaji)!: update to 26.9.4-edge and read every apiServer extraArgs entry (#4398)
## What this PR does

BREAKING CHANGE: a Kubernetes application whose
`controlPlane.apiServer.extraArgs` names a flag the tenant control plane
now owns fails to render until that entry is removed, except that an
`--enable-admission-plugins=` entry is moved into the new
`controlPlane.apiServer.admissionControllers` and a
`--kubelet-preferred-address-types=` entry into the new
`controlPlane.kubelet.preferredAddressTypes`.

Closes #3541.

Updates the Kamaji build this repo ships and rewrites the part of the
tenant Kubernetes chart whose correctness rested on the old one. The new
pin is `26.9.4-edge`, which is what `make update` picks.

Two upstream changes reach us. The tenant control plane no longer folds
the kube-apiserver argument list into a map keyed by flag name, so
entries supplied through the control plane spec arrive at the apiserver
verbatim, duplicates included. And the reset that rewrites those
arguments from scratch keys on a hash of the list rather than on its
length, so swapping one flag for another no longer strands the old one
on the running Deployment.

The OIDC block in the tenant chart read only the last entry of each
flag. That was right while earlier entries were dropped before the
apiserver saw them, and it is wrong now. An operator whose `extraArgs`
carried `--feature-gates=RemoteRequestHeaderUID=false` followed by an
unrelated `--feature-gates` entry got
`--requestheader-uid-headers=X-Remote-Uid` rendered next to a gate the
apiserver has switched off, and the control plane would not start. A
blanket `AllBeta=false` has the same shape, and so does an
`--emulated-version` entry naming kube followed by one naming another
component.

The chart now reads every entry with the accumulation each flag
performs. Occurrences of `--requestheader-uid-headers` append into one
list, so where a list of the operator's own omits `X-Remote-Uid` the
chart adds its own element beside it instead of refusing to render: the
union is what the apiserver validates, and their header keeps working.
Feature-gate terms collect per component and are applied in one pass, so
the chart carries only its own term, placed last where it wins. Naming
the kube component both bare and `kube:`-prefixed now fails the render,
because the apiserver refuses that configuration and nothing the chart
renders repairs it. `--emulated-version` accumulates too, and two
entries naming kube stop the apiserver outright, so the chart renders
nothing rather than pick a version that will not exist.

The same upstream change narrows what `extraArgs` can do, and the chart
now refuses the part that stopped working. Kamaji 26.9.4-edge sets a
list of kube-apiserver flags itself and drops any `extraArgs` entry that
names one before the apiserver sees the list, where `26.3.6-edge` merged
`extraArgs` last and let the entry win. A tenant with
`--enable-admission-plugins=NodeRestriction,AlwaysPullImages` loses both
plugins on the first reconcile after the upgrade, with nothing to say
so. The drop happens on the running control plane whatever the chart
renders, so the chart cannot prevent it; what it can do is refuse to
render such an entry, in every `oidc.mode`, and name it. The list
mirrors `buildKubeAPIServerCommand` at the pinned tag for the control
plane this chart renders (an etcd DataStore, konnectivity on, no
`dataStoreOverrides`), and the `controlPlane.apiServer.extraArgs`
description in the chart README carries it. A flag the control plane
only defaults, such as `--authorization-mode` or
`--requestheader-group-headers`, is still yours to set.
`--enable-admission-plugins` gets a replacement: the new
`controlPlane.apiServer.admissionControllers` passes a list through to
the KamajiControlPlane `spec.admissionControllers`, which the control
plane renders into that flag, and the application API checks each name
against the enum the KamajiControlPlane CRD declares. The list is
rendered only when non-empty, so a tenant that sets nothing renders
exactly as before, and a non-empty list replaces the control plane's
default set, as the `extraArgs` entry did under the previous build.
`--kubelet-preferred-address-types` gets the same treatment: the list
the chart used to hardcode on the KamajiControlPlane is now
`controlPlane.kubelet.preferredAddressTypes`, which renders that list
when left empty. Neither entry is refused where it can be moved: an
`--enable-admission-plugins=` or `--kubelet-preferred-address-types=`
entry is moved into its field while the field is empty, on every render,
and left out of the rendered `extraArgs`, so a tenant carrying one keeps
its setting across the Kamaji update with no edit and no ordering to get
right. The entry is refused beside a non-empty field, when it names
nothing, and in the two-entry spelling.

A separate defect in the same package is fixed here. The
TenantControlPlane schema reaches existing clusters through the
`kamaji-crds` subchart's templated CRDs, which every release upgrade
applies; `crds/` holds the copies a fresh install lays down first.
Directory-based renders lost that schema entirely, `helm template`,
`lint` and `package` alike, because the parent `.helmignore` matched the
subchart's `hack/` directory at any depth and that is where the
templates keep the schema they read with `.Files.Get`. Helm honours
`.helmignore` for a directory and skips it for an archive, so the
cluster never saw the difference and no local check could catch a schema
defect. The patterns are anchored to the chart root and a unit test
asserts the rendered CRDs carry their versions.

The bump is not cleanly reversible, and that is the one thing to know
before rolling it out. The args annotation holds a checksum once the new
build has written it, and the old build reads that annotation with
`strconv.Atoi`. The parse fails and `resetKubeAPIServerFlags` returns
early, ahead of both the args reset and the annotation write that would
put a number back. So after a rollback to `26.3.6-edge` the count-keyed
reset is disabled on every TenantControlPlane: a later removal of an
`extraArgs` entry no longer clears the flag from the running Deployment,
and it stays that way until someone writes a number into
`kube-apiserver.kamaji.clastix.io/args` by hand.

Findings from a live upgrade on a dev cluster, none of them caused by
the diff. The DNS-service-IP check moved out of a CEL rule on the CRD
into the admission webhook (`internal/webhook/handlers/tcp_dns.go`), so
it now depends on that webhook being reachable where the apiserver used
to enforce it by itself. The manager's readiness probe targets port 8081
while the webhook serves on 9443, so a Ready pod does not prove 9443
accepts connections; with `failurePolicy: Fail` on the
TenantControlPlane webhook, that window rejects writes rather than
letting them through unvalidated. The readiness probe this build adds to
kube-scheduler and kube-controller-manager survives a rollback, since
the old build neither sets nor clears those two; that one is benign.

Seven limits worth naming rather than leaving to be found. The
mixed-spelling refusal quotes the entry carrying each spelling, so one
entry holding both, `--feature-gates=Foo=true,kube:Bar=true`, is quoted
twice in its own message; the refusal is right and the message reads
oddly. The unit test that pins the rendered CRD schemas asserts on a
count, so a CRD missing from the render altogether fails on an empty
comparison rather than naming the CRD that went missing. There is one
window during the upgrade itself: the chart used to fold an operator's
own `--feature-gates` terms into the entry it rendered, because the old
build would otherwise drop them, and it now renders only its own term.
Nothing orders a tenant release after the Kamaji operator has rolled, so
a `v1.32` tenant that sets gate terms of its own with OIDC on can lose
them until the new operator reconciles it, which it then does on its
own. The same window takes a `--requestheader-uid-headers` list of the
operator's own that lacks `X-Remote-Uid`: the chart now renders its own
entry beside it instead of refusing, and the old operator keeps one
value per flag name, the chart's, so the operator's header is dropped
until the new operator reconciles. Which entry an error message quotes
is pinned only where one entry is eligible. Three variables choose
between equally eligible entries, the gate dictionary taking the last
setting for a name and the uid-headers ones the first, and no test
distinguishes first from last when two entries qualify, so a future edit
could make a message name the wrong entry of the two without a test
noticing. What the chart renders and what it refuses is pinned
exhaustively; it is the provenance in three message strings that is not.
The block is deliberately inconsistent about two shapes that stop the
apiserver equally hard: two `--emulated-version` entries naming kube
make it withhold silently, while a kube-prefixed and a bare gate
spelling in the same configuration make it refuse the render. The
refusal is right where the apiserver rejects the configuration whatever
the chart does; the withholding is right where the chart cannot read a
version it must render against. They are two rules, not one, and a third
that covers both is fair to ask for. Moving the refusal out from under
those conditions so it fires everywhere is the obvious third rule and it
is not free: the apiserver refuses the mix on `v1.31` too, but from
`k8s.io/apiserver/pkg/util/version` rather than from the component-base
`componentGlobalsRegistry.Set()` the message names, so hoisting it means
rewriting that sentence as well. Reading a flag and its value written as
two entries would be better than withholding for it, and is a change of
behaviour rather than a correction, so it is not here. And two sentences
about `--emulated-version` in `docs/oidc-tenant.md` sit thirty lines
apart and read as a retraction in sequence, though each is true at its
own layer: `toVersionMap` accepts an element naming a foreign component,
and the registry rejects it later.

What this leaves open, by name. The guard for a valueless flag at the
end of `extraArgs` covers `--requestheader-uid-headers`,
`--feature-gates` and `--emulated-version` only; any other flag that
takes a value swallows the appended `--authentication-config` the same
way, which is an input mistake the chart does not look for. The
managed-flag refusal matches names as written, as Kamaji does: an
underscore spelling such as `--secure_port` passes both, and the
apiserver normalises it to `--secure-port`, where the operator's value
wins as the later one, the same as before this change. Rendering
`--authentication-config` ahead of the operator's entries would close
that class for OIDC, since only the UID flags would then follow them; it
changes the rendered order, and so rolls every OIDC tenant that has
`extraArgs` of its own, and belongs in a separate change after the OIDC
floor move. One pre-existing comment needs checking: the note above
`extraArgs` in `templates/cluster.yaml` says that omitting the field
would leave the live array untouched under JSON merge patch, which reads
against Helm's `createPatch`, where a key the previous manifest carried
and the new one does not is patched to null; what that comment was
guarding against is still to be verified. The KamajiControlPlane CRD
accepts only the plugin names it lists, so a plugin outside that list
that the apiserver leaves off by default can no longer be enabled; that
limit is Kamaji's, since it owns the flag and restricts the field. A
name moved out of an `extraArgs` entry is not checked against that list
by the chart, so one outside it fails when helm-controller applies the
KamajiControlPlane, with an error that names it, rather than at render;
the chart keeps no copy of the list to check it with. A repeated address
type in a moved entry is dropped, keeping the first place it appears,
because the field is a set. With `oidc.mode: None` the chart does not
read `extraArgs` for the OIDC flags at all, and two shapes regress with
the bump there: a kube feature-gate term written both bare and
`kube:`-prefixed across entries, and two `--emulated-version` entries
naming kube. The old build kept one entry per flag, the new one passes
both, and the apiserver refuses both at startup. Next to a flag and
value written as two entries the chart keeps the refusals that hold
whatever the hidden value is, but it does not read positions: a hidden
`--feature-gates` term keeps the opt-out refusal off even when it sits
before the explicit opt-out and so cannot undo it.

What the live run could not exercise: the konnectivity datapath, because
the stand had no tenant workers, and `--requestheader-uid-headers`,
which comes from the tenant Kubernetes chart and no image swap reaches.
Both are covered by render and unit tests only.

One shape the old collapse was quietly repairing, with OIDC on, needs a
guard now that it is gone. A bare `--requestheader-uid-headers` in
`extraArgs` used to be overwritten by the entry of the same name the
chart renders, because the collapse kept one value per flag name; the
same held for a bare `--feature-gates` on `v1.32`, where the chart
renders a gate entry too. A bare `--emulated-version`, or a bare
`--feature-gates` on any other version, had no entry of the chart's to
be overwritten by and was emitted bare in sorted position, where it took
the next flag as its value, so it never worked. Without it those entries
reach the apiserver as written, and the apiserver gives a valueless long
flag the next argument whatever that argument looks like. What follows
the operator's last entry is the `--authentication-config` the chart
appends, so the flag swallows it and the AuthenticationConfiguration is
never applied. On `--requestheader-uid-headers` the path is accepted as
a header name and the control plane comes up with OIDC silently off,
failing every login for that tenant with nothing in the cluster to say
why; the other two reject it and the apiserver exits. The chart refuses
to render that shape and says which entry to fix.

Two things in the image build fall out of the bump. The datastore
deletion fix carried here as a patch landed upstream verbatim, so its
patch no longer applies, and one failing patch fails the whole apply of
the directory. The Go module overlay is obsolete for the opposite
reason: this tag already carries `golang.org/x/net`,
`golang.org/x/crypto` and `go.opentelemetry.io/otel/sdk` above the
versions the overlay forced, so keeping it would downgrade all three.

What I checked: `helm unittest` passes at 307 cases for the package,
with the OIDC suite carrying every shape above, and the image builder
stage builds end to end at the new tag with the one remaining patch.
Kamaji's own support for Kubernetes 1.36 and 1.37 does not move the
tenant version enum here; that is a separate update of the `kubernetes`
package. No image reference in `packages/system/kamaji/values.yaml` is
touched, since CI stamps that digest.

### Screenshots

No UI changes.

### Downstream repositories

I walked the trigger map file by file against the diff. The chart
templates, the vendored Kamaji chart, the image build and the docs reach
nothing downstream. The one generated file that reaches downstream is
`packages/apps/kubernetes/values.schema.json`, which gains
`controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` and a changed `extraArgs`
description; both new fields are optional with defaults, so a stored
Kubernetes object needs no change, no field is removed or renamed and no
version enum moves. The new fields are the one trigger the map fires on:
the Terraform provider keeps a hand-written schema.
`packages/apps/kubernetes/README.md` feeds the website's managed-apps
reference pages, which a bot regenerates on a stable tag.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: model `controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` on the kubernetes resource
(schema, model, expand/flatten).
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kamaji): update Kamaji to 26.9.4-edge. Tenant kube-apiserver arguments are now derived from a hash of the control plane's extraArgs instead of from its length, so on the first reconcile after the upgrade every tenant control plane rewrites those arguments from scratch and rolls once. The rebuilt list also orders them differently, Kamaji's own flags sorted first and the operator's kept verbatim after them, so the roll happens whether or not the tenant sets any extraArgs. A tenant with OIDC enabled takes a second roll when the chart's own flags land. Rolling the Kamaji image back to an earlier build does not restore the old behaviour: the older code reads that annotation as a number, gives up when it finds a checksum, and from then on never clears a flag you remove from extraArgs, on any TenantControlPlane, until the annotation is set back to a number by hand. The tenant Kubernetes chart reads every extraArgs entry the way the apiserver accumulates them: a --requestheader-uid-headers list that omits X-Remote-Uid now gets the chart's own element beside it instead of failing the render, a feature gate set in an earlier entry is honoured, and naming the kube feature-gates component both bare and kube:-prefixed fails the render, because the apiserver refuses that configuration outright. Writing --feature-gates or --emulated-version as a flag and a value in two separate entries now keeps the chart from rendering UID flags on its own, because an entry the chart cannot read may hide an opt-out; a --requestheader-uid-headers list of your own still gets X-Remote-Uid beside it, and write --flag=value to get the rest back. Action required for tenants that pass their own apiserver flags: Kamaji now sets a list of flags itself and drops a controlPlane.apiServer.extraArgs entry naming one, where the previous build let that entry win, and the Kubernetes application now refuses to render such an entry, except for the two it moves into a field. A tenant carrying one of the others stops reconciling after the upgrade, and no later change to that tenant applies, until the entry is removed; the entry no longer has an effect either way. The list is in the description of controlPlane.apiServer.extraArgs. An --enable-admission-plugins=value entry is moved into the new controlPlane.apiServer.admissionControllers, a list that replaces the control plane's default set as the entry did, and a --kubelet-preferred-address-types=value entry into the new controlPlane.kubelet.preferredAddressTypes, while that field is empty; nothing has to be edited for them. Before upgrading, list the affected entries with: kubectl get kuberneteses.apps.cozystack.io --all-namespaces --output json | jq --raw-output '["--advertise-address","--client-ca-file","--enable-admission-plugins","--service-cluster-ip-range","--kubelet-client-certificate","--kubelet-client-key","--kubelet-preferred-address-types","--proxy-client-cert-file","--proxy-client-key-file","--requestheader-allowed-names","--requestheader-client-ca-file","--secure-port","--service-account-key-file","--service-account-signing-key-file","--tls-cert-file","--tls-private-key-file","--etcd-servers","--etcd-cafile","--etcd-certfile","--etcd-keyfile","--etcd-prefix","--etcd-compaction-interval","--egress-selector-config-file"] as $owned | .items[] | .metadata as $m | (.spec.controlPlane.apiServer.extraArgs // [])[] | select((split("=")[0]) as $f | $owned | index($f)) | "\($m.namespace)/\($m.name): \(.)"' Entries it lists for --enable-admission-plugins or --kubelet-preferred-address-types need a change only when written as two entries, when they name nothing, or next to a non-empty field of their own.
```






<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Configure tenant control-plane admission controllers and kubelet
preferred address types.
- Choose a cert-manager issuer for webhook certificates; self-signed
certificates remain the default.
- **Bug Fixes**
- Improved OIDC argument handling for repeated, split, quoted, and
conflicting settings, with clearer render-time errors.
- DataStores are recognized as unused when no tenant control planes
reference them.
- **Validation**
- DataStore endpoints are checked for length, count, and IPv6 address
formatting.
- Chart-managed tenant control-plane arguments are rejected when
supplied through `extraArgs`.
- **Documentation**
  - Updated Kubernetes OIDC and certificate issuer guidance.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 22:26:41 +04:00
Aleksei Sviridkin 047c715775 chore: clean up stray console file and test fixtures
Drop a local kubeconfig that was committed by accident alongside the
console sources and ignore the filename so it cannot come back. Replace
environment-specific values in the vSphere migration fixtures with
synthetic ones: the tests only care about the shape of the inventory
record, not where it was captured.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 20:37:49 +03:00
Aleksei Sviridkin 3800f63513 fix(ingress): trust CF-Connecting-IP only from Cloudflare ranges
cloudflareProxy passed the Cloudflare ranges to the controller under
set_real_ip_from, which is an nginx directive and not an ingress-nginx
ConfigMap key, so the controller dropped it without a warning. With
use-forwarded-headers on, the realip module then trusted every source
from the default proxy-real-ip-cidr of 0.0.0.0/0, and any client that
reached the load balancer directly could set CF-Connecting-IP to an
address of its choice. That address is what whitelist-source-range
checks, so the tenant whitelist, and any per-Ingress one, could be
walked past with one header.

proxy-real-ip-cidr is the key the controller reads for the trusted
ranges.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:04:27 +03:00
Aleksei Sviridkin a92e89e969 docs(kubernetes): describe the OIDC UID header from v1.33
The tenant OIDC guide, the version field description and the e2e
helper comment still described a --feature-gates term rendered on
v1.32. Left as they were, they would tell an operator that the chart
switches that alpha gate on, which it no longer does. The guide now
says what a v1.32 tenant has to write itself to get the UID, and which
--emulated-version values fail the render, so an operator who meets
that refusal finds it explained. The breaking-changes note in the
README said the chart refuses the mixed gate spelling while OIDC is
on; with that refusal gone it would promise a render error that no
longer comes.

The version description returns to its earlier wording in values.yaml
and in the update-versions.sh template, because that template rewrites
the line on every version bump and would otherwise bring the old text
back. make generate carries it into the schema, README, API types and
ApplicationDefinition.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:10:53 +03:00
Aleksei Sviridkin 0078727620 fix(kubernetes): render the OIDC UID header only from v1.33
RemoteRequestHeaderUID is alpha and off on v1.32, so the chart had to
enable an alpha gate on v1.32 tenants with OIDC on to make
--requestheader-uid-headers start. From v1.33 the gate is beta and on
by default. No release has shipped the v1.32 path, so moving the floor
to an effective v1.33 now removes a behaviour no released chart had:
the chart renders no --feature-gates term of its own, and below the
floor it renders no UID flag and runs none of the UID checks, as it
already did on v1.31. The refusal of a list that ends in a bare flag
still applies there, because it guards the --authentication-config the
chart appends on every version.

The refusal of a kube gate spelling mixed between bare and
kube:-prefixed terms goes too. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it
renders, and the chart already passes other gate values the apiserver
refuses, unknown names and unparseable values, through to it.

--emulated-version is still read, because emulation moves the
effective version below the floor. Its kube version is now read from
the numbers at the start of the value, with the element split at its
first equals sign as toVersionMap splits it, so v1.32, 1.32.0 and
1.32-rc all read as 1.32. With OIDC on, an --emulated-version entry
without quotes whose kube version has a non-zero patch component fails
the render and names the entry: the apiserver refuses such a value and
would otherwise only crash-loop.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:10:53 +03:00
Aleksei Sviridkin 7d7f363cf2 docs(kubernetes): list the Kamaji update's extraArgs breaks
The Kamaji update changes what an existing tenant's extraArgs does, and
the package README is where an operator reads upgrade breaks. Record
there that an entry naming a flag the control plane owns stops taking
effect and, except for the two flags the chart moves into a field,
stops the release from rendering; which plugin names the admission
field accepts; and that repeated feature-gate spellings and kube
emulated versions, which used to collapse to the last entry, now reach
the apiserver and stop it from starting.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:13 +03:00
Aleksei Sviridkin 4e9f83ac0d fix(kubernetes)!: move or refuse extraArgs flags the control plane owns
From 26.9.4-edge the tenant control plane sets a list of kube-apiserver
flags itself and drops any extraArgs entry naming one before the
apiserver sees the list; 26.3.6-edge let such an entry win. An existing
tenant that set, for example, its own admission plugins through
extraArgs would lose them on the first reconcile after the upgrade with
nothing to say so, and a flag written with its value in a separate
entry leaves that value behind as a positional argument the apiserver
refuses.

Two of those flags have a KamajiControlPlane field the control plane
renders them from. An --enable-admission-plugins=value or
--kubelet-preferred-address-types=value entry is moved into that field
while the field is empty, and left out of the rendered extraArgs, so
the tenant keeps its setting across the upgrade without an edit. The
move is computed from the values on every render; the last entry wins,
as it did while the control plane kept one entry per flag name, and a
repeated address type is dropped because that field is a set. The
post-upgrade Job that patches extraArgs onto the KamajiControlPlane
reads the same list.

Every other entry naming an owned flag is refused at render time in
every OIDC mode, since the chart cannot stop the drop and the refusal
is where the operator learns of it. So are the two moved flags beside a
non-empty field, with a value that names nothing or is quoted, and in
the two-entry spelling. The list mirrors the Kamaji builder for the
shape of control plane this chart renders, and the extraArgs
description names the same flags.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:12 +03:00
Aleksei Sviridkin 0f9b8dc16c feat(kubernetes): expose admission controllers and kubelet address types
The KamajiControlPlane carries two lists the tenant control plane
renders into kube-apiserver flags: spec.admissionControllers into
--enable-admission-plugins and spec.kubelet.preferredAddressTypes into
--kubelet-preferred-address-types. The chart rendered neither from the
tenant values; the address types were a fixed list. Pass both through.

Admission controllers are rendered only when the list is non-empty, so
a tenant that sets none keeps the control plane's own default set, and
a list replaces that default rather than adding to it. An empty or
absent list of address types renders the list the chart rendered
before, so a tenant that sets nothing renders exactly as it did.

The names are declared as enums copied from the KamajiControlPlane
CRD at the provider pin, so the application API refuses a misspelled
name where the tenant edits it instead of at apply time. Both fields
are optional.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:11 +03:00
Aleksei Sviridkin 43cbcf9f46 fix(kamaji): anchor the ignore patterns to the chart root
Helm skips .helmignore when it reads a chart out of an archive and
honours it when it reads one out of a directory. A pattern with no
slash is matched against the base name at any depth, so `hack` reached
charts/kamaji-crds/hack, where the CRD templates keep the schema they
pull in with .Files.Get. The cluster, which gets the archive, has always
had the full CRDs; every local render, lint and package produced three
CustomResourceDefinitions with an empty spec, and no check that runs
before a cluster sees the chart could have caught a schema defect.

There is no hack directory at the top level of this package, so the
pattern excluded that subchart and nothing else. `images` is anchored
alongside it against the same trap; it matches only the intended
directory today and the packaged file list is unchanged by it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:01 +03:00
Aleksei Sviridkin 88264216b7 docs(kubernetes): restate the OIDC UID flags for the merged reading
The migration note told operators to move the extraArgs entry count in
the same edit so the running control plane would rewrite its apiserver
arguments; the control plane compares a hash of the list now, so any
change already forces the rebuild and no second edit is owed in either
direction. The paragraphs describing a value that survives only as the
last entry of its flag, and an empty value reaching the apiserver as a
bare flag, describe a build that is no longer shipped.

What replaces them is the accumulation each flag actually performs: a
uid-headers list the apiserver validates as the union of every entry,
feature gates settled across entries with the last term winning and one
spelling of the kube component throughout, and an emulated version that
two entries cannot both name.

One contract the earlier text sold is narrowed to what the chart
keeps. The render checks run only where the chart renders something, so
the three readings it refuses to guess at switch them off as well, and a
shape that would otherwise be refused passes through to a control plane
that does not start. A flag and value written as two entries is a
narrower case, and the page says which checks hold next to one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:59:01 +03:00
Aleksei Sviridkin 5b370d533f fix(kubernetes): read every extraArgs entry for the OIDC UID flags
The tenant control plane hands the kube-apiserver every entry of a flag,
so reading only the last one misses an opinion the apiserver acts on. An
earlier --feature-gates entry turning RemoteRequestHeaderUID off, or an
earlier --emulated-version naming kube, was previously dropped before the
apiserver saw it; now it decides, and rendering the uid header flag
against the later entry alone stops the control plane from starting.

Repeats of --requestheader-uid-headers append, so the chart adds its own
element beside a list that omits X-Remote-Uid instead of refusing to
render: the union satisfies the check the refusal existed for.
--feature-gates terms accumulate into one component bucket, so the chart
carries only its own term rather than a copy of the operator's, and
naming the kube component both bare and kube:-prefixed is refused
outright, which nothing the chart renders can repair.

A flag and its value written as two entries is read the same way the
apiserver reads it. pflag pairs them, so the value belongs to an entry
this code reads separately: the uid header flag counts as present, which
is what the gate-disabled check needs, while --feature-gates and
--emulated-version decide what is rendered and are not guessed at. What
holds whatever such a value hides still applies next to one: an empty
header element is refused, an explicit opt-out beside a header entry is
refused unless the hidden value is a gate term that could undo it, and
X-Remote-Uid is added beside an operator's own header list, which it can
only help. The chart's other additions wait for a reading.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:58:12 +03:00
Aleksei Sviridkin b31aa03a53 feat(kamaji)!: update to 26.9.4-edge
The tenant kube-apiserver argument list is no longer collapsed into a
map keyed by flag name: entries supplied through the control plane spec
reach the apiserver verbatim, duplicates included, and the reset that
rewrites them from scratch keys on a hash of the list rather than on its
length, so a same-count flag swap no longer strands the old flag on the
running Deployment.

The datastore deletion fix landed upstream verbatim, so its patch no
longer applies, and one failing patch fails the whole apply of the
directory. The module overlay is obsolete for the opposite reason: this
tag already carries golang.org/x/net, golang.org/x/crypto and
go.opentelemetry.io/otel/sdk above the versions the overlay forced, so
keeping it would downgrade all three.

The etcd.deploy override goes with it. The chart's dependency condition
is kamaji-etcd.deploy and the subchart is not vendored here, so the key
has never selected anything.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:58:12 +03:00
Aleksei Sviridkin 76b3652297 fix(kubevirt)!: evict non-migratable VMs instead of blocking the drain (#4403)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A node drain now completes when a VM on that node cannot live-migrate.
The VM is shut down and started again instead of holding the drain
forever. VMs that can migrate still migrate. The change is the
cluster-wide `evictionStrategy` in the KubeVirt CR, from `LiveMigrate`
to `LiveMigrateIfPossible`.

The drain is held by virt-api's eviction webhook, not by a
PodDisruptionBudget. With `LiveMigrate` and a VMI that is not
live-migratable, the webhook denies every eviction with HTTP 429
([pod-eviction-admitter.go#L147-L150](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L147-L150)).
`kubectl drain` retries a 429 until `--timeout`, which is infinite by
default. KubeVirt v1.9.0 does not create PDBs for VMIs at all; its
disruption-budget controller only deletes old ones
([disruptionbudget.go#L353-L406](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go#L353-L406)).
With `LiveMigrateIfPossible` the same webhook lets the eviction through
for such a VMI
([L152-L155](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L152-L155)).
The VM then comes back according to its `runStrategy`.

RWO disks are not the only case. A VM with a passthrough GPU is not
migratable either, unless the `VGPULiveMigration` gate is on and the GPU
is a mediated device
([vm.go#L1310-L1319](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-handler/vm.go#L1310-L1319)).
The gate is off in Cozystack, so any `vm-instance` with `gpus` set
blocked the drain of its node until now.

In KubeVirt a drain and a launcher update go through the same eviction.
After a KubeVirt upgrade the workload updater live-migrates outdated
VMIs that can migrate and evicts the rest, because
`workloadUpdateMethods` lists `Evict` as the fallback
([workload-updater.go#L421-L424](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L421-L424)).
The eviction call
([L614-L620](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L614-L620))
hits the same webhook as a drain. So a `vm-instance` VM that cannot
migrate either blocks both, which is what happens today, or is restarted
by both, which is what happens after this PR once the VM runs under the
new strategy.

This has a cost. A non-migratable VM now restarts on a drain. That
covers VMs on RWO storage and VMs with GPU or host device passthrough.
The webhook does not look at who asked for the eviction, so this holds
for any eviction through the API, not only a drain. A VM whose disk is
bound to the drained node, for example on a LINSTOR class without DRBD,
stays down until the node is back.

The upgrade itself restarts nothing, and existing VMs switch only after
their next restart. KubeVirt writes the cluster strategy into the VMI
spec when the VMI is created
([defaults.go#L208-L211](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/defaults/defaults.go#L208-L211)),
and the webhook reads that field first
([migrations.go#L116-L122](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/util/migrations/migrations.go#L116-L122)).
A VMI that was running before the upgrade keeps `LiveMigrate`, so a
drain still waits on it and the updater's eviction still gets 429. After
one restart the VMI carries `LiveMigrateIfPossible`. From then on every
KubeVirt bump shuts it down through the updater, in batches of 10 every
minute, the `batchEvictionSize` and `batchEvictionInterval` set in the
CR
([kubevirt-cr.yaml#L139-L140](https://github.com/cozystack/cozystack/blob/25d4a8c951b9fa7249b56ae79328d2cfd4f3a9c0/packages/system/kubevirt/templates/kubevirt-cr.yaml#L139-L140)).
Reading the source, a CPU socket increase on such a VM under
`vmRolloutStrategy: LiveUpdate` goes the same way, since the
`VCPUChange` condition sends it to the updater
([L320-L325](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L320-L325)).
I have not run that one.

I kept `Evict` even so, because `kubernetes-nodes` workers depend on it.
They set `evictionStrategy: External`, and for that strategy the webhook
marks the VMI for evacuation instead of evicting it
([pod-eviction-admitter.go#L156-L157](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L156-L157)).
CAPK sees the mark on a VMI that cannot migrate, drains the tenant node
and deletes the VMI
([machine.go#L414-L454](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/blob/v0.1.10/pkg/kubevirt/machine.go#L414-L454)).
The worker VM has `runStrategy: Always`, so KubeVirt starts it again on
the new launcher. That is how a GPU worker or any other non-migratable
worker gets the new launcher. Without `Evict` those workers would stay
on the old launcher for good. Worker behaviour does not change with this
PR, and the new test pins the method list so the fallback is not dropped
later.

`vm-instance` renders no `evictionStrategy` and has no value for it, so
a tenant has no way to opt out of either restart. KubeVirt would honour
a per-VM value over the cluster one. I filed #4419 to expose it.

I picked this over the options in the issue because KubeVirt already
decides migratability per VMI at eviction time. The platform CR does not
need to know anything about storage, and a chart shipping a VM does not
have to remember an override. Deriving the strategy from the disk access
mode in a chart would also miss the GPU case.

To get the new drain behaviour for a non-migratable VM that is already
running, restart it once after the upgrade. Until then a drain of its
node still blocks, as it does today.

The comment in the CR is on the same line as the key on purpose. `make
update` inserts blocks right before the `evictionStrategy:` line, so a
comment above the key would get separated from it.

A VM with `runStrategy: RerunOnFailure`, `Manual` or `Once` may stay off
after an eviction. If the guest shuts down cleanly on the ACPI request
within its grace period, the VMI ends `Succeeded`. `RerunOnFailure`
restarts only a failed VMI, and `Manual` and `Once` restart nothing on
their own
([vm.go#L1039-L1149](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go#L1039-L1149)).
`Once` stays off even after an unclean stop. Only the default `Always`
brings the VM back in every case.

What I did not verify. I have not drained a node with such a VM on a
live cluster, and I have not watched the updater restart them during an
upgrade. Everything above comes from the KubeVirt v1.9.0 and CAPK
v0.1.10 source. I did not check how `talosctl upgrade` handles a drain
that keeps getting 429. I also did not check how the cluster autoscaler
treats a node with a non-migratable VM now that its launcher pod can be
evicted. The pod has no `safe-to-evict` annotation, so for a VM on
node-local storage that likely comes down to the autoscaler's
`skip-nodes-with-local-storage` setting. Finally, I did not check
whether the upstream `VMCannotBeEvicted` alert is deployed in Cozystack.
For non-migratable VMs it stops firing after this change, since they are
evictable now.

Closes #4360

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubevirt)!: VMs that cannot live-migrate no longer block a node drain. The cluster-wide eviction strategy is now LiveMigrateIfPossible instead of LiveMigrate. VMs that can migrate still migrate. VMs that cannot, for example with a ReadWriteOnce disk or GPU or host device passthrough, are shut down and started again by their runStrategy. This applies to any pod eviction, not only a drain. A VM whose disk is bound to the drained node stays down until the node returns. A VM that is already running keeps the old strategy until its next restart, so restart non-migratable VMs once after upgrading if their nodes need to drain. The KubeVirt workload update uses the same eviction: once a non-migratable vm-instance VM runs under the new strategy, every later KubeVirt upgrade shuts it down, 10 VMs per minute. A VM with runStrategy RerunOnFailure, Manual or Once that shuts down cleanly on eviction stays off until it is started again. kubernetes-nodes workers are not affected.
```
2026-09-24 15:57:44 +04:00
Alexey Artamonov e8779e479c refactor(monitoring): keep the HelmRelease rationale in template comments
The timeout, waitStrategy and healthCheckExprs rationale in the
monitoring-system HelmRelease was written as `#` comments, which Helm
renders verbatim, so every tenant's HelmRelease carried about thirty lines
of prose. Move it into template comments like the traces paragraph beside
it: the reasoning stays in the chart source and the rendered object is
unchanged apart from losing the comments.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 14:41:27 +03:00
Alexey Artamonov 2682185bff docs(monitoring): warn that renaming a tracingStorages entry replaces its backend
A tracingStorages name is the identity of everything the entry renders:
the VTCluster/VTSingle, its Services and PVCs, and its Grafana datasource.
Renaming an entry therefore deploys an empty backend and removes the old
one, exactly like changing mode, but only mode said so. Give name the same
treat-as-immutable note, and state the limits the render guard enforces.

Reword the neighbouring mode and retentionDiskUsageBytes descriptions
without arrows and dashes, which render awkwardly in the dashboard and in
the generated README.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 14:40:36 +03:00
Alexey Artamonov 76445ba9b6 fix(monitoring): reject a negative tracingStorages storage
The storage guard only matched zero-shaped values, so -1Gi passed it and
the schema's quantity pattern alike, rendered into the VTCluster/VTSingle
spec, and left the release failing under retries: -1 once the API server
refused the PVC. Refuse a leading minus in the same guard, so the error
names the field before anything is applied and matches what the message
promises.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 14:39:39 +03:00
Aleksei Sviridkin 08c535c295 fix(monitoring): give kube-apiserver its own scrape size limit (#4427)
## What this PR does

kube-apiserver's `/metrics` grows with the number of resources and CRDs,
and on a full install it outgrows the 32MB `promscrape.maxScrapeSize`
that monitoring-agents sets globally. vmagent then refuses every scrape
of the API server, `up` drops to 0, and `KubeAPIDown` and `TargetDown`
fire against a healthy control plane while every rule over API server
metrics goes blind.

The apiserver `VMServiceScrape` endpoint now sets its own
`max_scrape_size` (128MB by default, `vmagent.apiserverMaxScrapeSize`).
vmagent uses a per-job value in place of the global flag for that job
only, so the 32MB guard for every other target is unchanged. The value
stays finite on purpose: vmagent reads the whole uncompressed response
into memory before parsing, `streamParse` included, so this limit is
also a memory ceiling.

A new `VMAgentScrapeSizeExceeded` warning fires on
`vm_promscrape_max_scrape_size_exceeded_errors_total`, so an oversized
target is reported as a size problem instead of only as a target that is
down. The counter has no target label, so the description points at the
vmagent targets page to find it.

Upgrade note: the per-job value replaces the global flag in both
directions. An install that raised `promscrape.maxScrapeSize` above
128MB through `vmagent.extraArgs` should raise
`vmagent.apiserverMaxScrapeSize` as well.

Verified on a dev cluster with the same operator (v0.68.4) and vmagent
(v1.136.0): a test vmagent with a 1MB global limit scraped all three API
servers with `max_scrape_size: 64MB` on the endpoint and refused all
three without it. The new counter reached storage with the `namespace`
and `pod` labels the alert uses, and the alert expression returned a
series. The `VMRule` passes the operator's admission webhook, which
rejects a malformed expression. Guest clusters get the same default,
since their HelmRelease overrides other `vmagent` keys only.

Fixes #4343

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring): kube-apiserver is scraped with its own 128MB size limit (`vmagent.apiserverMaxScrapeSize` in the monitoring-agents component) instead of the global 32MB, so a large `/metrics` no longer makes KubeAPIDown fire on a healthy API server. A new VMAgentScrapeSizeExceeded alert reports any target whose response exceeds the scrape size limit.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Kubernetes API server metrics can now use a configurable scrape-size
limit, set to 128 MB by default. Leaving the setting empty uses the
global limit.
* **Bug Fixes**
* Added a warning alert when vmagent reports scrape-size-limit errors.
The alert identifies the affected vmagent pod and service and provides
guidance for finding the target and adjusting its limit.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 14:30:46 +04:00
Aleksei Sviridkin 15035a915b fix(vm-instance): disable helm readiness wait on the VMInstance release (#3735)
## What this PR does

A VMInstance is served as a Flux HelmRelease that renders a KubeVirt
VirtualMachine. Since v1.5.0 (helm-controller v1.4.3 to v1.5.0) that
release is health-assessed by the kstatus poller. vm-instance sets no
`healthCheckExprs`, so the VirtualMachine goes through kstatus's own
rules in `fluxcd/cli-utils` (`pkg/kstatus/status`): `checkGeneration`
returns InProgress for any object whose `status.observedGeneration`
differs from `metadata.generation`, and past that check a `Ready=False`
condition is InProgress too, so a halted VM is never Current even with
its generation in sync. KubeVirt keeps a VirtualMachine's
`observedGeneration` behind its `generation` whenever a `spec.template`
change is pending a restart, the VM is stopped, or a live-updatable
change never converges (kubevirt/kubevirt#18700). So a healthy, Running
VM is read as permanently InProgress, the upgrade waits its full
timeout, and fails with `timeout waiting for: [VirtualMachine/...
status: 'InProgress']`. On releases still carrying the pre-1.5.0
`remediation.retries: -1` + rollback spec, the timeout then reverts the
previous manifest, dropping just-applied changes such as opened ports.

This sets `release.cozystack.io/helm-install-disable-wait: "true"` on
the vm-instance ApplicationDefinition, so cozystack-api generates the
HelmRelease with Install/Upgrade DisableWait. The release settles once
the VirtualMachine is applied, and VM health is read off the
VirtualMachine itself (`status.printableStatus` / `Ready`) and off the
WorkloadMonitor the chart already renders, which is what
`WorkloadsReady` on the Application reports. This restores the pre-1.5.0
behaviour, under which the VM was never a helm readiness gate. The same
mechanism is already used by `kubernetes-rd` and `kubernetes-nodes-rd`.

On its own that annotation does not reach a running cluster, so two gaps
in cozystack-controller are closed with it.

The first is the rollout. `applicationdefinition-controller` hashed each
definition's name and spec only, so a change confined to a
`release.cozystack.io/*` annotation left the hash unmoved and
cozystack-api kept serving the configuration it read at its last
start-up, which is the only place those annotations are ever read. The
hash now covers them too. Only that prefix, because kubectl and Helm
rewrite annotations of their own on every apply and hashing those would
roll the api Deployment on every reconcile.

The second is the releases that already exist.
`applicationdefinition-helm-reconciler` brings chartRef, valuesFrom and
labels on existing HelmReleases in line with their definition; it now
does the same for the readiness wait. Without it the annotation reaches
new instances only, which on an affected cluster is the wrong half:
#3734 reports 134 of 136 releases already looping upgrade and timeout,
and re-applying every VMInstance by hand was the only way out. The
setting is driven in both directions, since cozystack-api rebuilds the
whole spec on every write and emits `DisableWait=false` once the
annotation is gone. An absent install or upgrade block already behaves
as `DisableWait=false` and is materialised only when the definition asks
to disable the wait, so a release with nothing to change is not
rewritten.

Two alternatives were considered. A per-kind `healthCheckExprs` keyed on
`status.printableStatus` cannot work: setting it moves the
VirtualMachine from kstatus to the CEL status evaluator
(`fluxcd/pkg/runtime/cel`), which runs the same `observedGeneration`
check ahead of the expressions, and any `healthCheckExprs` added to this
kind later is a no-op while the wait is disabled. The typed
`spec.release.waitStrategy: legacy` is the narrower option, since the
legacy checker never waits on a custom resource, but it still gates the
release on the chart's Service, and with `external: true` that is a
LoadBalancer the legacy checker holds not ready until it has ingress. So
an external VM would still wait on IP allocation, which `WorkloadsReady`
already watches. Disable-wait avoids that and is the mechanism this
repository already uses for the same class of problem.

Tests: a helm-unittest suite (`tests/readiness_gate_test.yaml`) guards
that the annotation stays on the rendered ApplicationDefinition, and Go
unit tests cover both cozystack-controller paths: the hash moves on a
release annotation and stands still on an unrelated one, and the wait
lands on an existing release, is withdrawn when the annotation goes,
does not materialise empty install/upgrade blocks, does not rewrite a
release already in sync, leaves a release alone when the annotation does
not parse, and does not reach a release of another kind.

Note on existing clusters: no manual step. The definition changes on
upgrade, the config-hash rolls cozystack-api, and the helm reconciler
brings existing VMInstance releases in line. Releases that predate
v1.5.0 also carry a `remediation.retries: -1` + rollback spec, which is
left alone: with the wait disabled the upgrade no longer times out, so
that remediation no longer fires, and any write through the apps API
drops it.

Closes #3734.

### Downstream repositories

- [x] No downstream repository is affected by this change

<!-- Walked the trigger map against the diff. The change is a readiness
annotation on an ApplicationDefinition plus two cozystack-controller
paths. It does not touch openAPISchema, kind, plural, release.prefix,
any app package, any values.schema.json or any installer/platform values
key. The website's application-definitions.md describes the config-hash
as a checksum "over the sorted definitions", which stays accurate now
that the hash also covers the release.cozystack.io/* annotations, and no
page documents those annotations today. -->

### Release note

```release-note
fix(vm-instance): VMInstance HelmReleases no longer hang InProgress or roll back healthy VMs; the release no longer waits on VirtualMachine readiness, which helm-controller's kstatus poller cannot assess correctly for KubeVirt VMs. Existing VMInstance releases are brought in line automatically on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* VM installations no longer wait indefinitely when Kubernetes reports
the VM’s observed generation behind its current generation.
* Changes to release settings are now detected and applied to generated
Helm releases, including install and upgrade wait behavior.
* **Tests**
  * Added coverage for VM readiness and release-setting reconciliation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 14:30:23 +04:00
Alexey Artamonov cf7a5684d0 test(monitoring): make the single-mode VPA absence assertions able to fail
containsDocument with not: true but no any: true is scoped to a single
document, so the "no cluster VPA in single mode" and "no VPA when empty"
assertions passed regardless of what the render actually held. Add any: true
inside the matcher so they scan every document: a mutation that leaks the
cluster VPAs into single mode now turns the suite red, and it is green on HEAD.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 37789e1768 fix(monitoring): reject zero storage and over-long names in tracingStorages
Two tracingStorages values render cleanly and only fail once the operator
acts on them.

- A zero storage is silently destructive in single mode: the operator's
  isStorageEmpty gate checks IsZero() and mounts an EmptyDir instead of a
  PVC, the VTSingle reports operational and the release goes Ready, and every
  stored span is lost on the next reschedule. Type the storage field as a
  quantity so the schema refuses non-quantities at admission, and refuse
  every zero spelling ("", 0, 0Gi, 0.0, .0, 0e3, 0m, +0, -0Mi) in the render
  guard, since the quantity pattern accepts those.
- A name over 42 characters wedges cluster mode: the operator names the
  StatefulSet vtstorage-<name>, and each pod gets the label
  controller-revision-hash: vtstorage-<name>-<hash>, whose value is capped at
  63 with up to 10 characters of hash. Past that no pod is created, the
  VTCluster goes failed and the release stays not-Ready under retries: -1.
  Cap the length in the same guard.
- Quote the rendered name, so a numeric name such as 2024 stays a string
  instead of failing the apply on metadata.name.
- Rewrite the helper's rationale so it matches the trimmed CRDs and the
  pinned operator, and state in the mode field description that the API
  cannot enforce immutability on an array-item field, so the note is the only
  guard.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 05eebbdf4f refactor(monitoring): keep tracing rationale in template comments
The `#` prose in the traces datasource and in the HelmRelease health-check
block is not stripped at render the way `{{/* */}}` is, so it shipped into
every tenant's rendered GrafanaDatasource and HelmRelease. Move it into
template comments: the reasoning stays in the chart source but no longer lands
in the emitted objects.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 30fcfe6877 fix(monitoring): emit each vlogs GrafanaDatasource as its own document
The range over logsStorages emitted its documents with no `---` separator, so
two or more entries concatenated into a single document with duplicate keys,
each later entry overwrote the earlier one, and every log store but the last
silently got no datasource. Add the separator inside the range, matching the
traces datasource template.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov f20bd7b334 fix(monitoring): validate tracingStorages entries and flag mode/retention hazards
The installed VictoriaTraces CRDs are the trimmed ones (crds.plain: false),
so their spec is x-kubernetes-preserve-unknown-fields and a bad value reaches
the operator instead of being refused by the API server. Catch the ones a
render guard can, and document the one it cannot.

- name is used verbatim as the VTCluster/VTSingle and GrafanaDatasource
  metadata.name, so an invalid name is a late admission error and a duplicate
  silently overwrites the first entry (last write wins in the range). Guard it
  against an RFC 1123 label and a uniqueness set.
- retentionDiskUsageBytes maps to the operator's BytesString field, whose
  UnmarshalJSON rejects anything outside
  ^[0-9]+(kb|mb|gb|tb|KB|MB|GB|TB|KiB|MiB|GiB|TiB)?$, notably not the Gi
  grammar of the sibling storage field. The operator then marks the CR failed
  and the release fails well after the value was written. Refuse it at render
  with the upstream pattern instead.
- changing mode re-identifies the object instead of reconfiguring it, so
  switching single to cluster prunes the VTSingle and the operator
  garbage-collects its PVC, and cluster to single (or dropping the entry)
  leaks the vtstorage PVCs
  until the release is deleted. A stateless template cannot see the prior
  mode, so record the loss and the one-way migration in the field description.

The name, uniqueness, mode and retention checks share one pass in a helper;
the inline mode guard in vtraces.yaml moves there. The retention description
also notes that the cap is per vtstorage replica in cluster mode.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 796340fc79 fix(monitoring): correct stale 'render-rejected when empty' comments
The healthCheckExprs comment (and the mirroring test comment) still claimed
tracingStorages is render-rejected when empty and that the 'Ready with no
trace store' hole cannot be reached, but defaulting tracing off removed that
guard and made empty the default. Rewrite both to state the actual behaviour:
empty is the default (tracing off), so the exprs are usually inert (Flux
evaluates them only against CRs present in the inventory), and a Ready release
with no trace store is intentional and harmless because nothing produces
spans without a collector, so no fail-on-empty guard is needed.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 41aa2aad1d feat(monitoring): add VPA for VictoriaTraces backend components
vpa.yaml sized only metrics and logs components; the trace components got no
VerticalPodAutoscaler, so they sat at the operator's static defaults with no way
to size them (tracingStorages has no resource knobs and the template hardcodes
resources/replicaCount).

Add VPAs per tracingStorages entry, branching on mode: cluster mode gets vtinsert
and vtselect (Deployments) + vtstorage (StatefulSet), single mode gets
vtsingle (Deployment). Kinds verified against victoria-metrics-operator v0.68.4.
updateMode: Initial like the rest of the file (no eviction, because of the
Cilium IP-reuse race), which is also required for VTSingle's plain RWO PVC (an
evicting VPA would deadlock on multi-attach). Bounds are hardcoded, without the
dead min/maxAllowed conditionals the logs entries carry (tracingStorages
declares no such properties). Empty tracingStorages renders no trace VPA. Tests
cover the cluster three-VPA split, the single-mode single VPA with its
updateMode, and off-by-default.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 00ff19ec5c fix(monitoring): default tracing off and defer fail-on-empty guard
tracingStorages had two defaults (extra + system), so 'tracingStorages: null'
silently backfilled the inner default and a VTCluster rendered anyway, while
'tracingStorages: []' aborted the whole monitoring release. There was no clean
'off', and the default shipped a 6-pod VTCluster per tenant for a pipeline
that nothing writes spans into yet.

- Default tracingStorages to [] in both packages/system/monitoring/values.yaml
  and packages/extra/monitoring/values.yaml, so absence has nothing to fall
  through to and the generated schema advertises the shipped (off) state.
- Drop the fail-on-empty guard in templates/vtraces/vtraces.yaml. Nothing
  produces spans without a collector, so an absent backend drops no data. The
  guard belongs with the collector, where it becomes correct: an enabled
  producer with no backend really would drop spans, the logsStorages #3181
  rationale.
- healthCheckExprs stay unconditional (Flux evaluates them only against CRs
  present in the inventory, so they are inert when nothing renders).
- Tests: replace the empty-fails case with off-by-default (renders nothing);
  set tracingStorages explicitly where cases relied on the old default.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov b24c3e1bc4 test(monitoring): assert VTCluster/VTSingle readiness-gate entries
helmrelease_test.yaml pinned only healthCheckExprs[0] (VLCluster), not the
VTCluster [1] and VTSingle [2] entries that hold the release not-Ready until
a configured trace store reports operational. Assert both entries
(apiVersion/kind/current/inProgress/failed) so a dropped/reordered block or a
typo'd CEL expression fails a test instead of silently regressing the gate.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov dbe7a732ad test(monitoring): cover VTSingle PVC cleanup label + single-mode disk cap
Single mode stamps the release-scoping cleanup label via
spec.storageMetadata, a different field than the cluster path
(spec.storage.storage.volumeClaimTemplate), and it was untested, so a dropped
storageMetadata block would orphan the VTSingle PVC on tenant deletion with no
test to catch it. Add the VTSingle cleanup-label assertion and a single-mode
top-level retentionMaxDiskSpaceUsageBytes assertion.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 3774e09159 fix(monitoring): reject unsupported tracingStorages mode
tracingStorages[].mode accepted any string, so a typo (e.g. 'singel')
silently rendered a VTCluster instead of failing. Add a render guard that
fails on anything but cluster|single, restrict the schema property with a
TracingMode enum, and add a helm-unittest case asserting the rejected mode.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov 32f170949e test(monitoring): pin verified VictoriaTraces query URLs; jaeger stable default
Datasource service/port/path verified against upstream docs
(https://docs.victoriametrics.com/victoriatraces/querying/grafana/):
cluster vtselect:10471/select/jaeger, single vtsingle:10428/select/jaeger.
Jaeger is the stable, built-in Grafana core datasource; the Tempo API
(/select/tempo, type: tempo, VictoriaTraces v0.9.4+) is upstream-experimental,
kept as a documented alternative. Add helm-unittest assertions pinning both
the cluster and single-mode query URLs.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
Alexey Artamonov c2fce3d41d feat(monitoring): add VictoriaTraces backend + Grafana datasource for tracing
Phase 1 of the distributed-tracing design proposal (cozystack/community#38):
a per-tenant traces backend in the monitoring stack, mirroring the logs
(VLCluster) implementation. No new operator: VTCluster/VTSingle already ship
in victoria-metrics-operator.

- tracingStorages values (extra + system), per-entry mode cluster|single
- templates/vtraces/vtraces.yaml: VTCluster (default) / VTSingle (edge/dev),
  #3181-style empty-list render guard, retentionPeriod + optional disk cap
  (retentionMaxDiskSpaceUsageBytes), release-scoped PVC cleanup label
- templates/vtraces/grafana-datasource.yaml: Jaeger-typed traces datasource
- extra/monitoring readiness gate: VTCluster/VTSingle healthCheckExprs on
  status.updateStatus, alongside VLCluster
- helm-unittest suites (required/empty-guard, cluster-vs-single, datasource,
  PVC label) + regenerated values.schema.json / README / ApplicationDefinition

The per-tenant OTLP Collector, engine integrations, shared-central topology
and docs are out of scope and tracked in the tracing epic.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
2026-09-24 12:09:09 +03:00
myasnikovdaniil 9bec21461d feat(kubernetes): boot tenant worker disks from a shared golden Talos image (CDI clone) (#4171)
## What this PR does

Tenant worker boots from talos OS disk image that CDI streams over HTTP
from image factory, and it does that once per worker. Same bytes N
times, and N chances for a fetch to fail. In closed or rate-limited
environment that is wrong shape.

This PR adds opt-in catalog and per-pool selector that reads from it.

`packages/system/kubernetes-worker-image` imports one golden DataVolume
into `cozy-public` per `(schematicID, version)`, named
`talos-worker-<schematicID>-<version>`. Pool opts in with
`osImage.builtin` on its own `KubernetesNodes`, and its `disk-system`
DataVolume then carries `source.pvc` pointing at that golden, so CDI
does storage layer CSI clone instead of HTTP import. One import per
flavor, shared by every worker of every tenant.

Catalog is disabled by default, enable by adding
`cozystack.kubernetes-worker-image` to `bundles.enabledPackages`. Its
knobs (`imageFactoryURL`, `storageClass`, `storage`, `images`) are set
through `spec.components.platform.values.kubernetesWorkerImage` on
cozystack-platform Package. That needed optional package helper to learn
components argument, which `cozystack.platform.package.optional.default`
did not have, so every optional package registered through that wrapper
could name variant and nothing else. Thirteen other packages get same
path for free.

`osImage` has two mutually exclusive forms:

- `osImage.builtin` boots pool workers by CDI-cloning golden from
catalog. Clone is storage layer CSI copy, so OS image never crosses pod
network and factory answers once for whole fleet instead of once per
worker.
- `osImage.factory` keeps HTTP import and lets pool pin own
`imageFactoryURL`, `schematicID` and `version`. This is bring-your-own
path for operator running own factory or caching mirror, and it is what
pool gets when `osImage` is omitted.

In-guest talos installer written by reconcile job follows same
resolution, so pool that pins release boots it and upgrades into it
instead of being flipped back to cluster default halfway.

This is pool half of what #3950 asks for. That issue wants platform
level defaults for two channels, worker OS image and in-guest registry
mirrors. Here image channel gets per-pool source and shared cache,
platform default plumbing is left to the issue.

## Existing workers do not roll

With `osImage` omitted rendered `KubevirtMachineTemplate` is byte
identical to what chart produced before. KMT is named by hash of its own
content, so any drift would re-image every worker VM in fleet on next
upgrade. Stored render snapshot is untouched by this PR, that is the
evidence and not the claim. Editing `osImage` on existing pool re-images
that pool, like changing any other image reference.

## What fails at render

Clone that would silently degrade into something slower or wrong is
refused instead of rendered.

Golden absent, or its import in terminal `Failed` phase. Pool
`storageClass` different from golden one, because CDI cant CSI-clone
across storage classes and falls back to host assisted copy over pod
network, which is exactly the transfer `osImage.builtin` removes. Empty
class on either side resolves against cluster default class before
comparing, so pool at `storageClass: ""` is checked too. `diskSize`
below golden storage, because clone target cannot be smaller than
source. Talos release the matrix lists as unsupported, whether pool
reached it through `talos.version` or through `osImage.*` override.
Matrix passes talos minor it does not list, so what it promises is that
known-bad pairing is refused, not that every pairing is known.
`schematicID` that is not plain lowercase alphanumeric, or version that
is not `vMAJOR.MINOR.PATCH`, both are interpolated into DataVolume name
and image URL while `values.schema.json` types them as bare strings.

Golden that is only mid-import is not an error. CDI holds worker disk
until source populates, and failing there would abort whole pool
HelmRelease for length of a download. Same reasoning covers pool that
already cloned: absent or `Failed` golden fails render only for pool
still waiting for its first disk, otherwise reconcile of healthy pool
would abort over source it no longer reads.

Golden that declares no StorageClass is read from its PVC, not from
whatever default is in force today, because those differ once cluster
default moves. Cluster with two default classes resolves same way
Kubernetes does, newest wins. Pool that already cloned is identified by
annotation naming exactly that pool, not by disk name prefix, because
pool `md0-gpu` produces disks carrying `md0` prefix, and it reads its
own class from its own bound PVC, so admin moving
`storageclass.kubernetes.io/is-default-class` does not fail release of
pool whose disks are all on right class.

That resolved class is then written onto worker disk. Reading one class
and binding another is how pool at empty `storageClass` passed the
comparison and still took host assisted copy on its next worker, so
guard and disk now come from one value. HTTP path leaves empty class to
cluster as before, which is what keeps its render byte identical.

Catalog side refuses duplicate `(schematicID, version)` entries, entries
whose schematic or version cannot produce name pool can reconstruct, and
edit to `storage`, `storageClass` or source URL of entry whose golden
already exists. Last one is not patchable in place, so without guard it
lands as rejected patch on next upgrade.

Goldens carry `helm.sh/resource-policy: keep`, so dropping entry stops
managing golden instead of deleting it. Missing golden aborts whole
Kubernetes HelmRelease of every tenant pinned to it, not just the disk,
and each cloned worker disk annotates golden it came from, so "does
anything still clone this" is answerable before manual delete.

Three drift guards live in
`hack/check-worker-image-catalog-defaults.bats`, because values that
have to agree sit in files nobody edits together. One-sided talos bump
trips all three.

## Upgrade

Three pool values that never produced working image rendered before and
hard-fail now, so pool carrying any of them flips its HelmRelease to
`InstallFailed` on upgrade and not on next edit: `talos.version` in
`vMAJOR.MINOR` form, `schematicID` that is not plain lowercase
alphanumeric, and empty `talos.imageFactoryURL`, which used to render
schemeless `/image/...` source URL. None of the three could boot worker,
so this is failing earlier and not a regression, but it surfaces on pool
nobody touched and fix in each case is to correct the value. CR carrying
such value is admitted by API with value intact, and later unrelated
edit reaches `.spec.values` of HelmRelease while it stays
`InstallFailed`, so edit is taken everywhere and applied nowhere.

## Why e2e does not exercise the clone path

Clone is storage layer copy, so it needs StorageClass whose volumes are
reachable from any node. Merge-gating e2e has none: those lanes run on
talos containers, which share runner kernel and cannot load DRBD, so
node-local `local` is only class there. Nothing would arrange the
co-location such a class requires either, golden in `cozy-public` has no
consumer and binds where CDI importer landed, while worker disk binds
where its VM is scheduled.

So the suites keep importing over HTTP with `osImage` omitted, which is
also the render the no-roll guarantee rests on. This PR touches no e2e
harness file: `hack/` differs from main only by the two guards this
feature owns, one tying catalog default golden to pool chart talos and
StorageClass defaults, one on in-guest installer heredoc.

Switching suites over does not work on that substrate. Golden asks for
`replicated`, lane does not have it, CDI resolves no StorageProfile, and
catalog HelmRelease times out on DataVolume stuck in `ImportInProgress`.
Passing lane own class would fix the class and leave the co-location.

Nightly and tag build run same suites on QEMU with DRBD and
`replicated`, which is the substrate clone path was verified on. That is
where e2e coverage belongs, as its own change with that lane storage
headroom measured rather than assumed.

Worth being straight about what this leaves, because it bears on whether
the feature earns its place. The e2e argument was never the strong one:
golden fetches from image factory exactly once, and #3513 established
that node-join stall tracks host `kvm_amd` `vgif` flag rather than
registry egress. What clone genuinely removes is the fanout, N workers
streaming over pod network becomes N storage layer copies. That is
property of storage layer, so it is worth having where DRBD is and worth
nothing where it is not. This ships as production capability with the
constraint stated on the field, not as CI improvement.

## Testing

On this tree:

- full `make unit-tests` plus `make test-controllers`: 95 charts, 2520
helm-unittest cases, 1710 bats cases, `go test ./pkg/...` and
`./internal/...`, exit 0
- `pre-commit run --all-files`: exit 0, root `make generate` and per-app
generate both clean, so no generated-file drift
- render snapshot unchanged, which is what pins byte identical render
for pool that does not adopt `osImage`
- mutation pass over render guards, each term deleted or inverted in
turn and suite required to notice it by name. Restoring presence gate on
catalog StorageClass comparison reddens exactly one case, inverting its
`ne` to `eq` reddens five. Reverting worker disk to omit its resolved
class reddens three cases that pin where builtin disk binds, and forcing
that class non-empty on every path reddens HTTP-path case and all three
render snapshots, with snapshot diff naming the changed KMT hash

`E2E Tests` passed on run 35831830568. That run was on the previous
head, 4fd8e7ba, and this head differs from it only by one test comment
and the commit message.

## Verified on a live cluster

Throwaway three-node talos stand (OCI, talos v1.13.6, LINSTOR/ZFS,
`replicated` StorageClass) installed from this branch `packages/` tree.
This covers what neither `helm template` nor unit suites can answer. It
predates the change that writes resolved StorageClass onto worker disk,
and that stand ran with explicit `replicated` on both sides, which is
the value that change now renders.

- The catalog wires up from `bundles.enabledPackages` alone:
`PackageSource cozystack.kubernetes-worker-image` reconciled, its
HelmRelease installed, and the golden `talos-worker-<schematic>-v1.13.6`
imported to `Succeeded` in 108s (6Gi, RWX, `replicated`).
- A pool on `osImage.builtin` clones it on the storage layer. Both
worker DataVolumes went `CSICloneInProgress` then `Succeeded` in 43s
with `Running=False reason=Completed message="Clone Complete"`, carrying
`source.pvc` and no `source.http`, and no importer or upload Pod
appeared in the tenant namespace.
- The clone yields a bootable talos root disk. Both worker VMs reached
`Running`/`Ready`, joined the tenant cluster and went `Ready` about 66s
after creation, and the tenant cilium, coredns, csi, ingress-nginx and
metrics-server addons all installed on them.
- Access modes match across the hop: golden `ReadWriteMany` 6Gi to
worker `ReadWriteMany` 20Gi on the same `replicated` class, grown by
CDI.
- The written annotations are correct in practice: each worker disk
carries `kubernetes-worker-image.cozystack.io/golden` naming the golden
and `.../pool` naming its pool.
- helm-controller converges on a builtin pool: forcing a reconcile left
the content-hashed KMT name unchanged and the VMI and DataVolume UIDs
identical, so a reconcile does not roll the pool.
- Cross-namespace clone RBAC is in place as shipped: `create datavolumes
--subresource=source -n cozy-public` resolves to `yes` for a tenant
ServiceAccount through the `cdi-clone-dv` RoleBinding, and CDI runs with
`cloneStrategyOverride: csi-clone`. This PR adds no RBAC of its own.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

Trigger map points at two repositories.

Provider models every pool field by hand, so `osImage` needs schema,
model and expand/flatten pair there. Already tracked as
cozystack/terraform-provider-cozystack#39, filed when the field was
still called `image`.

`content/en/docs/next/operations/configuration/platform-package.md` is a
hand-written table of `spec.components.platform.values.*` keys and
nothing generates it, so `kubernetesWorkerImage` needs a row there.
Filed as cozystack/website#707.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/707
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/39
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): tenant worker pools can boot their system disk from a shared golden talos image instead of importing it from the Image Factory once per worker. Enable the opt-in `cozystack.kubernetes-worker-image` package via `bundles.enabledPackages` to publish one golden DataVolume per `(schematicID, version)` in `cozy-public`, then set `osImage.builtin` on a `KubernetesNodes` pool to have CDI clone it on the storage layer. A pool that omits `osImage` keeps importing over HTTP and renders exactly as before. The clone path needs the pool and the golden on the same StorageClass, and one whose volumes are reachable from any node, in practice a replicated/DRBD class.
```
2026-09-24 12:10:07 +05:00
Timofei Larkin 146e06e75e feat(api): take Application name limits from the application's own schema (#4367)
## What this PR does

The aggregated API server capped two kinds' names by knowing what their
charts render. A `Kubernetes` cluster name was held to 32 characters
because every worker pool is a `KubernetesNodes` release named
`kubernetes-nodes-<name>-<pool>`, and the default `md0` pool has to fit
the 53-character Helm release name limit. A `Kafka` name was held to 44
because the KRaft controller pod `kafka-<name>-c-<hash>-<id>` has to fit
the 63-character DNS-1123 label limit. The arithmetic is right, but it
is the charts' knowledge living in the platform: the next application
with a sub-resource budget needs another `if r.kindName == ...` branch,
and an application served from outside this repository — the whole point
of `ApplicationDefinition` — cannot get one at all.

An application now declares the limits on its own name in its own
schema, and the server applies whatever it finds there for whatever kind
the definition describes. The chart's `values.yaml` carries a `## @name`
annotation, `cozyvalues-gen` v1.7.0 publishes it at the root of
`values.schema.json` as `x-cozystack-name`, and `hack/update-crd.sh`
carries it into the `ApplicationDefinition`:

```yaml
## @name {string} - Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.
## @maxLength 32
```

```json
"x-cozystack-name": {
  "type": "string",
  "description": "Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.",
  "maxLength": 32
}
```

`maxLength`, `minLength` and `pattern` are honoured; `pattern` is
unanchored, as JSON Schema specifies. A declaration only tightens:
`prefix + name` must still fit the 53-character Helm limit. A rejected
name is answered with the chart's own description, or with "limit
declared by the <Kind> application" when the chart gives none.

The schema is untrusted input — definitions from other repositories are
written by hand — so a declaration that cannot be enforced as written
(not an object, a type error, an unknown keyword such as a misspelled
`maxLenght`, `maxLength` below 1, a negative `minLength`, `minLength`
above `maxLength` or above the Helm budget, a pattern that is not RE2)
refuses every create of that kind with `configuration error: the <Kind>
ApplicationDefinition declares an invalid x-cozystack-name: …`, and logs
the same at start-up. Dropping the declaration would admit names the
chart cannot render; applying it would reject every name and blame the
user. Other kinds are unaffected, and existing objects of the affected
kind can still be read, updated and deleted.

An upgrade brings `cozystack-api` up before `kubernetes-rd` and
`kafka-rd` re-render (`kubernetes-rd` depends on `cozystack-engine`), so
the new server can briefly read a definition that predates its
declaration. For exactly that case — a definition that declares nothing
— the two previous caps stay in `legacyNameCaps` as a floor, to be
removed once no supported upgrade starts from such a definition. That is
the only per-kind knowledge left in the name checks.

Also in this PR:

- The `pre-commit` job pins `cozyvalues-gen` v1.7.0, the first release
that understands `@name`
([cozystack/cozyvalues-gen#33](https://github.com/cozystack/cozyvalues-gen/pull/33)).
- The API review gate now diffs the root `x-cozystack-name` of every
definition as `metadata.name`, so a chart that lowers its cap or adds a
pattern is reported like any other tightened field. On this PR it
reports the Kubernetes and Kafka caps as added; their effective limits
are unchanged.
- The migration 56 comment that pointed at the removed
`maxKubernetesClusterName` now points at the chart's declaration.

Not in scope, because none of it is a static limit a chart can declare:
the `Tenant` name pattern in `pkg/apis/apps/validation`, the computed
tenant-namespace length check (it depends on the parent namespace at
request time), the tenant quota hierarchy, and the Phase 2 removed-field
warning for `Kubernetes`.

### How it is tested

`TestNameCapComesFromTheShippedSchema` reads the shipped `kubernetes`
and `kafka` ApplicationDefinitions off disk, builds storage through
`NewREST` as the server does, and checks the cap and the chart's
explanation; it then rewrites `maxLength` inside the shipped schema and
checks the limit moves with it, which a cap held in code would not.
`TestInvalidNameSchemaFailsClosed` covers each malformed declaration
through the same path, `TestLegacyNameCapsCoverUndeclaredDefinitions`
pins the upgrade floor and that a declaration replaces it, and
`TestClassifyNameDeclaration` covers the gate.

### Downstream repositories

Two repositories are affected and neither follow-up is filed yet, so no
box is ticked.

- `cozystack/website` — `content/en/docs/next/development.md` documents
the `values.yaml` annotations and needs `@name`;
`content/en/docs/next/cozystack-api/application-definitions.md`
describes `openAPISchema` and needs `x-cozystack-name`.
- `cozystack/external-apps-example` — it pins no `cozyvalues-gen`
version and regenerates against whatever is current; nothing in it
declares a name constraint, so it is unaffected until it chooses to.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(api): Application name limits now come from the application's own schema instead of being hardcoded per kind in the aggregated API server. A chart declares them with a `## @name` annotation in `values.yaml`, which cozyvalues-gen v1.7.0 publishes as `x-cozystack-name`; the API server enforces `maxLength`, `minLength` and `pattern` on `metadata.name` for any kind, answers a rejected name with the chart's own explanation, and refuses creates of a kind whose declaration it cannot enforce with a configuration error. The Kubernetes (32) and Kafka (44) name caps are unchanged in effect and are now declared by their charts.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Applications can declare minimum and maximum name lengths and
formatting patterns.
* Kubernetes cluster names are limited to 32 characters, and Kafka
cluster names to 44 characters, to meet generated resource naming
limits.
* Names that violate declared constraints are rejected with an
explanatory message. Tighter name constraints are also reported as
breaking changes.
* Declared maximum lengths cannot exceed the platform’s existing Helm
release-name budget.

* **Documentation**
* Added guidance on declaring and enforcing chart-specific resource-name
limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 10:29:27 +04:00
Andrey Kolkov ff52aeb44e chore(etcd-operator): bump to v0.5.6 (#4425)
## What this PR does

Bumps the vendored `etcd-operator` from **v0.5.5** to
**[v0.5.6](https://github.com/cozystack/etcd-operator/releases/tag/v0.5.6)**.

v0.5.6 is a controller-only bug-fix release; the two changes are:

- **Terminal-phase `EtcdMember` Pods are replaced**
([etcd-operator#370](https://github.com/cozystack/etcd-operator/pull/370)).
A graceful node shutdown SIGTERMs etcd, which exits 0, and the kubelet
parks the Pod in `Succeeded` without deleting it. The operator manages
bare Pods and only recreated on `NotFound`, so after a whole-cluster
reboot every member stayed down until the Pods were deleted by hand. The
reconciler now deletes an owned Pod in a terminal phase and lets the
next pass recreate it against the same PVC (or, for memory-backed
members, through the existing pod-loss re-add path).
- **`EtcdCluster` status stays truthful while etcd is unreachable**
([etcd-operator#369](https://github.com/cozystack/etcd-operator/pull/369)).
On a converged cluster the learner-promote and auth reconcile paths
returned their transient requeue before `updateStatus`, so
`Available`/`Degraded` and `readyMembers` froze at the last healthy
value and a fully down cluster kept advertising `Available=True`. The
status write now always happens, with the requeue carried through it.

Package changes are pin-only:

- **`etcd-operator`**: `appVersion` → v0.5.6 (image tag derives from
it), `ETCD_OPERATOR_REF` → v0.5.6, `tests/deployment_test.yaml` image
pins → v0.5.6.
- **`etcd-operator-crds`**: `ETCD_OPERATOR_REF` → v0.5.6. `make update`
at the new ref produces an empty diff: the five CRDs are byte-identical
to v0.5.5, and `charts/etcd-operator/files/manager-role-rules.yaml` did
not change, so `templates/rbac.yaml` is untouched.

Verified locally: `helm unittest` 18/18 (`etcd-operator`) and 1/1
(`etcd-operator-crds`), `helm lint` clean on both, `helm template`
renders `ghcr.io/cozystack/etcd-operator:v0.5.6`, and that tag resolves
in ghcr.io.

### Screenshots

n/a — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. The change is confined to
`packages/system/etcd-operator{,-crds}` and touches only version pins:
etcd-operator is a system component (not a
`packages/apps`/`packages/extra` package the website and TF provider
track), no CRD, schema, values, `hack/`, installer or platform values
changed.

- [x] No downstream repository is affected by this change

### Release note

```release-note
chore(etcd-operator): bump to v0.5.6, which replaces EtcdMember Pods left in a terminal phase after a node shutdown and keeps EtcdCluster Available/Degraded conditions truthful while etcd is unreachable
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Updated the etcd operator to version 0.5.6. This version is used
consistently across the operator deployment and its associated custom
resource definitions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 22:14:16 +04:00
IvanHunters 380c446cda fix(monitoring): pin the scrape size rendering and sharpen the alert
The per-job limit replaces the global flag downward as well, so an
install that raised promscrape.maxScrapeSize past it through extraArgs
has to raise this value too; say so where the value is set. Pin that a
numeric value still renders as a string, which the CRD field requires,
and pin the alert severity. Name the cluster in the alert and say where
the apiserver value lives, since a guest cluster cannot set it.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:56:07 +03:00
IvanHunters a17c5b2667 feat(monitoring): alert when vmagent refuses an oversized scrape
A target whose response exceeds the scrape size limit reports up == 0,
which is indistinguishable from the target being down: TargetDown and
KubeAPIDown fire and nothing names the real cause. Alert on
vm_promscrape_max_scrape_size_exceeded_errors_total so the first alert
says what is wrong. The counter has no target label, so the description
points at the vmagent targets page to find the target.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:28:29 +03:00
IvanHunters 4a268f6c01 fix(monitoring): give the kube-apiserver scrape its own size limit
kube-apiserver's /metrics grows with the number of resources and CRDs,
and on a full install it outgrows the global 32MB
-promscrape.maxScrapeSize. vmagent then refuses every scrape of it, so
up is 0 and KubeAPIDown and TargetDown fire against a healthy API
server while every rule over API server metrics goes blind.

Set max_scrape_size on the apiserver endpoint instead of raising the
global flag. vmagent uses it in place of the flag for that job only, so
the 32MB guard for every other target stays as it is. vmagent reads the
whole uncompressed body into memory even with streamParse, so the
per-job value is a memory ceiling and stays finite (128MB), exposed as
vmagent.apiserverMaxScrapeSize.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:27:00 +03:00
Andrey Kolkov e60fbc3f37 chore(etcd-operator): bump to v0.5.6
Controller-only release: EtcdMember Pods that reach a terminal phase
are replaced instead of being left behind, and EtcdCluster status is no
longer rewritten while etcd itself is unreachable. No CRD, RBAC or
values changes; re-vendoring the CRDs produces an empty diff.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-23 21:22:43 +04:00
IvanHunters 70ea242e1b docs(vm-instance-rd): trim the disable-wait comment and correct the mechanism
The annotation comment repeated the PR description and the test header;
two lines on why are enough. The hang comes from the kstatus generation
check in cli-utils, not the CEL status evaluator, since vm-instance sets
no healthCheckExprs. The config-hash rollout runs in cozystack-controller,
not cozystack-operator.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:18:06 +03:00
Aleksei Sviridkin 45f5698bce fix(cdi): validate kubevirt-cdi values against a schema (#4417)
## What this PR does

This gives the kubevirt-cdi chart a values schema, so a bad clone
strategy fails when the chart renders instead of when the CDI object is
applied. Today a typo like `csi_clone` renders fine, the CDI CRD refuses
it on apply, and the operator finds out from a failed HelmRelease.

The schema is generated by `cozyvalues-gen` from annotations in
`values.yaml`, through a new `make generate` target, the same way
`talos-log-collector` does it. There is no README here, so the target
only writes the schema.

The clone strategy is checked with a pattern, not with the CRD's enum.
The chart uses the empty string to omit the field and let CDI pick a
strategy per StorageProfile, and the CRD enum has no member for that.
`cozyvalues-gen` also can't emit an empty enum value. So the accepted
values are `copy`, `snapshot`, `csi-clone` and `""`. A null still works
as before. The key has a chart default, so Helm drops a null for it
before validation, and the template omits the field. `uploadProxyURL`
only has to be a string.

The new test suite refuses `csi_clone` and values of the wrong type, and
accepts each of the allowed values and null.

Closes #4141

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): kubevirt-cdi now validates its values against a schema, so an invalid `cloneStrategyOverride` such as `csi_clone` fails at render time instead of when the CDI object is applied.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration**
* Chart values are now validated to ensure the upload proxy URL is a
string, importer resources are provided as an object, and clone strategy
uses a supported value: `copy`, `snapshot`, `csi-clone`, or an empty
value.
  * Existing defaults for these settings remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 20:07:06 +04:00
Aleksei Sviridkin 5e460605c2 fix(cdi): drop the ExpandDisks gate from the CDI object (#4415)
## What this PR does

This drops a feature gate from the CDI object that CDI never reads.
`ExpandDisks` sits in `spec.config.featureGates` in
`packages/system/kubevirt-cdi/templates/cdi-cr.yaml`, but it is a
KubeVirt gate.

CDI v1.66.1 has no gate with that name. It looks a gate up by exact
name, so an extra entry is read and ignored. The CRD types the list as
plain strings, so nothing rejects it either. Disk expansion comes from
KubeVirt, where `ExpandDisks` is GA at v1.9.0, the release the platform
ships. I left the KubeVirt CR as is; #4029 tracks the GA gates in that
list.

The gate list now has its own test case, pinned whole to what upstream's
`cdi-cr.yaml` ships. Adding `ExpandDisks` back turns it red. `make
update` only checks the upstream file against its pinned hash and writes
nothing under `templates/`, so it can't bring the line back.

Closes #4112

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): stop listing the KubeVirt `ExpandDisks` feature gate on the CDI object, where CDI ignored it. Disk expansion is unaffected.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration**
* CDI now enables only the `HonorWaitForFirstConsumer` feature gate. The
`ExpandDisks` gate is no longer included in the configuration.
* **Tests**
* Updated configuration checks to verify the exact feature-gate list and
ensure it matches the gate CDI supports.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 20:06:31 +04:00
Aleksei Sviridkin 25d4a8c951 fix(kubevirt)!: evict non-migratable VMs instead of blocking the drain
The cluster-wide evictionStrategy was LiveMigrate. For a VMI that is
not live-migratable (a ReadWriteOnce disk, a passthrough GPU, a host
device), virt-api's eviction webhook answers every eviction with 429,
so a node drain retries forever and never completes.

LiveMigrateIfPossible keeps live migration for every VMI that can
migrate and lets the eviction through for the rest. The VM is shut
down and started again according to its runStrategy. A VM whose disk
is bound to the drained node stays down until the node returns.
KubeVirt writes the cluster strategy into a VMI's spec when the VMI
is created, so a VMI started before the upgrade keeps LiveMigrate and
still blocks the drain until it is restarted once.

A drain and a launcher update after a KubeVirt upgrade go through the
same eviction, so the workload updater's Evict fallback shuts down
non-migratable VMs started under the new strategy on every later
KubeVirt upgrade, where it used to get the same 429. Evict stays
anyway: for kubernetes-nodes workers, which use evictionStrategy
External, that eviction marks the VMI for evacuation, CAPK drains the
tenant node and deletes the VMI, and the VM starts it again on the new
launcher. The test pins the method list so the fallback is not dropped
to avoid the shutdowns.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 17:36:56 +03:00
Aleksei Sviridkin 9c97a7a7ac fix(cdi): validate kubevirt-cdi values against a schema
Without a schema, a cloneStrategyOverride outside the CDI enum, such as
csi_clone, renders a CDI object the API server refuses on apply, so the
mistake shows up as a failed HelmRelease rather than at the value.

The field is constrained by a pattern, not by the CRD's enum. The empty
string is the chart's way to omit the field and let CDI pick a strategy
per StorageProfile, the CRD enum has no member for it, and
cozyvalues-gen cannot emit an empty enum value. The key has a chart
default, so Helm drops a null for it before validating, and a null
keeps rendering as an omitted field.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 17:06:56 +03:00
Aleksei Sviridkin 0c358addfb fix(cdi): drop the ExpandDisks gate from the CDI object
ExpandDisks is a KubeVirt feature gate. CDI declares no gate by that
name and matches gate names exactly, so the entry was accepted and
ignored. Disk expansion is a KubeVirt feature, GA there at the release
the platform ships, so listing the gate on the CDI object points a
reader tracing where expansion comes from at the wrong component.

The feature-gate list is pinned whole in its own case, so a gate added
beside the one upstream ships turns the suite red.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 16:58:56 +03:00
Aleksei Sviridkin 2b5770cb4a fix(vm-instance)!: refuse resources that cannot size the VM (#4405)
## What this PR does

A vm-instance could ask for half a core and get one, with no error
anywhere. `resources.cpu` and `resources.sockets` are typed as
quantities, but they end up in `domain.cpu.cores` and
`domain.cpu.sockets`, which KubeVirt declares as unsigned integers. The
chart cast them with `int64`. That turns `500m` or `1.5` into `0`
without an error, and KubeVirt boots a zero core count as one core.

The chart now reads these two fields the way an integer field reads
them. It decodes the raw value as YAML and stops the render unless the
result is a positive whole number that fits in uint32. So `2`, `"2.0"`,
`"+2"` and `"1e1"` render as the integer they decode to. `500m`,
`2000m`, `1k`, `"1.5"`, `0`, `"0"`, `"-2"`, `"010"` and `"4294967296"`
stop the render with an error that names the field. A leading zero is
refused because YAML reads `010` as octal 8. The check looks at
presence, not truthiness, so an unquoted `0` is refused instead of being
read as unset. It is the same rule #4386 uses for kubernetes-nodes.

Memory gets a presence check too, which covers the case from the issue
thread. A quoted `"0"` was truthy and rendered `requests.memory: "0"`,
while an unquoted `0` was falsy and got dropped. Both spellings of zero,
and any negative amount, now stop the render. Any other value passes
through unchanged, because KubeVirt takes memory as a quantity. That
includes an amount below one byte, such as `500m` or `1e-3`, which
renders as it does on `main` today; this change does not refuse it.

The check lives in `virtual-machine.domainResources`. The VM spec, the
instance type decision and the update hook's patch all take their values
from that helper. The tests that pinned `cores: 0` as the old behaviour
now expect the refusal, and the helper comments about the truncation are
trimmed.

This is a breaking change. A release that carries one of these values
fails its next upgrade until the value is fixed: a cpu or sockets that
is not a positive whole number within uint32 or has a leading zero, and
a zero or negative memory. This includes a cpu or sockets set on its own
next to an `instanceType`, which used to be ignored whatever it held.

Some values change meaning instead of failing. A value like `"1e1"` or
`"2."` was truncated to 0 by the `int64` cast, so the VM booted with one
core. Now it gets the count the value spells, so that VM changes size on
upgrade.

I refuse these values instead of rounding them, because silent coercion
is how `500m` became one core in the first place. The issue asks for a
deprecation path, and I did not add a migration that rewrites stored
values. It would have to guess what `500m` meant, and both options (one
core, or dropping the field) hide the same mistake again. The
HelmRelease upgrade retries without uninstalling, so the running VM
stays up until someone fixes the value. #4386 makes the same trade-off.

The check runs at render time, not at admission. The schema still
accepts `500m`, so the API takes the object and the error shows up in
the HelmRelease status. Narrowing the schema to integers, as the issue
also proposes, would move the error to admission, but it needs a
migration of stored values. I left it for a separate change.

The field descriptions now say which values are accepted, and `make
generate` carried them into the README, the schema, the Go types and the
ApplicationDefinition.

The new `tests/resources_values_test.yaml` covers the accepted and
refused spellings for cpu, sockets and memory.
`tests/instancetype_resources_test.yaml` gets a case for a lone cpu next
to `cpuModel`. The whole vm-instance suite passes. Every refusal case in
the new file, and the `"1e1"` case, fails against the current `main`
template.

Closes #4104

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map against the diff. No field is added, removed or
renamed, and no default or enum changes. Only three field descriptions
change, and the website reference page is regenerated from the README at
release time.

### Release note

```release-note
fix(vm-instance)!: resources.cpu and resources.sockets must now be positive whole numbers, and resources.memory a positive quantity. A cpu or sockets that is not a plain whole number within uint32 (for example 500m, 1k or 1.5), or a zero or negative memory, now fails at render time with an error naming the field, instead of silently becoming a one-core VM or a zero memory request. A release carrying such a value fails its next upgrade until the value is fixed. A value such as 1e1 or 2., which used to be truncated to one core, now sizes the VM with the count it spells.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* VM sizing now accepts only positive whole-number CPU and socket values
within the supported range. Fractional, millicore, zero, negative, and
out-of-range values are rejected during rendering.
* Memory values must be positive quantities. A CPU or socket value
specified alone does not size the VM and is ignored; incomplete sizing
is rejected where required.

* **Documentation**
* Updated VM instance resource guidance to clarify CPU and socket
requirements, memory validation, and how socket values define vCPU
topology.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 17:58:36 +04:00
Aleksei Sviridkin 99de79ff52 fix(cdi): pin the CDI release kubevirt-cdi is reconciled against and stop update overwriting it (#4107)
## What this PR does

`packages/system/kubevirt-cdi` resolved its CDI version at run time from
GitHub's `releases/latest` redirect and wrote the result straight over
`templates/cdi-cr.yaml`. That file is not vendored. Upstream ships a
bare CDI object; ours adds Helm templating for `cloneStrategyOverride`,
`uploadProxyURL` and the worker-pod `podResourceRequirements`, the
`ExpandDisks` feature gate, a `CriticalAddonsOnly` toleration, and the
two RBAC documents that let a tenant clone from the golden-image
namespace. I diffed the two at v1.66.1 to write that list, so it is
complete as of that release rather than only hedged: upstream ships 283
bytes with one feature gate, two nodeSelectors and an `imagePullPolicy`,
and we are a strict superset of it. Every entry arrived by hand, so
`wget -O templates/cdi-cr.yaml` was not refreshing a vendored file, it
was overwriting a first-party one. Same loss as the `rm -rf templates`
that #4097 removes, arriving through the fetch instead of the wipe.

`update` now pins the release the object is reconciled against, and
fetches it only to confirm the bytes still hash to a digest I measured.
Nothing is written under `templates/`. At the next CDI bump whose asset
differs, that check fails, which is the point: upstream moving its own
defaults is when the local copy needs a human to reconcile it, not an
overwrite. The download is kept on a mismatch, since reconciling one
starts by reading it.

`RELEASE` is `v1.66.1`, which is the release `kubevirt-cdi-operator`
vendors, so the pin and the deployed version agree. What the platform
deploys is decided by that package, and it can move to another release
without this file changing, so this line is pointed at it by hand.
`update` then checks the two agree: it compares `RELEASE` with the
`cdi-operator` image tag that package vendors and fails naming both
versions when they differ, because otherwise its closing line would
report a release the platform may no longer run. That is a check and not
a derivation, so `RELEASE` stays a pin a human moves.

Upstream publishes no checksum for the asset, so the pin comes from a
download rather than from a published file:
`a497f90de608c1df26f9ee4095289373f17132d74a2042ca03e00c17c964f8a7`. It
catches an asset that moved under a tag that should be immutable. It is
not authenticity verification, and the comment beside it says so.

The digest command is chosen when `update` expands it, `sha256sum` when
it is on PATH and `shasum -a 256` otherwise. Which one ran here is worth
stating, because the usual claim about macOS is out of date: this
machine picked `sha256sum`, and that binary is `/sbin/sha256sum`, a
native Darwin one rather than GNU coreutils. It reports `sha256sum
(Darwin) 1.0` and prints the same `<digest> <path>` format.

The package already had a `test:` target, from `0418577f5`, so what it
gains here is the `.PHONY` line that commit did not add.
`hack/package.mk` declares only the nine targets it defines itself, so
without it a file or directory named `test` or `update` beside the
Makefile makes make consider that target up to date and skip the recipe
in silence. The new suite pins the `ExpandDisks` gate, the toleration,
`podResourceRequirements`, the `uploadProxyURL` override and the clone
RBAC, and renders the upload-proxy Ingress and TLSRoute, each under the
`gateway-enabled` value that selects it. The clone-strategy override is
pinned by `tests/clone_strategy_test.yaml`, which is already on `main`.

One consequence of the pin move is a claim rather than a number, so it
gets its own line. Upstream carried `WebhookPvcRendering` from v1.63.0
through v1.65.0 and dropped it by v1.66.0; the local object never
carried it, so the divergence ran both ways across those three releases
and one way since. The comment says to re-derive the delta at each bump
instead of trusting it, because that was a fact about a span of releases
and not about CDI.

`packages/system/kubevirt-cdi/Makefile` carried two identical `test:`
targets, one from `0418577f5` and one appended by a later merge, so make
printed `overriding commands for target 'test'` on every invocation in
the package. The second sat inside the span this recipe rewrites, so it
goes with the rewrite rather than being re-authored into it. The comment
kept above it is the reason a reader should not add another. Closes
#4157.

### Scope

An earlier revision also pinned and tested `kubevirt-cdi-operator`.
#3920 has since landed the two real bugs in that recipe, the patch strip
level and `patches/tolerations.diff`, so that half is dropped here
rather than merged twice. Pinning and testing the operator package is
#4190.

Named improvements left alone, so they read as choices.

- The two upload-proxy tests each render their route under the
`gateway-enabled` value that selects it and neither asserts the other is
absent; the templates already settle that with `ne "true"` and `eq
"true"` on the same variable, and `tests/standalone_render_test.yaml`
already asserts each route absent under the values that select the
other. So the package does cover it; these two cases just do not, and a
third copy would not add anything.
- The CDI object carries no version string for a rendered-output
assertion, so `RELEASE` is checked by `make update` rather than by `make
test`.
- Neither route case pins the backend service port, 443 in both
templates; that is outside what the suite claims.
- The template-loss guard still counts `cdi-cr.yaml` as named, because
the download line in `update` mentions the stem, so re-adding a wipe
here would have it report two files instead of three; the guard's header
already says a mention is not proof of a write, and the case needs a
wipe this recipe no longer has.
- The suite pins the local additions by design, and most of the
upstream-derived half is pinned by nothing: `imagePullPolicy` and the
two `nodeSelector` blocks have no assertion anywhere under `tests/`.
`HonorWaitForFirstConsumer` is the exception and only as a side effect:
`featureGates` is pinned whole, so that gate rides along with
`ExpandDisks`. The rest are maintained by the same hand reconciliation
the pin forces whenever the upstream asset changes, so a carry-over that
drops one of them is green. Three asserts would close what is left.
- One more, remote enough to name rather than code around: if the digest
tool is missing entirely, `$$actual` comes out empty, the comparison
against `CR_SHA256` fails, and the error text diagnoses a moved asset
when the real cause is a machine with neither `sha256sum` nor `shasum`.
Busybox ships the first and macOS ships the second, so nobody has hit
it. Closing it means an executable line, which would then owe mutation
coverage of its own, and that is disproportionate to a case with no
known instance. The mechanism is written here so the next reader does
not have to re-derive it from an error message that points the wrong
way.
- Separately, the per-package `.PHONY` gap this Makefile's own line
argues against is not specific to this package: most package Makefiles
do not declare their own `test:` or `update:` phony, and #4214 tracks
that.
- A key that does not exist yet is not held either: every item in the
delta is pinned whole, but a NEW key appearing at `spec` or
`spec.config` level passes unnoticed, and such a key can change
behaviour rather than sit inert. I confirmed it on a copy extracted from
this commit, appending `spec.config.filesystemOverhead.global: "0.99"`
and getting a green suite. That is the shape to recognise at the next
bump: not a changed value, which is caught, but a new sibling key beside
the ones we pin.
- And `spec.config.podResourceRequirements` is now pinned whole here and
by four leaves in `tests/importer-resources_test.yaml`: the whole-map
form is the stronger one and the leaves are a subset of it, so changing
the default takes two edits and the second failure reads as a surprise
until you know why both exist.
- If the operator's image reference ever gains a digest
(`:v1.66.1@sha256:...`), `${deployed##*:}` takes the digest's hex
instead of the tag, and `update` fails with a loud false mismatch rather
than passing silently.
- Deriving `RELEASE` with `$(shell ...)` from the sibling manifest was
suggested and declined on merits: a derived value is not a pin, the
point is that a human chooses which CDI release this object is
reconciled against, and computing it from whatever the neighbour last
vendored hides that choice from the diff.

The operator comparison lives in `update` and not in `test:`, even
though `kubevirt-operator` ties its own manifest to the tree from
`test:`. A mismatch calls for a hand reconciliation, and `update` is the
recipe that asks for one; it is also where the success line that would
otherwise name the wrong release is printed. I checked it on a copy
extracted from this commit. With the operator's image tag moved to
v1.67.0, `make update` exits 2 and names both versions. With the tags
equal it passes. With the check's `exit 1` removed, the moved tag passes
again, so the red comes from the check.

Worth knowing when reading the mismatch path: `kubevirt-operator`
deletes its download on a hash mismatch and this package keeps it. That
is deliberate here, since reconciling a moved asset starts by reading
it, but the two packages do read oppositely and neither file says so.

### Base and ordering

This stacks on #4097 and should land after it. Both edit
`packages/system/kubevirt-cdi/Makefile`, and this change builds directly
on that one's edit to the same recipe, so it is based on the head of
`fix/cdi-update-keeps-local-templates` rather than on `main`. GitHub
computes mergeability against the PR's base branch, which here is that
stack branch and not `main`. Both branches are rebased onto current
`main`, which has not touched these paths since the previous merge-base.


### Screenshots

No UI changes.

### Downstream repositories

Walked the trigger map against the diff. Two entries come close and
neither fires.

The website triggers on developer tooling and package Makefiles, but
`content/en/docs/next/development.md` documents `make update` only as
"Update Helm chart and versions from the upstream source", which still
holds, and its worked example is cilium, which this does not touch. Its
`make test` line refers to the e2e sandbox in `packages/core/testing`,
not per-package unit tests.

`ccp` triggers on `hack/` layout and on what a make target does, and
nothing under `hack/` is touched. Its `package-bump` skill already
assumes a package's `update` recipe hardcodes the upstream version
instead of resolving it at run time, and names cilium, kafka-operator,
opensearch-operator and postgres-operator as examples. `kubevirt-cdi`
was one of those counterexamples and now is not. Pinning
`kubevirt-cdi-operator` the same way is #4190 and out of scope here. The
skill handles vendored Helm charts, in-repo image builds and version
enums, and a raw manifest download is none of those. Everything else in
the map needs an app, a schema, an installer value, a variant or a
release asset.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): `make update` in kubevirt-cdi no longer overwrites the hand-maintained CDI object. It pins the CDI release that object is reconciled against instead of resolving GitHub's releases/latest at run time, and verifies the fetched asset against a pinned sha256 without writing under templates/.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Reliability**
* Updates now use verified, pinned CDI releases to prevent unintended
manifest changes.
* Failed or checksum-mismatched downloads no longer overwrite existing
configuration.

* **Bug Fixes**
  * Preserved local CDI customizations during release updates.
* Added validation for CDI resources, access controls, upload-proxy
routing, and TLS configuration.

* **Tests**
* Added automated checks for incomplete template refreshes and generated
Kubernetes resources.
* Added chart tests covering CDI behavior, clone permissions, and
upload-proxy networking.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-23 17:53:23 +04:00
Myasnikov Daniil bc8c54d6e6 feat(kubernetes): boot tenant worker disks from a shared golden Talos image
Every tenant worker VM imports the same Talos raw image from the Image Factory
over HTTP, once per worker: a 20-worker cluster fetches the identical ~4.15 GiB
artifact 20 times, and each fetch is a live dependency on the Factory at exactly
the moment a node joins.

packages/system/kubernetes-worker-image is a new opt-in catalog that holds one
golden DataVolume per (schematicID, version) in cozy-public, named
talos-worker-<schematicID>-<version>. A worker pool opts in with osImage.builtin
on its own KubernetesNodes resource, and its disk-system DataVolume then carries
source.pvc pointing at that golden, so CDI performs a storage-layer CSI clone
instead of an HTTP import -- one import per flavor, shared by every worker of
every tenant. Confirmed on a LINSTOR-backed cluster: the worker disk is a CSI
clone plus a resize, with no host-assisted upload pod.

Both halves are off unless asked for. The catalog is registered through the
platform's optional-package helper and disabled by default, so a cluster that
does not enable it creates no golden and keeps importing over HTTP. The disk
source is derived entirely from osImage, which is absent by default, so a pool
that does not set it renders byte-identical to the pre-feature chart and
adopting the feature never rolls existing workers. Editing osImage on an
existing pool re-images that pool, as changing any other image reference does.

What the render refuses, rather than degrading into a silent fallback:

- A pool whose golden has no catalog entry, or whose golden terminally failed to
  import. A golden that is merely mid-import is not an error; CDI holds the
  worker disk until the source populates.
- A pool whose StorageClass differs from its golden's. CDI cannot CSI-clone
  across classes and falls back to a host-assisted copy over the pod network,
  which is the transfer this feature exists to remove. An empty class on either
  side means the cluster default and is resolved before the comparison, and a
  pool that has already cloned reads its class from its own bound PVC rather
  than from today's default -- the golden's side is frozen the same way, and an
  admin moving storageclass.kubernetes.io/is-default-class must not fail the
  release of a pool whose disks are all on the right class. That resolved class
  is then written onto the worker disk, so the disk binds where the guard
  checked instead of on whatever the default has become by then -- reading one
  class and binding another is how a pool at an empty storageClass passed the
  comparison and still took the host-assisted copy. The HTTP path leaves an
  empty class to the cluster as before, which is what keeps its render, and the
  KubevirtMachineTemplate hash named after it, byte-identical.
- A diskSize below the golden's storage size, which a clone target cannot be.
- A schematicID that is not a plain lowercase alphanumeric identifier, or a
  version that is not a vMAJOR.MINOR.PATCH release. Both are interpolated into
  a DataVolume name and an image URL, and values.schema.json types them as bare
  strings, so a malformed one otherwise reaches the render. The catalog applies
  the same patterns, since both sides build the same name from the same fields.
- An images[] entry edited after its golden exists. A DataVolume spec cannot be
  patched in place, so the message names the entry and the supported move -- a
  new (schematicID, version) -- instead of leaving the API server to reject the
  patch on the next upgrade.

Goldens carry helm.sh/resource-policy: keep, so dropping an entry stops
managing a golden rather than deleting it: a missing golden aborts the whole
Kubernetes HelmRelease of every tenant pinned to it, not just the disk. Each
cloned worker disk annotates the golden it came from and the pool that owns it,
so "does anything still clone this?" is answerable before a manual delete
instead of after.

A builtin pool needs a StorageClass whose volumes are reachable from any node,
in practice a replicated/DRBD one. The golden in cozy-public has no consumer and
binds as soon as it imports, wherever CDI's importer landed, while a worker disk
binds where its VM is scheduled, and nothing here arranges for those to be the
same node. The platform ships cloneStrategyOverride: csi-clone, which rules out
the host-assisted strategy that would cross nodes. What a node-pinned class does
in that situation is not measured, so values.yaml states the requirement and not
a symptom.

ComputePlane pools forward only the per-pool shape fields, and osImage is
excluded there by name: the worker OS image, its Talos installer and its Image
Factory are operator decisions about the platform's own images.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-23 15:35:44 +03:00