admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-03 10:37:21 +09:00
Code Issues Packages Projects Releases Wiki Activity
6,188 Commits 540 Branches 294 Tags
feat/externalip-api
Commit Graph
388 Commits
Author SHA1 Message Date
Aleksei Sviridkin d685fb30ec fix(gateway)!: require tenant routes to declare their hostnames (#3475) 
## What this PR does

The tenant route hostname policy skipped the shape it most needed to
check. Its CEL opened with `!has(object.spec.hostnames) ||`, so an
HTTPRoute or TLSRoute declaring no hostnames satisfied the first operand
and got admitted with nothing compared against the tenant apex. An empty
list passed too: `has()` is true on `[]`, and `all()` over an empty list
is vacuously true.

Such a route is not inert. Under Gateway API it inherits the hostnames
of every listener it attaches to, and `parentRefs` can name a Gateway in
another namespace, so it covers whatever that listener covers. Reported
in #3467.

The predicate now requires the field to be present and non-empty before
validating it. Both policies rendered from that template share one CEL
string, so HTTPRoute and TLSRoute get the change together, and both
match `CREATE` and `UPDATE`, so an existing route that drops its
hostnames is denied exactly as a new one is.

The issue is titled after TLSRoute, but the reachable shape is mostly
HTTPRoute. TLSRoute allows a hostname-less route only on the deprecated
`v1alpha2`; at `v1` and `v1alpha3` the bundle this repo vendors marks
`spec.hostnames` required and rejects an empty list on `minItems: 1`,
while `v1alpha2` carries neither constraint. HTTPRoute `v1` carries
neither either, so it accepts both shapes. All four are checkable in
`packages/system/gateway-api-crds/templates/crds-experimental.yaml`.

Checking that turned up a stale claim in the template header, which said
TLSRoute had no `v1` in Gateway API 1.4 to 1.5 and used it to justify
the rule naming only `v1alpha2`. Gateway API 1.5.1, which this repo
vendors, serves TLSRoute at `v1` as the storage version. The rule still
covers every served version because `matchConstraints.matchPolicy`
defaults to `Equivalent`, and that default is now written down, because
setting `matchPolicy: Exact` reads like a tightening and would silently
stop the policy seeing `v1` and `v1alpha3` TLSRoutes. I corrected the
text and left the coverage alone: the wrong sentence predates this
branch, but my change is what makes the file contradict itself, so the
text is mine to fix while widening version coverage is a separate job.

A `ValidatingAdmissionPolicy` has no cross-object lookup, so it cannot
resolve the parent Gateway to see which hostnames a route would inherit.
Requiring the route to name its own is what keeps every admitted
hostname comparable against the apex. That has a cost: a route relying
on listener inheritance now has to spell its hostnames out, and a tenant
that hand-writes one gets a denial where it used to get an admission.

### The controller change belongs in the same PR

`renderHTTPRedirect` built the http-to-https redirect HTTPRoute with no
hostnames on purpose, to catch every host on port 80. It lands in a
`tenant-*` namespace, so the stricter policy denies it,
`reconcileHTTPToHTTPSRedirect` returns that error, and since it runs
before the status update the TenantGateway never reaches Ready. That
redirect is what keeps app routes attaching by hostname from serving
plaintext on port 80. Neither half of this change is safe alone.

The redirect now names the tenant apex and its wildcard. Both entries
are needed: a wildcard hostname is a suffix match over any number of
labels, so `*.<apex>` covers subdomains at every depth but not the bare
`<apex>`. `renderWildcardCertificate` already builds the same pair.

Coverage narrows in one direction. A request on port 80 with a Host
outside the tenant apex now falls through to a 404 instead of a 301.
That narrowing costs no covered host: the port-80 listener admits routes
only from the Gateway's own namespace and cert-manager's, and the
tightened policy requires every route in the tenant namespace to declare
its hostnames inside the apex, so any host published from this namespace
already falls under the apex or its wildcard.

An inheriting child tenant does not widen that set either. Its hostnames
would reach port 443 through a listener on this same Gateway, and
`cozystack-gateway-hostname-policy` compares every listener against the
host label of the Gateway's own namespace, so a child apex outside that
label never gets its listener: the Gateway write carrying it is refused
at admission whole, in `reconcileGateway` and before the redirect is
rendered. A child apex under this one passes that check and the wildcard
already covers it. That holds at merge base as well, so it is not
something this branch changes.

Deriving hostnames from `spec.apex` makes that field load-bearing for
admission, so it gains `MinLength=1`. An empty apex used to reconcile to
Ready, since the redirect declared no hostnames at all; it would now
render `""` and `"*."` and be rejected on the HTTPRoute hostname schema,
surfacing as a validation error naming the route rather than the field.
The renderer checks it too, because the CRD and the controller binary
roll out separately and an older CRD in the cluster will not carry the
bound.

That schema bound is a breaking change to an existing field and the API
review gate flags it as one, so here is what it actually invalidates.
The chart that renders these CRs already fails on an empty
`_namespace.host`, and in DNS-01 or existingSecret mode an empty apex
already broke earlier, in `renderGateway`, on a listener with `hostname:
"*."`. Nothing that works today becomes invalid. What the bound rejects
is a TenantGateway written by hand, past the chart, with an empty apex,
and I would rather that be refused at admission naming the field than at
reconcile time in controller logs, which is why the bound sits in the
CRD as well as the renderer.

Naming those hostnames also meant excluding the redirect from
`collectHostnameClaims`. A route that attaches to the Gateway and
declares hostnames otherwise reads as an app publishing them, and each
claimed hostname becomes an HTTPS listener named after its first label.
The first label of `*.<apex>` is `*`, which the Gateway CRD's
listener-name pattern rejects, so the Gateway write fails and the
reconcile stops before anything downstream. The exclusion matches on
namespace, name and controller ownership rather than on the labels the
renderer stamps: those labels are writable by anyone who can create a
route, and the claim set also drives conflict resolution, so a
label-keyed exception would let any route opt out of being marked
`Accepted=False` when it loses a hostname race. Name and ownership are
each required and each is pinned by its own test, since ownership alone
would exclude any other controller-owned route and name alone would
extend the exception to a route planted under that name by someone else.

### Radius

Only namespaces starting with `tenant-` are matched. The three
first-party TLSRoutes the platform ships, `kubernetes-api` in `default`
plus `cdi-uploadproxy` and `vm-exportproxy`, set hostnames explicitly
and none sits in a `tenant-*` namespace, so nothing changes for them.

I went through every producer of routes in the tree, in three places
rather than one. No Helm template emits an absent or empty hostname list
in any configuration a tenant can reach: the vendored openbao, harbor,
opencost and flux-operator route templates can render one, but each is
behind a flag that defaults off with no values path to turn it on, and
openbao would fail CRD validation first because its default route
apiVersion requires hostnames. Exactly one Go site builds a route, the
redirect fixed here. And one e2e fixture did rely on the old behaviour:
the Gateway suite's parentRef probe declared no hostnames and is applied
into `tenant-test`, which the policy matches, so it now reads the apex
from the namespace label and names a hostname under it, the same way the
neighbouring cases in that suite already did.

cert-manager also writes routes into tenant namespaces, since it creates
the Challenge in the Certificate's own namespace and the HTTP-01 solver
route beside it. That route declares the hostname it is validating, so
the policy admits it.

### Related issues

Two open issues have a text half that this PR happens to settle, and
neither of them closes here. #3788 reports that the port-80 allow-list
contains the namespace its own comment says it excludes; the comment and
the README paragraph now describe what that filter actually does, while
the behavioural exit it names stays open. #3789 reports that the
TLSRoute rule names only the deprecated API version; that premise is now
written down in the template header, including what has to change before
a bundle drops `v1alpha2`, but the rule still names `v1alpha2` alone.
#3787 covers the wildcard-hostname listener name that the apiserver
rejects, which is what makes the claim-set exclusion here load-bearing
rather than tidy.

### Screenshots

N/A (no UI changes).

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. No package is added, renamed or removed; no
`values.yaml` key, `values.schema.json` field, version enum or default
moves; no `kind` or `plural` changes; no namespace or release-asset name
changes; nothing under `hack/` moves, is renamed, or changes what a make
target does.

One row deserves a sentence rather than a blanket denial, because the
diff does change a CRD schema: `spec.apex` on
`gateway.cozystack.io/v1alpha1 TenantGateway` gains a `minLength` bound,
which lands in the generated `gateway.cozystack.io_tenantgateways.yaml`.
The provider hand-types `Package` under `cozystack.io` and
`Plan`/`RestoreJob` under `backups.cozystack.io`; `gateway.cozystack.io`
is not among them, and a tightening on an existing field adds no new
field for it to offer typed access to. The
`packages/extra/gateway/README.md` edit sits in the `## Security model`
section, below the `## Parameters` block that `cozyvalues-gen`
regenerates, so it survives `make generate`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(gateway)!: the tenant route hostname policy now requires an HTTPRoute or TLSRoute in a `tenant-*` namespace to declare a non-empty `spec.hostnames`. A route that omitted the field, or set it to an empty list, was previously admitted without any hostname being checked against the tenant apex, even though such a route inherits the hostnames of the listener it attaches to rather than matching nothing. A route in a `tenant-*` namespace relying on that inheritance is now rejected and has to name its hostnames. The controller-owned http-to-https redirect route names the tenant apex and its wildcard accordingly, so a request reaching port 80 with a Host outside the tenant apex now gets a 404 instead of a 301. `TenantGateway.spec.apex` additionally gains a `minLength: 1` bound, because the redirect derives its hostnames from that field; a TenantGateway written past the chart with an empty apex is now rejected at admission instead of failing later in the controller.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added edge-terminated gateway support with plain-HTTP listeners and
TLS passthrough handling.
- HTTP-to-HTTPS redirects now explicitly cover the tenant apex and
subdomains.
  - Gateway configurations require a non-empty apex hostname.
  - HTTPRoute, TLSRoute, and GRPCRoute resources must declare hostnames.

- **Bug Fixes**
  - Invalid hostname-less routes are now rejected consistently.
- Controller-managed redirects no longer trigger unnecessary hostname
claims, listeners, or certificates.
- Switching certificate modes now cleans up obsolete ACME resources and
route status.

- **Documentation**
- Updated security and gateway guidance for hostname validation, edge
termination, route compatibility, and certificate handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-07 18:49:04 +04:00
myasnikovdaniil 94a202bc5a test(e2e): run the merge-gating lanes on Talos containers (#4020) 
Replaces QEMU substrate with Talos containers on both lanes that gate a
merge. srv1-srv3 run as containers, so tenant worker sits at L2 instead
of L3.

Measured in CI on commits where both substrates ran the same tree:

| | qemu | container |
|---|---|---|
| `kubernetes-latest` / `-previous` | 1143-2040s, red on every
vgif-absent runner | 466-809s, green in both strata |
| whole job | | 24-49 min faster |

Almost all of it is in Chainsaw. Install takes about half an hour either
way.

Node-join soft-red gate goes with it. It was a prosthetic for nested
virt: a worker that registers in minutes elsewhere could miss any
deadline this test can afford, and #3513 established the discriminator
is host `kvm_amd` `vgif`, not anything about the product. Remove the
nesting level and there is nothing left to tolerate, so
`hack/e2e-node-join-soft-red.sh`, the marker and the `soft_red` outputs
are gone from all four workflows.

### What moves off the per-PR path

DRBD, and with it `replicated` StorageClass, its `Immediate` binding
mode, tenant StorageClass propagation (applies only to
remotely-accessible classes), and cozystack talos node image with its
extensions. None of it is lost - `nightly.yaml` and `e2e-tag.yaml` still
run the suite on QEMU, so all four keep nightly and release-candidate
coverage. Live migration is not on that list because nothing tests it
today anyway.

One more belongs on the list and was missing from it: the nocloud disk
build. `build-talos` no longer runs `make -C packages/core/talos
talos-nocloud`, so `images/talos/profiles/nocloud.yaml` is compiled by
`nightly.yaml` and by the tag build's `make assets` rather than by
anything a PR runs, and `promote-rc.yaml` still hard-fails a promotion
whose rc release has no `nocloud-amd64.raw.xz`. Nothing in that chain
breaks, because the tag build supplies the asset as before. What a PR no
longer catches is narrow: it still compiles `installer.yaml` through the
same imager, and the two profiles are generated by the same script and
differ only in `platform`, `kind`, `imageOptions` and `outFormat`.

### Product fixes

Four, all found by measurement while building the lane, each on its own
commit:

The linstor `drbd.enabled` gate. Without it drbd-logger sidecar exits,
satellite is never Ready, piraeus registers zero nodes and every PVC
hangs Pending behind a cluster that looks healthy.

linstor `--strict-topology`. Without it external-provisioner passes
every topology segment as `requisite`, so a node-pinned volume can be
provisioned away from its pod and the pod is then unschedulable forever
with nothing erroring anywhere.

kubevirt-cdi importer resources are configurable now. Default restates
CDI's own values exactly so production behaviour does not move, and the
lane raises the ceiling through its own Package.

vm-disk binds standalone disks immediately. A VMDisk on `local` silently
never populated, while the same disk on `replicated` did.

### Where the run stands

The lane has gone green twice, most recently on `cc70cef7b` where `E2E
(in-tree)` took 2h3m against its 215m cap with 45 Chainsaw tests
passing. The measurements quoted elsewhere in this description come from
the first of those runs, on a commit that has since been rebased away:
the two tenant-Kubernetes suites came in at 569.72s
(`kubernetes-latest`) and 470.19s (`kubernetes-previous`) against a 67m
operation. Both runs answer the four earlier reds, all of which were the
lua-protobuf segfault from #4012 once `vminstance`/`vmdisk` were fixed
on the branch.

ghcr.io mirror removal is carried here too and is the same change as
#4007, whichever lands first makes the other a no-op.

### What review changed

`drbd.enabled=false` now refuses to render together with
`talos.enabled=false`. It removed the logger sidecar only, while the two
DRBD initContainers piraeus contributes unconditionally are deleted by
the Talos configuration alone, so that combination produced a satellite
that can never reach Ready. `isp-full-generic` sets
`talos.enabled=false`, so it was reachable and not theoretical.

vm-disk keeps unconditional immediate binding, a standalone disk has no
consumer to wait for. What it costs on a node-pinned class is in the
chart README now, and the annotation is pinned per source type and on
the existing-DataVolume path where `helm upgrade` retrofits it.

The rest is the harness holding its own promises: HelmRelease gate reads
are bounded and its minimum-count half is pinned by a test that fails
without it, two Chainsaw ops sit above the deadlines they contain, the
fallback-StorageClass render restores `replicated` when it fails,
container lane compose project and pool files are per-checkout, and
three comments that said the opposite of the code now say what is true
(QEMU is still wired into nightly and e2e-tag, the lane replaces two
Packages and not one, the argument-count assertion cannot see upstream
drift).

### Release note

```release-note
fix(linstor): CSI provisioning is constrained to the node the scheduler chose (`--strict-topology`), and the DRBD sidecar can be switched off with `drbd.enabled` on substrates whose kernel cannot run DRBD. On an existing cluster the new default re-templates `spec.csiController.podTemplate`, so piraeus rolls `linstor-csi-controller` once during the upgrade. `drbd.enabled=false` requires `talos.enabled=true` and the chart now refuses the other combination at render time, because only the Talos satellite configuration removes the DRBD initContainers.
fix(kubevirt-cdi): CDI worker pod resources are configurable through `importerResources`, which reaches the importer, the uploader and the host-assisted cloner. The default restates CDI's own built-in values exactly, so an install that sets nothing behaves as before.
fix(vm-disk): every disk requests immediate binding, not only `upload` sources, so a standalone disk on a WaitForFirstConsumer StorageClass is populated instead of waiting for a consumer that may never arrive. On a node-pinned class such as `local` the volume then binds where CDI's worker pod was scheduled rather than where the VM will run; see the chart README before choosing such a class for VM disks. The default `replicated` is unaffected.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added container-based end-to-end testing with local storage support.
* Added optional backup-access preflight checks for database backup
examples.
* Added configurable CDI worker resources and LINSTOR storage settings.
  * Added configurable immediate storage binding for VM disks.
* Added stronger readiness checks, diagnostics, and parallel test
execution.

* **Bug Fixes**
* Improved cleanup failure reporting, capacity validation, and
missing-image detection.
* Improved FoundationDB health verification and storage placement
behavior.

* **Removed**
  * Removed GHCR mirror support and soft-red node-join tolerance.
  * Removed standalone OIDC render test suites.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-07 19:33:20 +05:00
Myasnikov Daniil c748c5b707 docs(e2e): drop the references the removed suites and carve-out left behind 
Convention 7 still sent readers to the node-join carve-out, while
section 1 of the same file -- rewritten on this branch -- says the
script, the marker and the soft_red output no longer exist. Restated as
what the deadline now is.

hack/e2e-chainsaw/README.md still listed kubernetes-oidc-system and
kubernetes-oidc-customconfig among the suites, and select-install.sh's
suite_to_source() still mapped both. The mapping is harmless today
because the round-trip pin walks existing directories, but select-e2e.sh
says the two functions have to stay in step, so a stale half is a trap
for whoever reads that promise.

sandbox-runner-headroom.bats keeps reading the QEMU sandbox file for
every lane, container lanes included: `prepare-env` also matches
`prepare-env-container`, and the compose file gives its nodes the same
8 CPU / 24576 MiB by design. The figures are therefore right for both
substrates, but only because the two agree -- and that they agree is
pinned in container-lane-capacity_test.bats rather than here. Said so in
the test, so the next reader knows what would have to change if the
container lane is ever sized differently.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
 2026-09-05 14:51:06 +05:00
Myasnikov Daniil 12157a2f93 docs(agents): describe the container lane as the merged standard 
The container lane moves out of "In-flight direction (not yet the merged
standard)" and into the body, because it is now what both merge-gating
lanes run. The node-join section is rewritten the other way round: it
described a deadline that failed the suite without blocking the lane,
and there is no longer any such carve-out to describe.

What the section gains is the part that was missing while the lane was a
spike -- what moves off the per-PR path with the substrate, and where it
still gets covered. DRBD and the replicated StorageClass were already
named; Immediate binding, tenant StorageClass propagation and the
Cozystack Talos node image were not, and all three now say nightly and
release-candidate rather than reading as lost. The host CPU and memory
visibility gap is stated too, since it is what a reader meets first on a
red run and it looks like a product bug.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-09-05 13:19:18 +05:00
Aleksei Sviridkin 6935df8c7f chore(kubevirt): update KubeVirt to v1.9.0 (#4015) 
## What this PR does

KubeVirt operator manifest goes from v1.8.4 to v1.9.0. The 1.9 release
turns every Beta feature gate on by default and adds
`disabledFeatureGates` to the CR for opting out, so the CR side needed a
few decisions.

`Template` is disabled. With it on, virt-operator deploys virt-template:
two Deployments, two APIServices, a webhook and a
ValidatingAdmissionPolicy that nothing here uses. One more aggregated
API server is one more way to break discovery. This one is not a knob.
`- Template` is written into the CR unconditionally and there is no
value that takes it back out, because nothing on the platform uses
VirtualMachineTemplate; if something starts to, the static entry becomes
a value then. Asking for it through `extraFeatureGates` is refused at
render time with an error that says so, rather than producing a CR
KubeVirt would reject.

`ImageVolume` is the host-dependent one: off below Kubernetes 1.35, left
on from 1.35, with a platform value for runtimes that cannot serve image
volumes even there (#3190 has the k3s case). Nothing the platform ships
uses containerDisk. The value is `kubevirt.disabledFeatureGates` on the
platform values, which the iaas bundle forwards into the emitted
`cozystack.kubevirt` Package as
`components.kubevirt.values.disabledFeatureGates`:

Closes #3190.

```yaml
kubevirt:
  disabledFeatureGates:
  - ImageVolume
```

The chart refuses to render a CR that names one gate in both lists.
KubeVirt's `kubevirt-update-validator.kubevirt.io` webhook rejects that
combination and runs with `failurePolicy: Fail`, so on a cluster that
already has the CR the write is refused and the `kubevirt` HelmRelease
stops converging with everything else the CR carries. The create
admitter runs no validation at all (its own comment says the update
validator is where that happens, and it only gets installed after the CR
exists), and `IsEnabled` in virt-config checks `FeatureGates` before
`DisabledFeatureGates`, so on a fresh install the same CR is admitted
and the gate silently stays on. Render-time is the only place left where
an operator gets told.

A name repeated inside one list is not an error: upstream compares the
two lists against each other, and both fields are
`x-kubernetes-list-type: atomic` in the CRD, so that CR is accepted and
the guard stays quiet. The gates the chart pins off can arrive a second
time from a supported values file, since the chart writes them itself
and an operator listing the gates their cluster declines has no reason
to know that, so the range skips them and the CR states each opt-out
once. The guard walks its own copy of the seven gates the chart enables.
That copy exists because the rendered list has to stay a literal list
for `make update` to find its `- VMExport` anchor, and a gate quietly
dropped from the copy would keep rendering under `featureGates` while
leaving the guard. The suite carries one case per gate, so removing any
single name turns exactly one of the seven red.

`HotplugVolumes` stays in the list even though 1.9 deprecates it. A
deprecated gate listed in the CR is still enabled, and while it is
enabled virt-controller skips the declarative hotplug path. Dropping it
would switch on declarative hotplug, but then virt-api rejects the
VMI-level `removevolume` subresource on VM-owned VMIs, and kubevirt-csi
uses that as a fallback during teardown. The CSI driver has to move off
that subresource first. Upstream names no removal release for it: the
`Deprecated` state's own doc comment says "discontinued in the following
release", but `MultiArchitecture` and `DisableMDEVConfiguration` have
been deprecated since 1.8.0 and are still registered at 1.9.0, so that
is a direction rather than a date.

What makes that opt-out list matter is a semantics change: on 1.8
`isFeatureGateEnabled` ended in `return false`, so a Beta gate nobody
listed was off; on 1.9 `IsEnabled` ends in `return fg.State == Beta`.
Sixteen gates are Beta at v1.9.0 and none of them is in this chart's
`featureGates`, so all sixteen arrive on with the bump. Two are switched
back off here, one is switched off below Kubernetes 1.35, and I went
through the rest:

- `Template`: off. Reason above.
- `ExternalNetResourceInjection`: off. With it on, virt-controller stops
calling `multus.NetworkToResource`, so a NAD's
`k8s.v1.cni.cncf.io/resourceName` no longer becomes a resource request
on the launcher pod. That is correct for clusters running a
network-resources-injector; this platform ships none, so a VM on an
SR-IOV NAD would schedule without its device.
- `ImageVolume`: off below Kubernetes 1.35, left alone from 1.35 on.
KubeVirt renders containerDisk images as `VolumeSource.Image` volumes
when the gate is on, and it does not check the host: `dropImageVolumes`
in the apiserver strips every image volume and the mounts referencing it
while Kubernetes' own `ImageVolume` gate is off, which is the default
until 1.35 (`kube_features.go` has it `Default: false` at 1.31 and
1.33). The pod is admitted without the disk and virt-launcher fails
looking for it. The management cluster is supported from 1.33, so on
1.33 and 1.34 the chart keeps the KubeVirt gate off and the
containerDisk sidecar path in use; from 1.35 the gate stands and
`extraFeatureGates` can name it. The apiserver version is a proxy for a
runtime capability, not the capability itself, so a cluster at 1.35 or
newer whose container runtime cannot serve image volumes (the k3s case
in #3190) switches the gate off through `kubevirt.disabledFeatureGates`;
that is what the value is for.
- `Snapshot`: on. `VirtualMachineSnapshot` needed the gate listed on
1.8; snapshot and export were exercised on the test cluster.
- `DeclarativeHotplugVolumes`: on and inert.
`handleDeclarativeVolumeHotplug` returns early while `HotplugVolumes` is
enabled, which this chart still does.
- `GPUsWithDRA`, `HostDevicesWithDRA`: on; each only lets a VMI request
a DRA-provisioned device, and nothing here creates one.
- `WorkloadEncryptionSEV`: on; needs SEV hardware and a VM that asks for
it.
- `PasstBinding`: on; needs a VM using the passt binding.
- `RebootPolicy`: on; adds a field on the VMI domain spec, unset means
the old behaviour.
- `VmiMemoryOverheadReport`: on; fills `status.memory.memoryOverhead`,
reporting only.
- `OptOutRoleAggregation`: on; it enables the `RoleAggregationStrategy`
field, and aggregation stops only for a CR that sets it to `Manual`.
This one does not.
- `KubevirtSeccompProfile`: on; installs the profile, and a workload
uses it only where `seccompConfiguration` says so.
- `NodeRestriction`: on; virt-handler node authorization, needs the
Kubernetes `ServiceAccountTokenPodNodeInfo` gate, GA since 1.32.
- `LibvirtHooksServerAndClient`: on; pre-migration hooks on the target
launcher, and no hooks ship here.
- `PodSecondaryInterfaceNamingUpgrade`: flips on; it adds
`--upgrade-ordinal-ifaces` to virt-launcher and a pre-migration hook
that renames the ordinal secondary interfaces of VMs still carrying
them. The test cluster's migrations exercised that path.

Five of the seven gates the chart enables (`ExpandDisks`,
`LiveMigration`, `AutoResourceLimitsGate`, `GPU`, `VMExport`) reached GA
by 1.9 and are inert: `IsEnabled` returns true for a GA gate whether or
not the CR names it. I left the list as it is; dropping the five is a
follow-up, because it changes what the CR says without changing what the
cluster does and it belongs in its own commit.

QEMU is 10.1.0 on both sides of this jump; the el9 build revision moves
from -10 to -20 and libvirt goes 11.9 to 11.10. A moving build revision
is the thing to check rather than a major:
[kubevirt/kubevirt#16386](https://github.com/kubevirt/kubevirt/issues/16386),
where every existing VM lost live migration, was 1.6.2 to 1.7.0 on
qemu-kvm 9.1.0-19 to 9.1.0-20, one build apart within the same QEMU
version. So I tested it: VMs running on the 1.8.4 launcher live-migrated
onto the 1.9.0 one on a three-node cluster.

`make update` in `kubevirt-operator` used to `rm -rf templates` before
the fetch and took the local PrometheusRule template with it. Now it
only overwrites the file it downloads.

`make update` in `kubevirt` now greps for every line of the new block
before it passes: the range directive, the `disabledFeatureGates:` key,
the three static entries, and the guard header that decides which static
entries render. Every insert in that target is guarded by one directive
it also inserts, so a hand-merge that keeps the directive and drops a
sibling line leaves the guard satisfied and the insert skipped. That
shape covers the directive and nothing else. What makes it worth a grep
per line here is the static entries: they are the only lines those
inserts add that are CR content in their own right rather than a wrapper
around a value, so they render on every install and their absence cannot
surface as a value that stopped taking effect. The CR just goes back to
deploying virt-template and to letting KubeVirt skip NAD resource
injection. The target cannot rebuild a partial block, so it names the
missing line and stops. The payload lines of the other four blocks are
exposed the same way; I checked, and both dropping `- {{ . }}` from the
`extraFeatureGates` range and dropping the `permittedHostDevices:` key
from its `with` render green and lose the operator's values. That is a
wider change than this PR and I left it out.

Two sharp edges I know about and left. `kubevirt.disabledFeatureGates:
[HostDevices]` on a cluster that also has `cozystack.gpu-operator` in
`enabledPackages` makes the bundle forward the same gate into both
lists, and the platform renders fine while the child kubevirt
HelmRelease then stops on the guard: fail-closed, with the gate named,
but diagnosed one level below where it was configured. And the
version-conditional opt-out is evaluated at render time, so a cluster
upgraded from 1.34 to 1.35 keeps `ImageVolume` disabled until something
re-renders the chart; the direction is the safe one, since the gate
stays off and containerDisk keeps the path it had on 1.8.4.

Two things I decided against doing here. The guard's copy of the gate
list is pinned against a name being dropped, not against a name being
added to the rendered list only; the copy exists because `make update`
needs the literal `- VMExport` anchor, so closing the other direction
means changing the self-heal contract. And
`hack/run-kubernetes-block-io_test.bats` still names a
`virt-launcher:v1.8.4` image in a fixture: inert, the test matches on
the container name, and the file is outside this change.

Tested on a three-node Talos v1.13.6 / Kubernetes v1.36.2 cluster with
Flux suspended. The upgrade took about ten minutes. virt-operator had
already deployed virt-template before the CR change landed and removed
it afterwards, CRDs included. Two VMs started on the 1.8.4 launcher, one
DataVolume-backed and one containerDisk-backed, live-migrated to the
1.9.0 launcher. VMDisk plus VMInstance through the apps API boots.
`addvolume` on the VM subresource reaches Ready, `removevolume` on the
VMI subresource is accepted. VirtualMachineSnapshot and
VirtualMachineExport from it reach Ready. Restore from snapshot and PVC
clone failed on that cluster inside the LINSTOR API layer, unrelated to
this change. CDI v1.66.0 from #3920 on top of it: HTTP imports to Block
and Filesystem, blank population and hotplug all work. `helm unittest`
is 26/26 for the kubevirt chart and 142/142 for the platform chart, and
the update idempotency script passes its eleven cases.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. No app added or renamed, no
variant, release asset, `ApplicationDefinition` or tooling contract
touched, and none of the `packages/core/platform/values.yaml` keys the
ansible role sets are in the diff.

One repository is reached. `packages/core/platform/values.yaml` gains a
top-level key, `kubevirt.disabledFeatureGates`, defaulting to the empty
list, and `operations/configuration/platform-package.md` on the website
is a hand-written table of `spec.components.platform.values.*` that
nothing generates. The row is in cozystack/website#678, which lands
after this PR merges.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/678
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(kubevirt): update KubeVirt to v1.9.0. Beta feature gates are on by default upstream now. The platform keeps `Template` off, which would deploy virt-template, and `ExternalNetResourceInjection`, which would hand `resourceName` injection for NetworkAttachmentDefinitions to an injector the platform does not run. `ImageVolume` is disabled on management clusters below Kubernetes 1.35: containerDisk-backed VMs are rendered with native image volumes when it is on, and an apiserver whose own ImageVolume gate is off, the default before 1.35, drops those volumes silently, leaving the VM without its disk. On 1.35 and newer the gate stands, and that is the first release where containerDisk-backed VMs are rendered with image volumes instead of sidecars; the bundled Talos ships Kubernetes 1.36, so this is the default there. A `kubevirt.disabledFeatureGates` list in the platform values opts out of any other gate a cluster cannot carry.
```
 2026-09-05 00:30:05 +04:00
Aleksei Sviridkin 3a369fed99 feat(redis): add opt-in TLS for Redis and Sentinel (#2729) 
## What this PR does

`tls.enabled` puts Redis and Sentinel on a TLS-only listener with a
per-release cert-manager chain, and hands the tenant the CA certificate
as `redis-<name>.tenant-ca`. Off unless set.

redis-operator v3.3.5 has no TLS support. The work is ported upstream in
freshworks-oss/redis-operator#85, which is open and not in a release, so
this carries it as three patches over the v3.3.5 tarball.
`images/redis-operator/patches/tls.diff` is the source half, cut from
that PR's commit `2ad2ed8d1`, which the patch header pins; it is applied
by the glob the image Dockerfile already runs, after the Dockerfile
strips the tarball's `manifests`, `charts`, `example` and `.github`, so
the image builds from source alone and the upstream tests that read the
CRD YAML skip instead of failing against the stale v3.3.5 copy.
`patches/crd-tls.diff` adds `spec.tls` to the vendored CRD, and with it
the CEL rule that fixes `tls.enabled` at creation; without the schema
block the apiserver prunes the field on admission and the patched binary
waits for something it never sees, and the rule cannot exist without the
block it references. `patches/rbac-tls.diff` grants Create, Get and
Update on Secrets, so the operator can publish the CA-only Secret it
addresses by name. That is the whole delta, and it is narrower than the
ported operator's own chart asks for: those three are the only Secret
calls in the patched source, and cert-manager Certificates get no grant
at all, because the operator only touches one under
`spec.tls.certManager` and this chart pins `certificateSecret` mode
instead. One change in `tls.diff` is not TLS and takes effect with TLS
off, named in the patch header: `MakeSlaveOfWithPort` connects to the
replica on its own port rather than on the master's. It changes a dial,
not a pod template, so shipping the rebuilt image rolls nothing; the
plaintext pod templates stay byte-identical to v3.3.5.

The image build registration this branch used to carry (`make -C
packages/system/redis-operator image` in the root `build:` target)
landed on main with the Valkey service (#3380), so the rebuilt operator
image is produced by CI from main as well; the digest in `values.yaml`
is still refreshed by release preparation, not here.

The chart patches re-apply at the end of `update:`, the same way
reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry
theirs. Otherwise `rm -rf charts` takes TLS back out on the next `make
update` and nothing fails. `make update` reproduces the committed tree
byte for byte.

TLS is chosen when the instance is created. The config gets `port 0` and
the listener moves to `tls-port`, so TLS replaces plaintext rather than
running beside it, and a running instance cannot change mode: a replica
restarted onto the TLS listener cannot replicate from a master still on
plaintext, and Sentinels already on TLS would fail over to a stale
replica. The RedisFailover schema in the carried CRD patch rejects a
change of `tls.enabled`, which on an existing instance surfaces as a
failed HelmRelease upgrade while the instance keeps serving what it did;
the README points at creating a new instance and migrating. That is also
why TLS is not inferred from `external`: an externally reachable
instance is the one most likely to have clients an upgrade must not
drop. Existing releases keep their plaintext listener and are not
restarted by the rebuilt operator image.

The tenant gets `ca.crt` and nothing else, through `tenantsecrets`. The
chart's own Secrets take a dot between the release and the suffix
(`<release>.ca-tls`, `<release>.tls`, `<release>.ca-cert`): the suffixes
are dot-free, so the split is unique and release `foo`'s CA Secret can
no longer be release `foo-ca`'s leaf, the collision the hyphenated form
had. The chart renders a `TenantProjection` pointing at
`<release>.ca-tls`, and the CA-extraction controller re-encodes the one
named key from the parsed certificate, so nothing that is not a
certificate gets through. Same path as postgres, mongodb, nats and
qdrant.

No Secret of this release is granted to the tenant by name beyond the
generated password, and that includes `<release>.ca-cert`, the CA-only
copy the operator publishes for itself. It holds no key today, but the
operator has Create and Update on Secrets cluster-wide, so what sits at
that name is the operator's call rather than the platform's, and RBAC
cannot filter by key. `hack/check-redis-rd-secrets.bats` pins the whole
rendered Secret surface instead of a list of known-bad names, because an
include with no `resourceNames` and no `matchLabels` restricts nothing.

The chainsaw suite for redis gains two cases. A TLS fixture asserts the
cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel
come up, the operator publishes its CA-only Secret under the widened
ClusterRole, the tenant receives exactly `ca.crt` through
`tenantsecrets` while the key-bearing Secrets stay off that path, and a
`redis-cli --tls` PING through the service name completes with the same
CA. A second case applies `tls.enabled` to a running plaintext release
and asserts the refusal: the HelmRelease upgrade fails with the schema's
message, the StatefulSet keeps its revision with both replicas Ready,
plaintext still answers and no TLS listener appears.

Certificate renewal rolls the pods from inside the operator. Redis and
Sentinel read the certificate once at startup, so the patched operator
stamps a hash of the TLS Secret's `tls.crt` and `ca.crt` into the pod
template of both the Redis StatefulSet and the Sentinel Deployment on
every reconcile, as `redis-failover.freshworks.com/tls-secret-hash`. A
renewal changes the hash; the operator's own roller replaces the Redis
pods one at a time and the Deployment controller rolls Sentinel. README
names the fallback: if the roll cannot run, the deadline is the old
certificate's expiry, past which the operator's own TLS connections fail
along with the clients.

`tls.authClients` maps to `tls-auth-clients` and defaults to `no`. The
platform mints no client certificate for the tenant and the CA key is
out of reach, so `yes` is only usable when the client certificate comes
from whoever also controls the CA. Changing the value on a running
release restarts the Redis and Sentinel pods: Redis reads the directive
from its config file only at startup, so the operator folds the
effective value into the certificate hash that drives the rollout, and
the change lands when it is made rather than at the next unrelated
restart.

Follow-ups in the carried operator patch go upstream first, to
freshworks-oss/redis-operator#85 or a successor there, and are picked up
here afterwards; the carried patch does not grow Go that upstream has
not seen. `tlsConfigFor` fetches the Secret on every call, tens of GETs
per reconcile. The renewal roll is static analysis so far: the live
check, `cmctl renew` on a running release while watching the `rfr-*` and
`rfs-*` pods roll, is still owed and belongs with the e2e follow-up.

Two facts about the shipped image rather than the patch.
`packages/system/redis-operator/values.yaml` keeps the digest of the
image built before the patch until release preparation rebuilds it, so
on `main` between this merge and the next release-prep, `tls.enabled`
renders the chain and `spec.tls` while the running operator ignores them
and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026
in `golang.org/x/net` v0.53.0, reached through
`PodService.UpdatePodLabels`; the port neither introduces nor fixes it,
upstream `master` is already on v0.55.0, so it lands with the next tag
bump rather than here.

Outside the patch: `cozyvalues-gen` renders `{}` in the README Value
column for `tls.authClients`: a child property without a default
inherits the parent object's `{}`, and `make generate` reproduces it
byte for byte, so it is a generator bug rather than something to
hand-edit here.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#37
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

I walked the trigger map against the diff file by file.
terraform-provider-cozystack is reached, tracked as
cozystack/terraform-provider-cozystack#37 rather than a speculative PR:
`values.schema.json` gains a `tls` object, and the provider hand-writes
a schema, model and expand/flatten pair per Kind. It already models
`tls` for postgres, kafka, nats and qdrant, not for redis. Redis needs
different wording from those four, whose descriptions say to omit the
block to follow `external`, which is the one thing redis does not do.

Nothing else is reached. The website regenerates this package's
reference page from `README.md` on a stable tag, so the README changes
need no PR there. No package added, renamed or removed.
`packages/core/platform/values.yaml`, variant and bundle composition and
release asset names are untouched, and so are `hack/package.mk`,
`hack/common-envs.mk` and `hack/update-crd.sh`, which is what ccp and
external-apps-example read. The two new files under `hack/` are bats
unit tests. No node prerequisite, installer value, telemetry metric or
cozy-proxy label changes.

### Release note

```release-note
feat(redis): add opt-in TLS for Redis and Sentinel. Set `tls.enabled: true` to serve both on a TLS-only listener backed by a per-release cert-manager chain; the tenant reads the CA certificate as the `<release>.tenant-ca` tenant secret. TLS is off unless set and is not inferred from `external`, so existing releases keep serving plaintext and are not restarted by the rebuilt operator image. TLS is chosen at creation: a running instance cannot switch modes, the change is rejected and surfaces as a failed HelmRelease upgrade, so add TLS by creating a new instance and migrating to it. Requires cert-manager, which the platform's `system` bundle installs on every variant. A renewed certificate rolls the Redis and Sentinel pods through the operator. TLS support is carried as a downstream patch over redis-operator v3.3.5 until freshworks-oss/redis-operator#85 lands in a release.
```
 2026-09-04 19:32:32 +04:00
Aleksei Sviridkin 900711567a fix(cozystack-basics): require route hostnames in tenant namespaces 
The route hostname policy short-circuited on an absent field. Its CEL
opened with `!has(object.spec.hostnames) ||`, so an HTTPRoute,
TLSRoute or GRPCRoute declaring no hostnames satisfied the first
operand and was
admitted without a single hostname compared against the tenant apex.
An explicitly empty list reached the same outcome by a different path,
because `all()` over an empty list is vacuously true.

A route that names no hostname does not match nothing. Under Gateway
API it inherits the hostnames of every listener it attaches to, so it
covers whatever that listener covers. Require the field to be present
and non-empty before validating it.

All three policies rendered by the template share one CEL string, so
HTTPRoute, TLSRoute and GRPCRoute pick the change up together. The denial message
now names the missing-hostnames case, and the Gateway e2e grep moves to
the part of that message common to every denial it issues.

The Gateway e2e suite carried the one fixture that relied on the old
behaviour: its parentRef probe declared no hostnames and is applied
into the suite's tenant namespace. It now reads the apex from the
namespace label and names a hostname under it, as the neighbouring
cases in that suite already did.

Radius is unchanged: the policy still matches only tenant-* namespaces.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-04 18:08:02 +03:00
Aleksei Sviridkin c91473fab1 docs(gpu): vGPU SR-IOV support ships with the bundled KubeVirt v1.9.0 
Two places also said a hand-crafted Package CR takes precedence over the
bundle render. It does not: the platform chart is the only writer of
those Packages, helm.sh/resource-policy: keep blocks deletion and not
replacement, so the next render puts the bundle's spec back. Both now
say the name has to be in bundles.disabledPackages.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-04 15:29:09 +03:00
Aleksei Sviridkin 5976ddf02f fix(gateway): normalise the GatewayClass name and refuse a blank one 
The class lists arrive over the values channel comma-joined, so every
reader trims each entry as it splits them. The scalar those entries are
compared against was not trimmed anywhere. A class name carrying a
stray space therefore failed `has` against its own padded self, and the
two readers of that scalar disagreed about the same class: the gateway
chart put the tenant on an edge-terminated class, while the
cluster-wide ACME issuers still pinned their solver to a listener an
edge Gateway does not render. That stops every HTTP-01 certificate in
the cluster from issuing, with nothing in any status naming the cause.

Normalise once at the single writer so both readers see a clean value,
and keep a defensive trim at each reader. The tenant's own choice does
not travel that channel — it comes from tenant.spec.gatewayClass — so
it is trimmed where the gateway chart reads it.

Unset and whitespace-only are different events and no longer share an
outcome. Unset means the choice was not made, and falling back to the
bundled class stays its documented behaviour. Whitespace-only means a
choice was made and is garbage, in the field that decides where TLS
terminates: resolving it quietly would take a tenant the operator meant
to put behind an edge terminator and publish it directly, with a real
certificate on a real address. Both the platform default and the
tenant's own choice now refuse it by name.

The kind check beside that refusal is part of the same fix, not a tidy
up. Through Helm's values path an unquoted numeric scalar arrives as a
float64, and the whitespace check compares it against a string, so
`className: 123` dies at that comparison — one line before the trim —
with an error naming neither the key nor the file. Refusing whitespace
without checking the kind first would therefore have made the numeric
case worse than it was. `kindIs "string"` is the right test because it
never has to name the type that actually arrived.

The order of the guard's steps is part of the guard. The value is read
before anything normalises it, so a default that fires on any falsy
value cannot launder 0 or false into the empty string and resolve them
to the bundled class ahead of the kind check. Unset is the one case
that still falls back, and each of those four inputs is pinned, so a
normalisation drifting back ahead of a check fails rather than quietly
covering only the loudest bad input.

Also correct a cleanup log line: the wildcard Certificate is deleted
for every mode that is not DNS-01, so naming HTTP-01 in that message is
wrong for existingSecret and for edge alike. And state the listener
name invariant plainly: the cap is 253, not the 63 of a DNS label, and
the longest name the builders produce is 84.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-09-04 14:33:35 +03:00
Aleksei Sviridkin 7851969448 docs(gateway): document GatewayClass selection and edge TLS termination 
An operator setting up a class whose provider terminates TLS upstream
had no page answering their questions: which platform values bound the
choice, what a tenant writes to make it, what a move between classes
does to certificates and to the replicated wildcard Secret, and how the
mechanism fails when a configured class name matches nothing installed.

Add docs/gateway-classes.md covering that surface. Describe the mode
and its consequences in the chart README alongside the other three,
including the one constraint stated nowhere else: the publishing tenant
cannot select a class whose TLS termination differs from the platform
default, because the cluster-wide ACME issuers derive their solver pin
from that default alone.

State the blast radius where it actually applies. The endpoints an edge
Gateway stops publishing exist only where those routes are rendered at
all, and both pages say what the transition does to the route
conditions the controller owns, so a reader is not left inferring it
from the code.

The security model names the selector each listener really carries. The
port-80 listener selects on the built-in kubernetes.io/metadata.name
label, which kube-apiserver writes and a namespace cannot forge; the
HTTPS and TLS-passthrough listeners select on
namespace.cozystack.io/gateway, written by the tenant chart and the
controller, so that guarantee does not extend to them. What closes the
exposure is that Namespace is cluster-scoped with no tenant verb on it,
and that the route-hostname policies bound what a labelled namespace
could publish.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-09-04 14:33:29 +03:00
Aleksei Sviridkin a2cb47c961 feat(gateway): update the CRD bundle to Gateway API v1.6.1 and gate GRPCRoute hostnames (#3872) 
## What this PR does

Bumps the vendored Gateway API CRD bundle in
`packages/system/gateway-api-crds` from v1.5.1 to v1.6.1. The package
Makefile pins the upstream ref, so the change is that pin plus the
re-rendered `templates/crds-experimental.yaml`.

The bundle version is not only bookkeeping: it is a compatibility signal
consumers read at runtime. Every CRD in the bundle carries a
`gateway.networking.k8s.io/bundle-version` annotation, and a Gateway API
controller can read it off the `gatewayclasses` CRD to decide whether
the installed bundle matches the one it was built against. Controllers
that do this typically accept any patch release but require an exact
major and minor match, and report `SupportedVersion=False` with reason
`UnsupportedVersion` on their GatewayClass when the minor differs, a
status condition that surfaces the skew rather than a functional break.
The Cloudflare Tunnel Gateway API controller proposed for the platform
in #3858 is built against Gateway API v1.6.1 and performs exactly that
comparison, so against the v1.5.1 bundle shipped today its GatewayClass
reports the bundle unsupported, and with this bump it reports supported.
Neither change depends on the other landing first: this PR only moves
the bundle, and the check lives entirely on the consumer side.

One consequence worth stating plainly, since it will outlive this PR: an
exact-minor check makes the shipped bundle version a coupled pin for any
such controller. A future bump to v1.7.x flips the same condition back
to `UnsupportedVersion` until that controller is rebuilt against the
newer bundle, so the two want to move together.

The bundle still emits the same twelve CustomResourceDefinitions and the
same two `safe-upgrades.gateway.networking.k8s.io` admission objects,
and every CRD still carries the `helm.sh/resource-policy: keep` stamp
the Makefile's post-processing step adds. One of the policy's two CEL
rules is rewritten, though, and it changes what a cluster accepts. That
has its own section below.

Two kinds change their version surface: TCPRoute and UDPRoute gain a
served `v1` that takes over as the storage version, and their `v1alpha2`
stays served with a deprecation marker. Every other kind keeps exactly
the versions it served at v1.5.1. That matters because the Cilium
operator resolves GatewayClass, Gateway, HTTPRoute and GRPCRoute in
`v1`, ReferenceGrant in `v1beta1` and TLSRoute in `v1alpha2` before it
enables its Gateway API controller; all six are still served by the
v1.6.1 bundle. TLSRoute in particular keeps `v1alpha2`, which the tenant
gateway controller, the platform TLSRoute templates and the tenant
hostname admission policy all address.

Schema changes are small and land on fields nothing in this repository
sets. `sessionPersistence.idleTimeout` is gone from HTTPRoute, GRPCRoute
and XBackendTrafficPolicy, removed upstream under GEP-1619 because no
implementation ever implemented it and HTTP cookies have no native
idle-timeout mechanism, so no working behaviour is lost even though the
field disappears from the schema; HTTPRoute gains a lower bound on
`retry.attempts` and now rejects duplicate `retry.codes`; ReferenceGrant
requires `spec` at the root. In the other direction, Gateway
infrastructure annotations and frontend CA certificate references go
from a limit of 8 to 16, and TLSRoute hostnames from 16 to 1024. No CRD
schema loses or changes a CEL validation rule, and the six added belong
exclusively to the new TCPRoute and UDPRoute `v1` schemas.

### The safe-upgrades admission policy accepts a different set of
bundles

The experimental kustomization ships a `ValidatingAdmissionPolicy` and
its binding, both named `safe-upgrades.gateway.networking.k8s.io`, next
to the twelve CRDs. The binding is `validationActions: [Deny]` with
`failurePolicy: Fail`, matching CREATE and UPDATE on
`apiextensions.k8s.io/v1 customresourcedefinitions` cluster-wide, and
Cozystack installs it in two places: on the host through
`cozystack.gateway-api-crds`, and inside every guest cluster that sets
`addons.gatewayAPI.enabled`.

Its second rule reads the `gateway.networking.k8s.io/bundle-version`
annotation off an incoming CRD. At v1.5.1 that rule was a blacklist:
reject anything matching `v1.[0-4].\d+` or containing `v0`, allow the
rest. At v1.6.1 it is a whitelist: allow `v0.0.0-dev` outright,
otherwise require the annotation to start with `v1.` and not match
`^v1\.[0-4](\.|$)`. Evaluating both expressions over a spread of
annotation values:

| bundle-version | v1.5.1 policy | v1.6.1 policy |
| --- | --- | --- |
| `v1.6.1`, `v1.5.1`, `v1.5.0-dev`, `v1.10.0` | allow | allow |
| `v1.4.0`, `v1.4.10`, `v1.0.0` | reject | reject |
| `v0.0.0-dev` | reject | **allow** |
| `v2.0.0` | allow | **reject** |
| `1.6.1` (no leading `v`) | allow | **reject** |
| `v1.4` (no patch) | allow | **reject** |
| `main`, empty string | allow | **reject** |

Nothing in this repository trips the new rejections: every CRD the
bundle ships is stamped `v1.6.1`. What is exposed is an operator
applying a Gateway API CRD set from somewhere else on a cluster that
carries the policy. A fork, a vendor build or a nightly whose annotation
reads `1.6.1` or `v2.0.0` used to go in and is now denied with `reason:
Invalid`, naming a policy the operator may not know is installed, and
whatever HelmRelease owns those CRDs then sits un-Ready. Uninstalling
the policy is the escape hatch, and the denial message says so.

The same hunk fixes an upstream inconsistency: at v1.5.1 the policy pair
was stamped `v1.5.0-dev` while its twelve CRDs said `v1.5.1`. At v1.6.1
all fourteen documents agree.

The Gateway and HTTPRoute objects the tenant gateway controller renders
were serialized across all three certificate modes and checked against
both the v1.5.1 and the v1.6.1 schemas; the two runs are identical, so
the bump does not change what the controller can create. `go.mod` still
pins `sigs.k8s.io/gateway-api` v1.4.1 and this PR deliberately leaves it
there. The CRD bundle and the Go client types have been on separate
versions since before this change, and nothing the controller emits
depends on the gap.

One thing to know before the next bump: from v1.6 on, the two things
upstream calls the experimental bundle disagree. This package follows
`config/crd/experimental`, the kustomization, which emits 12 CRDs at
v1.6.1; the published `experimental-install.yaml` for the same tag emits
13, the extra one being `xbackends.gateway.networking.x-k8s.io`, new in
v1.6 and not listed in the kustomization. At v1.5.1 the two agreed, so
there was nothing to choose between. Nothing in this repository
references XBackend and no vendored CRD depends on it, so following the
kustomization costs nothing today, but whoever bumps next inherits the
choice, and it should be made deliberately rather than by whichever
command gets typed.

The second commit corrects a comment this bump invalidated. It claimed
TLSRoute stays at `v1alpha2` through Gateway API v1.5 and that the
promotion to `v1` was still upstream-only; TLSRoute has been served at
`v1` since v1.5.0, so the claim was already wrong before this bump and
the version range it named is now wrong too.

The third commit closes a gap that correction exposed.
`cozystack-route-hostname-policy-tls` named `v1alpha2` alone while the
bundle serves `tlsroutes` at `v1`, `v1alpha2` and `v1alpha3`: three
served versions against one named. The gap is not new and this bump
neither creates nor widens it: the v1.5.1 bundle and the v1.6.1 one
serve the same three versions, with the same two marked deprecated.
Coverage of the two unnamed versions rested entirely on request
conversion, because `matchPolicy` is unset on both rules in that file
and so defaults to `Equivalent`, which rewrites a request submitted
under any served version into the one the rule names before the policy
sees it. That is why there is no bypass to demonstrate, and equally why
the arrangement is brittle: the conversion has nothing left to convert
into once no named version is served, and a rule that selects nothing is
not caught by `failurePolicy: Fail`, because nothing failed. The rule
now names all three, which is the shape the HTTPRoute rule above it
already used, and `hack/route-hostname-policy-version-coverage.bats`
reads the served versions out of the vendored bundle and fails when a
policy does not cover every one of them, so a future bundle move cannot
outrun the list silently, in either direction.

That guard carries negative fixtures rather than only watching a correct
tree pass. Three of them drive `assert_covers` itself, with the two
readers redefined in a subshell, against an omitted served version, an
empty served set and an empty named set. Each requires the specific
complaint text rather than merely a non-zero exit, so a helper renamed
out from under it fails for the right reason instead of reading as a
successful rejection. A fourth drives the comparison helper directly.

The fourth commit pins the vendored bundle version in the CRD package's
unit suite. Nothing outside the package Makefile named a Gateway API
version, so re-running `make update` against a different ref landed
green: twelve CRDs, every one stamped, document count unchanged, suite
passing. The new case reads the
`gateway.networking.k8s.io/bundle-version` annotation off every rendered
CRD and fails when it is not the version the Makefile pins.

The fifth commit extends the version-coverage guard to
`cozystack-gateway-hostname-policy`, which names the `gateways` versions
literally the same way the two route policies do, and whose version set
lives in the vendored bundle and moves on `make update` there. It
matched the served set already; the guard now says so on every run
instead of leaving one of the three hostname policies unpinned.

Two notes for whoever reads the release: the TCPRoute and UDPRoute
storage version moves from `v1alpha2` to `v1`, and while nothing in this
repository ships either kind and the CRDs declare no conversion block,
an upgraded cluster accumulates both entries in `status.storedVersions`
— inert today, load-bearing only if a later bundle stops serving
`v1alpha2`.

The sixth commit gives GRPCRoute the same hostname rule the other two
route kinds have. GRPCRoute carries `spec.hostnames` and it can reach a
tenant Gateway: the port-443 listeners pin `allowedRoutes.kinds` to
HTTPRoute and TLSRoute, but the port-80 listener leaves `kinds` unset,
and an unset list resolves to the kinds the listener protocol supports,
which for HTTP includes GRPCRoute. A GRPCRoute in a tenant namespace
could claim any hostname on the plaintext listener with nothing checking
it. Tenants hold no `gateway.networking.k8s.io` RBAC in Cozystack, so
this is the same defence-in-depth against a chart bug that the other two
rules are, not a tenant-user control. The rule is the HTTPRoute rule
with `grpcroutes` in place of `httproutes` and the one version the
bundle serves for it, and the coverage guard pins that list to the
bundle like the others.

Several places in the tree enumerate which route kinds the hostname
policies cover, and all of them named two: the port-443 kinds invariant
in the tenant gateway controller and the test that pins it,
`packages/extra/gateway/README.md`, and the threat model together with
the self-assessment. They name three now. The port-443 kinds set also
carries an accurate reason now. cilium#45559 governs only that every
port-443 listener carries the same set, not what is in it, so the
contents are a least-privilege choice: nothing the platform ships needs
gRPC, TCP or UDP routing on port 443. The README had also described
HTTPS and TLS-passthrough listeners as carrying different kinds, where
the controller gives both the same one.

One more commit makes the suite's existing assertions bite. A
ValidatingAdmissionPolicyBinding whose `policyName` points at another
policy renders without complaint and leaves its own policy inert, and
the suite pinned `validationActions` on the bindings while never pinning
`policyName` at all, so pointing the GRPCRoute binding at the HTTPRoute
policy kept every case green with the GRPCRoute control doing nothing.
All three bindings now pin the policy they name. In the same file,
`uncovered()` assigned to the variable names its caller reads back when
printing a mismatch, so a call site with the arguments swapped would
have printed a diagnostic naming the wrong side.

Two things are named rather than made. The version-coverage guard
collects `apiVersions` from every `resourceRules` entry mentioning the
resource without also matching on `apiGroups` or `operations`, so a rule
in another group, or one scoped to a single verb, would donate its
versions to the set coverage is measured against; the chart's own suite
pins the group and the resource on the rule it checks, which closes that
a layer below. And
`packages/system/cozystack-api/templates/api-tlsroute.yaml` renders
`gateway.networking.k8s.io/v1alpha2` while the bundle serves `v1` as
storage and marks `v1alpha2` deprecated, but the identical pin sits in
the kubevirt and kubevirt-cdi TLSRoute templates, and the pinned
`sigs.k8s.io/gateway-api` carries no `apis/v1` TLSRoute type for the
controller side to follow, so moving one of the three here would leave
the tree less consistent than leaving all three.

Closes #3789.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file: neither `packages/system/gateway-api-crds/Makefile`
nor its rendered CRD template matches any entry. Leaving every box empty
rather than ticking the "no downstream repository" line, because the
docs site carries a Gateway API version statement that is already
inaccurate and needs a decision that is not mine:
`content/en/docs/next/networking/gateway-api.md` and its v1.5 and v1.6
copies say Gateway API v1.5 ships TLSRoute at `v1alpha2` and that the
graduation to `v1` is still upstream-only, when v1.5.0 already shipped
TLSRoute `v1` as the storage version.

### Release note

```release-note
chore(gateway-api-crds): update the vendored Gateway API CRD bundle to v1.6.1. TCPRoute and UDPRoute gain a served v1 that becomes their storage version, with v1alpha2 still served and now deprecated; every other kind keeps the versions it served at v1.5.1. Upstream removed `sessionPersistence.idleTimeout` from HTTPRoute, GRPCRoute and XBackendTrafficPolicy; the field is not set by anything Cozystack renders, but a hand-written route that carries it loses it silently on its next write, with no admission error.

The bundle's `safe-upgrades.gateway.networking.k8s.io` ValidatingAdmissionPolicy changes which CRDs it accepts. Cozystack installs it cluster-wide on the host and in every guest cluster with `addons.gatewayAPI.enabled`. Its `gateway.networking.k8s.io/bundle-version` rule was a blacklist on two patterns and is now a whitelist on the `v1.` prefix with a `v0.0.0-dev` escape hatch. A bundle annotated `v0.0.0-dev` is now accepted where it was denied. A bundle annotated `v2.0.0`, `1.6.1` without the leading `v`, `main`, `v1.4` without a patch, or carrying an empty annotation is now denied where it was accepted. Nothing Cozystack ships is affected, since every CRD in the bundle is stamped `v1.6.1`, but applying a Gateway API CRD set from a fork, a vendor build or a nightly on such a cluster can now fail admission with `reason: Invalid`. Uninstalling the policy remains the documented escape hatch.

fix(cozystack-basics): the tenant TLSRoute hostname admission policy now names every served TLSRoute version (`v1`, `v1alpha2`, `v1alpha3`) instead of `v1alpha2` alone. Coverage of the other two rested on request conversion, which holds only while a named version stays served, and a rule left selecting nothing is not caught by `failurePolicy: Fail`.

feat(cozystack-basics): GRPCRoute hostnames in `tenant-*` namespaces are now gated by the same admission policy that gates HTTPRoute and TLSRoute. A tenant Gateway's port-80 listener leaves `allowedRoutes.kinds` unset, which admits GRPCRoute by protocol default, so a GRPCRoute could claim any hostname on the plaintext listener unchecked. Tenants hold no `gateway.networking.k8s.io` RBAC, so this is defence-in-depth against a chart bug rather than a tenant-user control.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
  - Added hostname admission validation for `GRPCRoute` resources.
- Expanded `TLSRoute` hostname validation to support all served Gateway
API versions.
- Ensured route hostname policies consistently cover HTTP, TLS, and gRPC
routes.

- **Security**
- Updated listener policies to restrict port-443 routes to supported
HTTP and TLS route types.
- Clarified fail-closed hostname validation behavior across Gateway,
Route, and Ingress resources.

- **Documentation**
- Updated security and threat-model documentation to include GRPCRoute
handling and validation coverage.

- **Maintenance**
  - Updated the bundled Gateway API CRDs to version 1.6.1.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-04 15:23:36 +04:00
Aleksei Sviridkin fb740db0b0 docs(security): separate the fail-open Gateway-listener policy from the fail-closed route policies 
The self-assessment summarised the whole hostname policy set as
fail-closed, while its own table row states that the Gateway-listener
policy passes a Gateway whose namespace carries no host label. Say
which policies are which. The port-443 kinds comment in the reconciler
test also tied the GRPCRoute admission rule to the port-80 listener;
the rule matches the kind wherever it attaches, and the port-443
exclusion is a least-privilege choice, not the reason the rule exists.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-04 01:18:07 +03:00
Aleksei Sviridkin af19aa933c docs(security): record GRPCRoute in the route hostname policy set 
The port-443 kinds set still excludes GRPCRoute, TCPRoute and UDPRoute.
cilium#45559 governs only that every port-443 listener carries the same
set, not what is in it, so the contents are a least-privilege choice:
nothing the platform ships needs gRPC, TCP or UDP routing on port 443.
TCPRoute and UDPRoute additionally carry no hostname and no admission
rule gates them.

The gateway README described HTTPS and TLS-passthrough listeners as
carrying different kinds. They carry the same one, which is what
cilium#45559 requires.

The threat model and the self-assessment enumerate which route kinds the
hostname policies cover, and both listed two.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-03 22:17:29 +03:00
Andrey Kolkov e94fcf8219 feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990) 
## What this PR does

Adds backup/restore support for the managed `RabbitMQ` application.
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so this backs up broker
**definitions** (vhosts, users, permissions, queues, exchanges,
bindings, policies, parameters) over the management HTTP API
(`GET`/`POST /api/definitions`) and stores them in the system bucket.
Message payloads and point-in-time recovery are intentionally out of
scope; a full-data backup would be a Velero volume snapshot, which is a
separate strategy.

It is delivered as a **first-class
`strategy.backups.cozystack.io/Rabbitmq` driver**, following the
Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

- `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's
`PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`,
pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a
`batch/v1.Job`, covering both restore flows: in-place (into the source
broker) and to-copy (into a separate, differently-named RabbitMQ in the
same namespace, authenticating with the target's own default-user
credentials and reading the source's object key).
- A **Go-read precondition** gates each run on the `RabbitmqCluster`
reporting `AllReplicasReady` (read through a small `rabbitmqtypes`
partial type), so a not-yet-ready broker surfaces a precise
`Ready=False` wait on the BackupJob/RestoreJob rather than a Job that
starts and fails against an unreachable management API.
- The produced `Backup` records the stored object's location on
`status.artifact.uri`, rendered from the strategy's
`spec.artifactURITemplate`. The `Backup` CRD has no status subresource,
so this persists on Create, mirroring the Etcd driver.

Dispatch is wired in both controllers (plus
`supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the
scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job
itself is already covered by the existing `batch/jobs` grant). It ships
`cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and
routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default`
BackupClass, which also enrols it in the `DefaultObjectsGate`. The
RabbitMQ application chart is left untouched — it had no prior backup
wiring, and the strategy carries the S3 coordinates itself.

Unit tests cover the driver's pure helpers (applicationRef validation,
cluster-name derivation, the readiness-message builder, artifact-URI
rendering, parameter round-trip) and the reconcile precondition gate
(defers with `Ready=False` and no Job until the cluster is ready, then
proceeds). A runnable example under `examples/backups/rabbitmq/` and an
honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a
sentinel vhost, back it up, drop it, and assert it round-trips through
object storage on both restore flows.

The Job pod reuses the existing `platform-migrations` image (curl +
ca-certificates) under a new
`backupStrategyController.rabbitmqBackupClientImage` key, stamped in
lockstep with `chBackupClientImage` in `packages/core/platform/Makefile`
so the repository stays pinned at a single digest
(`hack/image-pin-consistency.bats`). A dedicated slim image built by
this package is a follow-up.

**Validation:** `go build`/`go vet`/`gofmt`/`go test
./internal/backupcontroller/... ./internal/template/...`, deepcopy +
Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate
both on and off, `helm lint`, and `helm unittest` (18/18). The backup +
in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a
real management-API export and a real S3 `--aws-sigv4` upload/download
against SeaweedFS), so the driver's core contract is exercised on a real
cluster; the in-CI Chainsaw run is the remaining automated evidence. The
object-cleanup, terminating-namespace teardown and readiness-deadline
paths added during review are unit-tested (including `Reconcile`-level
finalizer retention and the terminating-namespace release) but not yet
cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled
on the `nano` preset, so the example/e2e use `c1.small`.)

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. This is an additive backup
strategy plus a new strategy CRD; no downstream repository is
mechanically forced by the change. A documentation page for RabbitMQ
backup on [cozystack/website](https://github.com/cozystack/website)
would be a reasonable follow-up, but it is not required by this diff —
leaving the boxes unticked for a human to decide rather than ticking "no
downstream" to clear the checklist.

### Release note

```release-note
feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added RabbitMQ backup and restore support for definitions, including
vhosts, users, queues, exchanges, bindings, policies, and related
settings.
  * Added in-place and restore-to-copy workflows with readiness checks.
  * Added scheduled and on-demand backup examples.
  * Integrated RabbitMQ backups with the default backup configuration.
* Backups store definitions only; message payloads and point-in-time
recovery are not included.
* **Documentation**
  * Added a complete RabbitMQ backup and restore demonstration guide.
* **Tests**
* Added end-to-end coverage for readiness and backup/restore round
trips.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-03 16:03:46 +04:00
Aleksei Sviridkin 61026b732c docs(agents): require the neutral Assisted-by: LLM trailer 
The attribution trailer exists for disclosure, not to advertise a
model. A model name in the trailer tells a reviewer nothing they can
act on, and it goes stale with every tool change, so the accepted form
is a single `Assisted-by: LLM` line however many tools took part.

A link to an assistant session is worse still: it rots, and it carries
nothing about the change itself, so no session trailer and no
transcript URL belong in a commit, a PR description, or a comment.

The commit body explains why a change is made; the diff already shows
what changed.

The hygiene rules for messages, trailers and in-code comments are
written down as review blockers for the same reason: a NOT LGTM on one
of them is then a stated rule, not a reviewer's taste.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-03 01:14:58 +03:00
scooby87 3a70f91423 fix(vm-disk): fail on storageClass/source edit of existing disk (#3969) 
## What this PR does

Fixes #2985. Editing a `VMDisk`'s `storageClass` or `source` after its
DataVolume already exists was silently ignored:
`packages/apps/vm-disk/templates/dv.yaml` looks up the existing
DataVolume and, when found, reused its `spec` verbatim (because
`DataVolume.spec` is immutable in CDI). The edit therefore rendered as a
no-op — the typed CR accepted it and the HelmRelease reported success,
while the DataVolume/PVC kept the original spec and the source of truth
silently diverged from the cluster.

The template now builds the desired `source` into a shared
`$desiredSource` variable (used both for rendering and comparison) and,
when a DataVolume already exists, calls `fail()` with an explicit
message if the requested `storageClass` or `source` differs from the
immutable existing spec — mirroring the explicit size handling already
done in `pvc-resize-hook.yaml`. Storage growth is unaffected and still
flows through the resize hook.

`source` is also marked `@immutable` in `values.yaml`, so the generated
schema now carries the same `x-kubernetes-validations` transition rule
(`self == oldSelf`) that `storageClass` already had; `types.go`,
`values.schema.json` and the generated `vm-disk-rd` CRD are regenerated
to match.

### Verification

- `helm unittest` covers both the immutability guard (via a mocked
existing DataVolume through `kubernetesProvider`, since `lookup` needs a
cluster) and initial-creation source rendering for every source type —
12/12 pass.
- Verified live on a dev cluster with `helm install --dry-run=server`
against a real DataVolume: an unchanged edit renders, a `storageClass`
change fails, a `source` change fails, and a storage grow still renders
the resize hook without tripping the guard.

### Scope note

This PR only makes the drift visible and marks `source` immutable. Two
deeper platform findings sit outside its scope: (1) #2657 (pre-existing)
— the `apps.cozystack.io` aggregated apiserver does not evaluate the
`x-kubernetes-validations` CEL rules on update, so `@immutable` is a
no-op at admission for every Application kind, which is exactly why the
chart-level `fail()` — not the CEL rule — is the substantive guard here;
(2) #3971 — that same apiserver does not honour `--dry-run=server` and
persists the change (surfaced while testing this fix).

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the diff
file by file. The diff touches only `packages/apps/vm-disk/` (template,
`values.yaml`, regenerated `values.schema.json`, tests), the regenerated
`packages/system/vm-disk-rd/cozyrds/vm-disk.yaml`, and
`api/apps/v1alpha1/vmdisk/types.go`. No package was added, renamed or
removed; no `values.schema.json` field was added, renamed or removed
(only a validation rule on the existing `source` field); no default
changed; `README.md` is unchanged; the `cozyrds` file was regenerated,
not renamed. The terraform provider models fields, defaults and enums
but not CEL immutability, so it is unaffected. No downstream repository
is reached by this change.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(vm-disk): editing storageClass or source on an existing VMDisk now fails with a clear message instead of being silently ignored; delete and recreate the disk to change these immutable fields.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for blank, HTTP, image, disk, and upload VM disk
sources.
* VM disk sources and storage classes are now protected from unintended
changes, while storage expansion remains supported.
* Existing disks preserve their configured source and storage settings.

* **Bug Fixes**
  * Invalid or conflicting source configurations are rejected.
* Prevented silent no-op updates and incorrect immutability errors for
existing disks.

* **Documentation**
* Documented source and storage class immutability, including the need
to recreate disks when changing either setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-02 18:40:24 +03:00
Aleksei Sviridkin d2fcf41fb8 docs(agents): drop redis-operator from the build-target violations 
The root Makefile build: list now carries the redis-operator image, so
the package stamps a reference from inside the list every other producer
iterates, and the image-refs rule no longer names it as a live violation.
flux-plunger, keycloak-operator and kilo still are.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-02 17:56:06 +03:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver 
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-01 18:21:10 +04:00
Andrei Kvapil 440c3b58f3 test(valkey): add e2e test and wire build, labeler and docs 
Add the Valkey e2e bats test (mirrors redis), build the compat image in the
top-level build target, map the valkey scope to area/database in the PR
labeler, and list valkey among charts with an immutable storageClass.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-09-01 11:44:07 +04:00
Alexey Artamonov 06624bacd0 fix(vm-disk): fail on storageClass/source edit of existing disk 
Editing a VMDisk's storageClass or source after its DataVolume already
exists was silently ignored: dv.yaml reused the existing DataVolume.spec
verbatim (DataVolume.spec is immutable in CDI), so the edit rendered as a
no-op while the HelmRelease reported success and the CR diverged from the
cluster.

When a DataVolume already exists, fail with a clear message if the
requested storageClass or source differs from what the disk was created
with. Both compare the operator's input recorded in an annotation at
creation (vm-disk.cozystack.io/storage-class and .../source), written only
at creation and preserved verbatim afterwards — never overwritten with an
unapplied edit. Recording the input rather than reading the live object
matters on both axes: the image-clone PVC name carries a chart-version
prefix (vm-image- was renamed to vm-default-images-, commit bec35e3a) that
drifts across upgrades with no edit, and the live PVC storageClassName of an
adopted or foreign DataVolume may legitimately differ from the CR default
without being a change. The annotation key presence (not its value) marks a recorded class, so a disk
created with the cluster default "" is recorded and still caught if later
changed to a concrete class.

Disks with no annotation (created earlier, or out-of-band) are handled
conservatively so a working release never deadlocks: source is compared
against the stored spec.source only when that stored source is a shape the
chart itself produces, so an adopted DataVolume using spec.sourceRef or a
non-chart source (registry, snapshot, http+secretRef, ...) is left alone
rather than deadlocked. The image-clone exemption fires only when the stored
source is a cozy-public pvc AND the request is itself an image, so a change
away from an image source to another type is still caught. storageClass has
no safe live fallback, so an un-annotated disk is not guarded here (apiserver
CEL, #2657, is the eventual guard). An unrecognized source type
now fails explicitly. Storage growth is unaffected and still handled by
pvc-resize-hook.yaml.

Mark source as immutable in values.yaml so the generated schema carries
the same x-kubernetes-validations rule already present on storageClass,
and document the contract, the exemption, and the source-annotation schema
coupling in docs/storage-immutability.md and the chart README.

Add helm-unittest coverage for the immutability guard (rejection incl.
clearing storageClass to empty and changing a non-image disk to image,
no-op reuse with annotations preserved, legacy image-disk drift, annotation
non-poisoning, un-annotated storageClass skip, adopted default-class disk,
unknown source) and initial-creation source rendering.

Fixes #2985

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-08-28 18:26:36 +03:00
IvanHunters 0fa6cf72a5 chore(ci): complete the backport label rename across all consumers 
The initial rename missed several literal consumers of the old label
names beyond labels.yml and backport.yaml:

- stale.yaml exempt-pr-labels still listed backport,backport-previous.
  actions/stale matches label names exactly and does not resolve
  label-sync aliases, so after the rename a kind/backport PR would lose
  its stale exemption and get auto-closed after inactivity. This is the
  one behavioural regression; the rest are documentation drift.
- PULL_REQUEST_TEMPLATE.md told contributors to apply the backport label.
- release.md label-target table and the gh pr list --search commands
  referenced the old names (the searches would silently return nothing).
- contributing.md claimed a backport label is auto-applied on the
  [Backport] prefix; pr-labeler only adds area/release, so the line was
  already stale and is corrected while renaming.

Historical narrative in release.md and the backport scope-to-area doc
rows (a Conventional-Commit scope, not the label) are left as-is.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-27 13:08:15 +05:00
Andrei Kvapil eef95f456d docs: add CNCF General Technical Review (GTR) questionnaire (#3572) 
## What this PR does

Adds the CNCF General Technical Review (GTR) questionnaire for
Cozystack, completed for the Sandbox → Incubation Due Diligence (tracks
cncf/toc#1916). It answers the Day 0 / Day 1 / Day 2 questions from the
CNCF TOC project-reviews template, with versioned links to the v1.6
documentation and to the repo's governance and security files.

**Status: DRAFT / work in progress.** The questionnaire itself is a
living document per the template's guidance. Everything repository
evidence settles has been answered and verified against the tree; the
banner at the top of the file names the seven substantive gaps that need
measurement, publication or a maintainer decision — load/scale results
and tested maximums, published SLOs/SLIs, measured platform baseline
overhead, the upgrade→downgrade CI matrix, whether upgrades trigger
Talos node reboots, the security-team rotation and diversity policy, and
publication of the CVE-scanning pipeline's policy documents.
Version-specific links should be pinned to the submitted tag before
final submission.

## Changes beyond the new file — please read this part

Review feedback found two factual errors that originate **outside** this
questionnaire, so fixing them only in the GTR would have left a reviewer
clicking through to the contradiction. Three existing files are
therefore touched:

- **`SECURITY.md` — supported-versions table. This is a policy change
and needs a policy review, not a wording check.** The table was dated
"As of March 17, 2026" and named `v1.1.x` as the current stable line,
five minors behind `v1.6.1`. It is rewritten to be keyed to **observable
patch activity on a line** — supported means a patch release within the
last 6 months, limited support 6 to 18 months, end of life beyond that —
so an adopter can determine their own status from the Releases page and
the table cannot drift again. Two intermediate versions of this PR got
the policy wrong in opposite directions and both are reverted: one
narrowed support to the latest two minors, which contradicts practice
since several `1.x` lines receive patch releases concurrently; the other
removed the end-of-life category altogether and extended discretionary
backports across all of `0.x`. What lands here restores an end-of-life
category, puts the `v0.41` cutoff back with its last-patch date, and
states explicitly that the six- and eighteen-month marks describe
current behaviour rather than committing the project to patch any line
for that long. `@kvaps`, as a CODEOWNER for this file: please review the
**policy**. If the project would rather state a guaranteed backport
window, or keep support entirely at maintainer discretion with no stated
marks, that is a governance decision and this PR deliberately does not
make it.
- **`SECURITY.md` — automated security analysis.** It said "two scanners
run continuously" and listed CodeQL and Scorecard. It now also describes
`zizmor` (GitHub Actions SAST, gating on PRs), the organization-wide
Trivy CVE-scanning pipeline, and Renovate's actual coverage. It also
records that CodeQL is **not** currently among the required status
checks on `main` — those are `pre-commit` and `E2E Tests` — so the
CodeQL gate is review convention rather than branch protection today.
- **`CONTRIBUTOR_LADDER.md`** — Reviewer nomination described adding a
username to an `OWNERS` file. No `OWNERS` file exists in this
repository; line 158 of the same document already refers to
`CODEOWNERS`. Corrected to name the files that exist. This does not
change the process.
- **`docs/security/self-assessment.md`** — brought in step with the
above on the same three points (CodeQL's gate status, the Trivy pipeline
and its limitations, Renovate's real coverage), so the GTR,
`SECURITY.md` and the self-assessment agree rather than contradict each
other in front of the TOC.

## Two things for `@kvaps` to decide, not to approve as wording

1. **Which release lines are supported — and the evidence points two
ways.** The six-month activity test in the rewritten table classifies
every `1.x` line from `1.0` to `1.6` as supported, because each has had
a patch release inside six months (`v1.1.7` as recently as 2026-06-03,
`v1.0.7` on 2026-03-30). But *recent* activity looks like three lines,
not seven: of the sixteen most recent backport branches, fifteen target
`release-1.4`, `release-1.5` or `release-1.6`, and only `1.4`, `1.5` and
`1.6` have shipped a release since 2026-07-03. Older backport branches
for `release-1.2` and `release-1.3` exist but are months old — and since
branches are never deleted, their existence proves nothing about the
present, which is the same trap the table's own lead-in now warns about.

So the concrete question is: **is the intended set `1.4`/`1.5`/`1.6`
supported with `1.0`–`1.3` on limited support, or is it genuinely all
seven?** If a critical fix would today be backported to `v1.1`, the wide
table is right. If it would not, the six-month window is the wrong
instrument and should be shortened. I have deliberately not chosen —
either answer is defensible and it is a maintainer call, not a
documentation one.

**A third file turns on the same answer.**
[`docs/security/incident-response.md`](https://github.com/cozystack/cozystack/blob/main/docs/security/incident-response.md)
step 5 says a fix ships "across all affected supported release lines",
pointing at the `SECURITY.md` table — so widening that table to seven
lines widens this commitment with it. This PR does not touch that file
and does not assert the claim, but the two should move together: if the
answer is "all seven", incident response is right and `SECURITY.md`
should stop hedging; if it is "1.4–1.6", the table, the window and that
sentence all change at once.
2. **Whether there is a backport rule at all.** No project document,
public or private, describes how a fix is chosen for backporting to a
given supported line. An intermediate version of this PR asserted one
(by severity, default-config reachability and adopter count); that was
invented and has been removed. The file now says the decision is taken
per fix and the rule is not documented. If a rule exists informally,
writing it down would answer a question the seven-supported-lines table
otherwise raises.

## Gaps this documents rather than fixes

Recorded here because they surfaced while verifying the questionnaire's
claims and each is a real finding:

- CodeQL is not a required status check, though two documents described
it as gating merges.
- There is no top-level `NOTICE` file, for a distribution bundling on
the order of fifty upstream components.
- No default Kubernetes audit policy is shipped; audit logging is opt-in
with the policy left to the adopter.
- Release artifacts are not signed and no issue tracks adding signing;
build provenance is explicitly disabled (`--provenance=false`); SBOM
generation is implemented but defaulted off.
- The CVE-scanning pipeline lives in a repository that returns 404 to
non-members. Its aggregate reporting is now published (below); its
policy documents and scanning scripts, which hold no finding data, are
not, so the method behind the numbers is still not independently
auditable.
- `nightly.yaml` has been failing since 2026-08-02 and `e2e-tag.yaml`
failed around the v1.6.1 release; current CI health is disclosed next to
the workflow links.

## New: published security reporting, and a backlog audit that changes
the SCA answer

`docs/security/reports/` now holds monthly vulnerability-handling
reports plus a [backlog
reconciliation](https://github.com/cozystack/cozystack/blob/main/docs/security/reports/reconciliation-2026-08.md).
The reconciliation exists because the CVE-scanning pipeline has no
write-back from this repository, so it has never marked a finding as
fixed and its counts were misleading in both directions. Audited against
what actually shipped:

- Remediation did happen — a clear majority of confirmed findings relate
to a component whose in-tree version now carries the fix — but as **one
wave in June 2026**, not a steady stream. The changes that did most of
the work name no CVE at all: re-vendoring the dashboard console removed
the vulnerable npm dependencies rather than upgrading them; refreshing
the Alpine bases cleared fourteen; bringing the Go toolchain current
satisfies all 62 `stdlib` findings.
- **57% of the backlog was never applicable or cannot be assessed.** The
largest block is packages containing no executable vulnerable code —
chiefly kernel headers, since a container never provides a running
kernel, which is the same reasoning the maintainers already applied 80
times to `kernel-headers` and never to `linux-libc-dev`. Reachable
exposure in a stock install is an estimated 2–5% of the tracker.
- **Four component bumps are available upstream and not taken**:
`nats-server`, the CoreDNS image, OpenBao, `fast-uri`.
- Seven defects in the pipeline itself, with a write-back specification
to fix the root cause. Two issues counted as findings are actually
quarterly-review tracking tasks.

The disclosure rule moved in the open direction: findings with no
upstream fix will be named with mitigation guidance, because under the
previous rule that class — a fifth of the backlog — stayed secret
permanently, and it is exactly where an adopter needs to know a
mitigation is the only control. Conversely, remediation is no longer
attributed to individual months, because crossing a count of still-open
findings with one month's publicly enumerable component bumps narrows
the search for unpatched exposure more than a bare count does.

Several findings warrant maintainer attention ahead of the backlog and
are being raised privately rather than named here.

### Downstream repositories

- [x] No downstream repository is affected by this change (docs-only; no
code, API, manifest or packaging changes).

### Release note

```release-note
docs: add CNCF General Technical Review (GTR) questionnaire for the Sandbox → Incubation review
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Added a comprehensive technical review covering planning,
installation, operations, architecture, security, and troubleshooting.
- Added April–August 2026 security reports and a complete CVE backlog
reconciliation.
- Expanded security self-assessment, scanning coverage, reporting
guidance, disclosure rules, and audit limitations.
- Clarified supported-version policies, release backport decisions, and
private vulnerability-reporting procedures.

- **Process Updates**
- Updated reviewer nomination requirements to use current ownership and
maintainer records.
- Clarified severity- and risk-based backport decisions for coordinated
security releases.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-26 16:53:10 +02:00
Timofei Larkin f5b957ee27 feat(kubernetes)!: manage worker pools as KubernetesNodes, adopt on upgrade (Phase 2, breaking) (#3315) 
## What this PR does

Phase 2 of the Kubernetes-app split
([cozystack/community#8](https://github.com/cozystack/community/pull/8))
— the **breaking** half. Stacked on #3314 (review that first).

Removes worker node pools from the `kubernetes` chart (they moved to
`kubernetes-nodes` in #3314) and adopts existing pools on upgrade:

- `kubernetes` chart is now **control-plane only**: drops the pool
`range` block, the `talos-reconcile` Job + RBAC (deleted; moved to
`kubernetes-nodes`), the preserve-old-KMT block, the
`kubernetes.nodeGroups` helper, and the per-group dashboard RBAC.
- **`spec.nodeGroups` is removed from the `Kubernetes` CR** (BREAKING).
Worker pools are now separate `KubernetesNodes` resources.
- **Migration 54** (bumps `targetVersion` to 55): pre-upgrade hook that,
for every `kubernetes-<cluster>` release, creates one
`kubernetes-nodes-<cluster>-<pool>` HelmRelease and re-annotates the
existing worker objects onto it with `helm.sh/resource-policy=keep`, so
the now control-plane-only parent upgrade does not prune running
workers. The default `md0` (implicit when `nodeGroups` was empty) is
materialised explicitly. Idempotent. Genuine infrastructure/read
failures fail closed (`exit 1`, retried on the next upgrade); hostile
stored shapes (a pool name over Helm's 53-char release-name cap, an
RFC-1123-invalid or whitespace/slash map key, a non-object `nodeGroups`)
are warned, pinned, and skipped rather than deadlocking the entire
fleet's upgrade.
- The `ingressNginx` addon no longer auto-provisions a node (the
implicit `md0` default is gone). If enabled without a pool carrying
`roles: [ingress-nginx]`, the controller DaemonSet simply schedules
nothing (degraded, not failed) — the requirement is documented in the
chart NOTES.txt rather than enforced at render time.

Because the child chart renders the pool objects **byte-identically** to
the old parent render (golden parity in #3314; the
KubevirtMachineTemplate content-hash name is preserved), adoption does
not churn live worker VMs.

### Migration number

`54` is contiguous with main (latest is `53`) and `targetVersion` is
`55`. The `52`/#3201 collision noted in earlier revisions has been
resolved: this branch was rebased and merged past it, so the numbering
is contiguous and `run-migrations.sh` has no gap to hard-fail on.

### Not yet done (blocking draft → ready)

- [ ] **dev e2e upgrade validation** (`cozystack-pr-test`): upgrade a
custom-`nodeGroups` cluster and a default-`md0` cluster; assert zero
worker-VM churn (KMT hash, Machine uid/creationTimestamp stable) and
that the adopted child HRs reconcile.
- [ ] downstream trigger-map walk (website docs for the `nodeGroups` →
`KubernetesNodes` change; terraform-provider for the removed API field).

### Downstream repositories

Deliberately unticked (draft) — the breaking API change affects docs and
the terraform provider; follow-ups to be filed before ready.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:

```release-note
Worker node pools of a managed Kubernetes cluster are now managed as separate `KubernetesNodes` resources. `spec.nodeGroups` on the `Kubernetes` CR is removed; existing pools are adopted automatically on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Worker pools are now separate `KubernetesNodes` resources with their
own lifecycle.
* Added a render-time compatibility guard to prevent worker pools from
running ahead of the parent control-plane minor version.
* Improved upgrade/uninstall behavior to better preserve already-adopted
worker objects.

* **Breaking Changes**
* Removed per-pool `nodeGroups` and worker node health settings from the
`Kubernetes` chart/CR; pool configuration now lives in
`KubernetesNodes`.
* Ingress requires at least one `KubernetesNodes` pool with `roles:
[ingress-nginx]`.
* `cluster` and `storageClass` are immutable after creation; pool
`version` may lag but must not be ahead of the parent minor.

* **Documentation**
* Updated READMEs, notes, and schemas to reflect the `KubernetesNodes`
split and the immutability/version rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-26 12:06:18 +04:00
Aleksei Sviridkin ad62a8d62b test(e2e): stop blocking merges on the tenant node-join deadline 
The tenant node-join wait no longer discriminates. Nested
virtualisation on the shared runners degrades far enough that a guest
which registers in minutes on real hardware can miss any deadline this
test can afford, so a red on that one wait says as much about the
machine the run landed on as about the product, and it teaches a
reader to re-run the job.

The suite still fails on it. That is what keeps Chainsaw's catch
running: the previous-instance container logs, the host snapshot, the
data-plane capture and the event dump are all keyed on the test
failing, and the report stays honest. What changes is one layer up.
The wait writes a marker beside its diagnostics, and a new lane-side
gate reads that marker to decide whether the job blocks. It opens only
when every failed test in the run carries one, and blocks on anything
it cannot attribute: a report it could not read, a failed test the
report does not name, a failure with no marker.

Only the lanes that block a merge ask it. The two pull-request lanes
do; nightly and the tag lane run the same suites and stay hard. A soft
pass on the tag lane would ship a release candidate whose tenant
clusters brought up no worker nodes, and nightly is the only place an
e2e failure reaches anyone who did not open a pull request, since no
lane in this tree notifies anywhere.

A softened red finishes the job green, so the job's result stops
answering whether the suite passed, and both kinds of consumer are
handed the decision instead of inferring it. A step gated on the job
failing would silently stop running: the SSH breakpoint is that case,
and it still opens for a pull request marked for debugging. A job that
reports the result would silently start reporting a pass: the required
commit status now says the deadline was missed and that the merge is not
blocked on it, rather than that the suite passed.

The deadline goes to 29m. That is the test's own budget and nothing
more: its clock starts after the machinedeployment wait, the
LoadBalancer assignment and the healthz probe, while the chart's
autoscaler and health-check timers have been running since the Machine
was created, so no ordering against them is claimed. The chart's 20m
and 30m give a scale to pick against, and a budget over the first
leaves room for a remediation cycle without promising that one fits.

The operation ceiling both suites give the script rises with the
deadline, and the diagnostics phase budget rises inside it, because
the collectors gated ahead of the serial console cost more between
them than the old budget covered. Both budget guards read their terms
from the sources that set them. The e2e job cap rises with them, and
its guard holds what a guard can hold: the pair's operation and catch
ceilings, and one figure across every lane that runs the suite. It
also records that the suites' full summed ceilings have never fit any
cap in that file, so the margin above the bound is taken from a
measured run instead. The prose that restates the cap elsewhere moves
with it, and the sweep that holds the deadline's restatements now
reads the figure spelled in words as well as in minutes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-21 12:52:42 +03:00
Aleksei Sviridkin 9ccb133b66 fix(ci): de-escalate select-e2e and name every escalation (#3817) 
## Why

I replayed the 150 most recently merged pull requests through
`hack/select-e2e.sh` with every escalation branch instrumented. 118 of
them (78.7%) ran the full 21-suite Chainsaw run, and the causes, counted
once per escalating PR with the count where that cause was the only one
in brackets, were `FULL_PATTERN` 88 [68], `NO_SUITE_FOR_GROUP` 24 [24],
`CHAINSAW_SHARED` 18 [1] and `UNCLASSIFIED` 7 [4]. `YQ_BROKEN`,
`NO_GRAPH_OWNER` and `BACKSTOP_EMPTY` never fired.

Two problems come out of that. Most of the escalation is real but
unnecessary. Whole classes of path that provably cannot affect a
Chainsaw suite were escalating because no rule claimed them. And the
commonest cause of a full run was absent from the log, so "why did this
pull request run everything" could only be answered by re-deriving the
selection by hand.

## What changes

**Every escalation now names its cause on stderr.** Seven branches could
reach the full suite and four did it in silence, `full_suite_pattern`
among them, the commonest cause by a wide margin, so the usual answer
was the one the log never carried. The lines go to stderr and must stay
there: stdout is the suite list and both e2e lanes parse it, so a reason
line on stdout would be read as a suite name. A unit test asserts every
message, verified by muting each of the eleven in turn and confirming
the file goes red for all of them. A reason that regresses to silence
changes no selection and is otherwise invisible.

**`cozystack.etcd-application` gains its missing `dependsOn:
cozystack.etcd-operator`.** `packages/extra/etcd` renders `kind:
EtcdCluster` from `etcd-operator.cozystack.io/v1alpha2`, and this was
the only operator-backed application source in the tree with no edge to
its operator. It is a latent install bug as much as a selection gap:
without the edge `etcd-rd` registers the ApplicationDefinition as soon
as the engine is up, so a tenant can create an `Etcd` before the
operator exists and its HelmRelease fails on `no matches for kind
"EtcdCluster"`, and `hack/select-install.sh`'s forward closure for the
`etcd` suite omitted the operator for the same reason. On the selection
side `cozystack.etcd-operator` reached no runnable suite, so every
change to the operator ran all 21.

**`vm-disk-application` maps to the `vminstance` suite.** It resolved to
a `vm-disk` suite that does not exist, `intersect_suites()` dropped the
name, and the source reached nothing runnable. Neither did
`cozystack.kubevirt-cdi` above it, whose only other dependents are
`vm-default-images` and `vm-disk-application` itself, so every CDI
change ran all 21. The coverage was there and the table did not know it:
the `vminstance` suite creates a `VMDisk` and asserts the `DataVolume`
behind it (`hack/e2e-chainsaw/vminstance/vmdisk.yaml`,
`vmdisk-vmi.yaml`). Same defect class as the etcd edge above, on the
mapping side rather than the graph side, and with no production blast
radius of its own, since `src_to_suites` is read by the selector and by
nothing else.

**`cozystack.cozystack-basics` joins `cozystack.cozystack-engine` as a
propagation hub excluded from the reverse-dependency walk.** Every edge
into it exists so a namespace or a platform-wide policy is in place
before the dependent installs, which `kubevirt-cdi` states in its own
source ("Depend on cozystack-basics so the target namespace exists
first"), and that is install ordering rather than behaviour. The damage
runs opposite to the engine's: the engine fans one change out to every
app, basics narrows instead. It reaches no suite today so it escalates,
but it sits upstream of `kubevirt-cdi`, so the first suite to land under
CDI silently converts the platform's namespace-and-policy package from
the full run to that one suite. Not hypothetical — on #3426 enabling the
site-router suite takes `cozystack-basics` from 22 suites to
`site-router` alone, and nothing in the output says coverage was lost.
Dropping the reverse edges keeps the package reachable, stops it
propagating, and leaves a change to it running everything through the
per-path escalation.

**`hack/e2e-apps/<name>.bats` maps to the `<name>` suite**, off the
basename, exactly as the per-suite rule takes the name off a
`hack/e2e-chainsaw/<app>/` directory. Deliberately mapped rather than
marked inert: what remains there is wired to nothing after the Chainsaw
migration, and inert would bake that orphan status into the rule and go
quietly wrong the day a lane runs those files again. The basename is
membership-tested against the suite list on the spot, so an unmatched
one escalates there and then rather than having the verdict deferred to
the final intersection, where the rest of the diff would decide it.

**`packages/tests/` and three `.gitattributes` join
`inert_config_pattern`,** which is what the script's own header says to
do with a genuinely inert path instead of widening the fall-through.
`packages/tests/` is a helm-unittest fixture chart, and changing a test
*of* `cozy-lib` does not change `cozy-lib`, no PackageSource lists those
paths as a component, and nothing installs them, while a change to the
library itself still escalates through `packages/library/`. The
`.gitattributes` entries are enumerated rather than matched by filename,
because the justification is what those files contain (only
`linguist-generated` markers) and the name does not carry it:
`.gitattributes` can also set `filter`, `eol`, `working-tree-encoding`
and `export-subst`, each of which changes what lands in the working tree
and therefore what gets built.

**`full_suite_pattern` escalates only `hack/e2e-*.bats`, not every
`hack/*.bats`.** The root `Makefile` is the authority on the split and
draws it at exactly that prefix, `BATS_UNIT_FILES := $(filter-out
hack/e2e-%.bats,$(wildcard hack/*.bats))`, so the 60 files it keeps are
the unit lane and the e2e sandbox runs none of them, while the three it
filters out are what `packages/core/testing`'s recipes execute and those
still escalate. Being inert here does not leave them untested: `make
unit-tests` is gated on the `plan` job's `code` output, which
`pull-requests.yaml` computes as "any changed path outside `docs/`" and
never from `select-e2e.sh`, so a bats-only pull request still runs the
whole unit lane, plus install and the OpenAPI tests, since the `e2e` job
reads the same output rather than the selection.

## Measured effect

The before-and-after comparison below is a second measurement over a
slightly different population, stated separately because the two are not
interchangeable: it replays the last 150 first-parent merge commits on
`main` (`git diff --name-only $sha^1 $sha`) through the base selector
with base sources versus this branch with its own, where the histogram
above walks the 150 merged pull requests as GitHub lists them. The
populations overlap heavily and the direction is the same, and the
totals differ by two commits.

| | full suite | scoped | nothing |
|---|---|---|---|
| before | 120 | 20 | 10 |
| after | 106 | 25 | 19 |

Full-suite rate 80.0% → 70.7%, verdict changed on 15 commits. Fourteen
de-escalated and I read every one of their file lists: they are
unit-lane bats files, non-e2e workflows, `docs/`, `CODEOWNERS`,
helm-unittest fixtures and linguist markers. The fifteenth gets one
suite *wider*: a `cert-manager` change now also selects `etcd`, which is
correct, since `cert-manager` is a dependency of `etcd-operator` and the
etcd app now genuinely reaches it through the new edge.

That replay predates the two reachability rules, and replaying the same
150 first-parent merges across those two on their own — the selector at
`73bc4c0ae` against the selector at `adf231513`, same sources on both
sides — changes no verdict at all. Two of the 150 touch
`cozystack-basics` and both ran the full suite before and after, which
is the hub exclusion keeping something true rather than failing to do
anything, and none of the 150 touch `kubevirt-cdi` or `vm-disk`, so the
mapping shows up only in a direct selection.

A few individual selections, before → after:

| changed files | before | after |
|---|---|---|
| `hack/select-e2e_test.bats` | 21 suites | *nothing* |
| `hack/select-e2e_test.bats` +
`packages/apps/redis/templates/redis.yaml` | 21 suites | `redis` |
| `hack/e2e-install-cozystack.bats` | 21 suites | 21 suites |
| `packages/tests/cozy-lib-tests/**` | 21 suites | *nothing* |
| `packages/system/.gitattributes` | 21 suites | *nothing* |
| `hack/e2e-apps/postgres.bats` | 21 suites | `postgres` |
| `packages/system/etcd-operator/values.yaml` | 21 suites | `etcd` |
| `packages/system/kubevirt-cdi/values.yaml` | 21 suites | `vminstance`
|
| `packages/apps/vm-disk/values.yaml` | 21 suites | `vminstance` |
| `packages/system/cozystack-basics/values.yaml` | 21 suites | 21 suites
|

## Blast radius of the `dependsOn` edge

A `dependsOn` on a PackageSource is an install-ordering edge in
production, not only a test-selection hint, so this is the part worth
reviewing hardest. Both Packages are emitted by
`packages/core/platform/templates/bundles/system.yaml` under the single
`bundles.system.enabled` guard with no intervening conditional between
them, in every variant, so no supported configuration has the app
without the operator. `hack/select-install.sh --validate` reports the
graph still has no cycle and nothing dangling. The one new exposure is
an admin who lists `cozystack.etcd-operator` in
`bundles.disabledPackages` while keeping the app, which now leaves it
`DependenciesNotReady`. That is the same exposure postgres, mariadb,
kafka and redis already carry, with `Package.spec.ignoreDependencies` as
the escape hatch. On upgrade `etcd-rd` waits for the operator's two
HelmReleases, and an unhealthy `etcd-operator` already means broken etcd
apps.

## Testing

Every rule added or changed has a test in `hack/select-e2e_test.bats`,
the install-side change has one in `hack/select-install_test.bats`, and
the new graph edge has a helm-unittest guard at
`packages/core/platform/tests/sources_etcd_application_dependson_test.yaml`.
The round-trip test that walks every suite through both mapping tables
stays green, `vminstance` included now that two sources map onto it.

The two reachability rules were each checked by removing the rule and
confirming its own test reddens: without the mapping the CDI test falls
back to the full-suite escalation, and without the hub exclusion
`cozystack-basics` selects `redis vminstance` instead of everything. The
hub test has to seed its own hazard, since no suite sits downstream of
basics in this tree yet, so it gives `redis-application` an edge onto
basics in a copied sources directory and pins that a redis change still
selects redis, or the assertion would pass against a graph that resolved
nothing.

`make bats-unit-tests` aborts on the first failing file (`for f in …; do
… || exit 1; done`), and `hack/ghcr-mirror_test.bats` fails on `main`
today and sorts before `select-e2e_test.bats`, so the aggregate target
never reaches these tests. I ran the 60 unit files individually instead:
59 pass, and the one failure is that pre-existing one, byte-identical to
the base commit and referencing nothing this branch touches. Also green:
`make helm-unit-tests`, `make rd-presets-check migrations-target-check
test-check-readiness`, `hack/select-install.sh --validate`, and
`pre-commit run --all-files` leaving a clean tree.

## Two things a reviewer should know, neither addressed here

`assert_selection` in `hack/select-e2e_test.bats` aborts the whole file
with `exit 1` rather than failing a single test. That is the only thing
that works under `hack/cozytest.sh`, which provides no real `run`,
`$status` or `skip`, and it is the idiom the existing tests in that file
already use. It becomes wrong when #3497 and #3498 flatten these onto
real `bats(1)`, where a failing assert should end one test and let the
rest report. Worth converting with that work rather than ahead of it.

`cozystack.mongodb-application` carries the identical missing operator
edge: `packages/apps/mongodb` renders `psmdb.percona.com/v1` while the
source depends only on `cozystack.networking` and
`cozystack.cozystack-engine`, so `cozystack.mongodb-operator` still
reaches no runnable suite and every change to it escalates to all 21.
Same defect class and same fix shape as the etcd edge, but it is a
second production install-ordering change and belongs in its own pull
request.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- The default etcd application now includes its operator and required
platform dependencies, ensuring the EtcdCluster provider is available.

- **Bug Fixes**
- Improved end-to-end test selection for application-specific changes,
shared components, inert paths, and unresolved or unclassified files.
- Full-suite escalations now provide clear reasons through standard
error.

- **Tests**
- Expanded coverage for test selection, dependency reachability, and
etcd application dependency declarations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-20 14:12:56 +03:00
IvanHunters f035da8a0f Merge branch 'main' into feat/kubernetes-nodes-phase2b 
Resolve conflicts introduced by main:
- Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in
  favour of the GPU type main added; main removed the fluxcd addon wholesale.
- Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's
  redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57.
- Accept the branch's removal of the in-chart talos-reconcile Job; it moved to
  the kubernetes-nodes chart, main only touched its comment.
- Regenerate kubernetes values.schema.json / README and the kubernetes-rd
  ApplicationDefinition schema.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-20 14:04:32 +03:00
Aleksei Sviridkin 467e431765 chore(ci): add area/cilium label and map the cilium PR scope to it 
Cilium has enough open issues of its own (version pins and bumps, the
Gateway API implementation, network policy, IPAM next to kube-ovn,
LB-IPAM) that area/networking no longer narrows a search. Declare
area/cilium in labels.yml, route the cilium scope of PR titles to it
instead of area/networking, and update the scope table in the
contributing guide.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-19 16:28:50 +03:00
IvanHunters cd72d81bb4 Merge branch 'main' into feat/kubernetes-nodes-phase2b 
Reconcile the phase-2b split with main's configurable worker-health
feature (MHC condition timeouts, maxNodeProvisionTime, podCpuLimit/
podCpuRequest), which landed on both charts after this branch forked:

- kubernetes (parent): pool rendering stays removed; main's parent
  worker-pool additions do not apply post-split.
- kubernetes-nodes (pool): absorb the feature; keep the directional
  parent-version skew guard and the immutable cluster/storageClass rules.
  KubevirtMachineTemplate content hashes are unchanged (render snapshot).
- migration 54: carry podCpuLimit/podCpuRequest (KMT-hash inputs) and the
  cluster-wide readyUnknownTimeout/readyFalseTimeout/maxNodeProvisionTime
  into adopted pools, so adoption neither rolls nor resets a tuned pool.
  Covered by a new adopt-values bats case.
- drop the parent chart's obsolete MHC-timeout suites; the
  kubernetes-nodes twins cover the feature.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-19 13:14:11 +03:00
Myasnikov Daniil 12e892325a docs(release): warn that v1.6.0's naming audit fails open 
v1.6.0's own upgrade notes tell operators to run
hack/seaweedfs-naming-audit.sh, and the copy that shipped in v1.6.0 silences
every kubectl call with 2>/dev/null. A timeout or an RBAC denial therefore
prints an empty "all clean" table indistinguishable from a genuinely clean
fleet, and that output gates a runbook step that deletes PVCs. Anyone
following those notes today can still get a false clean.

Verified per tag: v1.6.0 has 11 blanket redirections and no run_kubectl
wrapper; v1.6.1, v1.6.2 and v1.5.4 have 4 and 18 respectively, the
fail-closed shape from #3436 and its backport #3474.

Published changelogs are historical records and are not normally edited.
This one is an exception on purpose, because the instruction it carries is
still live and still wrong.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-19 14:59:43 +05:00
Myasnikov Daniil eb8c19d4a1 docs(release): enrich the v1.5.4 changelog 
The generated changelog is a good inventory but carries no operator-facing
layer: no required-actions section, no runnable pre-upgrade checks, and the
SeaweedFS 4.31 rename left as three isolated fix bullets that never say what
to do about it. Every generated bullet is preserved; this adds to them.

Widen the base to v1.5.2. v1.5.3 was tagged but its release was never
published and its changelog PR was closed unmerged, so docs/changelogs/
v1.5.3.md exists on no ref and a v1.5.3 base leaves #3212 (persistent
EFI/TPM state) and #3194 (filer postgres2 connection pool) documented
nowhere at all. Coverage is now 82/82 commits in range.

Correct three claims. The slot 45 divergence is a skip rather than an
ordering problem: v1.5.4 is the first 1.5.x stamped targetVersion 46, and
run-migrations.sh loops seq CURRENT (TARGET - 1), so a cluster at 46 runs
46..53 on the way to 1.6 and never executes 1.6's own slot 45. Its chart-side
half was missing too, and it is not redundant -- a fresh v1.5.4 install is
stamped 46 having never run any slot, so only the chart can reach it. The
S3 checksum bullet named a barman-cloud plugin path that does not exist on
this branch.

The audit re-run warning names v1.6.0, not earlier releases. The script does
not exist at v1.5.2; v1.5.4 is the first 1.5.x to carry it and carries the
fail-closed version. v1.6.0 shipped the copy that silenced every kubectl
failure, so a clean table from it is not evidence.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-19 14:59:03 +05:00
cozystack-ci[bot] c6238e1544 docs: add changelog for v1.5.4 
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
 2026-08-19 09:38:23 +00:00
cozystack-ci[bot] be6e66786f docs: add changelog for v1.6.2 
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
 2026-08-19 07:56:28 +00:00
Aleksei Sviridkin b76cb2091c test(e2e): record the runner VM shape and region in the e2e report 
An e2e lane asks for a runner class, not for a machine. A flex shape is
filled from whatever capacity the pool has at the time, so one lane can
be served by two processor generations, and the region is a preference
with a fallback. None of that reached the report, so a red run and a
green one could not be compared by the hardware they ran on.

The read is taken on the runner VM, before the sandbox container is
created, because the metadata service answers on a link-local address
the container has no guaranteed route to: an "unreachable" answered from
in there would look exactly like a capture that had tried. It is wired
into the sandbox apply target rather than into a workflow step, so every
lane picks it up through `make prepare-env` on both the passing and the
failing path, and cozyreport folds the file into the artifact from
inside without going near the endpoint.

The keys are fetched one at a time and the root instance document never
is. That document carries the instance and compartment OCIDs, the image
OCID, the display name, the hostname and the user metadata bag, and the
report is a public artifact, so asking per key makes that property
something the wire settles rather than something a filter has to be
trusted for. It also matches where the shape sizing is published: it has
its own subpath and is not in the root document at all. From the sizing
object only the numbers are kept, so the rule holds for keys nobody has
seen yet, and an answer that is not one scalar is named rather than
printed.

Every read carries a wall-clock ceiling, since a read that never returns
costs the step its whole budget without reaching any exit status, and the
first key doubles as the probe so a machine that is not an OCI instance
pays one ceiling rather than one per key. Nothing here is fatal: this
runs inside the one retried step in the job, so a runner with no
metadata service leaves an "imds unreachable" line and an exit status
of 0.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-18 03:31:42 +03:00
IvanHunters 5297df5cf3 Merge remote-tracking branch 'origin/main' into pr-3315-lgtm 
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>

# Conflicts:
#	api/apps/v1alpha1/kubernetes/types.go
#	api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go
#	hack/e2e-chainsaw/_lib/run-kubernetes.sh
#	packages/apps/kubernetes-nodes/tests/render-parity.sh
#	packages/apps/kubernetes/README.md
#	packages/apps/kubernetes/templates/cluster.yaml
#	packages/apps/kubernetes/values.schema.json
#	packages/apps/kubernetes/values.yaml
#	packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
 2026-08-17 11:25:29 +03:00
myasnikovdaniil 9f5550abdd fix(release): publish rewritten packages during RC promotion (#3795) 
## What this PR does

Fixes #3477.

- Publishes a new packages OCI candidate after rewriting RC image tags
to stable tags.
- Pins the candidate digest in the stable installer without rebuilding
container images.
- Verifies the candidate against the prospective merge and original RC
artifact before creating stable names.
- Refuses promotion when the target maintenance branch lacks the
candidate-aware pipeline.
- Protects candidates referenced by open release PRs from retention
cleanup.
- Leaves already published v1.6.0 and v1.6.1 unchanged.

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Testing

- `bats --no-tempdir-cleanup hack/promote-packages-artifact_test.bats
hack/verify-promoted-packages_test.bats
hack/promote-packages-contract.bats
hack/promotion-retention-contract.bats`
- `actionlint .github/workflows/promote-rc.yaml
.github/workflows/pull-requests.yaml
.github/workflows/pull-requests-release.yaml
.github/workflows/retention.yaml`
- `sh -n hack/promote-packages-artifact.sh
hack/verify-promoted-packages.sh`
- `git diff --check origin/main...HEAD`

### Release note

```release-note
fix(release): publish the rewritten stable packages artifact when promoting a release candidate
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Release candidates now use temporary, immutable package artifacts
verified before stable publication.
- Stable promotions validate package contents, installer references,
container digests, and release metadata before tagging.
- Temporary promotion artifacts are retained while needed and cleaned up
after expiration.

- **Bug Fixes**
- Improved handling of release labels, fork pull requests, and promotion
pull requests.
  - Added end-to-end verification before release promotion.
- Prevented invalid, ambiguous, or untrusted package references from
being promoted.

- **Documentation**
- Updated release and promotion guidance for candidate verification,
digest pinning, and transactional promotion.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-17 12:52:19 +05:00
Myasnikov Daniil adf231513f fix(ci): correct two reachability defects in the selection graph 
Both are cases of the walk reaching the wrong set of suites for a
package, and both were found by enabling the site-router suite on #3426,
which is the first suite in the tree to land downstream of kubevirt-cdi.

`vm-disk-application` now maps to the `vminstance` suite. It mapped to a
`vm-disk` suite that does not exist, which intersect_suites() drops, so
the source reached nothing runnable and neither did kubevirt-cdi above
it: every CDI change ran all 21 suites. The vminstance suite creates a
VMDisk and asserts the DataVolume behind it
(hack/e2e-chainsaw/vminstance/vmdisk.yaml, vmdisk-vmi.yaml), so it is the
suite that covers both. A CDI change now selects `vminstance` alone.

`cozystack.cozystack-basics` joins `cozystack.cozystack-engine` as a
propagation hub excluded from the reverse-dependency walk. Every edge
into it exists so a namespace or a platform-wide policy is in place
before the dependent installs -- kubevirt-cdi says exactly that in its
own source -- which is install ordering, not behaviour. The damage runs
opposite to the engine's: the engine fans one change out to every app,
while basics narrows down instead. It reaches no suite today so it
escalates, but it sits upstream of kubevirt-cdi, so the first suite to
land under CDI silently converts the platform's namespace-and-policy
package from the full run to that one suite. On #3426 that is 22 suites
down to `site-router` alone, with nothing in the output to say coverage
was lost. Dropping the reverse edges keeps the package reachable, stops
it propagating, and leaves a change to it running everything through the
per-path escalation.

Both rules have a test, each verified to redden with its rule removed:
the mapping test falls back to the full-suite escalation, and the hub
test -- which seeds the hazard by giving redis-application an edge onto
basics, since no suite sits downstream of it in this tree yet -- selects
`redis vminstance` instead of everything.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-16 10:44:44 +05:00
Aleksei Sviridkin cf2d9c6d13 test(e2e): split the tenant worker CPU time across QEMU threads 
The node-join diagnostics phase samples the tenant worker's CPU
counters twice on failure, but it reads them from the compute
container's cgroup, and that cgroup holds every thread QEMU runs:
the guest's vCPUs, QEMU's main thread, and its IO and worker
threads. A container burning most of its quota while the guest
reports no progress is two incompatible findings wearing one
number, and no container-level counter separates them.

Add a collector that execs into each Running worker's compute
container, finds the QEMU process and copies every
/proc/<pid>/task/<tid>/stat line into the artifact verbatim, with a
legend beside them. Nothing is computed here. The legend names
where utime and stime sit and why counting fields over whitespace
reads the wrong column for exactly the threads the split is about:
a KVM vCPU thread is named "CPU 0/KVM", that space is inside the
comm parentheses, and every field after it shifts by one.

It runs inside the sampling loop the two existing subjects already
share, so it costs no second wait: both readings ride the interval
already being paid, and the rate comes from the two files plus the
stamps each one carries.

Every way the read can come up short is a named outcome rather than
an empty directory, and the legend and the pairing instruction are
withheld from each of them: telling a reader to subtract two files
asserts that both hold a whole reading. The probe shells out to
nothing but getconf, whose absence it names, because a probe that
needed cat would report a container missing coreutils as a
container whose QEMU had exited, and it counts what it managed to
read so that having read no process at all stays separable from
having read them all without finding QEMU. Its stderr is suppressed
on the walk rather than on either read: redirections are applied
left to right, so a 2>/dev/null after an input redirect is not in
place yet when opening the input fails, and a thread that exits
between the glob and the read would otherwise put a shell
complaint in the artifact beside a healthy capture. It also reports
a QEMU that was named and then gone before its task files could be
opened, because a process heading alone is enough to make the
capture non-empty and would otherwise carry a legend describing
columns the file does not have.

The Pod listing the cAdvisor captures already share now takes its
whole query as a parameter, so this collector reuses that read
rather than carrying a third copy of it, and asks for the Running
Pods only: a launcher left over from an earlier attempt costs the
sibling caller nothing and would cost this one a bounded exec and a
slot under its cap. The guard that holds the sampling group against
the room the tenant snapshot needs now counts every capped walk in
the loop rather than the two spelled the same way, so the new
subject is inside the arithmetic rather than beside it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-16 05:17:16 +03:00
Aleksei Sviridkin 959e90c9ab test(e2e): time the guest console capture and baseline it on a green run 
A tenant worker's guest console is stamped in seconds since that guest's
own kernel started, so a log ending mid-boot says how far the guest got
and nothing about when. Whether it ends there because the guest started
late or because the guest stopped printing decides which failure is being
looked at, and neither instant was recorded: the read's own instant was
nowhere in the file, surviving only as the artifact's mtime, which is
metadata rather than content and so says nothing to anyone reading the
log; and the guest's start only inside a describe dump that a capture
with no silent console never triggers. Across three collected runs every
capture exited zero and every log began at the first kernel line, so the
reads were complete at both ends and the logs end where the guests
stopped -- and which of the two happened could not be read off any of
them.

Each console log now carries the instant its own read started and
returned, and CAPTURE-TIMELINE.txt beside it carries each Pod's phase,
its start time and the start of its compute and console containers, from
one bounded read taken once for the whole selector. Those subtract to how
long a guest had been alive when it was read, which is what the log's
last kernel stamp is then compared against. The console container is
asked for in both status lists, since KubeVirt runs it as an init
container and reading only one of them would report a container that
started fine as one that never ran. A read that failed, a read cut off
part way and a read that answered with nothing are recorded as
themselves rather than left as a missing file, and the note names the
error log only when there is one.

The same capture also runs on the passing path, before the tenant is torn
down and the Pod holding the stream goes with it. Every console this tree
has collected came from a failure, so what a healthy boot looks like has
never been measured, while "slower than usual" is the claim the failure
keeps resting on. That caller asks for the pool's minimum rather than its
maximum, since two workers booting is the same measurement as ten, and it
warns in the job log rather than failing a suite that has already proved
everything it exists to prove. Both paths write the same directory, so
the capture also records which of them produced it.

That capture costs real wall clock at the tail of an operation with a
fixed ceiling, which is a way for a run that passed to be killed while
collecting a debugging aid. So it is declined out loud when the ceiling
no longer has room for it, against a reserve derived from its own cost
and held in step with the timeout both suites give the operation.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-15 23:33:33 +03:00
Myasnikov Daniil c28d7804af fix(ci): enumerate the inert .gitattributes instead of matching by name 
The rule was broader than its justification. It read `(^|/)\.gitattributes$`,
so it claimed every present and future .gitattributes anywhere, while
the argument for it was about content: every one in this tree holds
nothing but `linguist-generated` markers, which reach no build, chart or
test.

The filename does not carry that argument. .gitattributes can also set
`filter`, `eol`, `working-tree-encoding` and `export-subst`, each of
which changes what lands in the working tree and therefore what gets
built, so a by-name rule would eventually make a live file inert with
nothing to notice it.

Nothing is lost by narrowing, because only three files ever reached the
rule: api/.gitattributes and internal/crdinstall/manifests/.gitattributes
are escalated by full_suite_pattern first, so entries for them would be
dead. Measured against the base selector, those three are exactly the
paths whose classification this branch changes, and they still select
nothing.

Same trade as the workflow enumeration a few lines up, and the same
upkeep, which the comment now states: after adding or moving one,
re-check with `git ls-files '*.gitattributes' '.gitattributes'` and
confirm the contents are still only linguist markers. A .gitattributes
not on the list is classified by whatever rule its path falls under --
the graph inside a package, the unclassified fall-through at the root --
and both fail safe. A test pins that root case, so a future widening
back to a by-name match turns the file red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-15 17:58:08 +05:00
Myasnikov Daniil b2702621d3 fix(ci): decide the per-app bats rule per path, not after the merge 
Two defects in the hack/e2e-apps/ arm added by the previous commit.

The arm set trigger_any=1 for whatever basename it derived and left the
verdict to the final intersection. That makes the escalation depend on
the REST of the diff: hack/e2e-apps/monitoring-oidc-system.bats alone
emptied the selection and the backstop escalated, but the same file
beside packages/apps/redis/templates/redis.yaml selected `redis` with
nothing on stderr, where the pre-change selector ran all 21 suites.
That is the merge-before-escalate shape #3330 removed from the graph
walk, reintroduced one rule over. The basename is now
membership-tested against the suite list on the spot and an unmatched
one sets trigger_full=1 and names the file.

POSIX case matches `/` with `*`, so the arm also sees nested paths, and
the unanchored capture turned hack/e2e-apps/fixtures/postgres.bats into
the "suite" fixtures/postgres -- right outcome by accident, reason line
printing a path where a suite name belongs. The capture is anchored to
one segment, and what the arm does with anything that is not a
top-level <name>.bats is now stated rather than left to fall out of a
regex: it is shared material for those suites the way
hack/e2e-chainsaw/_lib/ is for the Chainsaw ones, so it escalates for
the same reason.

Both claims that said otherwise are corrected -- the arm's own comment
and docs/agents/e2e-testing.md both asserted the deferred escalation
was equivalent to the old fall-through, which held only for a
single-file diff. The backstop comment drops the e2e-apps clause it
gained last commit, since that route no longer reaches it. The header's
"four of the seven" becomes count-free: this commit adds two escalating
branches, so a hardcoded total was already stale.

Tests: the mixed-diff case is the regression pin (the isolated case
passes with the escalation deferred and cannot see the bug), plus a
nested-path case and a non-.bats case. Both fixes are
mutation-verified -- restoring either defect turns the file red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-15 17:58:08 +05:00
Myasnikov Daniil 4d4f5edf92 fix(ci): escalate only the e2e bats files, not the whole unit lane 
full_suite_pattern took every hack/*.bats, so all 63 of them ran the
whole Chainsaw suite -- including the 60 that are the unit lane and
that the e2e sandbox never executes.

The root Makefile is the authority on the split and draws it at
exactly the prefix this narrows to:

  BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))

The 60 files it keeps run under `make unit-tests`; the three it
filters out -- e2e-prepare-cluster, e2e-install-cozystack,
e2e-test-openapi -- are the ones packages/core/testing's recipes
execute inside the sandbox, and those still escalate. So no Chainsaw
suite can regress from an edit to one of the 60.

Inert here is not a green gate with nothing behind it. `make
unit-tests` is gated on pull-requests.yaml's `code` output, computed
in the workflow as "any changed path outside docs/" and never from
select-e2e.sh, so a bats-only pull request still runs the whole unit
lane. The `e2e` job reads the same output, so it still installs the
platform and runs the OpenAPI tests before skipping Chainsaw.

Two tests: one pins the rule on both sides of the prefix and asserts
an inert bats file beside an app path does not mask the selection, and
one walks every hack/*.bats in the tree and requires the selector's
verdict to agree with the Makefile's split, so a file named to fit
neither lane surfaces here rather than on the next PR that touches it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-15 17:58:07 +05:00
Myasnikov Daniil 33428671fa fix(ci): classify hack/e2e-apps/ and packages/tests/ in select-e2e 
Three path classes matched no rule in hack/select-e2e.sh and escalated
to all 21 Chainsaw suites through the unclassified fall-through -- 7
of the last 150 merged pull requests between them.

hack/e2e-apps/<name>.bats now maps to the <name> suite, off the
basename exactly as the per-suite rule takes the name off a
hack/e2e-chainsaw/<app>/ directory. Not marked inert: what is left in
that directory after the Chainsaw migration is wired to nothing, and
inert would bake that orphan status into the rule and go quietly wrong
the day a lane runs those files again. A basename no suite carries
drops out of the intersection at the bottom of the script and
escalates through the backstop, which is the outcome those paths
already had, so mapping optimistically loses nothing and gains the
correct answer for every name that is a suite.

packages/tests/ and every .gitattributes join inert_config_pattern,
which is what the script's own header says to do with a genuinely
inert path rather than widening the fall-through. packages/tests/ is a
helm-unittest fixture chart: changing a test OF cozy-lib does not
change cozy-lib, no PackageSource lists those paths as a component,
and nothing installs them -- a change to the library itself still
escalates through packages/library/. .gitattributes is matched by name
wherever it lives because every one in this tree is a
linguist-generated marker, which reaches no build, chart or test; one
under a tree full_suite_pattern already covers still escalates, since
that pattern is checked first.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-15 17:58:07 +05:00
Myasnikov Daniil 3c3f9450ae fix(ci): name the cause on every select-e2e escalation 
Seven branches in hack/select-e2e.sh escalate to the full Chainsaw
suite and four of them did it in silence: the shared
hack/e2e-chainsaw/_lib/ and .chainsaw.yaml case, the
full_suite_pattern match, a packages/ path no PackageSource claims,
and the empty-selection backstop.

"Run everything" is the same 21 suite names whichever rule produced
it, so a reason line is the only thing that tells the seven apart.
full_suite_pattern is the commonest cause of a full run by a wide
margin -- 88 of the last 150 merged pull requests, 68 of them with no
other cause -- so the usual answer to "why did this pull request run
the whole suite" was the one the log never carried, recoverable only
by re-deriving the selection by hand.

Each new line names the path that caused it rather than only the rule,
and the backstop names the directly-selected names it could not
resolve. They go to stderr, never stdout: stdout is the suite list and
both e2e lanes parse it, so a reason line there would be read as a
suite name.

A unit test asserts each of the four messages. A reason that regresses
to silence changes no selection, so it is invisible to every other
test in the file.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-15 17:58:07 +05:00
Aleksei Sviridkin ef96292b64 fix(ci): give the nightly e2e runner headroom over its guest vCPUs (#3833) 
## What this PR does

The e2e sandbox boots three QEMU nodes at `-smp 8`, so 24 guest vCPUs
are committed before any tenant VM exists. Four jobs boot that sandbox.
Three run on the 32-vCPU class; nightly ran on the 24-vCPU one, where 24
guests against 24 host vCPUs leaves nothing for QEMU's own emulation and
IO threads, containerd, the kernel and the runner agent. The other lanes
moved to the larger class at some point and nightly was not moved with
them.

The sandbox keeps its 24 vCPUs. That guest count is part of what e2e
claims to test, and a smaller nightly cluster stops matching the one PRs
run against.

This is hygiene, not a fix. Measured boot-phase data across 18 runs says
the load a workflow puts on the host dominates guest starvation, not the
runner class: the lanes already on 32 vCPU boot guests slower than
nightly did on 24, and class and workflow kind are fully confounded
there, so neither separates out. Do not read this as making anything
faster or as addressing a particular red run. What is left is the
arithmetic. A 1:1 commit has no headroom at all, and two lanes running
the identical sandbox on different classes is drift nothing was
watching.

The rest is a check, because the guest demand and the runner classes
live in different files and nothing compared them.
`hack/sandbox-runner-headroom.bats` reads `-smp` and `-m` times the loop
count out of `hack/e2e-prepare-cluster.bats`, reads the runner class of
every job that calls the sandbox make targets, and wants the runner
strictly larger on both axes. Neither side is a hardcoded list, so
bumping `-smp`, adding a management node, or adding a fifth lane on a
smaller runner turns it red on the same commit. Memory is in there
because the same argument applies to it and `-m 24576` sits on the same
line as `-smp 8`, not because a class that violates it exists today.

It stops instead of guessing when it cannot read something, since a
guess comes out smaller than the real demand and passes a runner that
cannot hold the sandbox. Refused: a missing or non-literal `-smp` /
`-m`, a launch written outside any `@test` block or inside one left open
at end of file, a `for` opener the strict pattern does not match, a
`for` list holding anything but literal words, a `while` sharing the
block, two launches or two loops in one block, and a `runs-on` with no
`oracle-vm-<n>cpu-<m>gb` in it. Unparsed loop openers are counted
separately from parsed ones, so moving `do` to the next line is refused
rather than silently multiplied by one. There is also a cross-check that
every workflow calling the sandbox make targets yields at least one
matched job, so a renamed target turns the parser red instead of quietly
shrinking what it covers.

The derivation runs against the live tree, which passes whether or not
those refusals still work, so each one is pinned by a fixture that feeds
a synthetic file to the same program. That matters more than it sounds:
two of the refusals above exist because the first version of this file
did not have them, and a launch moved into a file-level helper would
have halved the measured demand and stayed green.

The numbers come from:

```bash
sed -n '98,107p' hack/e2e-prepare-cluster.bats    # three nodes, -smp 8, -m 24576
grep -rn 'prepare-env' .github/workflows/         # the four jobs that boot it
grep -rn 'runs-on: oracle-vm-' .github/workflows/ # their runner classes
```

The rule also goes into `docs/agents/e2e-testing.md` next to the other
environment invariants, so whoever adds the fifth lane picks its runner
class from the docs instead of from a red unit suite.

Checked that it bites, twice over. Fourteen mutations of the inputs:
reverting the runner class, `-smp 11`, a fourth and fifth management
node, `-smp cpus=8`, `-m size=24576`, `do` on the next line, `$(seq 1
3)` as the list, a `while` loop, the launch moved into a file-level
helper, renaming the make target, `ubuntu-latest`, a matrix expression
in `runs-on`, a hypothetical `32cpu-64gb` class, and a new lane on a
4-vCPU runner. Then eleven mutations of the parser itself, one per
refusal, each of which kills exactly the fixture that names it and
nothing else. Green on this tree, and the same numbers out of gawk, mawk
and BSD awk.

Closes #3831.

### Screenshots

No UI change.

### Downstream repositories

Walked the trigger map against the diff. It changes one `runs-on` line
plus a comment in `.github/workflows/nightly.yaml` and adds one file
under `hack/`. The `ccp` trigger is moving or renaming something under
`hack/`, or changing what a make target does; `hack/package.mk` and
`hack/common-envs.mk` are untouched, and the new test is picked up by
the existing `hack/*.bats` wildcard without the target changing. The
`ansible-cozystack` and `talm` triggers are node prerequisites in
`hack/e2e-prepare-cluster.bats`, which this reads and does not modify.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(ci): run the nightly e2e suite on a runner class larger than the sandbox's guest vCPU demand, and add a unit check that fails when an e2e lane loses that headroom
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Updated nightly end-to-end testing to run on a higher-capacity runner,
improving support for multiple QEMU guests and required resource
headroom.
* Added validation to prevent sandbox runners from being undersized or
oversubscribed.

* **Documentation**
* Documented the resource requirements and deterministic conventions for
end-to-end sandbox testing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-15 12:59:55 +03:00
Aleksei Sviridkin 527d4ea00f test(e2e): measure tenant worker CPU as a rate, and the sandbox nodes under it 
Two instruments for the node-join failure the kubernetes-* suites keep
hitting. Neither changes what the suites assert.

The worker CPU capture now reads container_cpu_usage_seconds_total
beside the CFS counters, and reads them twice a fixed interval apart
instead of once. The throttled counters say a container met its ceiling
and cannot say how much CPU it got, which are the two candidate
explanations for a stalled worker: a guest held at a one-core ceiling,
and a guest whose vCPU thread is rarely scheduled at all. Both leave
throttled periods behind. Every counter involved accumulates from
container start, so a single reading is an average over the whole uptime
and no rate at all, and the profile under investigation is a guest that
freezes for tens of seconds and then runs flat out: the average comes
out low, the instantaneous figure comes out at the cap, and only the
second separates it from a guest that is simply capped.

The sandbox nodes are a new subject. The worker captures describe a
cgroup, so a guest that is runnable and never scheduled looks from there
like a guest that asked for nothing. Whether the sandbox node itself was
given its turn is recorded only in the steal column of its own
/proc/stat: no kubelet endpoint carries it, and crust-gather cannot
reach it because PodSecurity refuses its node-shell Pod here, so the
read goes over the Talos API, which is already how the report reaches
those nodes for dmesg. The rows are captured verbatim with their column
legend beside them rather than reduced to a percentage, because
/proc/stat has no header, steal and guest sit next to each other, and
guest is large on a node running VMs and already counted inside user.

Both subjects are read on either side of one shared wait, so the pair
pays it once. The wait is not the interval either subject's counters
span, though: each pass walks its own nodes and the other subject's pass
falls between a subject's two readings, and on the run this exists for
those passes are the slow part. Each capture therefore stamps both ends
of its read, so a rate comes from subtracting stamps rather than from
assuming the knob, and the width of those brackets is the uncertainty.
That wait is a measurement interval rather than a stand-in for a
condition nobody wrote a wait for, so the e2e conventions carve it out
by name instead of marking it as a fixed timeout to be replaced.

The pair is gated below the serial console rather than above it. Its
ceiling is sixteen bounded reads plus the interval, which is most of the
phase budget, and admission gates only when a collector may start, so
above the console a run that hit those bounds would leave the console
never started rather than merely cut short. The console is the only
capture that survives a worker which never reached apid, which is the
shape this failure usually takes. That overrun is inside the term the
budget derivation already allows for one late admission, and a guard
holds the pair's ceiling against it.

No collector here may leave an empty directory, since an empty directory
reads as a healthy node: a missing binary, a missing talosconfig, a
listing that never answered, a node with no address and a read that was
cut off each land as a named note or a COLLECTION-FAILED.txt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-15 10:19:19 +03:00
Aleksei Sviridkin db2ddfab2b fix(ci): give the nightly e2e runner headroom over its guest vCPUs 
The e2e sandbox boots three QEMU management nodes at `-smp 8`, so 24
guest vCPUs are committed before any tenant VM starts. The nightly lane
ran that sandbox on a 24-vCPU runner, a 1:1 commit that leaves the host
nothing for QEMU's emulation and IO threads, containerd, the kernel and
the runner agent. The tag, fork and pull-request lanes already run the
same sandbox on the 32-vCPU class, so move nightly there too.

This aligns headroom and claims no performance effect. Measured boot
data shows the load a workflow puts on the host, not the runner class,
dominates guest starvation, and the two are confounded in that data; no
red run is attributed to the class here. The case is the arithmetic: a
1:1 commit leaves no headroom, and two lanes running the identical
sandbox on different classes is drift nothing was watching.

Add hack/sandbox-runner-headroom.bats to keep the two sides paired. It
derives the guest vCPU and memory demand from the sandbox bats file, and
the runner class of every job that drives the sandbox make targets, then
requires the runner to be strictly larger on both axes. Neither side is
a literal list, so raising `-smp`, adding a management node or adding a
lane on a smaller runner turns it red instead of drifting apart
silently.

Input the derivation cannot read in full is refused rather than guessed:
a guess produces a demand smaller than the real one, which passes a
runner that cannot hold the sandbox, while a refusal names the line.
That covers a launch written outside any `@test` block, a block left
open at end of file, a `for` opener the strict pattern misses, a
non-literal loop list, and a `while` loop sharing the block.

The derivation runs against the live tree, which passes whether or not
those refusals still work, so each one is pinned by a fixture feeding a
synthetic file to the same program. Loosening the parser turns a fixture
red rather than quietly widening what the check accepts.

Record the rule alongside the other environment invariants in
docs/agents/e2e-testing.md, so a new lane picks its runner class from
the documentation rather than from a failing unit suite.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-15 02:41:19 +03:00
Myasnikov Daniil a37a3178e5 docs(release): name all four sourcers of the image-ref enumeration 
The library header still claimed three call sites. There are four:
this PR made hack/verify-promoted-packages.sh source it to compare
the candidate's refs against the rc artifact's. The historical
drift story stays with the original three, since the verifier has
only ever read the shared enumeration.

The consumer table gained hack/promote-packages-artifact.sh under a
bullet requiring every consumer of references to read all three
shapes through the library, while the exception below it still named
only hack/overlay-main-images.sh. The publisher sources nothing and
needs nothing: it hands the whole tree to `flux push artifact` and
touches one reference, the installer's own self-pin, by key with yq.
Say that, rather than implying a gap it does not have -- the real
exception remains overlay-main-images.sh, which does read references
and does walk the tree itself.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-14 18:37:59 +05:00
Myasnikov Daniil 04fa8471fe docs(release): say what the gates actually guarantee 
Three claims that were stronger than the code, plus one legibility fix.

The PR gate's comment said a `packages/` edit "on the PR or its base"
fails the status before merge. The base half is not true: GitHub fires
no `pull_request` event when the base branch advances, it only
recomputes refs/pull/N/merge, so a change landing on the base after this
went green is never re-verified and the PR merges on the stale result.
Checking the merge tree does make base drift visible WHEN the job runs,
which is worth keeping — but finalize is what actually catches the drift
case, after the merge and before any stable name. Say both, in the
workflow and in the reviewer checklist, so nobody reads the green as
covering more than it does.

"Transactional at every consumer-visible name" is a judgement call
dressed as an absolute, inherited from the "this dispatch performs NO
registry mutation" it replaced. The dispatch now does write to the
registry: the `promotion-*` candidate is a real, publicly listable tag.
What it creates is no *stable* name — no vX.Y.Z, no :latest, no stable
installer. Stated that way the sentence stops needing the reader to
already know which writes were meant.

The retention paragraph listed two of the conditions the workflow
enforces. It now lists all of them, including the ones the workflow
gained here: same-repository, bot-authored, `release-X.Y.Z` branch, the
`release` label, exactly one valid digest pin, and protection from a
base-branch tip as well as an open PR.

Also: `flux pull artifact` reads its credentials from
~/.docker/config.json, which is why finalize now runs `docker login` —
but its runner class is not one of the docker-using build ones and the
setup step installs everything except docker. Check for it there, where
the message can name what is missing, instead of failing with "command
not found" halfway down the release path. And the verifier's exclusion
list is a restatement of flux's `excludeOCI`, not a derivation — the CLI
does not expose it — so the coupling and the pinned version are now
written down next to the list.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-14 18:37:58 +05:00
Myasnikov Daniil f21d277eba fix(release): retain live promotion candidates 
Protect package candidates referenced by open release PRs while pruning abandoned run-specific promotion artifacts. Document the stable-candidate lifecycle and maintenance-branch rollout requirement.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-14 18:37:57 +05:00
Aleksei Sviridkin b7b56530b9 docs(platform): explain system component memory limits (#3496) 
## What this PR does

Adds `docs/operations/system-memory-limits.md`, recording why system
components carry memory limits — which is not recoverable from the
values alone, and is the kind of thing that gets "cleaned up" by a
well-meaning future change.

The Talos userspace OOM handler (v1.12+) selects victims by ranking
cgroups that lack `memory.max`. Three consequences are each easy to get
wrong independently: only a limit grants immunity (a request just moves
BestEffort to Burstable, which under `strictCgroupClassOrdering` means
"killed second"); every container in the pod needs one, init containers
included, because kubelet only sets pod-level `memory.max` when they all
have it; and `PriorityClass` — `system-node-critical` included — has no
effect whatsoever, because the handler walks cgroupfs and never talks to
the API server. That last one is the trap, since setting it looks like a
fix.

The page also covers the two installer knobs and the constraint that
makes them dangerous to lower (a defaulted limit below a container's own
request fails admission), how to verify a pod cgroup actually carries
`memory.max` rather than trusting the QoS class, what is deliberately
left uncovered (Kubernetes static pods are Talos-managed; a handful of
vendored containers expose no upstream resources knob), and how to relax
the global-PSI backstop clause of the trigger for a cluster that trips
it persistently.

Ordering note: this documents behaviour introduced by the LimitRange and
DaemonSet PRs, so it should not merge ahead of them.

### Downstream repositories

Walked the trigger map against the diff. This adds one file under
`docs/operations/`, alongside the existing pages there; nothing in the
map covers that path, and the website keeps its own copy of user-facing
documentation rather than generating it from this directory. A
maintainer may still want a version of this on the website, which is a
judgement call rather than a forced follow-up.

- [x] No downstream repository is affected by this change

### Release note

```release-note
docs(platform): document why system components carry memory limits, how the Talos userspace OOM handler selects victims, how to tune or disable the system-namespace LimitRange, and why PriorityClass does not influence any of it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added guidance on Talos userspace out-of-memory behavior and system
component memory limits.
* Documented namespace boundaries, installer tuning, validation
constraints, rollout behavior, and verification steps.
* Added instructions for inspecting OOM actions, identifying uncovered
workloads, evaluating trigger expressions, and configuring optional
overrides.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-14 13:49:46 +03:00