Commit Graph
2787 Commits
Author SHA1 Message Date
Aleksei Sviridkin 1ffc8580a4 build(keycloak-operator): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:01:00 +03:00
Aleksei Sviridkin e2757ba87b build(redis-operator): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Two of the files the patches edit are upstream integration tests behind
the integration build tag, which go test compiles none of, so each
directory is also vetted with that tag. Doing it per derived directory
rather than for the one directory those tests live in today keeps a
later patch to a tagged test elsewhere covered.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:00:59 +03:00
Aleksei Sviridkin 0a606815ab build(kilo): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:00:59 +03:00
Aleksei Sviridkin a3a7d66bdf build(linstor): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:00:58 +03:00
Aleksei Sviridkin a73b91e232 build(bucket): run the tests of the packages the patches touch
The image build ran go build only, and go build never compiles a
_test.go, so nothing ran the tests of the code the patches change. Run
go test in the builder stage on the packages the patches touch. The list
is read from the patches themselves, so a new patch is covered without
an edit to the Dockerfile.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:00:58 +03:00
Aleksei Sviridkin 293a780938 build(opensearch-operator): run the tests of the packages the patches touch
The image build ran go test on two package trees listed by hand, and go
build never compiles a _test.go. A patch that reaches a directory
nobody listed brings tests that are proven once, by hand, when the
patch is cut, and nothing notices them failing after the next version
bump.

Run go test in the builder stage on the packages the patches touch. The
list is read from the patches themselves, so a new patch is covered
without an edit to the Dockerfile. It replaces the hand-picked go test
line, keeping its -race flag: two lines testing the same packages is
one too many. The hand-picked line also covered pkg/reconcilers/util,
which no patch touches.

controllers/ is left out. The stall-tracing patch touches it, and its
suite starts an envtest control plane from etcd and kube-apiserver
binaries the builder does not have, so testing it fails the build.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:00:57 +03:00
Aleksei Sviridkin bf03c84711 chore(deps): move to the Kubernetes 1.37 libraries without the apimachinery fork (#4473)
## What this PR does

This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, and drops the
`k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and
#4472; those were a stack of the same commits.

The four commits are meant to be read one by one, and each builds and
passes the tests on its own:

1. **Plain watches stop getting the initial-events-end bookmark.** The
apiserver defaults `sendInitialEvents=true` onto any watch with an empty
or `0` resourceVersion and leaves bookmarks off, and our aggregated
storages took that flag alone as a WatchList request. From
`k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api
panics on that bookmark, so `kubectl get -w` on any cozystack-api
resource would die after the bump. The marker now needs both flags, the
same condition the watch cache uses. There are tests per storage and one
through the real handler chain.
2. **The apimachinery fork goes.** Every apps kind shares the
`Application` Go type, and stock `Scheme.ConvertToVersion` picks the
first kind registered for it. The fork patched that; the apps group now
encodes through a serializer that keeps the kind the object carries. A
handler-chain test covers create, get, list, update, watch, apply and
table for two kinds and fails without it.
3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the
plugin is on by default and refuses every write until it has synced
`mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and
which clusters before 1.36 do not serve. Without this, every create,
update and delete would fail after the bump. It is also re-applied in
`Complete`, since `--disable-admission-plugins` replaces the list.
4. **The bump itself**, with the regenerated clients, openapi and
strategy CRDs.

Commits 2 and 4 are where the review effort goes; 1 and 3 are small.

Known changes in behaviour:

- Without the fork, the row objects of a Table requested with
`includeObject=Object` carry the first registered app kind. kubectl asks
for that only with `--sort-by`, and it sorts on fields, not on the kind.
Nothing at this layer can fix it, because the handler converts through
`APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks
and ValidatingAdmissionPolicy still see the right kind.
- The alpha PodSpec field `workloadRef` became `schedulingGroup`
upstream. Once the new strategy CRDs are applied, the apiserver prunes
`workloadRef` from stored strategies it serves, and their next write
erases it. It only ever worked behind an alpha feature gate.

The branch is rebased onto the Renovate rollup (#4447), with `go mod
tidy` and codegen rerun on top of it.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application API responses now preserve the requested resource kind
across operations, including lists and watches.
* Backup strategy resources support additional Kubernetes pod settings,
including probe protocols, volume ownership options, and scheduling
groups.
* Backup strategy schemas include updated Kubernetes pod fields and
descriptions.
* **Bug Fixes**
* Initial-event watch bookmarks are sent only when requested; plain
watches continue without an ending bookmark.
* Mutating Admission Policy is disabled by default, including when
admission plugin options are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 21:45:51 +03:00
Aleksei Sviridkin 9809b22280 fix(vm-disk): report DataVolume readiness through WorkloadsReady (#4380)
## What this PR does

Nothing on a vm-disk application tells whether CDI ever finished
populating the disk. Its WorkloadMonitor has no pods to count, and
`Ready` follows the Helm release. So an import or clone that never
completes can leave a green application over a disk nobody can use.
cozystack/cozystack#2908 is one way to get there: a clone into a
StorageClass that cannot place a replica where the source data sits. The
expensive part there was the silence, not the stuck clone.

The disk state was in the DataVolume phase all along, and nothing read
it. With this change the WorkloadMonitor in cozystack-controller reads
it. The monitor lists the DataVolumes its selector matches and reports
not operational while any of them is in flight, Failed, Unknown or has
no phase. An upload counts as in flight until it completes. That
surfaces on the application as `WorkloadsReady=False` with reason
`DataVolumeNotReady` and a message that names the DataVolume and its
phase, for example `DataVolume vm-disk-x is ImportInProgress`.

`Ready` still follows the HelmRelease, as for any other kind
(`convertHelmReleaseToApplication` in
`pkg/registry/apps/application/rest.go`), and for a vm-disk it no longer
waits on the DataVolume at all. So the disk state has to be visible
outside `Ready`, and this PR puts it where operators look by default:

- `kubectl get` on any application kind gets a `WORKLOADS` column next
to `READY`. It shows the `WorkloadsReady` status, or `<none>` when the
application has no WorkloadMonitor.
- The console badge turns amber and reads `DataVolumeNotReady` while
`Ready` is True and `WorkloadsReady` is not True for that reason. A
failing `Ready` still wins. The console shows only the disk case: other
kinds need a separate decision, because a stopped VM legitimately has no
pod and reports `WorkloadsReady=False` too.

A Helm readiness gate on the DataVolume would be the obvious
alternative, and it is not used on purpose. helm-controller runs that
wait inside its reconcile, so an importing disk would hold one of the
shard's worker slots for the whole install window, and a disk stuck for
good would keep taking the slot back. cozystack/cozystack#4145 has the
details. The monitor costs no slot, keeps watching after install, and
covers existing disks without anyone writing to them.

Without any gate, the install would still wait on the DataVolume.
helm-controller's default poller falls back to a resource's `Ready`
condition for kinds it has no rule for (`checkReadyCondition` in
fluxcd/cli-utils `pkg/kstatus/status/status.go`), so a vm-disk install
could hold a worker slot for the whole install timeout. The
ApplicationDefinition therefore gets a `healthCheckExprs` entry for
DataVolume with `current: "true"`, the same idiom postgres-rd uses for
its ScaledObject. With it Helm never waits on the DataVolume. The entry
is pinned by a helm-unittest in `packages/system/vm-disk-rd`, which gets
a `test` target for it.

Release settings reach a HelmRelease only when the application is
created or updated, so an existing disk gets the entry on its next
update. The reconciler that syncs definitions onto existing releases
copies chartRef, valuesFrom and labels only. That affects any kind with
release settings, so it is filed as cozystack/cozystack#4409 and not
fixed here. `WorkloadsReady` covers every disk right away, since the
monitor needs no write to the application.

CDI copies the DataVolume labels onto the PVC it creates and makes the
DataVolume its controller, so the monitor also publishes a `pvc-<name>`
Workload for the disk. That Workload reads not operational while its
DataVolume is not ready, even though the PVC is already Bound, so
`kubectl get workloads` agrees with `WorkloadsReady`. While the
DataVolumes cannot be read it keeps the verdict it has, like the
monitor.

The WorkloadMonitor CRD now requires at least one selector label. An
empty selector would match every Pod, PVC, Service and DataVolume in the
namespace. No chart renders one.

When an application has several monitors that are not operational and
only some of them name a cause, `WorkloadsReady` carries the named
messages and reason only. A vm-disk has one monitor, so it does not hit
this.

### Which phases count as ready

Settled phases pass: `Succeeded`, `PendingPopulation`,
`WaitForFirstConsumer`, `Paused`. A disk with `bindImmediately: false`
on a `WaitForFirstConsumer` class rests in `PendingPopulation` or
`WaitForFirstConsumer` until a VM consumes it. A multi-stage import
rests in `Paused` until its next checkpoint, and the vm-disk chart
renders no checkpoint source. These report ready while holding no data,
because that is where such a disk is supposed to sit.

`UploadReady` is not one of them. CDI enters it when the upload server
pod is running and ready, and leaves it for `Succeeded` only when that
pod succeeds (`updateStatusPhase` in
`pkg/controller/datavolume/upload-controller.go` at v1.66.1), so the one
phase covers a disk nobody has uploaded to and a half-uploaded one. A
`source: upload` disk therefore reports `WorkloadsReady=False` from
creation until `virtctl image-upload` completes.

A phase the controller does not know also reads as ready, so a CDI
release that adds or renames a phase cannot turn working disks red. The
cost goes the other way: a renamed in-flight phase silently reads as
settled. The list wants a re-read whenever the CDI version moves. It
matches the named phases of the CDI v1.66.1 API, the version vendored in
`packages/system/kubevirt-cdi-operator`.

### Clusters without CDI

CDI comes with the IaaS bundle, while cozystack-controller runs
everywhere, and CDI can be installed while the controller is running. A
watch on a kind whose CRD is absent never syncs, and then the whole
controller does not start. So the watch is not set up at startup. A
runnable adds the DataVolume watch once the kind is served. It waits for
the first reconcile, because only from then on does a new watch start at
once, and then checks discovery right away and every 30s after that. It
installs the DataVolume reader only after the DataVolume informer has
synced within a timeout, because a reconcile context has no deadline and
a List on an informer that never syncs would block every WorkloadMonitor
reconcile. The watch source is started once: a controller-runtime `Kind`
source adds an event handler to the informer and never removes it, so on
a timeout the next check waits on the source it already has instead of
starting another one. RBAC needs no change: the controller's ClusterRole
already has read on everything.

When the DataVolumes cannot be read, because the reader is not installed
yet or the List fails, the monitor still publishes its replica counts
and keeps its last DataVolume verdict, and a List error is returned so
the request is retried and recorded as a `DataVolumesUnavailable`
Warning event on the monitor. The status of a monitor without
DataVolumes is not affected by a failing List. A disk whose verdict is
already recorded does not read as ready in the window after a controller
restart. A monitor with no recorded verdict yet, which is every monitor
right after the upgrade that brings this change and any disk created
right after a restart, reads ready until the DataVolume watch has
synced. Because the watch starts on the first reconcile rather than on
the next discovery tick, that window is the time to sync the DataVolume
informer plus at most one 5s requeue, not a 30s poll interval. The
window is not zero, since a monitor reconciled in it is published before
its DataVolumes can be read. From then on, a reconcile that runs before
the watch has synced is requeued after a few seconds, so the DataVolume
events that queue it are not lost. A pass that changes nothing in the
monitor status does not write it, so that requeue sends no status writes
to the API server. Once the reader is installed, every monitor that
holds a DataVolume verdict is queued once more. So a verdict whose
DataVolume was deleted while the controller was down is recomputed then,
not at the periodic resync.

### Tests

The table test in `workloadmonitor_controller_test.go` covers the named
phases of CDI v1.66.1, plus Failed, Unknown, an empty phase and a
missing status. A second table checks the message and reason for an
importing, a failed, a phaseless and a populated disk, and a third test
lists two stuck DataVolumes in both orders and expects one message. It
also covers a DataVolume outside the selector, the kind not being
served, a List error that still publishes the replica counts, and a
missing reader that keeps the last verdict. The watch tests check that
the watch starts on the first reconcile and not before it. A source that
never syncs leaves no reader and is retried on the same source, with the
reconciles in between still requeued. A List failure records the Warning
event, the sync window records none, and an unchanged pass writes no
status. A manager that stops during the wait gets the cancellation back,
and a reconcile that runs while the watch syncs is requeued without an
error. A table test covers the PVC Workload: importing, populated, a PVC
with no DataVolume controller or a controller of another kind or group,
and the unread cases. A helm-unittest in
`packages/system/cozystack-controller` pins the selector's
`minProperties`. `rest_table_test.go` pins the `WORKLOADS` column,
`rest_workloads_test.go` the condition message and reason, and
`status.test.ts` the console badge, including a halted VM that stays
green.

In `hack/e2e-chainsaw/vminstance` the existing VMDisk scenarios now wait
for the DataVolume to be Ready, then for `Ready` and `WorkloadsReady` on
the VMDisk itself. That replaces the wait on the HelmRelease. Those
asserts would pass even if the monitor ignored DataVolumes, so a new
scenario imports from an `.invalid` URL. It first waits for the importer
to have failed, which CDI reports on the DataVolume as `Running=False`
with reason `Error`, and then for `Ready=True` with
`WorkloadsReady=False` and reason `DataVolumeNotReady` on the VMDisk.
Without the first wait the VMDisk assert passed in about a second, on a
DataVolume that had no phase yet. The e2e runs on a fresh install only,
so nothing here exercises an existing release across an upgrade.

### What it does not do

#2908 lists possible fix directions, and none of them is here. The
cross-class clone still never completes, it just stops being invisible.
Keep the issue open.

While the DataVolume watch syncs, every monitor is still requeued. Only
the ones whose selector reaches a DataVolume need it, but the controller
cannot tell which those are before it can read DataVolumes.

Closes #4449

### Screenshots

There is no screenshot: I had no running console to take one from. The
badge change is pinned by the vitest cases in
`packages/system/dashboard/images/console/apps/console/src/lib/status.test.ts`:
a VMDisk whose `WorkloadsReady` has reason `DataVolumeNotReady` goes
amber, and a halted VMInstance with the generic reason stays green.

### Downstream repositories

I walked the trigger map against the diff.
cozystack/terraform-provider-cozystack is affected: its `wait_for_ready`
reads only `Ready`, so on a `cozystack_vmdisk` it returns before the
disk is populated once this lands. That needs a decision on the provider
side, so it is an issue there. The release-time bot regenerates
`packages/apps/vm-disk/README.md` into the website's virtualization
pages, and `vm-disk` is in the `VMS` list that generator reads, so the
website needs no follow-up. The ApplicationDefinition keeps its kind,
plural, prefix and path.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/59
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-disk): report whether CDI finished populating a disk in the WorkloadsReady condition, which names the stuck DataVolume and its phase. kubectl get shows WorkloadsReady in a new WORKLOADS column for every application kind, and the console badge turns amber while a disk is not populated, so a disk whose import or clone never completes no longer looks healthy. Ready of a new or updated vm-disk no longer waits for the DataVolume, so wait on WorkloadsReady to know the disk is populated.
```
2026-10-02 19:47:02 +03:00
IvanHunters 11910e0b2c fix(kubernetes-nodes): refuse a resources.cpu that cannot be a core count (#4386)
## What this PR does

A worker pool's CPU value could land in the VM as something KubeVirt
cannot accept. `resources.cpu` is typed as a quantity, but on the
kubevirt substrate the template wrote it unchanged into
`domain.cpu.cores`, which is `uint32` in KubeVirt and an integer in the
KubevirtMachineTemplate CRD. With `resources.cpu: 500m` the chart
rendered `cores: 500m`. That string was truthy, so it also counted as
explicit sizing and dropped the pool's `instanceType` from the VM, and
then the API server refused the template anyway.

The render now decodes the value as YAML, the same way `cores: <raw>` is
decoded, and stops unless the result is a whole number from 1 to
4294967295. `500m`, `"1.5"` and `"1e10"` now fail at render time with an
error naming the field, before the value reaches the VM template or the
instancetype decision. `"0"` and `"-2"` were already refused on `main`
and still are.

Only inputs that could not produce a working pool are refused. Every
value that rendered a pool before renders the same bytes now, so the
content-hashed template keeps its name and nothing rolls. That includes
`2`, `"2.0"`, `"+2"` and `"1e1"`, and also two odd cases I kept on
purpose because pools may already carry them. An unquoted `0` is falsy,
so the pool renders without `domain.cpu` and is sized by its
instancetype, as on `main`. A leading-zero value such as `"010"` is
still emitted as written and keeps the reading it gets on `main` today,
where YAML treats it as octal. I checked both against a render of
`main`, and the tests pin the template names that `main` produces for
them.

This is where kubernetes-nodes and vm-instance now differ. Since #4405,
vm-instance refuses a leading zero and an unquoted `0`, and this chart
does not. Stored worker pools can carry these values, because migration
56 copies `resources` verbatim, so refusing them here without a
migration would fail those pools' releases on their next upgrade. #4554
tracks the migration and the refusal after it.

The proxmox substrate is not checked. It converts the quantity to
millicores and rounds up to whole cores, so `500m` is a valid size there
and becomes one core. The field description now says what the value
means on each substrate.

Memory is left alone. It goes to `domain.memory.guest`, which is a
`resource.Quantity`, so a quantity string is valid there.

The issue suggested coercing to an integer, as vm-instance did before
#4405. I didn't, because `int64` turns `500m` into `0` without an error.

The new helm-unittest suite covers each input form, both preserved
cases, the upper bound, the instancetype case and the proxmox path.

Two neighbouring descriptions still say only "CPU available.":
`nodeGroups[].resources.cpu` in `packages/extra/computeplane` and in the
parent `kubernetes` chart. computeplane passes `resources` straight into
this chart, so the new check applies to its pools as well. I left both
alone to keep this change to one chart.

Closes #4150

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes-nodes): on the kubevirt substrate, a worker pool whose resources.cpu cannot be a whole vCPU core count (for example 500m, 1.5 or a value above 4294967295) now fails at render time with a clear error, instead of rendering a VM template that is refused later or dropping the pool's instanceType. Values that rendered a working pool before render unchanged.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* KubeVirt CPU settings now accept whole-core values from 1 to
4,294,967,295. Fractional, millicore, negative, and larger values are
rejected; an unquoted `0` is treated as unset.
* Proxmox rounds fractional and millicore CPU values up to whole cores.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 15:57:11 +03:00
IvanHunters 339be8d42d fix(vm-default-images,kubernetes-worker-image,vm-disk): wait for golden images to import and refuse cloning a failed one (#4392)
## What this PR does

Both golden image catalogs, `vm-default-images` and
`kubernetes-worker-image`, went Ready before any image had imported.
Each one renders a DataVolume per image with immediate bind, so every
import starts at install, but neither PackageSource had
`healthCheckExprs`. The HelmRelease went Ready as soon as helm applied
the DataVolumes, and an image that failed or stalled stayed green.

This PR makes both releases wait for the imports. Each component's
`install:` block now has a DataVolume health check. `current` holds only
on `Succeeded`, and `failed` holds on `Failed`. Any other phase,
`Unknown` included, matches neither, and flux keeps waiting. A
DataVolume with no `status` key makes the CEL evaluation error out. Flux
reports that as Unknown and keeps waiting too.

The other settled phases are safe to treat as not ready for these
charts. `PendingPopulation` and `WaitForFirstConsumer` are where a
volume waits for a consumer, and immediate bind removes that wait.
`UploadReady` and `Paused` belong to upload and multi-stage sources, and
both charts render only `source.http`.

There is no per-component timeout, so I did not set one. The window is
the operator-wide `--helmrelease-install-timeout`, 10m by default. The
operator sets `RetryOnFailure` on both install and upgrade, so a
download that outlasts the window is retried as an upgrade until the
import finishes. Nothing uninstalls the DataVolumes in between, and the
import keeps running.

One image that never imports keeps its whole catalog not Ready until the
image imports or its entry is removed. There is no time limit, and I
kept it that way on purpose. A group/kind health check cannot tell one
image from another, so it cannot be made per image. A per-component
timeout would be an API change for one opt-in package. Both charts'
`values.yaml` document this. Nothing on `main` lists either package in
`dependsOn`, and the images that did import stay usable in the meantime.
Tenant worker pools look up each golden directly and never wait on the
catalog's status.

The price is a helm-controller slot. The operator labels the generated
HelmRelease with `cozystack.io/package`, plus `cozystack.io/privileged`
when privileged. The flux-shard-operator acts only on HelmReleases that
already carry `sharding.fluxcd.io/key`. So both catalogs go to the
system helm-controller, which picks up releases without that key and
runs with `--concurrent=5`. A DataVolume in `Failed` ends each attempt
at once and costs next to nothing. An import that stalls without
reaching `Failed` is the expensive case, for example an importer that
keeps retrying an unreachable URL. Each attempt waits out the full
window, and the next one starts 30s later. So one of those five slots
stays busy about 95% of the time for as long as the image is stuck.
#4145 describes the same cost on the shard controllers.

The status change alone does not protect a vm-disk, because vm-disk
clones from a golden without checking it. A golden still importing is
fine, since CDI holds the clone until the source is populated. A golden
whose import has Failed is never populated, so the clone would wait
forever. vm-disk now refuses to render a new disk on a Failed golden and
names the image. This matches the check the worker pools already have. A
disk that already exists is exempt, so a golden that fails later does
not break that disk's upgrades. The check fires only when CDI sets
Failed, and at CDI v1.66.1 an import reaches Failed only on a checksum
mismatch. The goldens carry no checksum, so a broken URL keeps the
importer retrying and the new disk still waits, with only the catalog's
not-Ready status pointing at the image. #4564 tracks that gap. A golden
that does not exist at all is a separate gap, tracked in #4548.

Tests pin the expressions for both catalogs. A helm-unittest case checks
them on the rendered PackageSource. A Go test builds the HelmRelease
spec from the shipped file and checks that they arrive with the poller
wait strategy. The repo has no harness that evaluates flux CEL, so the
phase behaviour above is argued from flux's evaluator and not exercised
by a test. The vm-disk check has helm-unittest cases for a Failed, a
Succeeded, and a still-importing golden, for a render with no cluster,
and for an existing disk.

Closes #4169

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-default-images, kubernetes-worker-image): the golden image catalogs now report Ready only after their images have finished importing, and a failed or stalled import keeps the catalog not Ready. A vm-disk created from a golden image whose DataVolume is in phase Failed is now refused with an error naming the image, instead of waiting forever on a clone that cannot complete.
```
2026-10-02 15:44:28 +03:00
Aleksei Sviridkin 10a6cf2a0b test(tests): make absence-only helm-unittest cases fail when they check nothing (#4664)
## What this PR does

Twenty-one helm-unittest cases asserted only that something is absent
from a document, and could pass without ever looking at the document
they name. This PR makes each of them fail when that happens.

1. Thirteen cases picked their document by position: mongodb (4), nats
certmanager (3), opensearch (3), qdrant certmanager (2), seaweedfs
cluster_scoped_names (1). When a template gains or reorders a document,
the index points at some other object that also lacks the field, and the
case stays green. These now use `documentSelector` on `kind` or
`metadata.name`, like the earlier harbor conversion.
2. Seven cases had no selector at all, on templates wrapped in a
top-level `if`: kubernetes cilium_gateway_alpn, mongodb restore,
linstor-scheduler admission, seaweedfs s3_traffic_distribution and three
in vertical-pod-autoscaler. If a values change turns the condition off,
the render is empty and the absence check passes on nothing. A
`documentSelector` turns an empty render into `document not found`.
3. The redis null-namespace case checked that the in-cluster name is
present, but never that the external name is absent. It now asserts that
neither `my-redis.` nor `my-redis.example.com` (the suite-level host
that `_namespace: null` must override) is in the SAN list.

The `kubernetes cluster` entry in the #3634 catalog no longer applies,
since `packages/apps/kubernetes/tests/cluster_test.yaml` has been
removed from main. I did not add the `my-redis.redis.` check that #4621
mentions, because the per-engine SAN it refers to has not landed.

Each case was checked by mutation. I broke the template in a way the old
case could not see, then ran the chart's `make test` with the new test
file and with the one from main. For group 1 I removed the guard and put
an extra document in front of the target. For group 2 I turned off the
top-level condition. For group 3 I let the empty-host branch emit the
bare SAN. Every time the main version stayed green and the new one went
red, and the fixed cases were the only extra failures. All templates are
back to their committed bytes.

Follow-ups, not in this PR:

- The same absence-only shape has appeared in more cases since #3634 was
filed: mongodb `tls_test.yaml` (8 cases), mariadb `tls_test.yaml` (6),
redis `tls_test.yaml` (2 more), kubernetes
`kamaji_managed_args_test.yaml` (2), and one each in kafka
`dashboard-resourcemap_test.yaml`, cert-manager-issuers
`solver_test.yaml` and seaweedfs `volume_resize_hook_test.yaml`. They
need a separate issue. The four platform `bundles_proxy_protocol*` cases
are #3864, handled in #3929.

Closes #3634

Closes #4621

### Screenshots

Not applicable, test-only change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Updated chart validation to identify rendered resources by their type
or name rather than relying on document order, improving consistency
across Kubernetes applications.
* Expanded Redis TLS checks to verify that certificates omit generic and
external hostnames when no namespace data is provided.
* Kept existing expected behavior checks for gateway, database,
messaging, search, storage, and autoscaling resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 14:08:07 +03:00
Aleksei Sviridkin 9f72f0bda9 feat(api): refuse a WorkloadMonitor with an empty selector
An empty selector matches every Pod, PVC, Service and DataVolume in the
namespace, so one such monitor goes non-operational on any import there.
No chart renders one, but the schema marked the field required without
requiring any label, and the API accepted it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:37 +03:00
Aleksei Sviridkin 6170c1615f feat(dashboard): turn the status badge amber while a disk is not populated
The console badge followed Ready alone, and Ready follows the Helm
release. A vm-disk install does not wait on its DataVolume, so a disk
CDI never populated was painted green.

The badge now shows the WorkloadsReady condition while Ready is True
and WorkloadsReady is not True with reason DataVolumeNotReady. Other
causes of WorkloadsReady=False keep the badge as it was: a stopped VM
has no pod and reports not operational on purpose, and turning it
amber for every kind needs its own decision. A failing Ready still
wins, because it is the more specific cause.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:34 +03:00
Aleksei Sviridkin 99f3d3b2b7 feat(controller): tag a DataVolume cause with a condition reason
WorkloadsReady=False covers different situations: a stopped VM has
no pod and its monitor reports it not operational on purpose, while a
disk whose DataVolume never populates is broken. A consumer that wants
to react to one and not the other had only the message text to go by.

The WorkloadMonitor status gains a Reason next to Message, set to
DataVolumeNotReady by the DataVolume check, and the Application API
carries it into the WorkloadsReady condition in place of the generic
reason whenever a monitor that is not operational names one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:33 +03:00
Aleksei Sviridkin 48ece79ffc feat(controller): name the DataVolume that keeps a disk not ready
For a vm-disk, WorkloadsReady is the one condition that reports
whether CDI populated the disk, and its generic "One or more workloads
are not operational" gives an operator nothing to act on: a disk has
no workloads, and the stuck DataVolume and its phase are exactly what
they need to know.

The WorkloadMonitor status gains a message that the controller fills
with each selected DataVolume that is not ready and its phase, and the
Application API uses the messages of the monitors that are not
operational in WorkloadsReady instead of the generic text whenever one
of them names a cause.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:32 +03:00
Aleksei Sviridkin f4b1f08adb fix(vm-disk): keep the Helm install from waiting on the DataVolume
Helm's default poller falls back to a resource's Ready condition for
kinds it has no rule for, so a vm-disk install could wait on the
DataVolume's Ready and hold a helm-controller worker for the whole
install timeout while an import runs or never completes. Whether CDI
populated the disk is reported by the WorkloadMonitor, so the release
now treats the DataVolume as current and Ready of a vm-disk no longer
depends on the import. Release settings reach a HelmRelease when its
application is created or updated; the reconciler that syncs
definitions onto existing releases copies the chart reference,
valuesFrom and labels only, so an existing disk picks this up on its
next update.

The vm-disk-rd package had no unit test target; it gets one, and a
test that pins the entry.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:01:13 +03:00
Aleksei Sviridkin 360f03bb8b feat(monitoring,tenant): shared-central tracing topology (phase 4) (#3775)
## What this PR does

Adds the opt-in shared-central traces topology from the distributed
tracing design (cozystack/community
`design-proposals/distributed-tracing`, rollout item 4): a tenant can
send its spans to one VictoriaTraces store in tenant-root instead of
running its own, with both write and read isolation between the tenants
that share it.

**Switching it on.** A tenant opts in with the new Tenant field
`tracingCentral`, which needs the tenant's own Monitoring. The tenant
chart publishes it to the tenant's monitoring charts as
`_namespace.tracingCentral` in the `cozystack-values` Secret, which the
apps API does not let an application write, and renders the two network
paths the topology needs under the same condition. tenant-root decides
whether it hosts the store, with the new Monitoring field
`tracingCentralHost` next to a cluster-mode `tracingStorages` entry
named `generic`.

**Isolation.** VictoriaTraces takes the tenant account from the
`AccountID`/`ProjectID` headers and does no authorization of its own, so
other tenants reach the shared store only through a vmauth in
tenant-root. Each central tenant gets a VMUser that pins an account
derived from its namespace name and UID, for both its collector's writes
and its Grafana reads, and vmauth overrides any account a client sends.
The collector and Grafana log in with credentials the chart generates
and keeps. The tenant egress rules allow only pods carrying the
collector's or Grafana's label to reach the vmauth proxy port, and a pod
that carries such a label without the tenant's credentials gets only 401
from it; the vmauth internal routes, whose `/metrics` lists every
tenant's user, are served on a port tenants cannot reach. The
tenant-root namespace itself is inside the trust boundary, as it already
is for the metrics of tenants that rely on a parent's monitoring: its
workloads reach the store directly, and one that learns a tenant's
account ID can read and write that account, and whoever tenant-root
admits to its Grafana reads the store through a datasource that bypasses
vmauth (with the System oidc mode, tenant-root's own groups, which
already hold the same access level in every tenant under it); the
`tracingCentralHost` description says so.

**Sharing the store.** Each tenant's VMUser is capped at four requests
in flight per vmauth replica, reads and writes together, and each
collector replica exports with two senders, so more collector replicas
do not raise a tenant's throughput; above the cap vmauth queues for up
to 10s and answers 429, which the exporter retries. The store has one
disk and one retention for all of its tenants and nothing reserves space
per tenant; `tracingCentralHost` states that, since hosting is
tenant-root's decision. The vmauth runs two replicas spread across nodes
under a PodDisruptionBudget, so draining one node does not stop every
central tenant at once. Every traces store is also disk-bounded now, as
the design requires: without `retentionDiskUsageBytes` it drops its
oldest day once its filesystem is 80% full. That alone does not keep one
tenant from filling the shared disk, since VictoriaTraces always keeps
the last two days.

**States that would drop spans silently are refused, one race aside.** A
live render fails, with a message naming the prerequisite, when a tenant
turns central on while tenant-root hosts nothing, and when tenant-root
stops hosting while central tenants remain; tenant-root's render also
refuses `tracingCentralHost` without a store to host. The second check
does not wait on a VMUser that is already being deleted, since its
tenant is leaving. Each check reads the other side's objects when it
renders, so a tenant turning central on while tenant-root stops hosting
can pass both and export to a vmauth that is gone; the
`tracingCentralHost` description names this race and its recovery. At
runtime, tenant-root's Monitoring release is not Ready while the vmauth
has failed. These checks run on a render, so removing the tenant-root
Monitoring altogether is not refused, and from then on every Monitoring
render of a central tenant fails until it turns central off; the
`tracingCentralHost` description says to turn central off on the tenants
first. Turning central on keeps the tenant's own local stores and their
datasources, so traces stored before the switch stay readable until the
tenant removes them; turning it off again leaves the spans written
meanwhile in the shared store, unreadable from the tenant until it turns
central on again, and spans sent while the collector switches back may
be dropped; a deleted tenant's spans likewise stay until retention drops
them. Right after a tenant turns central on, or after its credentials
Secret is re-created, spans its collector sends before the vmauth has
loaded the new login get a 401, which the exporter does not retry, so
they are dropped; the tracing guide in #3772 covers it.

**Where this departs from the design.** The design has the per-tenant
collector inject the account headers and adds exactly one egress rule.
Here the account is pinned by the tenant's VMUser at the vmauth and the
collector sends none, because the collector runs in the tenant namespace
and whatever runs with its pod label could otherwise choose an account;
vmauth is also what the design already requires for reads, so both
directions go through it. That needs a second egress rule, for the
tenant's Grafana to reach the vmauth, beside the collector's. The
per-tenant write rate the design gives the collector is not in this PR:
no OpenTelemetry distribution has a rate-limiting processor, so #4648
builds the collector with one and holds each central tenant to a rate
tenant-root publishes. Until it lands, a tenant writing faster than the
disk cap allows over two days can fill the disk the other tenants share.

**Consent is per host, not per tenant.** `tracingCentralHost` lets every
tenant that sets `tracingCentral` in, nested tenants included, and each
gets its own VMUser and its own request cap; there is no allow-list. A
tenant-root that needs to limit who shares its store keeps
`tracingCentralHost` off.

**Upgrade.** Tenants that do not opt in render the same objects as
before, and their `cozystack-values` Secret is unchanged. The monitoring
charts render the same objects too, with one exception: a VTSingle or
VTCluster without `retentionDiskUsageBytes` gains
`retention.maxDiskUsagePercent: "80"` in its `extraArgs`, so its spec
changes and its storage pods restart once, and a store already more than
80% full drops its oldest days when it starts. The inner monitoring
HelmRelease's values gain the new `tracingCentralHost: false` default,
as with any new field, and its `healthCheckExprs` gain a `VMAuth` entry,
which matches nothing outside a hosting tenant-root. Both change the
HelmRelease spec, so every tenant's monitoring release upgrades once,
rendering the same objects. A tenant-root that has a cluster-mode
`generic` store gets no vmauth until it sets `tracingCentralHost`.

**Testing.** helm-unittest covers the three charts the change touches
(tenant, monitoring, extra monitoring) and every guard was checked by
mutation; a bats test covers what helm-unittest cannot, that the
generated password and the collector's rollout checksum agree on a first
install. On a test cluster: a spoofed `AccountID` from a tenant is
overridden by vmauth, one tenant cannot read another's spans, the vmauth
starts under the real tenant network policies (and stays in init without
its apiserver label), the guards refuse as described, and the rendered
collector config passes `otelcol validate` in the pinned image. There is
no end-to-end test yet; one that covers cross-tenant isolation is
planned as a follow-up.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: covered by cozystack/cozystack#3772, the tracing guide
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#64
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The Tenant gains the user-facing field `tracingCentral` (bool, default
false), which the trigger map sends to terraform-provider-cozystack: its
hand-written Tenant schema and model need the field. The follow-up is
cozystack/terraform-provider-cozystack#64. The Monitoring gains
`tracingCentralHost` (bool, default false); the provider's
`cozystack_tenant_module` is a marker without values, so it needs no
change. The website reference pages for both are generated from the
READMEs on release. The tracing user guide in #3772 covers the
per-tenant backend only. A central-mode section in the tracing guide of
#3772 (hosting and opting in, the shared disk and retention, turning
central off on the tenants before tenant-root stops hosting, and the
recovery from the opt-in/stop-hosting race) also takes the lifecycle
details the `tracingCentral` description no longer carries.

### Release note

```release-note
feat(monitoring,tenant): add an opt-in shared-central traces store. A tenant sets tracingCentral to send its spans to one VictoriaTraces store in tenant-root instead of its own, and tenant-root opts in to hosting it with tracingCentralHost on its Monitoring. Tenants sharing the store are isolated on write and read through a vmauth in tenant-root, and each is capped in the requests it can hold in flight.
```
2026-10-02 10:07:12 +03:00
Aleksei Sviridkin fccb7fee06 test(tests): fail absence-only cases when the template renders nothing
Each of these templates is wrapped in a top-level condition, and the
case asserts only that a field is absent. If a values change turns the
condition off, the render is empty and the absence holds trivially, so
the case passes without looking at anything. A document selector makes
an empty render fail with "document not found".

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:27:17 +03:00
Aleksei Sviridkin ef5ef06806 test(tests): select documents by identity in absence-only test cases
These cases picked a document by position and then asserted only that
something is absent from it. When a template gains or reorders a
document, the index lands on a different object that lacks the field
just as well, and the case stays green while the property it names is
no longer checked. Selecting by kind or metadata.name keeps each case on
the object it names, and fails with "document not found" when that
object stops rendering.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:27:17 +03:00
Aleksei Sviridkin 4583ee595f fix(kubernetes-nodes): refuse a resources.cpu that cannot be a core count
On the kubevirt substrate a worker pool's resources.cpu, although
schema-typed as a Kubernetes quantity, is written verbatim into
domain.cpu.cores, which KubeVirt declares as uint32 and the
KubevirtMachineTemplate CRD as an integer. A value such as 500m or
"1.5" therefore rendered a template the API server refuses, and
because the same raw string was truthy it also counted as explicit
sizing and dropped the pool's instancetype matcher first. A whole
count past the uint32 ceiling rendered too, although KubeVirt cannot
hold it.

The render now decodes the value the way the emitted cores field is
decoded and stops unless that is a whole number between 1 and the
uint32 maximum, before the value reaches the VM template or the
precedence decision. Only inputs that could not produce a working
pool are refused. Every value that rendered one still renders the
same bytes, so no pool rolls: a bare numeric zero stays falsy and
means unset, and a leading-zero spelling is still emitted as written.

The proxmox substrate is left out of the check, because it converts
the quantity to millicores and rounds up to whole cores, so 500m is a
valid size there. The field description now says what the value
means on each substrate.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:10:40 +03:00
Aleksei Sviridkin 548d33f01f build(images): fall back to direct and retry Go module downloads
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.

Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.

The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
Mattia Eleuteri f714ada567 fix(postgres): refuse a physical restore across image flavors
A recovery replays the source's data directory, so it carries the same
risk as switching the image of a live cluster: the postgresql and
postgis images are built on different Debian releases, and the glibc
change invalidates collatable indexes, while a postgis-to-postgresql
move also loses the libraries behind every PostGIS object. The chart
only guarded the live cluster, so a fresh release bootstrapped from a
backup of the other flavor went through, and the guard's own advice was
to restore a backup, which is that very move.

The backup-controller now records the source flavor in the snapshot it
stores on each Backup and fails a RestoreJob whose target runs another
flavor, before the purge deletes the target's Cluster and PVCs. A
snapshot without a flavor predates flavors and so comes from a
postgresql cluster. For the chart-managed bootstrap, the chart compares
against the recovery source's Cluster while it still exists. Both paths,
and the live-cluster guard, point a flavor change to a logical dump.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 18:31:18 +02:00
Mattia EleuteriandMatthieu dc2971264d feat(postgres): add a postgis image flavor
Tenants migrating spatial data need PostGIS, but the chart always runs
the CloudNativePG PostgreSQL image, which does not ship it, and exposes
no way to pick another one. CloudNativePG publishes a PostGIS operand
image; a closed `flavor` enum selects it without handing tenants a free
image field.

The flavor pins the same PostgreSQL minor as the default one, on the
standard-trixie variant: backups go through the barman-cloud plugin,
so the deprecated system variant is not needed. update-versions.sh now
derives the postgis map from the minors it picks, and a bats check
fails if the two maps drift apart.

The default images are built on bullseye and the postgis ones on
trixie. Switching a live cluster between them would swap glibc under
an initialised data directory and silently break every collatable
index, and CNPG would roll it out as a plain image update. The chart
therefore refuses a flavor change on an existing Cluster.

Co-authored-by: Matthieu <matthieu@hidora.com>
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 18:30:37 +02:00
Alexey Artamonov 68ee97300b docs(monitoring): say a central tenant's collector still needs enabling
The tracingStorages description said a tenant with shared-central
tracing gets the collector without a local entry, which reads as if
tracingCentral alone deploys it. tracingCollector.enabled still
decides whether it renders.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-01 18:02:22 +03:00
Alexey Artamonov 69a16bd2da docs(tenant): shorten the tracingCentral description
The description ships in the godoc, the schema, the README, the
ApplicationDefinition and the dashboard's field help, and at about
2,800 characters it buried what an operator needs to decide on the
field. It now says what the field does, what tenant-root has to host
first, and who is inside the shared store's trust boundary; the
lifecycle details move to the user guide.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-01 18:02:22 +03:00
Aleksei Sviridkin 889bb79236 feat(harbor): restrict the portal and API to listed source ranges (#4300)
## What this PR does

Adds `managementWhitelist` to the Harbor app, a list of IP addresses or
CIDR ranges allowed to reach the web portal and the Harbor API, and I
made it a typed list on purpose. The obvious alternative, a free-form
`ingress.annotations` map, gives tenants a way into the shared ingress
controller. The platform ingress-nginx runs with
`allowSnippetAnnotations: true` (the kubernetes app needs a
`configuration-snippet`), and v1.11.5 defaults `annotations-risk-level`
to `Critical`, so it accepts any annotation, `server-snippet` included.
Tenants can't create Ingress objects themselves, but they can write any
field of an app spec. An annotation map on the Harbor spec would let any
tenant put arbitrary nginx config into a controller that serves every
tenant. A typed list is also what the repo already does for this:
`filer.whitelist` in `extra/seaweedfs`, `whitelist` in `extra/ingress`
and `tcp-balancer`.

The use case is a registry that serves customers: pulls and pushes come
from anywhere, the admin UI and API should answer only to an office or
VPN range. Right now the portal, `/api/v2.0` and `/v2/` sit on one
Ingress and one public host, and a tenant can't restrict any of them.

With a non-empty list the existing Ingress gets
`nginx.ingress.kubernetes.io/whitelist-source-range`, and `/v2` and
`/service/token` move to a second Ingress, `<release>-registry-ingress`,
on the same host. ingress-nginx applies that annotation only to the
locations of the Ingress that carries it, so those two stay open. They
are what a registry client needs: Harbor answers an anonymous `/v2/`
request with a Bearer challenge whose realm is `/service/token` on the
same host. Everything else under `/service/` is
`/service/notifications/*`, internal job callbacks that the bundled
nginx already answers with 404, so it stays on the restricted side. The
second Ingress uses the same TLS secret and has no cert-manager
annotations, otherwise ingress-shim gets two owners for one Certificate.
The HTTP-01 solver is its own Ingress and is not restricted. With the
default empty list the chart renders exactly what it renders today.

An HTTPRoute can't express a source restriction, so in Gateway API mode
a non-empty list fails the render. Dropping it silently would leave the
portal public while the spec says it is restricted.

Each entry has to be an IP address or CIDR range, and a bad one fails
the render. ingress-nginx reads an empty `whitelist-source-range` as no
restriction and turns an unparseable one into a 503 for the whole
location, and the values schema generator can't validate list items.

Three things users should know. Harbor-to-Harbor replication that pulls
from this registry goes through its API, so the other registry's address
has to be on the list. The check uses the client address that
ingress-nginx sees: with `externalTrafficPolicy: Local` or PROXY
protocol that is the real client, behind another proxy it is the proxy.
And for the portal and API the list replaces the tenant ingress
`whitelist` rather than narrowing it.

Disabling the portal is not here. The vendored Harbor chart (1.19.1)
always renders the portal Deployment and its internal nginx always
proxies `/` to it, so there is no small switch to wire.

The new Ingress is added to the dashboard Role and to the ingress
selector in the ApplicationDefinition.

### Screenshots

No UI code change. The field reaches the dashboard form through the
regenerated schema and `keysOrder`.

### Downstream repositories

The terraform provider models Harbor by hand, so the new field needs a
follow-up there: cozystack/terraform-provider-cozystack#46. The website
reference page is generated from `README.md` on release. Nothing else in
the trigger map is touched: no package added or renamed, no platform,
installer or `hack/` change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/46
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(harbor): add `managementWhitelist`, a list of IP addresses or CIDR ranges allowed to reach the Harbor web portal and API. Only `/v2/` and `/service/token` stay reachable from any address. The default is an empty list, which keeps the current behaviour. Not supported with Gateway API publishing: a non-empty list fails the render there. Each entry must be an IP address or CIDR range; for the portal and API the list replaces the tenant ingress `whitelist`.
```
2026-10-01 17:39:56 +03:00
Aleksei Sviridkin 54159736bc feat(talos)!: bump Talos to v1.14.2 (#4598)
## What this PR does

Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a
newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and
v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4.
ZFS moves from 2.4.3 to 2.4.4.

On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning
on a failed send (EPIPE). What followed was RCU stalls, processes stuck
in D-state, a hanging `drbdsetup disconnect` and two-phase-commit
timeouts. The [9.3.4
ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has
fixes for a sender thread pinning a CPU while its connection is down,
for two-phase-commit issues and a hanging `drbdadm disconnect`, and for
two nodes ending UpToDate with different data after a reconnect. None of
them are in 9.3.3. I matched the trace to those entries by their
descriptions only, not against the commits.

The base installer image had to change. Talos v1.14 no longer publishes
`ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses
`ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager
profiles. I built the amd64 installer from the new profile with
`imager:v1.14.2` locally and it builds without errors.

PR CI will not boot the new image. It builds the installer and matchbox
images, but the e2e runs on the upstream `talos:v1.13.5` container from
`hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD
and ZFS is the nightly after merge, which builds the nocloud disk from
these profiles and runs it in QEMU.

On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on
demand. After upgrading a node from v1.13.6, `drbd` was loaded and the
transport module was not, `drbdsetup new-peer` failed with "Failed to
create transport (drbd_transport_xxx module missing?)", and every DRBD
resource on the node stayed in Connecting. Listing the module in
`machine.kernel.modules` fixed it without a reboot. The cause is
[siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565), in
Talos since v1.14.0: the kernel is built with an empty modprobe path, so
it no longer loads modules on `request_module()`, which is how DRBD asks
for its transport. Any module that used to load that way now has to be
listed explicitly (reported upstream as siderolabs/talos#14501). Outside
DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2
kernel builds as modules, so LVM-thin or multipath users on Talos need
them listed too. The e2e node config now loads `drbd_transport_tcp`
explicitly, and the talm preset (cozystack/talm#252) and the install
docs (cozystack/website#720) get the same line. The module ships in the
drbd extension on v1.13 too, so the extra line is safe before the
upgrade.

The order matters for host nodes. Either update the cozystack preset in
the talm project to a version with cozystack/talm#252 and re-render and
apply the node config, or add `drbd_transport_tcp` to
`machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2.
A node upgraded first loses DRBD replication until the module is added.

The host network policy now also denies port 2383 to `world`. Talos
v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from
2379 to a separate listener on 2383, and the policy only blocked 2379
and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes
would be reachable from outside, protected only by client mTLS.

Two smaller changes come with the bump. The multus
`talos-cni-plugins-checked-against` marker moves to v1.14.2, because the
pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as
multus. The system memory limits doc now describes the v1.14 OOM
trigger, since v1.14.0 dropped its global memory PSI clause
([siderolabs/talos#13895](https://github.com/siderolabs/talos/pull/13895)).

Before upgrading hosts, operators should know a few things from the
[v1.14.0 release
notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the
Talos compatibility code:

- Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports
Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a
cluster on Kubernetes 1.31 has to upgrade Kubernetes first.
- etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to
2383. The Cozystack etcd scrape proxy reads a separate metrics listener
on `127.0.0.1:2381`, which this change does not move. Anything else that
scraped or health-checked etcd on 2379 has to switch to 2383, and a
firewall outside the cluster that blocked 2379 should block 2383 too.
- etcd and kube-apiserver now require TLS 1.3, so a client that can only
speak TLS 1.2 stops connecting.
- Workload isolation (`SecurityProfileConfig`) stays off on upgraded
clusters, but `talosctl gen config` turns it on for new ones. I haven't
tested Cozystack with it enabled.
- `talosctl apply-config --mode=reboot` is gone.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#720
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
cozystack/talm#252
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website PR refreshes the `next` version pins for v1.14.2 and adds
the module to the Talos install pages. The talm PR adds it to the
cozystack preset.

### Release note

```release-note
feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Upgraded host node support to Talos v1.14.2 across machine images,
with refreshed firmware and storage extensions.
* Updated memory-pressure guidance for Talos v1.14, including
version-specific behavior and configuration recommendations.
  * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383.
* Updated cluster preparation to load the TCP transport module required
for DRBD.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:03:30 +03:00
scooby87 031ded71ee feat(seaweedfs): alert before an instance stops taking writes (#4573)
## What this PR does

A SeaweedFS instance that runs out of room stops taking writes without
anything paging. A volume server below its `minFreeSpacePercent` reserve
turns every volume it holds read-only; before that, it can run out of
slots, and a bucket whose volumes are all full then cannot get a new
one, even with half the disk free. Backups archiving into the instance
fail in turn, and a Postgres cluster doing continuous WAL archiving
keeps the segments on its own volume until that fills too.

The generic `KubePersistentVolumeFillingUp` alert does not catch this:
it pages below 3% free, while a volume server stalls at its 5% reserve
or, when slots run out, with the disk still half empty; and its early
warning needs the disk to keep filling, which stops once writes fail.

This PR adds per-instance rules to the SeaweedFS system chart, on
signals the servers already publish and the platform already scrapes,
rendered under the same `monitoring.enabled` gate as the chart's
ServiceMonitors:

- `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less
than 10% of its disk left above the reserve.
- `SeaweedFSVolumeServerFull` (warning): a volume server cannot host
another volume, because it is at the reserve or has no slot left.
- `SeaweedFSWritesFailing` (critical): the master has failed to place a
write in every 5-minute window for 15 minutes. Writes may still land in
between; the usual cause is a bucket with no writable volume left that
the master cannot grow. A new bucket's first writes fail a few picks
while its volumes grow, which the 15-minute hold does not reach unless
new buckets keep arriving.

Every expression is pinned to the release namespace, because the root
vmalert evaluates each instance's rules against every namespace's
metrics.

It also adds `docs/operations/seaweedfs-capacity-for-backups.md`: how to
size an instance for continuous WAL archiving, what the rules mean, and
how to grow an instance.

The expressions were validated live rather than by unit tests, since the
repository has no rule-evaluation tooling: on a test cluster, a
standalone SeaweedFS 4.45 on a 1 GiB volume was scraped by the
platform's vmagent and the rules rendered from this chart were evaluated
by the root vmalert. All three loaded healthy and each reached `firing`:
`SeaweedFSVolumeServerDiskLow` with 5.7% of the disk left above the
reserve, `SeaweedFSVolumeServerFull` both at the reserve and on slot
exhaustion, which stopped one bucket at 50% disk use, and
`SeaweedFSWritesFailing` under continuous failing writes, resolving once
they stopped. A new bucket's first successful writes on a free disk did
fail a few picks, which is why that rule holds for 15 minutes. On the
healthy platform instance all three stay quiet. helm-unittest pins what
a render can break: the namespace selector of each rule, the shared
label set of the two arms of the full-server rule, the gate, the
critical rule's hold, and that alert templating reaches vmalert
unrendered; each assertion was checked to fail when its line is mutated.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches a system chart template, its test and a document under
`docs/operations/`; no app values, platform values,
ApplicationDefinition, tooling or `hack/` path from the trigger map.

### Release note

```release-note
feat(seaweedfs): alert when a SeaweedFS volume server nears its free-space reserve, cannot host another volume, or keeps failing to place writes, and document how to size an instance for continuous WAL archiving.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added namespace-scoped SeaweedFS monitoring alerts for low or
exhausted disk capacity, unavailable volume slots, and write-placement
failures. Capacity alerts use a 10% disk threshold, and alerts trigger
after 15 minutes; write failures are marked critical.
- **Documentation**
- Added guidance on estimating SeaweedFS storage needs for Postgres
backups, interpreting capacity alerts, and expanding storage when
supported by the StorageClass.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 10:29:35 +03:00
Aleksei Sviridkin 1e81164d0e fix(kubevirt): route the export proxy to the Service KubeVirt creates
virt-operator names the export proxy Service virt-exportproxy. The
Ingress and the TLSRoute publishing vm-exportproxy.<host> named a
vm-exportproxy Service instead, which nothing creates, so both routes
had no backend. The public hostname stays vm-exportproxy.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 03:11:22 +03:00
Alexey Artamonov d5fc723208 feat(monitoring): bound every traces store by disk usage
The distributed tracing design requires each traces backend to be
disk-bounded, not only age-bounded, so a full volume cannot block
ingest. retentionDiskUsageBytes was optional, so a store without it
filled its volume and stopped taking spans; with a shared-central store
that stops ingest for every tenant at once.

A store without a byte cap now drops its oldest day once its
filesystem is 80% full. VictoriaTraces refuses to start with both
limits set, so a byte cap, when given, replaces the percentage one.
It always keeps the last two days, which the values text now says.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 23:29:41 +03:00
Alexey Artamonov 2d10565873 feat(tenant): let a tenant opt into shared-central tracing
tracingCentral turns the shared-central topology on for a tenant with
its own Monitoring: it publishes _namespace.tracingCentral for the
tenant's monitoring charts and renders the only two cross-boundary
paths that topology needs, the collector and Grafana each to the
tenant-root vmauth. Both come from one condition, so a tenant never
exports to the shared store without the path, or gets the path without
opting in, and tenant-root, which hosts the store, gets neither. The
flag is written only for tenants that opt in: every application in a
tenant reads that Secret, and a new key for all of them would
re-upgrade them all on the platform upgrade.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 22:23:47 +03:00
Alexey Artamonov 27b7376bb9 feat(monitoring): add a shared-central tracing topology
A per-tenant traces backend costs every tenant its own VictoriaTraces
stack. A tenant can now send new spans to one shared store in
tenant-root instead, switched on by _namespace.tracingCentral, which
the tenant chart publishes and the apps API does not let an
application set. tenant-root decides whether it hosts that store, with
tracingCentralHost next to a cluster-mode store named generic.

VictoriaTraces takes the tenant account from request headers and does
no authorization of its own, so other tenants reach the store only
through a vmauth in tenant-root; tenant-root itself, with whoever it
admits to its Grafana, stays inside the trust boundary. Each central
tenant gets a VMUser that pins an account derived from its namespace
name and UID for both writes and reads and caps its requests in
flight, which the collector's two export senders stay under; its
collector exports and its Grafana reads with credentials the chart
generates and keeps, and vmauth overrides any account a client sends.
The chart owns those credentials because the operator publishes a
VMUser's own Secret only once a VMAuth selects it; the collector rolls
when they change, since it reads them at start. The vmauth internal
routes, whose /metrics names every tenant's user, are served on a port
tenants cannot reach, and the vmauth is sized by a VPA, spread across
nodes under a disruption budget, has a WorkloadMonitor like the other
monitoring workloads, and may reach the apiserver, where its
config-reloader reads its config Secret.

Renders refuse the states in which spans would be dropped with every
object Ready: a central tenant while tenant-root hosts nothing,
tenant-root dropping the hosting while central tenants remain, and
the host flag without a store to host. Each refusal reads the other
side at render time, so a concurrent opt-in and stop-hosting can pass
both; the values text names that race and its recovery. At run time
tenant-root's release is not Ready while the vmauth has failed.
Only the collector's export moves: local stores the tenant still lists
keep rendering with their datasources, because the switch belongs to
the parent tenant and the stores to this one, and the one-store limit
of the collector holds in both modes, since turning central off makes
it export locally again. tenant-root never counts as central.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 22:23:47 +03:00
Aleksei Sviridkin c983b5c404 fix(linstor): use severity warning instead of warn in piraeus-datastore alerts (#3799)
## What this PR does

Seven of the piraeus-datastore alert rules label themselves `severity:
warn`. Alerta, which the monitoring package ships as the alert sink,
validates that value and `warn` is not on its list. Every webhook
delivery for these alerts dies with a 500.

```
Severity (warn) is not one of security, critical, major, minor, warning,
indeterminate, informational, normal, ok, cleared, debug, trace, unknown
```

Alertmanager retries seven times and drops the notification, and the
alert never shows up anywhere. The affected rules are the LINSTOR/DRBD
ones, such as storage pool at capacity, lost quorum, unintentional
diskless and stuck resync. These are exactly the alerts you least want
silently dropped.

Found this on a v1.4.6 cluster where `linstorStoragePoolAtCapacity` was
genuinely firing. The cluster had 153 dropped notifications in 40
minutes and nothing in Alerta. Renaming the label to `warning`, which
the other three rules in the same file already use, stopped the failures
immediately and the alerts came through.

The same labels sit on `main` and in `v1.6.1`, so a backport to
`release-1.6` would be appreciated.

The test commit adds a bats guard so the next one gets caught in CI. It
walks every `PrometheusRule` and `VMRule` under `packages/` outside
vendored `charts/`, helm templates included, and fails on a severity
outside Alerta's default alarm model, which is exactly where `warn`,
`info`, `error` and `fatal` fall. Helm actions are stripped before the
file is read, so a rule behind a `{{- if }}` or an unquoted `{{
.Release.Namespace }}` is checked rather than skipped, and a file that
still does not parse fails the contract instead of thinning it out. On
the current tree that is 194 alerting rules in 41 scanned files.

The guard found one more on `main`. `DatabaseAutoscalerAtMax` in
`packages/system/keda/templates/alerts.yaml` carries `severity: info`,
which Alerta refuses the same way. It is renamed to `informational`, the
value the model has for that level and the one seven other rules in the
tree already use.

Moving that alert off `info` also stops `InfoInhibitor` in
`packages/system/monitoring-agents/alerts/general.rules.yaml` from
firing. That rule matches any `severity="info"` alert and carries
`severity: major`, so on `main` it usually reached Alerta whenever
`DatabaseAutoscalerAtMax` fired, even though the keda alert itself was
refused. After this change no rule carries `info`, so operators no
longer see that `InfoInhibitor` notification.

Fixes #2411.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. It touches severity labels
inside `packages/system/piraeus-operator/alerts/` and
`packages/system/keda/templates/`, plus a bats file under `hack/`. No
package is added or renamed, and there is no values schema, CRD, tooling
the satellites gate on or node contract change. Nothing downstream
restates these labels.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(linstor): LINSTOR/DRBD alerts carried severity `warn`, which Alerta rejects, so they were never delivered. They now use the standard `warning` severity, and the keda at-max autoscaler alert moves from `info` to `informational` for the same reason.
```
2026-09-30 21:44:57 +03:00
Aleksei Sviridkin 2c2f2c194e fix(dashboard, foundationdb): show the connection ConfigMap to tenants (#4612)
## What this PR does

Fixes #4611.

Since #4148 a tenant may read the `<release>-config` ConfigMap that
holds the FoundationDB cluster file, but the console has no view for
ConfigMaps, so the connection string was still reachable only with
kubectl.

Chart:

- `<release>-resourcemap` is rendered regardless of
`monitoring.enabled`, because the console now reads it;
- the tenant Role grants `get`, `list` and `watch` on
`<release>-resourcemap` next to `<release>-config`, still by
`resourceNames`, with no wider ConfigMap access.

Console:

- the application page gets a ConfigMaps tab when the application's
resource map lists at least one ConfigMap in the same namespace;
- each listed ConfigMap is read and watched with a `metadata.name` field
selector, which RBAC checks against `resourceNames`, so a tenant never
needs `list` on all ConfigMaps;
- values are shown as plain text with a copy button, the cluster file
needs no decoding;
- a resource map that is missing or forbidden hides the tab, a listed
ConfigMap that cannot be read shows the API error instead of looking
empty, and a malformed map or an invalid name is reported.

The tab is generic, but today only FoundationDB ships a resource map, so
it is the only application that gets it.

#3490 keeps `monitoring.enabled` as the switch for the resource map and
asks whether the map should render unconditionally. This PR answers that
for FoundationDB by rendering it always, so the wording in #3490 would
need to follow if both land.

Tests: `helm unittest` for the chart (6 tests), `tsc --noEmit`, `vitest
run` (438 tests in 60 files) and `eslint` on the changed files pass. The
new tests cover reading only the listed names in the active namespace,
the watch, copying, a clipboard failure, a read denial, and malformed
maps. The chart renders with `monitoring.enabled` true and false.

### Screenshots

The ConfigMaps tab of a FoundationDB application, rendered by the
console from this branch against a local stub API with synthetic data:

![ConfigMaps
tab](https://raw.githubusercontent.com/yankawai/cozystack/29ae3c94e539edc05b29700788a9b875b5a13d54/foundationdb-configmaps-tab.png)

### Downstream repositories

Walked the trigger map against the diff: it changes a FoundationDB chart
template and its test, and the console. No `values.yaml`,
`values.schema.json`, README, CRD or `hack/` file is touched. The
website FoundationDB page is generated from the chart README and says
nothing about connecting, and the Terraform provider does not read these
ConfigMaps. The console is vendored in this repository.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(dashboard, foundationdb): the application page shows the FoundationDB connection ConfigMap with the cluster file to tenants who may read it
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Application details now include a ConfigMaps tab when configuration
maps are available. View their values and copy them to the clipboard.
  * The tab displays loading, empty, unavailable, and error states.
* **Bug Fixes**
* The resource map remains available when monitoring is disabled, and
its access permissions are included.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 21:23:05 +03:00
scooby87 51155f1003 feat(postgres-operator): alert when the WAL archive backlog does not drain (#4572)
## What this PR does

`LastFailedArchiveTime` compares a cluster's last failed WAL archival
with its last successful one, so it resolves as soon as any segment gets
through. Archiving that fails intermittently, or runs slower than the
cluster writes WAL, keeps clearing that alert while the queue of
segments waiting on the Postgres volume grows, until the volume is full
and the database stops. The alert is also only a warning.

This PR adds `CNPGWALArchiveBacklog`, a critical alert that fires when a
primary's `archive_status` queue has not been empty once in 30 minutes.
A healthy archiver empties the queue within seconds of each WAL switch,
so a queue that never drains means point-in-time recovery is falling
behind, whatever the archiver reports. Idle clusters and replicas have
an empty queue and stay quiet.

Validated live, since the repository has no rule-evaluation tooling: on
a test cluster, two CNPG clusters archived into an object store that
refused connections and into one that accepted them and never answered,
while writing WAL continuously. The rule, evaluated by the root vmalert,
fired for both. Over the previous seven days of the same cluster's
metrics, none of its healthy clusters would have matched; those clusters
carry light load, so a cluster whose archiver runs near its limit all
the time has not been observed and would page, which is the intended
reading of a queue that never drains. The unit test pins only that the
rule ships in the rendered PrometheusRule as critical, since the rules
reach the cluster through a file glob that drops a moved file silently.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches an operator chart's alert rules and its test; no path
from the trigger map.

### Release note

```release-note
feat(postgres-operator): page when a Postgres cluster's WAL archive queue has not drained for 30 minutes, which the existing archiving alert misses while some archives still succeed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a critical alert for sustained PostgreSQL WAL archive backlogs.
It triggers when the minimum observed ready archive status remains above
zero over a 30-minute window, then stays pending for 10 minutes before
firing. The alert describes potential impacts on point-in-time recovery
and storage usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 21:21:22 +03:00
Aleksei Sviridkin 97fa2a619f feat(dashboard): show a connection hostname for LoadBalancer services
Users who provision a service through the dashboard had no hostname to
connect to: the Services tab showed only the raw LoadBalancer IP, while
the pre-1.4.0 UI composed <service>.<root host> for every Service. Render
<service>.<suffix> under the External IP of every LoadBalancer Service
(also while the IP is still Pending), and nothing for ClusterIP Services.

The suffix is config.json serviceDomain, set by the platform value
publishing.serviceDomain. There is deliberately no fallback: the app
charts fix their external name as <release>.<tenant host> while the
LoadBalancer Services are named <release>-external-write and similar,
so a name derived from the tenant apex would match neither the DNS nor
the certificate. Without a configured suffix nothing is rendered.

Runtime config previously reached only the Shell as a prop; it is now
published through AppConfigContext and read from there by the Shell and
the Services tab alike, so a single test on the rendered version pins
the wiring.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 18:17:38 +03:00
Aleksei Sviridkin 67adfbce3c feat(dashboard): surface service-domain in the console config.json
The console loads config.json once at startup and treats a missing
serviceDomain key as "show no hostname", so the chart copies
_cluster.service-domain into config.json only when it is non-empty rather
than shipping an empty string the SPA would have to special-case. The
helm-unittest suite pins the unset, empty and set cases.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 18:17:37 +03:00
Aleksei Sviridkin 0bc3f9f25c fix(kubernetes): name TalosConfigTemplate by content hash (#3523)
## What this PR does

Fixes #3515.

`TalosConfigTemplate.spec` is immutable — the
`vtalosconfigtemplate.cluster.x-k8s.io` webhook denies any mutation with
`TalosConfigTemplate.Spec is immutable` — but the `talos-reconcile` Job
applied it under a fixed `<release>-<group>` name on every reconcile.
Once the template existed, **no change to the rendered worker machine
config could ever reach the cluster**: the apply was rejected
server-side while the HelmRelease stayed `Ready`. Reported with
`machine.network.searchDomains` after a `clusterDomain` correction; the
same held for `certSANs`, `machine.registries` and nameservers.

- Name the `TalosConfigTemplate` `<release>-<group>-<6-char hash of its
spec>`, mirroring the `KubevirtMachineTemplate` content-hash idiom the
chart already uses, and point
`MachineDeployment.spec.template.spec.bootstrap.configRef` at that name.
A content change now creates a **new** template instead of attempting a
rejected mutation, and CAPI rolls the workers onto it.
- Extract the machine config into a single named template
(`_talosconfigtemplate.tpl`) shared by the two consumers that have to
agree on the name byte-for-byte: the Job, which embeds it in the heredoc
it `kubectl apply`s, and the `MachineDeployment`, which hashes it into
`configRef`.
- Assemble the node group once, in `kubernetes-nodes.group`, and read it
back at both call sites. Sharing the rendering was not enough: each file
also built its own `$group` dict, so the two names agreed only by
coincidence of the keys the spec helper happens to read.
- The runtime values the Job substitutes (`${TALOS_CA_B64}`,
`${SVC_IP}`, `${COREDNS_IP}`, tokens) stay shell placeholders and are
deliberately **not** part of the hash: they are facts of an
already-provisioned cluster, not chart inputs, so a certificate rotation
or a Service re-address does not roll workers.

### What the shared spec carries

Worker node pools live in the `kubernetes-nodes` chart, which owns
`_talosconfigtemplate.tpl` outright: `make update` copies nothing from
the parent chart.

The helper renders everything `main` puts into the worker machine
config, so the hashed spec is the one the cluster gets:

- The heredoc escaping from #3513: backslash, dollar and backtick
escaping on `talosVersion`, `installerRepository` and `schematicID`, the
escaped `machine.registries.mirrors` block, and the `INVARIANT` comment
above `data:`, which belongs in that file since it is where the next
field will be added. The spec is still emitted from an unquoted heredoc
in a host-cluster Pod, so an unescaped copy would be command injection.
- The per-pool install image. `_talosconfigtemplate.tpl` calls
`kubernetes-nodes.resolveOsImage` itself instead of reading
`talos.schematicID` and `talos.version`, so a pool on `osImage.builtin`
or `osImage.factory` reinstalls the schematic and version it actually
booted. The install image is part of the hashed spec, so the hash has to
be taken over the resolved image, or an `osImage` change would render a
different config under an unchanged name.
- The Proxmox branches: `nameservers` from `proxmox.dnsServers` where a
KubeVirt pool points at the cluster's CoreDNS, and the kubelet's
`cloud-provider: external`, without which the Proxmox
cloud-controller-manager never initialises the Node.
- `machine.kernel.modules`, emitted only when the pool resolves to a
non-empty list.

`kubernetes-nodes.group` carries every key those read, `podCpuLimit`,
`podCpuRequest`, `osImage`, `proxmox` and `kernelModules` included. A
key missing there is the failure this PR exists to prevent: with
`kernelModules` read at one call site only, an explicit module list
hashes to two different names and the `MachineDeployment` points at a
template nothing creates.

The `kubectl apply` stays `--server-side --force-conflicts` (#4163), so
the Job is not OOMKilled on the merge path; with a content-hash name the
apply is normally a create or a no-op, and server-side apply keeps the
no-op cheap too.

### The replace-on-immutable path and the Role

The runtime values stay outside the hash, so one failure is left that
the hash cannot absorb: a new apiserver address renders a different spec
under an unchanged name, and the webhook rejects it. The replace path
from #4089 handles exactly that case and is kept, with its `kubectl
delete` pointed at `${TCT_NAME}`.

The talos-reconcile Role pins its TalosConfigTemplate `get`, `patch`,
`update` and `delete` rights to the pool's own template, so it names the
hashed template too. A server-side apply of a template that does not
exist yet is still a PATCH on its name: a rule left on
`<release>-<group>` would leave the Job Forbidden and the pool with no
template to scale onto, with nothing turning red. The name-agreement
guard below therefore covers the Role as well.

## Upgrade impact — please read

**Existing clusters roll their Talos workers once.** The `configRef`
moves from `<release>-<group>` to the hashed name; CAPI creates a new
`MachineSet` for it and rolls machines onto it under the existing
`RollingUpdate` strategy (`maxSurge: maxReplicas`, `maxUnavailable: 1`)
plus the group's `MachineHealthCheck`. The rendered machine config
itself is byte-identical to `main` for the same values, so this
particular roll delivers no config change — it is the one-time cost of
adopting the content-hash name, the same cost `KubevirtMachineTemplate`
already paid. Proxmox pools roll the same way: the
`ProxmoxMachineTemplate` name does not move, but the `configRef` does.

**Until the new template exists, the pool cannot scale.** The Helm
upgrade points the `MachineDeployment`'s `configRef` at the hashed name
before the `talos-reconcile` Job has created that template, and there is
no render-time ordering fix, since the template needs values Helm does
not have. Existing workers are untouched in that window, but the pool is
frozen: in CAPI v1.10.1, which this repository ships,
`getTemplatesAndSetOwner` records the missing bootstrap template, and
`rolloutRolling` then fails in `getAllMachineSetsAndSyncRevision` with
`cannot create a new MachineSet when templates do not exist` and returns
before it scales either the new or the old `MachineSet`. A replica
change from the cluster-autoscaler therefore reaches no `MachineSet`:
scale-up and scale-down both wait. On an upgraded cluster the Job's
inputs already exist, so the window is normally a few seconds.

**A failed `talos-reconcile` Job now blocks autoscaling, and nothing
turns red.** After `backoffLimit: 30` the Job is not retried until the
next Helm upgrade of the pool release, and the HelmRelease stays `Ready`
meanwhile, as it did in #3515. Before this PR the same failure left the
pool on its old template and still scaling; with it, the pool cannot
scale at all until the template exists. Where to look:

- the `MachineDeployment` `kubernetes-<cluster>-<group>`: its
`ScalingUp` condition under `status.v1beta2.conditions` reads `Scaling
up from N to M replicas is blocked because TalosConfigTemplate does not
exist` (or `Scaling up would be blocked because …` while no scale-up is
pending), and the capi-controller-manager logs carry `cannot create a
new MachineSet when templates do not exist` for it;
- whether a `TalosConfigTemplate` named after its
`spec.template.spec.bootstrap.configRef.name` exists in the tenant
namespace;
- the `talos-reconcile` Job's status and Pod logs — but the Job has
`ttlSecondsAfterFinished: 600`, so a failed Job and its logs are deleted
ten minutes after it gives up. Past that, the `MachineDeployment`
condition and the missing template are the only evidence left.

The pre-existing fixed-name template is **not** deleted — the outgoing
`MachineSet` still references it — and is garbage-collected with the
`KamajiControlPlane` when the cluster is deleted. Hashed templates
therefore accumulate, one per distinct machine config. Pruning them
safely is filed as #4355 rather than widened into this PR.

**The one roll is not the only one.** Hashing the whole rendered spec
makes anything inside it a roll trigger, including two things that are
not edits to this cluster's resource. For a pool sized by
`instanceType`, the kubelet reservations come from a live `lookup` of
the `VirtualMachineClusterInstancetype`, so a later platform-side edit
to that instancetype rotates the name and rolls the pool again — and the
`KubevirtMachineTemplate` hash this mirrors does not have that property,
since it embeds the instancetype by name and never its resolved size.
And `talos.registryMirrors` is inside the hashed spec, so setting or
changing a mirror now rolls the pool rather than applying only to nodes
created later; the field's documentation said nothing about this and now
says it. The same holds on Proxmox for `proxmox.dnsServers`, and on
every substrate for `kernelModules`: both are inside the hashed spec, so
changing either rolls the pool, and the `kernelModules` documentation
now says so instead of describing the fixed name as the current state.

**The roll is not free on a node group holding a scarce resource.** From
exercising this same rollout path on a production fleet of 13 tenant
clusters, by rotating the `KubevirtMachineTemplate` hash on a GPU node
group. `maxSurge: maxReplicas` is dangerous when the resource is
constrained: with 6 GPUs and 1 free, CAPI created all 5 new machines at
once and 3 sat `Pending` on `Insufficient nvidia.com/...`, each having
already imported a 200 GiB disk. It converges, but it wastes a lot of
import. The natural reflex, `maxSurge: 0` with `maxUnavailable: 1`,
deadlocks instead: CAPI computes `maxScaledDown = available - (desired -
maxUnavailable)`, which was `4 - 4 = 0`, so it removed no old machine
while the new ones waited for exactly those machines to release their
GPUs; nothing moved until `maxUnavailable` was loosened to 2. And the
`MachineHealthCheck` amplifies both — at `maxUnhealthy: 100%` and
`nodeStartupTimeout: 10m`, machines queuing for a GPU were declared
unhealthy and remediated in a loop, three deleted and recreated while
perfectly healthy, each iteration re-importing the disk. The
per-node-group `nodeStartupTimeout` override fixes that, but the default
is shorter than a plausible queue wait. Anyone upgrading a fleet with a
GPU pool hits all three, and this PR is what starts the roll.

## Validation

- `helm unittest packages/apps/kubernetes-nodes` passes, snapshots
included. Pointing `configRef` back at the fixed name fails
`nodegroup_test.yaml`, the snapshot and the name-agreement bats;
dropping `osImage` from `kubernetes-nodes.group` fails the
`worker_image_*` suites; removing the Proxmox nameserver branch fails
`substrate_branches_test.yaml` and the Proxmox heredoc case.
- `hack/check-talosconfigtemplate-name-agreement.bats` is new. The
unittest suites pin each name against its own literal in its own file,
so a divergence shows up as one literal having moved, which reads as a
stale fixture and invites re-pasting the value. The guard renders the
chart once and compares the Job's `TCT_NAME`, the Role's
TalosConfigTemplate `resourceNames` and the `MachineDeployment`'s
`configRef.name` against each other, so the failure states the invariant
rather than a number. Asserted on a GPU-shaped pool as well as the
default one, because the default renders `gpus []`, `kubelet {}` and
`instanceType ""` as falsy and a consumer reading a subset of the group
is indistinguishable there from one reading all of it; on an explicit
`kernelModules` list, which neither of those shapes sets; and a last
case pins that the shapes hash differently, so the others cannot hold
vacuously. Verified by injecting each divergence: dropping `gpus` at one
call site, dropping `kernelModules` at one call site, and pointing the
Role back at `<release>-<group>` each fail it, the last one also failing
the Role cases in `talos_tct_immutable_test.yaml`, as does pointing the
`delete` back at the fixed name.
- `hack/cozytest.sh hack/talos-reconcile-heredoc_test.bats` passes, the
Proxmox cases included. This is the execution-level evidence that the
escaping survived the move into `_talosconfigtemplate.tpl`: it renders
with a hostile `registryMirrors` endpoint and a hostile
`installerRepository`, runs the extracted heredoc through a real shell,
and asserts the values come out literal rather than command-substituted;
hostile `schematicID` and `version` values, and `proxmox.dnsServers`
entries that are not addresses, are asserted refused at render; and the
Proxmox heredoc is asserted to emit the nameservers it was given. A
rendered-string regex cannot catch a heredoc the shell refuses to emit;
this does.
- `make generate` in `packages/apps/kubernetes-nodes` — no diff.
- The "byte-identical to `main`" claim was measured against `main` on
six pool shapes: default, GPU, an explicit `kernelModules` list,
`kernelModules: []` on a GPU pool, `osImage.factory` and Proxmox. For
each, the `TalosConfigTemplate` manifest the Job applies differs only in
its name (`${RELEASE}-${GROUP_NAME}` → `${TCT_NAME}`); the spec is
unchanged, the Job's `TCT_NAME` equals the `MachineDeployment`'s
`configRef.name`, and the `KubevirtMachineTemplate` and
`ProxmoxMachineTemplate` names do not move.
- **Not verified:** no live-cluster or e2e run (needs bare-metal nodes).
The upgrade sequence above, the scaling freeze included, comes from
reading the CAPI v1.10.1 `MachineDeployment` controller source and the
chart's existing `KubevirtMachineTemplate` precedent, not from watching
it end to end.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Every box is left empty on purpose, including the first, because I am
not confident about one line of the map and would rather a maintainer
decided than have me tick it. The change adds no field and changes no
default: the only schema movement is the description text of
`talos.registryMirrors` and `kernelModules`, regenerated from their
`values.yaml` doc comments. Nothing else downstream restates this
chart's templates, the hash naming is internal to the chart, and no
image reference, CI behaviour or CRD shape moves. The open question is
whether `terraform-provider-cozystack` restates field descriptions as
well as field shapes; if it does, those description changes reach it and
want a follow-up.

## Release note

```release-note
fix(kubernetes): name the worker TalosConfigTemplate after a hash of its content so machine-config changes (searchDomains, certSANs, registries, nameservers) actually propagate instead of being silently rejected by the immutability webhook. Upgrading rolls existing Talos workers once, on KubeVirt and Proxmox pools alike. Until the talos-reconcile Job has created the new template, the pool cannot scale, so a failed talos-reconcile Job now blocks cluster-autoscaler scale-up and scale-down while the HelmRelease stays Ready; check the MachineDeployment's ScalingUp condition ("blocked because TalosConfigTemplate does not exist") and the Job status, which is deleted ten minutes after it fails. On a node group holding a scarce resource such as GPUs, review the MachineDeployment surge settings and the MachineHealthCheck nodeStartupTimeout first: `maxSurge: 0` can deadlock the roll and the default startup timeout can trigger spurious remediation while machines queue for the resource.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Worker Talos configuration templates now use content-hashed names for
consistent reconciliation and bootstrap references.
* Changes to worker machine configuration, including GPU and registry
mirror settings, automatically roll existing workers.

* **Documentation**
* Updated registry mirror documentation to explain that configuration
changes roll all workers in the pool.

* **Tests**
* Added coverage for hashed template names, configuration-triggered
worker rotations, and matching references across worker resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-30 17:43:09 +03:00
Yan Bondarenko 2fa7fea29b fix(dashboard): show ConfigMaps listed in the application resource map
FoundationDB keeps its connection string in a ConfigMap that its operator
writes, and tenants may read it, but the console had no view for
ConfigMaps. Tenant roles grant ConfigMaps by name only, so the tab takes
the names from the application's resource map and reads each one with a
metadata.name field selector, which RBAC checks against resourceNames.
An application without a readable resource map shows no tab, while a
listed ConfigMap that cannot be read is reported instead of looking
empty.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-30 17:29:49 +03:00
Yan Bondarenko 17aaf3e038 feat(monitoring): add optional email alert delivery
Platform alerts need an email delivery path alongside Alerta without
exposing arbitrary namespace Secrets through tenant-controlled values.
SMTP credentials therefore use a fixed Secret name.

Alertmanager parses the addresses and connects to the SMTP host only
when it sends a notification, so a malformed value there drops alerts
instead of failing the release. The chart validates these fields when
it renders. Addresses must be literal dot-atom mailboxes, and a domain
literal must hold a dotted-quad IPv4 address, because IPv6 literal
syntax differs across the supported Go mail parsers. A bracketed
smarthost must be a valid IPv4 or IPv6 address, and its port must be
between 1 and 65535 without leading zeros. The repeat interval follows
the Alertmanager duration grammar and range, so a value that renders is
one Alertmanager loads.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-30 13:55:12 +03:00
Yan Bondarenko 0cf6e8c98b fix(keda): use severity informational instead of info in the at-max alert
Alerta validates the severity label against its alarm model, which has
no info in it, so the DatabaseAutoscalerAtMax notification is refused
the same way the piraeus warn alerts were. informational is the value
the model carries for this level, and the one the other rules in the
tree already use.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-30 13:53:06 +03:00
Yan Bondarenko 3ade16a380 fix(linstor): use severity warning instead of warn in piraeus-datastore alerts
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-30 13:53:06 +03:00
Kirill Ilin 218384ed32 feat(keycloak): let Keycloak and its database proxy spread across zones
Neither the Keycloak StatefulSet nor the keycloak-kms-proxy Deployment
exposed any placement, so the scheduler was free to put both replicas of
each in one failure domain. For Keycloak that is worse than it looks: a
StatefulSet does not replace a pod stuck on an unreachable node, so losing
that domain takes every replica for as long as it stays lost.

Expose affinity and topologySpreadConstraints for both, empty by default
so nothing changes unless an installation opts in, and give the proxy a
PodDisruptionBudget once it runs more than one replica.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-09-30 14:07:22 +05:00
Alexey Artamonov 2c5c230435 fix(postgres-operator): count every scraped empty queue in the WAL backlog rule
The per-pod subquery stepped once a minute and took the latest sample
at each step, so with a 30-second scrape it looked at half the samples.
A queue that emptied only in a skipped sample did not count, and the
rule could page on a cluster that was keeping up. While a recreated pod
overlapped its old series, taking the maximum also hid the new
series' zero. The minimum of each raw series, then the minimum per pod
name, sees every sample and still keeps the window across a new
instance label.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 11:11:03 +03:00
Mattia Eleuteri 9afe7f1293 fix(kubernetes): assemble the node group once in kubernetes-nodes
The TalosConfigTemplate name is a hash of the machine-config spec, and two
templates have to agree on it: talos-reconcile-job.yaml creates the object,
nodegroup.yaml points MachineDeployment.bootstrap.configRef at it. Sharing the
rendering was not enough — each file also assembled its own $group dict, one
carrying every pool value and one only the few the spec reads, so the two
names agreed only by coincidence of the keys the spec helper happens to read.
Dropping "gpus" from the Job's dict diverged them on a GPU pool with the whole
suite still green, which is the "templates do not exist" deadlock this fix
exists to make impossible.

Move the assembly into a single kubernetes-nodes.group helper, read back with
fromYaml at both call sites, so there is one input as well as one rendering.

Cover it too, since the seam can be reintroduced. The chart's helm-unittest
suites pin each name against its own literal in its own file, which catches a
divergence but reports it as one literal having moved — indistinguishable from
a stale fixture, so re-pasting the new value is the obvious and wrong repair.
hack/check-talosconfigtemplate-name-agreement.bats renders the chart once and
compares the two values against each other, so the failure states the invariant
instead of a number. It asserts on a GPU-shaped pool as well as the default
one: the default renders gpus [], kubelet {} and instanceType "" as falsy, so a
consumer reading a subset of the group is indistinguishable there from one
reading all of it. kernelModules gets a case of its own, because it has no
default and stays unset on both of those shapes. A last case pins that the
shapes hash differently, so the others cannot hold vacuously.

The same guard compares the talos-reconcile Role's resourceNames on
TalosConfigTemplates against that name. The Job applies server-side, and
that is a PATCH on the name even when it creates the object, so a Role
naming anything else leaves the Job Forbidden and the pool without a
template, with the same silent outcome.

Document that talos.registryMirrors and kernelModules sit inside the hashed
spec, so changing either rolls the pool's existing workers instead of only
affecting nodes created later. The kernelModules text described the
fixed-name template as the present state and the content-hash name as a
future one.

No rendered output changes: the golden render snapshots are unchanged.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-09-30 08:59:03 +02:00
Aleksei Sviridkin 4570672186 fix(cilium): deny world ingress to etcd's HTTP port on Talos v1.14
Talos v1.14 serves etcd's /metrics, /health and gRPC-gateway JSON API
on a dedicated listener on 2383 instead of the client port, and its
release notes ask for 2383 to be blocked wherever 2379 was. The host
policy denied 2379 and 2380 to world, so on v1.14 nodes the JSON KV
API would be reachable from outside behind client mTLS alone.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 04:23:38 +03:00
Aleksei Sviridkin e002f8d5e5 feat(talos)!: bump Talos to v1.14.2
Talos v1.14.2 ships DRBD 9.3.4 and ZFS 2.4.4. The v1.13 line tops
out at DRBD 9.3.3, which lacks the 9.3.4 fixes for a sender thread
pinning a CPU while its connection is down, for two-phase-commit
retries and hangs that leave drbdadm disconnect stuck, and for two
nodes ending UpToDate with different data after a reconnect.

Talos v1.14 no longer publishes ghcr.io/siderolabs/installer, so the
imager profiles take installer-base as the base installer, which is
also what upstream's own v1.14 imager profiles use.

BREAKING CHANGE: Talos v1.14 no longer loads kernel modules on demand,
so every node must list drbd_transport_tcp in machine.kernel.modules
before it is upgraded, or DRBD cannot connect to its peers.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 04:23:38 +03:00
Aleksei Sviridkin a09eded9ec fix(kubernetes-worker-image): wait for goldens to import before Ready
The Talos worker image catalog renders its goldens the same way the VM
image collection does: DataVolumes with immediate bind whose import
starts at install. With no health check the release went Ready as soon
as helm applied them, so a golden that never imported left the catalog
green.

Gate it on the DataVolume phase with the same expressions. Worker
pools read each golden directly and do not wait on the catalog status,
so the gate changes what the status reports and delays no pool.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 22:15:59 +03:00
Aleksei Sviridkin 51ac18776e fix(vm-default-images): wait for golden images to import before Ready
Every image is a DataVolume with immediate bind, so its import starts at
install, but nothing waited for it: the HelmRelease went Ready as soon as
helm applied the DataVolumes. An image that failed or stalled left the
component green, and the failure surfaced later on a vm-disk cloning
from it, far from the cause.

Gate the release on the DataVolume phase. Only Succeeded counts as
ready, and a Failed phase, when CDI sets one, ends the wait at once.
Every other phase, and a missing status, keeps the wait open until the
install window expires. There is no per-component timeout, so a download
longer than that window is retried as an upgrade until it finishes, and
the retry strategy never uninstalls the volumes.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-09-29 22:15:58 +03:00
Marian Koreniuk d902421f47 feat(kubernetes): deploy the Proxmox cloud-controller-manager with the cluster
On this substrate the worker kubelet runs with --cloud-provider=external, which
taints every node node.cloudprovider.kubernetes.io/uninitialized until a cloud
provider clears it. Without a cloud-controller-manager the taint is permanent:
nothing but tolerating DaemonSets schedules, and the CSI node plugin — which
reads topology.kubernetes.io/region and zone off its own Node and exits fatally
without them — never starts.

It runs in the management cluster for the same reason as the CSI controller
beside it: it authenticates to the hypervisor, and the tenant is cluster-admin
of its own cluster. Its whole job is to write on the tenant's Nodes, and it
does that over the tenant's admin kubeconfig from out of cluster, which also
means there is no tenant-side RBAC left to install.

capmox mode, not the default one. The default resolves a node to its VM by name
and writes providerID as proxmox://<node>/<vmid>, while capmox reads the CAPI
Machine and writes proxmox://<SMBIOS UUID> — the form capmox itself puts on the
Machine. With the default the Node and its Machine carry two different
identifiers for the same VM.

The tolerations are spelled out rather than a blanket `operator: Exists`. That
one also matches node.kubernetes.io/unreachable:NoExecute with no
tolerationSeconds, which stops admission adding the bounded 300s one and lets
the Pod outlive the Node it runs on.

`proxmox.insecure` defaults to false: the controllers verify the hypervisor's
certificate unless an operator turns that off, and both run here, so the value
describes what the management cluster trusts, not the tenant. The README gains
a section on what the two tokens can reach — the controller acts on whatever
VMID and volume handle the tenant's objects name — and on the per-tenant
Proxmox scope that contains it: a storage of the tenant's own and one ACL on
the resource pool proxmox.pool puts its workers in, which follows every VM
capmox creates — VM.Audit there for the CCM too, which deletes a NotReady Node
whose VM it cannot see.

ccm, csi and insecure are optional in the schema. Each has a default, and what
the substrate needs is enforced at render time — a proxmox cluster without
ccm.credentialsSecretName does not render — so a Kubernetes object written
before these fields existed stays valid, and adding them is not a breaking
change to the API.

The kubelet flag is asserted on both substrates: present on proxmox, absent on
kubevirt, where nothing would ever clear the taint it causes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:05 +02:00