Commit Graph
4153 Commits
Author SHA1 Message Date
Aleksei Sviridkin 72ec1189fd feat(velero): build the KubeVirt plugin image for amd64 and arm64
Upstream publishes quay.io/kubevirt/kubevirt-velero-plugin for amd64
only, so on an arm64 cluster the velero server never starts: its
plugin init container fails with "exec format error". Rebuild the
plugin in-tree from the v0.9.0 release tag, cross-compiled on the
build platform, into an image that behaves like upstream's: it copies
the plugin binary into the plugins volume at /target.

The image target stamps the built reference by digest into
values.yaml, and the package joins the root build list so every build
path refreshes that pin. Until the first stamp is committed the
upstream reference stays in place, which the chart test accepts
alongside the digest-pinned in-tree one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 20:30:40 +03:00
Aleksei Sviridkin 084ac47237 feat(build): build published images for amd64 and arm64 by default
PLATFORM defaulted to empty, so buildx built for the host alone. On an
arm64 workstation `make image` published an arm64-only digest, which
amd64 nodes then fail to pull; the cilium pin shipped exactly that
until it was rebuilt by hand. A build now targets linux/amd64 and
linux/arm64 unless PLATFORM is set, and the digest a package stamps is
the index digest.

LOAD=1 keeps building for the host only. The classic docker image store
cannot load a multi-platform index, and loading is the documented local
workflow.

This changes local pushes: a `make image` on the default docker driver
without the containerd image store now fails with "Multi-platform build
is not supported for the docker driver", where it used to build for
the host. Such a build needs a docker-container builder (BUILDER), or
LOAD=1 or an explicit PLATFORM for a single architecture.

talos and testing pin linux/amd64 with `override`, placed before the
include because common-envs.mk expands BUILDX_ARGS immediately. The
Talos assets are amd64: matchbox bakes the kernel and initramfs into
its image, and the e2e sandbox boots the disk under qemu-system-x86_64.
An arm64 variant of either would still serve or boot amd64 files.

CI pins linux/amd64 on every image build. Nothing there sets up QEMU,
fork builds export a single-platform archive, and tags.yaml builds on
the default docker driver, so the default would break or emulate those
jobs. Native arm64 builds belong in a leg of their own.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:14:12 +03:00
Aleksei Sviridkin d9cbbec5ff fix(build): cross-compile image builders on the build platform
The remaining Go builder stages, plus the kube-ovn, console and
piraeus-server builders, ran on the target platform. A multi-platform
build therefore ran the whole compiler under QEMU: slow, and for Go
unreliable, as the migration-controller Dockerfile already records a
"found bad pointer in Go heap" crash from exactly that. The builders
now run on $BUILDPLATFORM and produce the target architecture from
TARGETARCH.

Two builders only produced the right architecture because they were
emulated. token-proxy ran `go build` with no GOOS/GOARCH, and kube-ovn
calls upstream `make build-go`, which hardcodes GOARCH=amd64 even on an
arm64 build. On a native builder both would have shipped host binaries
inside the other architecture's image, and nothing fails until a node
of that architecture starts the container. token-proxy now passes the
target explicitly; kube-ovn picks build-go or its arm64 twin
build-go-arm from TARGETARCH.

The console builder emits static JS and linstor-server's debian/control
declares every package Architecture: all, so both run natively without
a target.

A bats test requires every stage that compiles to sit on a BUILDPLATFORM
FROM and to read TARGETARCH or TARGETPLATFORM on a RUN or ENV line. A
bare ARG declaration does not count, which is what let token-proxy
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:05:06 +03:00
Aleksei Sviridkin 623113f920 fix(build): pass the shared buildx flags from every image target
Three package Makefiles spelled out their docker buildx flags instead
of using $(BUILDX_ARGS) from hack/common-envs.mk: flux-plunger,
kubeovn-plunger and both dashboard images. kubeovn-plunger landed just
before the change that moved the other Makefiles onto BUILDX_ARGS,
flux-plunger was modelled on it, and dashboard kept its own flags when
that change touched it.

Those targets ignored PLATFORM, SBOM and BUILDX_EXTRA_ARGS. On an arm64
workstation the plungers came out as arm64 images, which fail on amd64
nodes with "exec format error"; CI builds on amd64 and never saw it.
Dashboard and metallb hardcoded --platform=linux/amd64 instead; for
metallb that sat next to $(BUILDX_ARGS), so setting PLATFORM gave buildx
two platforms and a multi-platform build.

All of them now build through $(BUILDX_ARGS) alone. CI sets only
BUILDER, which these targets already honoured, and runs on amd64, so
its builds are unchanged. Local builds of dashboard and metallb now
follow the host architecture like every other package: on an arm64
machine build them for an amd64 cluster with PLATFORM=linux/amd64.

A bats test dry-runs every package image target with a probe platform
and a marker in BUILDX_EXTRA_ARGS, and requires each buildx call to
carry the marker, so $(BUILDX_ARGS) is really passed, and that
platform and no other.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 14:32:44 +03:00
Aleksei Sviridkin 2c2c0301f7 chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447)
## What this PR does

This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.

Before the bumps:

- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.

Held back on purpose, with the reasons in the commit messages:

- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.

govulncheck reports nothing reachable in any of the five modules,
standard library included.

Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.

Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.

Closes #4487.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added a restore resource policy option for Velero strategies.
  * Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
  * Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 10:17:29 +04:00
Aleksei Sviridkin 4a2dea2d08 feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages (#4087)
## What this PR does

**Series behind #3481** — three separable pull requests:

1. **#4087 — Cluster API providers** (capmox + in-cluster IPAM)
2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on #4088.

---

This is the provisioning layer #3481 asks to specify first: the part
that
creates the VMs. Neither of the other two is needed to review it.

### What it adds

Two Cluster API providers, packaged the way the existing KubeVirt
provider is —
vendored components, `gzip -9 -n` for a reproducible artifact, a `make
components` target that regenerates it from the upstream release and
checks
the downloads against the SHA-256 sums the Makefile pins:

| Package | Version | Contract |
|---|---|---|
| `capi-providers-infraprovider-proxmox` | capmox v0.7.7 | v1beta1 |
| `capi-providers-ipam-in-cluster` | CAIP v1.0.3 | v1beta1 |

capmox v0.7.7 rather than the current v0.9.x on purpose: from v0.8 it
declares
contract `v1beta2`, which the CAPI core shipped here (v1.10.1) does not
serve.

Both join the `iaas` bundle as **opt-in** packages, where the KubeVirt
infrastructure provider is unconditional. The asymmetry is deliberate:
capmox
drives a hypervisor outside the cluster and needs a Secret holding a
Proxmox API
token, which capi-operator reads before it installs anything — with no
Secret the
provider is never deployed at all. Enabling it by default would ask
every
Cozystack install that has no Proxmox to supply hypervisor credentials.

```yaml
bundles:
  enabledPackages:
    - cozystack.capi-provider-infra-proxmox     # brings IPAM with it
    - cozystack.capi-provider-ipam-in-cluster   # also enableable on its own
```

Enabling capmox emits the IPAM provider automatically — capmox allocates
worker
addresses from an `InClusterIPPool` and has no DHCP mode, so on its own
it never
gets a `ProxmoxCluster` to `Ready`. IPAM stays independently enableable,
since
assigning addresses is a generic Cluster API concern.

Naming capmox in `enabledPackages` while disabling IPAM stops the
render. The
cluster would catch that combination on its own — the capmox
`PackageSource`
depends on the IPAM `Package`, so with that `Package` absent capmox goes
`Ready=False` with `DependenciesNotReady` and no `HelmRelease` is
created — but
it would not say why. Failing the render names the two conflicting list
entries
at the moment someone writes them, instead of leaving a condition to
decode.

### Why the earlier attempt did not produce a VM

#1927's blocker — "ProxmoxMachine does not produce a VM" — was not a
capmox
incompatibility. It was four independent causes, each of which only
became
visible once the previous one was cleared:

1. **`PROXMOX_URL` included `/api2/json`.** capmox appends that itself,
so every
   call went to `/api2/json/api2/json/…` and returned 501.
2. **No IPAM provider.** `no matches for kind "InClusterIPPool"`; the
`ProxmoxCluster` never reached `Ready` and every `ProxmoxMachine` sat in
   `WaitingForClusterInfrastructure`.
3. **`templateSelector.matchTags` is set equality, not containment**
(`slices.Equal` after sort). A template carrying four tags is not
selected by
   naming one of them: `found 0 VM templates with tags`.
4. **capmox waits for the qemu-guest-agent, which Talos does not ship.**
   `error waiting for agent: timed out` — cleared with
   `checks.skipQemuGuestAgent` and `skipCloudInitStatus`.

None of these are in the provider. They are packaging and configuration,
which
is why the earlier work could look complete and still never boot a
machine.

### Evidence

Run on Proxmox VE 9.2.11 with Talos v1.13.6 and Cozystack's CAPI stack
(core v1.10.1, CABPT v0.6.12, Kamaji control-plane provider v0.19.0):

```
$ kubectl get proxmoxmachine,machine -n tenant-proxmox
proxmoxmachine/kubernetes-px-md0-…   Ready=True   VMProvisioned=True
machine/kubernetes-px-md0-…          Running      proxmox://0ef88cc1-6aea-4563-af64-344d8060d5c0

$ qm list
106 kubernetes-px-md0-jmzds-h9npx  running
```

The machine boots Talos from the nocloud ISO, reads the machineconfig
capmox
writes, and joins. That is the layer #69 called a hard blocker.

### What this PR does not claim

- One Proxmox node, one tenant cluster, manual runs — there is no e2e
coverage
  for Proxmox in CI, and this PR does not add any.
- It does not make Proxmox a supported platform. It makes the
provisioning layer
reviewable, which #3481 names as the precondition for that conversation.
- It does not touch `capi-providers-core`. The same drift check, pointed
at that
package, shows that the `startupProbe` from #2946 never reached the
shipped
blob; that fix changes what every new install runs and is #4483, with
its
  own release note.
- Nothing here exposes `ProxmoxCluster.spec.credentialsRef` to a tenant,
and
nothing should: `capmox-manager-role` grants get/list/watch/patch on
Secrets
  cluster-wide and that field takes a namespace. Whatever renders a
`ProxmoxCluster` for a tenant in the rest of the series has to keep the
field
  out of tenant hands.

### Testing

`make unit-tests` — exit 0. New suites:
`packages/core/platform/tests/sources_capi_provider_proxmox_test.yaml`
and
`bundles_proxmox_wiring_test.yaml` (both bundle paths, the
enable/disable
precedence, and the refusal above).

Every negated `containsDocument` in the wiring suite carries `any:
true`;
without it the negation passes whether or not the document is in the
render.
Three mutations, each reverted afterwards — emitting the capmox
`Package`
unconditionally, emitting the IPAM `Package` unconditionally, and
dropping the
`disabledPackages` check from `$proxmoxEnabled` — each turn red the
tests named
after the property they break, and the unmutated chart stays green (44
suites,
198 tests).

`hack/capi-provider-components_test.bats` covers both new provider
packages:
the blob and the YAML are the same bytes; the Makefile version, the
ConfigMap
name and `spec.version` are one value — `applied-spec-hash` covers the
provider
spec and the config Secret but not the ConfigMap, so a bump that
regenerates the
files while `spec.version` stays behind installs the old provider
silently; and
the vendored files match the SHA-256 sums the Makefile pins, so the pin
holds
even when nobody runs `make components`.

The render was compared against `main` for five configurations
(defaults,
`isp-full`, `isp-full-generic` with paas, an unrelated
`enabledPackages`, and
`disabledPackages` naming IPAM). No line is removed in any of them and
the
`Package` count is unchanged; the only additions are the two
`PackageSource`
objects, which the chart renders for every package, optional or not.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/website — matches, no follow-up yet.**

- `packages/core/platform/values.yaml` changed.
`content/en/docs/next/operations/configuration/platform-package.md` is a
hand-written table of the `spec.components.platform.values.*` keys; this
PR does not add a key, but it documents two new values that
`bundles.enabledPackages` now accepts, and the table is where a reader
would look for them.
- Two platform components are added
(`capi-providers-infraprovider-proxmox`,
`capi-providers-ipam-in-cluster`) → `guides/platform-stack/_index.md`,
`operations/configuration/licenses.md`.
- The `iaas` bundle gains two optional packages →
`operations/configuration/variants.md`.

**Everything else: no match.** Nothing under `packages/apps/` or
`packages/extra/` (terraform-provider, website app lists), no
`packages/core/installer/values.yaml` or `networking.*`/`publishing.*`
key the Ansible role sets, no `hack/` file or package-layout change
(ccp, external-apps-example), no `ApplicationDefinition` semantics, no
telemetry metric, no node prerequisites (talm, ansible), no
commit-convention change (community).

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(cluster-api): add optional Cluster API packages for Proxmox — the capmox infrastructure provider and the in-cluster IPAM provider it requires. Neither is deployed unless named in `bundles.enabledPackages`, since capmox drives a hypervisor outside the cluster and does nothing without Proxmox API credentials. Enabling the Proxmox provider brings the IPAM provider with it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added optional Proxmox infrastructure provider support for IaaS
clusters.
- Added in-cluster IP address management, usable independently or
alongside Proxmox.
- Proxmox deployments require a pre-created credentials Secret and
automatically include in-cluster IPAM.
  - Added validation for incompatible Proxmox and IPAM settings.

- **Tests**
- Added coverage for provider dependencies, package enablement,
disabling behavior, invalid configurations, and packaged provider
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 01:48:39 +04:00
Aleksei Sviridkin e883f6f7fc fix(capi): ship the core provider manifests the repository actually holds (#4483)
## What this PR does

`capi-providers-core` ships the Cluster API core manifests twice: as
`files/core-components.yaml`, which a reviewer reads, and as
`files/components.gz`, which the chart mounts into the cluster. #2946
added a `startupProbe` to `capi-controller-manager` and edited only the
YAML; the blob was never regenerated, so the probe never reached a
cluster. This regenerates the blob from the YAML (`gzip -9 -n -c
files/core-components.yaml > files/components.gz`), so the decompressed
blob matches the YAML byte for byte, and the only change it carries is
the six lines of #2946.

### What this reaches, and what it does not

New installs only. capi-operator re-applies a provider when the hash of
its spec and its config Secret changes (`applied-spec-hash`); the
ConfigMap holding the components is in neither, so a cluster that
already runs `capi-providers-core` keeps its manager without the probe
until the provider's version moves. Whether existing clusters should get
the probe, and how, is left open on purpose: moving `spec.version` and
the ConfigMap name together would force the re-apply, but that is a
version decision for the package, not a side effect of regenerating a
blob.

### Also in this diff

- `.helmignore`: `files/.*-components.yaml` is a glob, so it matched
only names beginning with a dot, and `helm package` shipped the readable
YAML alongside the blob that duplicates it. Now
`files/*-components.yaml`.
- `hack/capi-core-components_test.bats` holds the two files to the same
bytes from now on; it goes red on a YAML edited without regenerating the
blob.

Found by the drift check written for the Proxmox provider packages in
#4087, pointed at the package that was already here; split out because
it changes what every new install runs and deserves its own release
note.

### Testing

`bats hack/capi-core-components_test.bats`: 1/1; red with a line
appended to the YAML, green again after `gzip -9 -n -c`. `helm package`
of the chart now contains `files/components.gz` and
`files/metadata.yaml` only.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(cluster-api): ship the `startupProbe` from #2946 in the packaged core provider components. New installs get it; a cluster that already runs the core provider keeps its manager without the probe until the provider's version moves, because capi-operator re-applies components only when the provider spec or its config Secret changes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Added a check that the compressed core components match the YAML
source, with a diff provided when they differ.
* **Chores**
* Updated chart packaging exclusions to cover component YAML files
regardless of whether their names begin with a dot.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 22:43:55 +04:00
Aleksei Sviridkin a285a5ef36 feat(kubernetes): add a proxmox substrate for managed clusters and worker pools (#4088)
## What this PR does

**Series behind #3481** — three separable pull requests:

1. #4087 — the Cluster API providers (capmox + in-cluster IPAM)
2. **#4088 — the `substrate` switch** in `kubernetes` and
`kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on this one.

---

#4087 adds the Cluster API providers; this PR teaches the two
managed-cluster
charts to point at them.

### The shape

A single value, `substrate: kubevirt | proxmox`, on both charts — not a
parallel
set of charts. The control plane is Kamaji either way; what changes is
the
infrastructure half:

| | `kubevirt` (default) | `proxmox` |
|---|---|---|
| `Cluster.infrastructureRef` | `KubevirtCluster` | `ProxmoxCluster` |
| worker template | `KubevirtMachineTemplate` | `ProxmoxMachineTemplate`
|
| apiserver Service | `ClusterIP` | `LoadBalancer` |
| tenant CCM/CSI | in-cluster (`kccm`, kubevirt-csi) | rendered off; the
Proxmox pair is #4089 |

The KubeVirt path renders byte-for-byte what it rendered before. Checked
by
rendering `kubernetes-nodes` with default values at `main` and at this
head and
diffing the output, not only by the snapshots — the snapshots cover
`nodegroup.yaml` alone, and an earlier revision of this branch changed
the
reconcile Job while they stayed green. The parent chart's default render
differs
only in the random fields of `talos-secrets`, which differ between two
renders of
`main` as well.

`osImage` (#4171) is refused on the proxmox substrate. Only half of it
could be
honoured there — `factory.version` reaches the machineconfig's installer
image
while the worker disk keeps coming from the Proxmox template — and a
field that
half-works is worse than one that says no. `talos.version` and
`talos.schematicID` still set the installer target.

### Three things that only show up once workers are off-cluster

**The apiserver endpoint.** A KubeVirt worker runs inside this cluster
and
reaches the tenant apiserver by ClusterIP. A Proxmox worker does not: it
lives on
the hypervisor's L2 and cannot route to `10.96.0.0/16`. The machine
provisions
fine, `VMProvisioned=True`, `providerID` set — and then never joins,
with no CSR
to show for it. On the proxmox substrate the Kamaji Service is therefore
a
`LoadBalancer`, and the reconcile Job reads
`.status.loadBalancer.ingress[0].ip` instead of `.spec.clusterIP`.

**And DNS, for the same reason.** The worker machineconfig carried
`nameservers: ${COREDNS_IP}` — the management cluster's CoreDNS
ClusterIP,
equally unreachable. The proxmox path takes its own resolvers
(`proxmox.dnsServers`) and the chart refuses to render without them,
rather than
writing an address that cannot answer.

**`externalManagedControlPlane` and the managed-by annotation.**
capmox's
webhook nil-derefs on a `ProxmoxCluster` whose control-plane endpoint is
empty
unless `spec.externalManagedControlPlane` is set, which is correct here
— Kamaji
owns the endpoint. Separately, the `cluster.x-k8s.io/managed-by: kamaji`
annotation that the KubeVirt path carries must **not** be set on the
Proxmox
object: with it, CAPI never sets an ownerRef and capmox stops at
`Waiting for Cluster Controller to set OwnerRef`, leaving the address
pool
uncreated and every worker waiting on infrastructure.

### Linked clones by default

`proxmox.full: false` — worker disks are linked clones of the template
rather
than full copies. Measured on a ZFS-backed store: **8 KiB per worker
instead of
10.2 GiB**. Operators who want independent disks set `full: true`; the
value is
plumbed straight through to `ProxmoxMachineTemplate`.

`proxmox.storage` is a full-clone parameter and is refused with a linked
clone:
a linked clone always lives on the template's storage, and both Proxmox
(`parameter 'storage' not allowed for linked clones`) and capmox's own
CRD
(`Must set full=true when specifying storage`) reject the pair. The
chart says
so at render time, naming both values, instead of failing at apply time
with an
error about a `ProxmoxMachineTemplate` nobody wrote.

### Sizing

`ProxmoxMachineTemplate` takes `numCores` and `memoryMiB`, not
Kubernetes
quantities, so the pool's `resources.cpu`/`memory` are converted
(millicores →
cores, bytes → MiB, both rounded up). The rounding is pinned in
`tests/substrate_branches_test.yaml` with `1500m` and `8G`, where
rounding up and
down disagree; `tests/substrate_proxmox_sizing_test.yaml` covers the
refusal to
size a pool that gives neither.

### Known upstream behaviour this surfaces

`TalosConfigTemplate.spec` is immutable by schema, so the reconcile Job
cannot
rewrite an endpoint it has already published — `kubectl apply` fails
with
`TalosConfigTemplate.Spec is immutable`, and the template has to be
deleted for
the Job to recreate it. A fresh install never meets it; it appears when
the
apiserver address changes on an existing cluster. Handling it changes
the
KubeVirt path too, so it is not in this PR: it is a separate change on
`proxmox/tct-replace`, with its own upgrade note.

### Evidence

```
$ KUBECONFIG=<tenant> kubectl get nodes -o wide
kubernetes-px-md0-jmzds-ndnl6  Ready  <none>  75s  v1.35.6  10.0.0.160  Talos (v1.13.6)

$ KUBECONFIG=<tenant> kubectl get pods -A
cozy-cilium  cilium-…            Running
kube-system  konnectivity-agent  Running
kube-system  coredns-…           Running
```

No management-cluster address remains in the rendered
`TalosConfigTemplate`.

### Testing

`make unit-tests` — exit 0. `apps/kubernetes` 26 suites / 314 tests;
`apps/kubernetes-nodes` 29 suites / 160 tests / 12 snapshots, all
unchanged on
the default path. Each of the ten commits passes both suites on its own.

Every substrate branch is asserted on both sides, and every assert goes
red on
the mutation it exists for: inverting the `SVC_IP` condition, inverting
the
`nameservers` condition, `ceil`→`floor` in the sizing, removing the
proxmox
gate from each of the kccm and csi templates (the two RBAC bindings
included),
pinning the retention lookup back to `KubevirtMachineTemplate`, widening
the
IPv6 class of the `dnsServers` validation, dropping the parent chart's
`dnsServers` validation, the storage-with-linked-clone refusal, and the
`osImage` refusal. Negated `containsDocument` asserts carry `any: true`.

`hack/talos-reconcile-heredoc_test.bats` additionally runs the rendered
heredoc
through a real shell: a `dnsServers` entry with a substitution is
refused and
named, an IPv6-shaped one with a substitution is refused too, and a real
address survives the shell unchanged.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/terraform-provider-cozystack — matches, no follow-up yet.**

- `packages/apps/kubernetes/values.schema.json` and
`packages/apps/kubernetes-nodes/values.schema.json` both gain fields:
`substrate`, and a `proxmox` object (`allowedNodes`, `dnsServers`,
`ipv4Config`, and on the pool side `templateTags`, `network`, `storage`,
`full`). The provider hand-maintains a schema, a model and an
expand/flatten pair per Kind, with no codegen from these files.
- `values.yaml` gains defaults for those fields (`substrate: kubevirt`,
`proxmox.full: false`, `prefix: 24`). The provider restates defaults it
models and sends its own, so a stale one wins silently rather than
showing as a diff.

**Everything else: no match.** No app added, renamed or removed (website
app lists); no version enum, `kind`, `plural`, `release.prefix` or
output Secret/Service name changed; `packages/core/platform/values.yaml`
untouched; the `*-rd` files are regenerated, not renamed (ccp); no
`hack/` change; no node prerequisites (talm, ansible); no telemetry
metric; no commit-convention change (community).

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): add a `substrate` value to the `kubernetes` and `kubernetes-nodes` applications, selecting whether worker VMs run on KubeVirt in this cluster (`kubevirt`, the default and unchanged) or on an external Proxmox VE cluster through capmox (`proxmox`). On the Proxmox substrate the tenant apiserver is published through a LoadBalancer and worker disks default to linked clones.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for choosing KubeVirt or Proxmox for Kubernetes clusters
and node pools.
* Added Proxmox configuration for nodes, networking, storage, DNS,
templates, cloning, and IPv4 allocation.
* Proxmox deployments now use the appropriate cluster, machine,
networking, and control-plane resources.

* **Validation**
* Added checks for supported substrates, required Proxmox settings,
sizing, and incompatible options.

* **Documentation & Tests**
* Documented the new settings and expanded coverage for rendering,
validation, cloning, sizing, and template replacement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 22:09:23 +04:00
Aleksei Sviridkin 09d02fdc6c docs(apps): correct the shared resources and preset paragraphs (#4394)
## What this PR does

Corrects two sentences of the boilerplate shared by the app READMEs,
both raised in the review of #3936.

- "When left empty, the preset defined in `resourcesPreset` is applied"
describes an all-or-nothing rule, but
`cozy-lib.resources.defaultingSanitize` merges per key: every resource
`resources` leaves unset is taken from the preset. Twelve READMEs
carried the sentence.
- The legacy flat names were called aliases of their "1:1 instance-type
equivalents", which reads as the `c1` series listed just before it.
`nano` through `small` equal the `t1` sizes, and `medium` equals
`c1.small` rather than `c1.medium`, the trap
`docs/operations/resource-presets.md` already warns about. Ten READMEs
carried the sentence.

Only the hand-written section changes. `make generate` for all twelve
packages leaves the diff at exactly those 22 lines.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: README prose only, no values,
schema, default or package change. The website regenerates the app
reference pages from these READMEs on release, so nothing to open there.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified how resource settings work across several app READMEs: any
unset resource field now inherits its value from the preset, rather than
requiring the entire resource block to be empty.
* Updated legacy preset name guidance to explain that deprecated flat
names keep their original sizes, including `nano`–`small` matching `t1`
sizes and `medium` matching `c1.small`.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 20:17:41 +04:00
Aleksei Sviridkin b16edec3f3 fix(tcp-balancer): rename backends and pin HAProxy so the chart starts on 3.3+ (#4366)
## What this PR does

`haproxy:latest` now resolves to 3.3 and newer. Since 3.3 a backend
named like a frontend is a fatal config error, so every tcp-balancer pod
fails at config parse (#1966). We hit it on Cozystack 1.6.1 with
`haproxy:3.4.4` behind the tag.

- backends get a `_backend` suffix and every `default_backend` follows;
the rename is the one from #2321 by @officialasishkumar
- the image is `haproxy:{{ .Chart.AppVersion }}` with `appVersion:
3.4.4`, the current 3.4 LTS tag
- the whitelist `acl`, `accept` and `reject` lines of each frontend
render inside one guard
- `tests/` holds helm-unittest suites, and the package `Makefile` gets
the `test` target the sweep looks for
- the `whitelistHTTP` description says it needs a non-empty `whitelist`

Two inputs change behaviour on upgrade. On `main` both fail at config
parse with `no such ACL : 'whitelist'` on every HAProxy version, because
the `acl` line sits inside the `with` and `accept if whitelist` does
not:

- `whitelistHTTP: true` with an empty `whitelist` now fails the render
with `whitelistHTTP is enabled but whitelist is empty`. Its HelmRelease
upgrade fails and the release stays on its old revision, which is
already crash-looping.
- `kubernetesAndTalos` with an empty `whitelist` now starts, and ports
6443 and 50000 accept any source. With `external: true` the Service is a
LoadBalancer and carries both ports. `whitelist` is the only switch
those frontends have, so an empty list means no filtering.

Every other input renders the same config apart from the backend names.
The image and the `checksum/config` annotation change, so the Deployment
rolls, and for releases already crash-looping on `latest` the rollout is
the fix.

Validation: renders of `main` and of this branch for the default,
whitelist and empty-whitelist inputs were checked with `haproxy -c` on
2.9.7 and 3.4.4. `helm unittest .` passes 9/9, and removing the `reject`
or `accept` line from any single frontend turns a test red.

Fixes #1966

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: chart templates, `Chart.yaml`,
the package `Makefile`, `tests/`, and a description-only change in
`values.yaml` with its generated carriers. No field is added, removed or
renamed and no default changes, so nothing in the map fires.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tcp-balancer): rename HAProxy backends and pin the image to the chart appVersion so the balancer starts on HAProxy 3.3+. whitelistHTTP with an empty whitelist now fails the render; kubernetesAndTalos with an empty whitelist leaves ports 6443 and 50000 unfiltered.
```
2026-09-25 19:29:39 +04:00
Marian Koreniuk 8722462066 fix(kubernetes-nodes): refuse osImage on proxmox instead of honouring half of it
#4171 gave a worker pool an osImage, choosing where its system disk comes from:
a CDI clone of a shared golden image, or an HTTP import from an Image Factory.
A Proxmox worker's disk comes from neither. capmox clones a VM template that
already carries Talos, and nothing in this chart can change what that template
holds.

Half of the field would still appear to work, which is why it is refused rather
than ignored. osImage.factory.version reaches the machineconfig's installer
image and changes the reconcile Job, so a pool could set it, watch the Job
roll, and end up running one Talos while booting another: v1.12.6 moves the
installer to .../installer/<schematic>:v1.12.6 and leaves the
ProxmoxMachineTemplate untouched. osImage.builtin has no meaning here at all.

Nothing is lost by refusing it: the installer target on this substrate is
settable through talos.version and talos.schematicID, which do reach it.

The branches suite gains the refusal case. The substrate parameter's description said changing the value "rolls the
pool". The chart refuses the change on a live pool, and the description says
so; it reaches the README, the values schema and the CRD.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 4bc6621a5c test(kubernetes,kubernetes-nodes): assert both sides of every substrate branch
Every `if substrate` in both charts is asserted on both sides, so an inverted
condition cannot pass:

- the reconcile Job reads SVC_IP from the LoadBalancer status on proxmox and
  from the ClusterIP on kubevirt;
- the machineconfig carries the pool's nameservers on proxmox and none on
  kubevirt;
- the proxmox gate on each kubevirt-only template in the parent chart, the two
  RBAC bindings included, so a proxmox cluster renders no binding to a Role,
  ClusterRole or ServiceAccount it does not render;
- the sizing conversion at 1500m and 8G, where ceil and floor disagree
  (2 cores and 7630 MiB against 1 and 7629);
- the old-template retention loop keeps a ProxmoxMachineTemplate a MachineSet
  still references;
- the proxmox worker monitor carries no Pod threshold while the kubevirt one
  does;
- a live cluster and a live pool refuse a substrate switch while a first install
  still renders;
- both charts refuse a dnsServers entry that is not an address, the parent
  chart included, where the value reaches ProxmoxCluster.spec.dnsServers.

The negations carry `any: true`: without it a negated containsDocument passes
whether or not the document is in the render.

hack/talos-reconcile-heredoc_test.bats gains the dnsServers cases: the render
refuses a command substitution and names the entry; an entry with a colon in it
is refused too, which pins the IPv6 branch of the validation and not only the
IPv4 one; and a real address survives the Job's shell unchanged. The heredoc is
extracted by shape — `cat <<EOF` at the start of a line or right after `=$(` /
`="$(` — so the same case keeps working whichever way the Job writes it.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 517baf945d fix(kubernetes,kubernetes-nodes): validate dnsServers, refuse a substrate switch, drop the Pod threshold on proxmox
Three faults in the proxmox substrate, none of which the render refused.

1. proxmox.dnsServers reached the reconcile Job's shell unescaped.

The Job applies the worker TalosConfigTemplate through an unquoted heredoc, so
every byte of its body is subject to parameter expansion and command
substitution at Job runtime. The INVARIANT comment above that block says each
tenant-controlled value must be shell-escaped or pattern-validated; the new
nameserver list was neither. `quote` there is YAML quoting.

Rendered with `$(id -u)` and a backtick pair in dnsServers, extracted the way
hack/talos-reconcile-heredoc_test.bats does and run under sh, the shell executed
both and put their output in the worker's resolver list. The Job's pod holds the
parent cluster's Talos token, cluster id, cluster secret and bootstrap token in
its environment, and its Role can read the Talos CA and patch the parent
KamajiControlPlane.

Each entry is now validated as an IPv4 or IPv6 address at render time, in both
charts. Validation rather than escaping because Talos takes addresses here: an
address cannot carry a metacharacter, so the property holds by construction, and
the ordinary mistake of writing a hostname is caught in the same move. IPv6 is
matched on its character set rather than its full grammar — what matters is which
bytes may reach the shell, and the apiserver rejects a malformed address anyway.

2. Nothing stopped a live cluster or pool from being switched between substrates.

`substrate` had no immutability of any kind, and the `## @immutable` marker this
adds is dashboard-enforced only, so both charts now read the live object.

Switching a cluster removes the KubevirtCluster, the cloud-controller-manager,
the kubevirt-csi controller and the in-tenant -csi HelmRelease with all their
RBAC from the render. Flux then uninstalls the CSI driver behind every PVC the
tenant has. CAPI does not prevent it — its Cluster webhook forbids removing
spec.infrastructureRef but admits a change of kind.

Switching a pool rolls every worker to the other hypervisor and drops the
machine template the draining MachineSet still references, because the retention
loop looks up only the kind the new substrate uses. A pool whose substrate
differs from its parent's was not rejected either, though the README says the
two must match.

Both guards are inert offline, like assertNoForeignPool and assertParentVersion
beside them: with no apiserver the lookup is empty and a first install renders.

3. The worker WorkloadMonitor reported a proxmox pool as broken.

The controller lists Pods matching the monitor's selector and sets
Operational=false while fewer than minReplicas are ready. On kubevirt those Pods
are the virt-launcher Pods. A proxmox worker is a VM on another machine with no
Pod here, so the count is always zero and any threshold pinned the pool — and
through the lineage labels its whole application — to WorkloadsReady=False while
every node was healthy. The monitor still renders for the dashboard and the
lineage; what it no longer carries on proxmox is a threshold counted in Pods.

Two smaller things. The wait-loop log line is written per substrate, so a
default kubevirt render of kubernetes-nodes is byte-identical to main and no
existing pool gets a new reconcile Job on upgrade. And the comment claiming a
rename of $kubevirtmachinetemplate would roll every live worker VM is replaced:
the hash is taken over the rendered template, not the variable name, so renaming
it leaves the render byte-identical.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 33e1de52d2 test(kubernetes-nodes): pin the linked-clone default and three silent guards
Nothing failed when `full` was wrong: the pool came up, the workers joined, and
each one quietly cost the whole template disk instead of the increment — 10.2
GiB against 8 KiB on a ZFS store. A default whose only symptom is disk usage is
exactly the kind that drifts, so both it and its override are now asserted.

The bridge and dnsServers guards were two of the guards on this path with no
coverage. Both fire on absence, so the suite omits each and the tests supply the
other one back; declaring either at suite level would make it impossible to take
away, which is why the sizing guard already lives alone.

The storage guard is the third: `storage` with a linked clone is refused and the
message names both values, while `storage` with `full: true` reaches the
ProxmoxMachineTemplate.

Each assert goes red on the edit it exists for: flipping the default reds the
clone suite, deleting a guard reds the guard suite.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk c5996c1d0f fix(kubernetes,kubernetes-nodes): give Proxmox workers an endpoint they can reach
A worker on the proxmox substrate is a VM on the hypervisor network, not a Pod
in this cluster. It was handed two addresses only an in-cluster consumer can
route to: the apiserver as a ClusterIP, and the management cluster's CoreDNS
ClusterIP as its resolver. The VM booted, the Machine reported Ready with a
providerID, the disk was where it should be — and no CSR ever arrived. Nothing
in that picture points at an address.

The control-plane chart now publishes the tenant apiserver as a LoadBalancer on
the proxmox path, putting the endpoint on the network the workers share, and
the pool's reconcile Job reads the Service's LoadBalancer address instead of its
ClusterIP for the host entry the workers dial. The nameservers follow the same
reasoning: proxmox pools carry their own resolvers rather than pointing at a
CoreDNS they cannot reach, and the chart refuses to render a pool that has not
named any.

KubeVirt keeps the ClusterIP on both counts. Its workers are in this cluster,
where a ClusterIP is the right answer and a LoadBalancer address would be an
unnecessary one.

Verified end to end on a real tenant cluster: the node joins, its CSR is
approved, and cilium, konnectivity and coredns come up in the tenant.

One operational note this surfaced, not caused by it: TalosConfigTemplate.spec
is immutable, so the reconcile Job cannot rewrite an endpoint that has already
been published — the template has to be deleted for the Job to recreate it.
That matters only when an existing cluster's apiserver address changes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk d78af928fc feat(kubernetes-nodes): default Proxmox worker disks to linked clones
A full clone on a ZFS pool configured with `sparse 0` reserves the whole
volsize the moment the VM is created: 10.2G for a worker whose Talos image
occupies 337M. A linked clone writes only its own increment — measured at 8K
for the same worker on the same host — because it shares the template's base
snapshot until it diverges.

The cost is that the clone holds that snapshot, so the template cannot be
rotated while any linked clone still references it. That is bounded rather
than permanent: consolidation detaches each worker 24 hours after creation, so
template rotation is blocked for at most a day.

Consolidation is not a chart concern, but two of its constraints are worth
recording where the flag is set, because both were discovered by testing on a
live host rather than from the docs. A linked clone can only be detached by
moving its disks to a *different* storage — Proxmox refuses a same-storage
move outright. And both disks must move: leaving efidisk0 behind keeps the
template pinned just as effectively as the system disk would.

Set full: true for a pool that must be independent of the template from the
first second.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:14 +02:00
Marian Koreniuk 135eb6bac4 fix(kubernetes): make the Proxmox cluster object reconcilable
Two corrections found by building a real tenant cluster on Proxmox rather than
by rendering the chart. Each produced a symptom that pointed away from its
cause.

capmox skips control-plane endpoint validation only when
spec.externalManagedControlPlane is set. Kamaji owns the endpoint here, so the
field was semantically true all along and merely absent — and on an absent
controlPlaneEndpoint the webhook does not return a validation error, it panics
(internal/webhook/proxmoxcluster_webhook.go:105). The install failed with a
nil-pointer stack trace out of an admission webhook, which reads like a broken
provider rather than a missing field.

The cluster.x-k8s.io/managed-by annotation now renders only on the KubeVirt
path. It tells Cluster API the infrastructure object is owned by something
outside CAPI, so the Cluster controller never sets an ownerRef on it — and
capmox's reconciler exits at "Waiting for Cluster Controller to set OwnerRef on
ProxmoxCluster" without ever creating the InClusterIPPool its workers draw
addresses from. Harmless on KubevirtCluster, which nothing reconciles;
load-bearing on ProxmoxCluster, which capmox must.

With both in place the object reaches Ready=True, the pool is created, and the
Cluster reaches Provisioned.

The proxmox suite's ProxmoxCluster case asserts both: the field is set and
the annotation is absent.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:14 +02:00
Marian Koreniuk db839d1198 feat(kubernetes): add a proxmox substrate for managed clusters
Companion to the same switch in kubernetes-nodes. The control plane is Kamaji
on either substrate, so this changes exactly one object — the one
Cluster.infrastructureRef points at — plus the two tenant addons that exist
only because the workers are KubeVirt VMs living in this cluster.

A KubevirtCluster needs no spec: its workers are in the same cluster and reach
the apiserver over the pod network. A ProxmoxCluster addresses an external
hypervisor, so it carries what capmox cannot infer — which Proxmox nodes may
host VMs, which addresses they may take, and who resolves names for them.
controlPlaneEndpoint stays absent on both: the cluster.x-k8s.io/managed-by
annotation hands that to Kamaji.

The address pool is required rather than defaulted. capmox has no DHCP mode; it
allocates from an InClusterIPPool built out of ProxmoxCluster.spec.ipv4Config,
so an unset pool is not a smaller configuration but a ProxmoxCluster that never
reaches Ready and worker Machines that wait on it forever. Refusing at render
time turns that into a message instead of a silence.

The KubeVirt cloud-controller-manager, its cloud-config and the kubevirt-csi
driver are gated off on the proxmox path. They reach the workers'
VirtualMachines and DataVolumes in this cluster; on Proxmox those objects do
not exist, and leaving them rendered would put controllers in the tenant
reconciling nothing. Their Proxmox equivalents are separate packages and land
separately.

The default path is unchanged: rendering it before and after this commit
differs only in the per-render random bootstrap tokens, and all 19 existing
suites pass untouched.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:29:09 +02:00
Marian Koreniuk 85f576756b feat(kubernetes-nodes): add a proxmox substrate for worker pools
Worker pools are already Talos machines created by Cluster API; only the
substrate underneath them is fixed to KubeVirt. This adds `substrate:
kubevirt|proxmox` and swaps exactly two things: the machine template object
and every reference to it. Autoscaler capacity annotations, kubelet
reservations, MachineDeployment, MachineHealthCheck, WorkloadMonitor and the
talos-reconcile Job are substrate-agnostic and rendered once, which is why
this is a branch in the existing chart rather than a second copy of it.

The default path is unchanged down to the bytes: the render snapshot and all
15 existing suites pass untouched, and the hash-named template keeps its
variable name so no live worker VM rolls on upgrade.

Five things the Proxmox path must do differently, each learned from a
failure that pointed somewhere else:

- Skip capmox's guest-agent and cloud-init-status probes. Talos ships
  neither, and left on they leave the VM booted and the Machine stuck on
  "error waiting for agent: the operation has timed out". This is the same
  class of problem `virtualMachineBootstrapCheck.checkStrategy: none` solves
  for CAPK's SSH probe; capmox's CRD names TalOS in the skipCloudInitStatus
  description.

- Clone a template carrying the nocloud image, not openstack. capmox injects
  bootstrap data as a nocloud ISO, and an openstack image ignores it exactly
  as silently as a nocloud image ignores the configdrive the KubeVirt path
  depends on.

- Pass templateTags through verbatim. capmox compares the template's tag set
  with slices.Equal after sorting, so a selector listing a subset matches
  nothing and reports "found 0 VM templates with tags ..." while the tags are
  plainly present.

- Require resources.cpu and resources.memory. Proxmox has no
  VirtualMachineClusterInstancetype, so the lookup that normally supplies
  effective CPU and memory has nothing to read — and those two numbers feed
  the kubelet reservations and the autoscaler capacity annotations. A pool
  that cannot state them would be scheduled against invented capacity, so it
  is refused at render time. KubeVirt-only knobs (gpus, logSerialConsole,
  podCpuLimit/Request) are refused too rather than accepted and ignored.

- Refuse proxmox.storage unless full is true. A target storage is a full-clone
  parameter: a linked clone always lives on the template's storage, and both
  Proxmox ("parameter 'storage' not allowed for linked clones") and capmox's
  own CRD ("Must set full=true when specifying storage") reject the pair.
  Left to them, a pool with storage set and a linked clone fails at apply time
  with an error about a ProxmoxMachineTemplate the operator never wrote;
  failing the render names the two values instead.

The Talos image is not a chart input on this path: capmox clones a template
that already exists on the hypervisor, so building and rotating that template
is an operator responsibility with no equivalent on the KubeVirt side.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:29:08 +02:00
Marian Koreniuk 9934014908 fix(capi): ship the core provider manifests the repository actually holds
#2946 added a startupProbe to capi-controller-manager so it survives the
certificate provisioning delay. It edited
capi-providers-core/files/core-components.yaml, which is the copy a reviewer
reads. The chart mounts files/components.gz, and that blob was never
regenerated, so the probe never reached a cluster.

The blob is regenerated from the YAML it is supposed to carry:

  gzip -9 -n -c files/core-components.yaml > files/components.gz

The decompressed blob now matches the YAML byte for byte, and the only change
it carries is the six lines of #2946.

This reaches new installs only. capi-operator re-applies a provider when the
hash of its spec and its config Secret changes; the ConfigMap holding the
components is in neither, so a cluster that already runs capi-providers-core
keeps its manager without the probe until the provider's version moves.
Whether existing clusters should get the probe, and how, is left open here on
purpose: moving spec.version and the ConfigMap name together would force the
re-apply, but that is a version decision for the package, not a side effect of
regenerating a blob.

.helmignore is corrected in the same change: `files/.*-components.yaml` is a
glob, so it matched only names beginning with a dot, and `helm package`
shipped the readable YAML alongside the blob that duplicates it.

hack/capi-core-components_test.bats holds the two files to the same bytes from
now on, so the next edit of one without the other fails in CI.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:26:13 +02:00
Marian Koreniuk 18b83a72b3 feat(platform): make the Proxmox substrate installable from a bundle
Until now the two Cluster API packages behind the Proxmox substrate existed but
no bundle emitted them, so a Cozystack install shipped neither and the substrate
was reachable only by applying Package CRs by hand.

They join the iaas bundle as opt-in packages, where the KubeVirt infrastructure
provider is unconditional. The asymmetry is the point: capmox drives a
hypervisor outside this cluster and needs a Secret holding a Proxmox API token,
which capi-operator reads before it installs anything — with no Secret the
provider is never deployed at all. Enabling it by default would ask every
Cozystack install that has no Proxmox to supply hypervisor credentials.

Enabling capmox brings the in-cluster IPAM provider with it — capmox allocates
worker addresses from an InClusterIPPool and has no DHCP mode, so on its own it
never gets a ProxmoxCluster to Ready. IPAM stays independently enableable,
since assigning addresses is a generic Cluster API concern rather than a Proxmox
one.

Naming capmox in enabledPackages while disabling IPAM stops the render. The
cluster would catch that combination on its own: the capmox PackageSource
depends on the IPAM Package, so with that Package absent capmox goes
Ready=False with DependenciesNotReady and no HelmRelease is ever created. What
it would not do is say why. Failing the render names the two conflicting list
entries at the moment someone writes them, instead of leaving a condition to
decode later.

The negated containsDocument asserts in the wiring suite carry `any: true`.
Without it a negated containsDocument passes whether the document is in the
render or not, so the two properties the suite exists for — off by default,
and disabledPackages winning — could not fail.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:25:00 +02:00
Marian Koreniuk 67db200530 feat(cluster-api): add Proxmox infrastructure provider package
Packages ionos-cloud/cluster-api-provider-proxmox (capmox) the same way
capi-providers-infraprovider packages CAPK: vendored components gzipped into
a ConfigMap, an InfrastructureProvider that fetches them by label.

Two details differ from the KubeVirt package on purpose.

The fetchConfig label is `infraprovider-provider: proxmox`, not the shared
`infraprovider-components: cozy`. A shared label would put both providers in
one match. capi-operator would still install the right one — it keys every
matched ConfigMap by its name as the version and takes the one equal to
spec.version — but it builds that repository in a single pass and returns an
error for the whole list as soon as one match is malformed: a name that does
not parse as semver, or missing metadata. Sharing the label would let a broken
ConfigMap belonging to one provider stop the other from being installed. The
bootstrap packages already separate on a `bootstrap-provider` key; this
follows that.

components.gz is produced with `gzip -n`. Without it the header carries a
timestamp, so regenerating from identical input yields a different blob and
the committed artifact appears to change when nothing has. `make components`
records the version pin, the source URLs and the flag, so refreshing the
vendored manifests is reproducible rather than remembered. It also checks
both downloads against the SHA-256 sums recorded in the Makefile before
packaging them: a release tag is mutable, so the pin is on the bytes, and
bumping CAPMOX_VERSION means recording the new sums next to it.

Pinned to the v0.7 series: from v0.8 capmox declares the v1beta2 Cluster API
contract, while capi-providers-core here is v1.10.1, which serves v1beta1.

The package source depends on the in-cluster IPAM provider and deliberately
not on cozystack.kubevirt: the KubeVirt provider needs it because its VMs run
inside this cluster, whereas a Proxmox hypervisor is external and reached
over its API with the credentials named by `credentialsSecretName`.

That Secret is a precondition rather than an optional extra, and the chart
does not create it — a hypervisor token belongs to the platform's secret
management, not to a Helm release. capi-operator reads the Secret before it
installs the provider, so while it is missing nothing is deployed at all.
values.yaml carries what an operator needs at that point: that PROXMOX_URL is
the bare https://<host>:8006 — capmox appends /api2/json itself, and a URL
that already carries it sends every call to /api2/json/api2/json/... and gets
501 back — the full user@realm!tokenid form of PROXMOX_TOKEN, and the fact
that a rotated token reaches the manager only after the next provider
reconcile and a restart of its pod.

Both edges are asserted in tests/sources_capi_provider_proxmox_test.yaml;
losing the IPAM edge reproduces a failure mode whose symptom points nowhere
near its cause.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:23:45 +02:00
Marian Koreniuk 86834d30a9 feat(cluster-api): add in-cluster IPAM provider package
Cluster API ships address management as a separate provider, and Cozystack
has never needed one: KubeVirt workers lease their addresses from the
cluster network's DHCP. An infrastructure provider that allocates addresses
instead of leasing them cannot work without it.

capmox is such a provider. It builds an InClusterIPPool out of
ProxmoxCluster.spec.ipv4Config and hands every VM a static address from it.
With no IPAM provider installed, the pool kind does not resolve:

  Reconciler error: no matches for kind "InClusterIPPool"
                    in version "ipam.cluster.x-k8s.io/v1alpha2"

ProxmoxCluster then never reaches Ready and every ProxmoxMachine sits in
WaitingForClusterInfrastructure indefinitely — indistinguishable, from the
outside, from a provider that is simply broken.

Pinned to the v1.0 series rather than v1.1.0: both serve InClusterIPPool
v1alpha2, but v1.0 declares the v1beta1 contract that matches
capi-providers-core v1.10.1.

`make components` checks both downloads against the SHA-256 sums recorded
in the Makefile before packaging them. A release tag is mutable, so the pin
is on the bytes rather than on the tag; bumping CAIP_VERSION means recording
the new sums next to it.

The package carries no edge to any infrastructure provider. The dependency
belongs on the consumer's side, which keeps this reusable for any future
provider with the same requirement.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:23:45 +02:00
scooby87 cffe509dcf feat(monitoring): per-tenant OTLP collector (tracing phase 2) (#3763)
## What this PR does

Phase 2 of distributed tracing (design cozystack/community#38, epic
#3761), built on the phase-1 backend (#3762, merged): a per-tenant
**OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It
lives in the tenant namespace (like a per-tenant vmagent), so
app→collector traffic is intra-namespace and never crosses the tenant
boundary. No OTel operator exists in the platform, so this is a plain
Deployment + ConfigMap + Service (the fluent-bit pattern).

## Changes

- **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`,
HTTP `:4318`) → `memory_limiter` + head sampling
(`probabilistic_sampler`, default 10%) + optional `attributes/redact`
(PII/secret control point) + `batch` → `otlphttp` exporter to the
`tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428`
single, endpoints per upstream docs). PSS-restricted securityContext,
digest-pinned image via `cozy-lib.image`, config-checksum roll.
- **app-facing Service `otel-traces`** (4317/4318).
- **`tracingCollector` values** — `enabled` / `replicas` /
`samplingPercentage` (schema-bounded 0-100) / `redactAttributes` /
`resources`. A **fail-guard** rejects an enabled collector fronting more
than one `tracingStorages` backend (it exports to a single backend by
design).
- **`otel-traces` WorkloadMonitor** and dashboard resource-map entry,
rendered under the collector's own gate. The same resource-map Role also
gets the pre-existing `alermanager` typo corrected to `alertmanager`, so
the alertmanager WorkloadMonitor is readable from the dashboard.
- helm-unittest coverage for the collector and its WorkloadMonitor;
regenerated `values.schema.json` / `README.md` / `monitoring-rd`
ApplicationDefinition.

## Verification

- `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all
pass.
- `helm template` renders cleanly; `make generate` idempotent.
- Live check on a test cluster (single-mode backend): the collector
starts with `redactAttributes` containing a numeric, a colon-bearing and
an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted
and is queryable from the backend through the Jaeger API, with the
redacted attributes removed and the others kept.
- Local `/branch-review` — **LGTM** (one blocker + four recommendations
fixed, each with a test: multi-backend guard, memory_limiter cgroup
limit, configurable replicas, sampling bounds, trust-boundary wording).

## Scope / deferred (epic #3761)

`AccountID`/`ProjectID` header injection for the shared-central topology
(phase 4) and engine integrations (phase 3) are follow-up PRs.

### Screenshots

N/A, no UI change.

### Downstream repositories

The diff touches only `packages/extra/monitoring`,
`packages/system/monitoring` and `packages/system/monitoring-rd`. The
website's monitoring reference page is regenerated from the package
README by the release bot, and the Terraform provider models only the
tenant's `monitoring` flag, not these values, so no follow-up is needed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage in cluster or single-instance modes,
with retention settings and Grafana datasources.
* Added an enabled-by-default OpenTelemetry collector with configurable
replicas, sampling, attribute redaction, and resource limits.
* Added workload monitoring and automatic resource recommendations for
configured trace components.
  * Added readiness health checks for both tracing backend modes.

* **Bug Fixes**
* Corrected dashboard permissions for alert monitoring and trace
collector resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 16:27:45 +03:00
Aleksei Sviridkin 368d00b5d4 test(e2e): remove unused monitoring suites and audit BATS runners (#4019)
## What this PR does

I removed the two monitoring OIDC BATS suites that had no runner after
the Chainsaw migration, and brought the runner audit and wrapper OIDC
test from [#3778](https://github.com/cozystack/cozystack/pull/3778).

Also added checks for audience mapper and `OIDC_MODE=System`. The named
`KeycloakRealmGroup` test documents coverage the document-count and kind
assertions already provided.

Runner audit reports unused BATS suites, missing files in runner paths,
and unresolved Make variables (including `$(E2E_SUITE)`).

Live Monitoring API admission, Grafana operator reconciliation and mode
transitions are not tested here.

### Validation

- `make -j4 unit-tests SHELL=/usr/bin/dash` passed.
- Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart
tests passed.
- Runner audit: 12 cases passed under dash and Bats 1.14.0. The full
shell syntax and BATS sweep passed under dash too.
- Three chart mutations and seven runner mutations failed as expected.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```
2026-09-25 16:55:05 +04:00
scooby87 03e2da5805 feat(postgres,mariadb)!: drop plaintext user passwords (#4078)
## What this PR does

Removes the plaintext `users[].password` field from the `apps/postgres`
and `apps/mariadb` charts (and the mariadb `root` account), so
managed-database passwords can no longer be pinned in tenant values or
committed to git. Passwords are always chart-generated into the
`<release>-credentials` Secret and preserved across reconciles via
`lookup`, matching the model `redis`/`valkey` already use. This closes
the "User passwords" item of the portal upstream epic
(aenix-org/cozyportal#1060).

The generated password never lands in the init-script Secret or the SQL
it runs: the postgres init-job reads each user's password from the
mounted `<release>-credentials` Secret at run time and applies it
through a psql variable (`ALTER ROLE … WITH PASSWORD :'pw'`), so it does
not linger in the init-script manifest a tenant can read. It does live
in cleartext `stringData` in the `<release>-credentials` Secret itself,
which is a chart manifest and therefore kept in Helm release history for
`MaxHistory` revisions like any chart-generated secret — that residue is
exactly why rotation cannot be built on the chart (see below). A
`password` or a `passwordRotation` key left over in a tenant's values is
accepted-but-inert and draws an admission warning, so an operator
editing it is told it has no effect rather than getting a silent 200.

Backup/restore into a copy keeps the target's chart-managed credentials
working: the CNPG driver clears `bootstrap.enabled` once recovery is
healthy so the init-job re-applies the generated passwords onto the
recovered roles, and the MariaDB driver excludes `mysql.global_priv`
from the dump (`ignoreGlobalPriv`) for backups captured from this
release onward, so restoring one of those into a copy does not clobber
the target's chart-managed users or `root`. Excluding the grant table
has a flip side, and it applies to every MariaDB restore, not only a
to-copy one: only accounts declared through `User` CRs (the chart's
`users` map) are captured, so an account created out of band directly in
MySQL is in no backup and a restore brings back its data and `mysql.db`
grants but not the account itself. This is also a backup-side fix only:
a restore from a backup taken **before** this change still replays the
source's `mysql.global_priv` and overwrites the target's users and
`root`, and `root` then needs an out-of-band reset (the operator applies
`rootPasswordSecretKeyRef` only at datadir bootstrap, so there is no
in-band path). See `docs/operations/backup-classes.md` for the
convergence window, the out-of-band-account limitation, and this
old-backup caveat.

Password **rotation** is deliberately **not** in this PR. A
chart-rendered credentials Secret keeps its cleartext in Helm release
history for `MaxHistory` revisions, so a rotation built on it cannot
revoke a leaked password; rotation is being reworked as a controller
that owns the Secret — design proposal in cozystack/community#72.

Migration is seamless: existing releases keep their current passwords on
upgrade (the value already lives in `<release>-credentials`, which
`lookup` re-reads and preserves); a `password` still present in a
tenant's values is ignored, so no existing HelmRelease fails validation.

Verification: `helm unittest` passes for both charts (postgres 87,
mariadb 126, including render-time username guards that reject any name
k8s would later reject as an invalid Secret key or User resource name);
`go build ./...`, `go vet`, the backup-controller tests and the
apps-apiserver tests are green; `make generate` was run with the
CI-pinned `cozyvalues-gen` v1.6.0 so `values.schema.json`, the READMEs,
the API types, and the `cozyrds` ApplicationDefinitions are all in sync
(the branch was rebased onto `main`, which reconciled cleanly with the
read-replica autoscaling feature that landed there).

### Screenshots

N/A — no UI change.

### Downstream repositories

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: affected — `content/en/docs/next/cozystack-api/go-types.md`
is hand-written and still shows `Password:` in the Go examples, and the
managed-app reference pages regenerate from these chart READMEs, so both
document `users[name].password` until refreshed (not yet filed).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: affected — the provider hand-maintains the postgres/mariadb
schemas and per-field defaults; this PR removes `users[].password`. A
follow-up issue/PR is needed there (not yet filed).

`terraform-provider-cozystack` and `website` are reached by this diff;
both boxes are left unticked because the follow-ups are not yet filed.
Same-repo future work this change suggests: the controller-owned
password rotation (design in cozystack/community#72), and extending the
"passwords only in the Secret" model to the other engines that still
accept plaintext (`clickhouse`, `mongodb`, `nats`, `opensearch`,
`rabbitmq`, `vpn`).

### Release note

```release-note
feat(postgres,mariadb)!: passwords for managed PostgreSQL and MariaDB users can no longer be set via `users[].password` — they are always auto-generated and stored only in the `<release>-credentials` Secret, never rendered into the init-script or the SQL it runs. Existing releases keep their current passwords on upgrade; a `password` (or a `passwordRotation`) left in values is ignored with an admission warning. Password rotation is deferred to a follow-up controller design (cozystack/community#72). MariaDB backups now exclude the `mysql.global_priv` grant table (`ignoreGlobalPriv`), so a restore recreates only accounts declared through the chart's `users` map — an account created out of band directly in MySQL is not captured and must be recreated after a restore.
```
2026-09-25 15:49:53 +03:00
Alexey Artamonov 9d6fea7017 fix(monitoring): correct the alertmanager name in the dashboard Role
The Role listed the WorkloadMonitor as 'alermanager', so the real
'alertmanager' monitor was never granted: a tenant dashboard is
forbidden from reading its status and the alertmanager health tile
breaks. The test pins the correct name and the absence of the typo.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 13:35:34 +03:00
Alexey Artamonov 60b1b05434 feat(monitoring): add per-tenant OTLP collector for distributed tracing
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.

The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.

redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.

The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 13:35:34 +03:00
Aleksei Sviridkin 2d2186d1f0 feat(monitoring): VictoriaTraces backend + Grafana datasource (tracing phase 1) (#3762)
## What this PR does

Phase 1 of distributed tracing (design: cozystack/community#38; epic:
#3761): a **VictoriaTraces backend and Grafana traces datasource** in
the monitoring stack, mirroring the logs (`VLCluster`) implementation.
No new operator: `VTCluster`/`VTSingle` already ship in
victoria-metrics-operator.

- **`tracingStorages`** values in `extra/monitoring` (annotated,
schema-generating) and `system/monitoring` (defaults), per-entry `mode:
cluster|single`. **Empty by default, so tracing is off out of the box**:
nothing produces spans until the collector ships, so no backend is
provisioned until an operator adds an entry.
- **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or
`VTSingle` (edge/dev/small); `retentionPeriod` plus optional
`retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup
label**, reused by the existing post-delete hook, so no hook change is
needed.
- **Render-time validation** (`_helpers.tpl`): the installed
VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise
reach the operator and fail after admission. Each `tracingStorages`
entry is checked before anything renders: `name` must be an RFC 1123
label of at most 42 characters (the `vtstorage-<name>-<hash>` pod
revision label is capped at 63) and unique across the list, `mode` must
be `cluster` or `single`, `storage` must be a positive quantity (a zero
size makes the operator mount an EmptyDir in single mode, and a negative
one fails when the PVC is created), and `retentionDiskUsageBytes` must
be a VictoriaMetrics `BytesString` (the operator rejects the
Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage`
field is also typed as a quantity in the schema, and the rendered name
is quoted so a numeric name stays a string.
- **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces
datasource per entry (built-in Grafana core datasource, no plugin).
- **VPA**: a `VerticalPodAutoscaler` for each trace component
(`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle`
Deployment), matching the metrics/logs sizing, with `updateMode:
Initial`.
- **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on
`status.updateStatus`, alongside the existing `VLCluster` gate.
- **VictoriaLogs datasource fix**
(`templates/vlogs/grafana-datasource.yaml`): the range over
`logsStorages` emitted no `---` between documents, so two or more
entries collapsed into one document and only the last datasource
survived. Each entry is now its own document.
- **Tests and schema**: helm-unittest suites for off-by-default, cluster
vs single, datasource type and both query URLs, retention, PVC label,
VPA targets, every validation guard with its boundaries, and per-entry
vlogs datasource documents, plus regenerated `values.schema.json`,
`README.md` and the `monitoring-rd` ApplicationDefinition.

#### Upgrade impact

With default values the only rendered object that changes is the
`monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle`
`healthCheckExprs` and `tracingStorages: []`. Flux evaluates those
expressions only against objects in the release inventory, so they are
inert while tracing is off. No VictoriaTraces object is created on
upgrade. Installs with two or more `logsStorages` entries additionally
gain the `GrafanaDatasource` objects that were previously dropped (see
the VictoriaLogs fix above); nothing else under VictoriaLogs or
VictoriaMetrics changes.

#### Verification

- `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring`
(11 tests) pass. Each validation guard and each branch of the
zero-storage pattern was mutation-checked: reverting it turns its own
test red.
- `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly.
`tracingStorages: []` (the default) renders nothing. An invalid,
duplicate or over-long `name`, a zero or negative `storage`, or a bad
`retentionDiskUsageBytes` (`8Gi`) is refused at render with an
actionable message; a numeric `name` renders as a string. `make
generate` is idempotent.
- **Datasource URLs verified against upstream docs** ([VictoriaTraces
and
Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)):
cluster `vtselect:10471/select/jaeger`, single
`vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana
datasource; the Tempo Query API (`/select/tempo`, `type: tempo`,
VictoriaTraces v0.9.4+) has a richer TraceQL UX but is
upstream-experimental, so Jaeger is the default. Both URLs are pinned by
unit tests.
- The `retentionDiskUsageBytes` pattern, the zero-storage check and the
name limit were checked against victoria-metrics-operator v0.68.4
(`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>`
StatefulSet name) and the Kubernetes `ControllerRevisionName` naming.

#### Scope / deferred (tracked in #3761)

Per-tenant OTLP Collector, engine integrations, shared-central topology,
and docs are follow-up PRs. This PR intentionally does **not** enable
the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and
the gRPC listener lands with the collector.

Builds on the now-merged design proposal cozystack/community#38.

### Screenshots

Not a UI change.

### Downstream repositories

Walked against the diff, no box ticked yet:

- cozystack/website: `monitoring` is in the website `Makefile`
`SERVICES` list, so its reference page is regenerated from
`packages/extra/monitoring/README.md` automatically on the next stable
tag. A user guide for tracing belongs to the docs follow-up in #3761; no
website issue or PR exists for it yet.
- cozystack/terraform-provider-cozystack: the provider has no
`Monitoring` resource (only the tenant's `monitoring` flag), so the new
schema field does not reach it.
- The remaining repositories in the trigger map are not reached by this
diff: no platform values, variants, installer values, `hack/` tooling,
node prerequisites, ApplicationDefinition semantics or telemetry names
change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage with cluster or single-node
deployment modes.
* Supports retention periods, persistent storage sizing, storage
classes, and optional disk-usage limits.
* Automatically configures Grafana tracing data sources for each enabled
trace backend.
* Monitoring readiness reflects the health of configured trace storage
services.
  * Invalid trace storage settings are rejected during configuration.
* **Bug Fixes**
* Every VictoriaLogs storage now gets its own Grafana datasource; with
several log storages, all but the last were previously dropped.
* **Tests**
* Added coverage for storage rendering, retention settings, validation
guards, Grafana endpoints, health checks, and cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 14:21:26 +04:00
IvanHunters 0bae19fccc feat(lineage-webhook): refuse a second ApplicationDefinition for a kind
A definition declaring an application kind that another definition already
declares now fails when it is applied, instead of sitting next to the first
one where it has no effect. An update that keeps its kind is allowed, so a
duplicate that predates the check keeps upgrading, and so is a second
definition from the same Helm release, since a release that renames its
definition object creates the new one before it deletes the old one.

The webhook is registered with failurePolicy Ignore: ApplicationDefinitions
ship with the platform itself, so an outage of the webhook must never block
a bootstrap or an upgrade, and a failure to read the existing definitions
lets the request through with a warning rather than answering with an error,
which Ignore would not cover. It is the feedback layer; which definition
manages a kind is decided by the ownership rule whatever it lets through.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:39:35 +03:00
Alexey Artamonov 237f2e2bbd fix(postgres): say how to get past a refused extension clause
Before the charset guard, an extension value was spliced into CREATE
EXTENSION whole, so a release can carry a clause after the name, such
as earthdistance CASCADE, and that is the usual way to pull in a
dependency. The guard refuses such a value on upgrade too, and keeping
it that way is right, but the tenant needs to know what to write
instead. The reject message now says: drop the clause, which changes
nothing for an extension that is already installed because the script
never drops one and CREATE EXTENSION IF NOT EXISTS leaves it alone, and
list the dependencies first, since extensions are created in list
order.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 00:03:22 +03:00
Aleksei Sviridkin 564f540f46 fix(kubernetes): render the OIDC UID header only from v1.33 (#4407)
## What this PR does

Stacked on #4398; review the diff against `fix/kamaji-pin-hash-reset`.

The tenant Kubernetes chart now renders
`--requestheader-uid-headers=X-Remote-Uid` only from an effective
Kubernetes version of v1.33, where `RemoteRequestHeaderUID` is beta and
on by default. It no longer renders a `--feature-gates` term of its own
on any version. On v1.32 it used to add a term enabling that gate, which
is alpha there, whenever it could read the operator's entries and they
left the gate unset; no release tag contains that path, so dropping it
now changes nothing a user has run.

The chart still reads `--emulated-version`, because emulation moves the
effective version: `--emulated-version=1.32` on a v1.35 tenant puts the
gate back to alpha-and-off, and the header flag is not rendered there.
It still reads the operator's `--feature-gates` entries for an opt-out
(`RemoteRequestHeaderUID=false`, or `AllBeta=false` with no explicit
`=true`), so the header flag never lands next to a disabled gate.
`AllAlpha` is no longer read, since it says nothing about a beta gate.

An unquoted `--emulated-version=value` entry whose kube version has a
non-zero patch component now fails the render and names the entry. The
rule mirrors upstream rather than the regex in #4396: `toVersionMap`
parses each value with `version.Parse` and refuses only `ver.Patch() !=
0` (`staging/src/k8s.io/component-base/compatibility/registry.go` lines
320-326 at v1.35.0, the same code in
`staging/src/k8s.io/apiserver/pkg/util/version/registry.go` at v1.31.0).
`version.Parse` takes `^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$`
(`versionMatchRE` in `k8s.io/apimachinery/pkg/util/version/version.go`,
identical at v0.31.1 and v0.35.0). So `1.32`, `v1.32`, `1.32-rc`,
`1.32.0` and `1.32.0-rc` all start an apiserver emulating 1.32, and
`1.32.4` and `1.32.4-rc` do not. A `^[0-9]+\.[0-9]+$` check would have
refused the first four. The chart extracts the numeric run with the same
pattern, reads the minor from it, and fails only on a non-zero third
component. A quoted value is still left unread.

Two deliberate narrowings:

- The chart no longer refuses a mix of bare and `kube:`-prefixed
`--feature-gates` terms. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it renders.
The apiserver still refuses the mix
(`component-base/compatibility/registry.go:405` at v1.35.0), and the
chart now passes it through like the other fatal gate values it already
leaves to the apiserver: unknown names, unparseable values, quoted
terms.
- On v1.32 the chart now neither renders nor judges the UID flags, as it
already did on v1.31. An operator's own `--requestheader-uid-headers`
entry without the gate reaches an apiserver that refuses it, which is
how every released chart version behaved. A cluster built from `main`
that already runs a v1.32 tenant with OIDC on loses the chart's
`--requestheader-uid-headers` and gate term on the next render; no
release carried them. That holds only while no release is cut from
`main` before this merges; a release that carries the v1.32 gate term
first would turn this into a breaking change for v1.32 tenants, whose
migration is to write both flags into `extraArgs` themselves.

An element such as `kube=1.32=x` is now split at its first `=` only, the
way `toVersionMap` splits it with `SplitN(compVer, "=", 2)`. The chart
used to skip that element, which the apiserver reads as 1.32, and so
rendered the header flag against an emulated 1.32. That gap predates
this branch.

Rebased onto #4398 at `fb2e57c3b`, which reads a two-element
`--feature-gates` and `--emulated-version` separately. That behaviour
carries over unchanged under the v1.33 floor: behind a hidden
`--emulated-version` only an explicit `RemoteRequestHeaderUID=false` is
refused beside a header entry, while a blanket `AllBeta=false` waits for
a reading, because a lower emulated version can make the gate alpha
again; nothing is refused as an opt-out behind a hidden
`--feature-gates` value; the empty-element refusal applies behind both;
and behind either one `X-Remote-Uid` is added only beside a uid-headers
list of the operator's. The README breaking-changes note from #4398 now
says the chart refuses neither the mixed gate spelling nor two kube
emulated versions. In `docs/oidc-tenant.md` the paragraphs that list
refusals now name the managed-flag move and refusal from #4398 and no
longer carry a count, and the two-element paragraph now says what the
chart renders behind a hidden value.

Tests: `helm unittest` goes from 308 on #4398 to 298. 26 cases are
deleted. 25 of them pinned the v1.32 gate term, `AllAlpha` handling or
the mixed-spelling refusal. One emulated a version below 1.32, which the
below-floor emulation case now covers. v1.32 cases that tested parsing
rather than the gate are retargeted to v1.34. Sixteen cases are added:
the patch-level refusal, the version parsing below the floor and above
it (on v1.35 emulating `1.33-rc`, `v1.33`, `1.33.0` and `kube=1.33=x`,
where only a correct minor renders the header), the first-`=` split, a
pin that a mixed gate spelling now passes through, emulation landing
exactly on 1.33, and a v1.32 opt-out beside a header entry passing
through untouched.

Coverage notes, not applied: no case pins that the patch-level refusal
stays off with `spec.oidc.mode: None`, where the block that holds it
does not run. The case "splits a component element at its first equals
sign only" pins that the kube element of `kube=1.32=x` is read. It does
not pin the split itself, because the version pattern ignores what
follows the numbers either way.

The `extraArgs` field description still calls the field the escape hatch
for apiserver flags unrelated to OIDC. It is unchanged here, and it
stays accurate for the v1.32 advice in the guide, because the UID flags
belong to the aggregation layer rather than to OIDC.

Optional, not applied: on v1.32 the chart could refuse an operator's
`--requestheader-uid-headers` entry that comes without the gate, since
the gate is off by default there and the apiserver will not start. The
same skip also drops two refusals the branch history had at 1.32, a
header entry beside `RemoteRequestHeaderUID=false` and a header list
ending in a comma; both now reach an apiserver that refuses them. It
stays out so that v1.32 matches v1.31 and the released charts, where the
chart judges none of these flags.

Not applied: for a value with a zero-prefixed major such as `01.32.4`,
the patch refusal suggests `01.32`. The apiserver refuses both values
over the leading zero, so the configuration fails either way and only
the suggested fix is off.

Why the patch refusal stays while the mixed-spelling refusal goes: the
patch refusal is about `--emulated-version`, which the chart reads
itself to decide what to render. The mixed spelling concerns
`--feature-gates`, which the chart no longer writes to.

Not split out: the `--emulated-version` parsing and the patch-level
refusal share the fix commit with the floor move, because both change
how the same value is read. The change also removes a comment from
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` that became false, so
select-e2e runs the full suite for this PR.

Default path: renders of the parent and this branch through
`tests/values/common.yaml` and `tests/values/oidc-system.yaml` at v1.31
to v1.35 are byte-identical, once the randomly generated Kamaji token
fields are masked, except for OIDC System at v1.32, which loses exactly
the two UID lines.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only field that changes is the description of `version`, which
returns to its earlier wording. The website's kubernetes reference page
is regenerated from the README on the next tag, and the Terraform
provider keeps its own copy of that description, which already reads
"Kubernetes major.minor version to deploy.".

Closes #4373

Closes #4396

### Release note

```release-note
fix(kubernetes): the tenant chart renders the aggregation-layer UID header flag with OIDC on only from an effective Kubernetes v1.33, and never renders a --feature-gates term of its own. v1.32 tenants render no UID flags, which matches every released version, so nothing regresses. This supersedes the v1.32 clause of the #3539 release note. An unquoted --emulated-version entry with a patch-level kube version such as 1.32.4, which the kube-apiserver refuses, now fails the render and names the entry.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* OIDC-enabled clusters on Kubernetes 1.33 and later now receive the UID
request-header setting without an added feature-gate flag.
* Chart rendering now checks Kubernetes and Talos version compatibility
and prevents rendering when worker deployments do not meet ownership or
retention requirements.
* **Bug Fixes**
* Emulated Kubernetes versions with non-zero patch components now fail
with a clear error; supported version formats are recognized more
consistently.
* **Documentation**
  * Updated version and OIDC guidance to reflect current chart behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 00:01:56 +04:00
Aleksei Sviridkin 339cffe70f feat(kamaji)!: update to 26.9.4-edge and read every apiServer extraArgs entry (#4398)
## What this PR does

BREAKING CHANGE: a Kubernetes application whose
`controlPlane.apiServer.extraArgs` names a flag the tenant control plane
now owns fails to render until that entry is removed, except that an
`--enable-admission-plugins=` entry is moved into the new
`controlPlane.apiServer.admissionControllers` and a
`--kubelet-preferred-address-types=` entry into the new
`controlPlane.kubelet.preferredAddressTypes`.

Closes #3541.

Updates the Kamaji build this repo ships and rewrites the part of the
tenant Kubernetes chart whose correctness rested on the old one. The new
pin is `26.9.4-edge`, which is what `make update` picks.

Two upstream changes reach us. The tenant control plane no longer folds
the kube-apiserver argument list into a map keyed by flag name, so
entries supplied through the control plane spec arrive at the apiserver
verbatim, duplicates included. And the reset that rewrites those
arguments from scratch keys on a hash of the list rather than on its
length, so swapping one flag for another no longer strands the old one
on the running Deployment.

The OIDC block in the tenant chart read only the last entry of each
flag. That was right while earlier entries were dropped before the
apiserver saw them, and it is wrong now. An operator whose `extraArgs`
carried `--feature-gates=RemoteRequestHeaderUID=false` followed by an
unrelated `--feature-gates` entry got
`--requestheader-uid-headers=X-Remote-Uid` rendered next to a gate the
apiserver has switched off, and the control plane would not start. A
blanket `AllBeta=false` has the same shape, and so does an
`--emulated-version` entry naming kube followed by one naming another
component.

The chart now reads every entry with the accumulation each flag
performs. Occurrences of `--requestheader-uid-headers` append into one
list, so where a list of the operator's own omits `X-Remote-Uid` the
chart adds its own element beside it instead of refusing to render: the
union is what the apiserver validates, and their header keeps working.
Feature-gate terms collect per component and are applied in one pass, so
the chart carries only its own term, placed last where it wins. Naming
the kube component both bare and `kube:`-prefixed now fails the render,
because the apiserver refuses that configuration and nothing the chart
renders repairs it. `--emulated-version` accumulates too, and two
entries naming kube stop the apiserver outright, so the chart renders
nothing rather than pick a version that will not exist.

The same upstream change narrows what `extraArgs` can do, and the chart
now refuses the part that stopped working. Kamaji 26.9.4-edge sets a
list of kube-apiserver flags itself and drops any `extraArgs` entry that
names one before the apiserver sees the list, where `26.3.6-edge` merged
`extraArgs` last and let the entry win. A tenant with
`--enable-admission-plugins=NodeRestriction,AlwaysPullImages` loses both
plugins on the first reconcile after the upgrade, with nothing to say
so. The drop happens on the running control plane whatever the chart
renders, so the chart cannot prevent it; what it can do is refuse to
render such an entry, in every `oidc.mode`, and name it. The list
mirrors `buildKubeAPIServerCommand` at the pinned tag for the control
plane this chart renders (an etcd DataStore, konnectivity on, no
`dataStoreOverrides`), and the `controlPlane.apiServer.extraArgs`
description in the chart README carries it. A flag the control plane
only defaults, such as `--authorization-mode` or
`--requestheader-group-headers`, is still yours to set.
`--enable-admission-plugins` gets a replacement: the new
`controlPlane.apiServer.admissionControllers` passes a list through to
the KamajiControlPlane `spec.admissionControllers`, which the control
plane renders into that flag, and the application API checks each name
against the enum the KamajiControlPlane CRD declares. The list is
rendered only when non-empty, so a tenant that sets nothing renders
exactly as before, and a non-empty list replaces the control plane's
default set, as the `extraArgs` entry did under the previous build.
`--kubelet-preferred-address-types` gets the same treatment: the list
the chart used to hardcode on the KamajiControlPlane is now
`controlPlane.kubelet.preferredAddressTypes`, which renders that list
when left empty. Neither entry is refused where it can be moved: an
`--enable-admission-plugins=` or `--kubelet-preferred-address-types=`
entry is moved into its field while the field is empty, on every render,
and left out of the rendered `extraArgs`, so a tenant carrying one keeps
its setting across the Kamaji update with no edit and no ordering to get
right. The entry is refused beside a non-empty field, when it names
nothing, and in the two-entry spelling.

A separate defect in the same package is fixed here. The
TenantControlPlane schema reaches existing clusters through the
`kamaji-crds` subchart's templated CRDs, which every release upgrade
applies; `crds/` holds the copies a fresh install lays down first.
Directory-based renders lost that schema entirely, `helm template`,
`lint` and `package` alike, because the parent `.helmignore` matched the
subchart's `hack/` directory at any depth and that is where the
templates keep the schema they read with `.Files.Get`. Helm honours
`.helmignore` for a directory and skips it for an archive, so the
cluster never saw the difference and no local check could catch a schema
defect. The patterns are anchored to the chart root and a unit test
asserts the rendered CRDs carry their versions.

The bump is not cleanly reversible, and that is the one thing to know
before rolling it out. The args annotation holds a checksum once the new
build has written it, and the old build reads that annotation with
`strconv.Atoi`. The parse fails and `resetKubeAPIServerFlags` returns
early, ahead of both the args reset and the annotation write that would
put a number back. So after a rollback to `26.3.6-edge` the count-keyed
reset is disabled on every TenantControlPlane: a later removal of an
`extraArgs` entry no longer clears the flag from the running Deployment,
and it stays that way until someone writes a number into
`kube-apiserver.kamaji.clastix.io/args` by hand.

Findings from a live upgrade on a dev cluster, none of them caused by
the diff. The DNS-service-IP check moved out of a CEL rule on the CRD
into the admission webhook (`internal/webhook/handlers/tcp_dns.go`), so
it now depends on that webhook being reachable where the apiserver used
to enforce it by itself. The manager's readiness probe targets port 8081
while the webhook serves on 9443, so a Ready pod does not prove 9443
accepts connections; with `failurePolicy: Fail` on the
TenantControlPlane webhook, that window rejects writes rather than
letting them through unvalidated. The readiness probe this build adds to
kube-scheduler and kube-controller-manager survives a rollback, since
the old build neither sets nor clears those two; that one is benign.

Seven limits worth naming rather than leaving to be found. The
mixed-spelling refusal quotes the entry carrying each spelling, so one
entry holding both, `--feature-gates=Foo=true,kube:Bar=true`, is quoted
twice in its own message; the refusal is right and the message reads
oddly. The unit test that pins the rendered CRD schemas asserts on a
count, so a CRD missing from the render altogether fails on an empty
comparison rather than naming the CRD that went missing. There is one
window during the upgrade itself: the chart used to fold an operator's
own `--feature-gates` terms into the entry it rendered, because the old
build would otherwise drop them, and it now renders only its own term.
Nothing orders a tenant release after the Kamaji operator has rolled, so
a `v1.32` tenant that sets gate terms of its own with OIDC on can lose
them until the new operator reconciles it, which it then does on its
own. The same window takes a `--requestheader-uid-headers` list of the
operator's own that lacks `X-Remote-Uid`: the chart now renders its own
entry beside it instead of refusing, and the old operator keeps one
value per flag name, the chart's, so the operator's header is dropped
until the new operator reconciles. Which entry an error message quotes
is pinned only where one entry is eligible. Three variables choose
between equally eligible entries, the gate dictionary taking the last
setting for a name and the uid-headers ones the first, and no test
distinguishes first from last when two entries qualify, so a future edit
could make a message name the wrong entry of the two without a test
noticing. What the chart renders and what it refuses is pinned
exhaustively; it is the provenance in three message strings that is not.
The block is deliberately inconsistent about two shapes that stop the
apiserver equally hard: two `--emulated-version` entries naming kube
make it withhold silently, while a kube-prefixed and a bare gate
spelling in the same configuration make it refuse the render. The
refusal is right where the apiserver rejects the configuration whatever
the chart does; the withholding is right where the chart cannot read a
version it must render against. They are two rules, not one, and a third
that covers both is fair to ask for. Moving the refusal out from under
those conditions so it fires everywhere is the obvious third rule and it
is not free: the apiserver refuses the mix on `v1.31` too, but from
`k8s.io/apiserver/pkg/util/version` rather than from the component-base
`componentGlobalsRegistry.Set()` the message names, so hoisting it means
rewriting that sentence as well. Reading a flag and its value written as
two entries would be better than withholding for it, and is a change of
behaviour rather than a correction, so it is not here. And two sentences
about `--emulated-version` in `docs/oidc-tenant.md` sit thirty lines
apart and read as a retraction in sequence, though each is true at its
own layer: `toVersionMap` accepts an element naming a foreign component,
and the registry rejects it later.

What this leaves open, by name. The guard for a valueless flag at the
end of `extraArgs` covers `--requestheader-uid-headers`,
`--feature-gates` and `--emulated-version` only; any other flag that
takes a value swallows the appended `--authentication-config` the same
way, which is an input mistake the chart does not look for. The
managed-flag refusal matches names as written, as Kamaji does: an
underscore spelling such as `--secure_port` passes both, and the
apiserver normalises it to `--secure-port`, where the operator's value
wins as the later one, the same as before this change. Rendering
`--authentication-config` ahead of the operator's entries would close
that class for OIDC, since only the UID flags would then follow them; it
changes the rendered order, and so rolls every OIDC tenant that has
`extraArgs` of its own, and belongs in a separate change after the OIDC
floor move. One pre-existing comment needs checking: the note above
`extraArgs` in `templates/cluster.yaml` says that omitting the field
would leave the live array untouched under JSON merge patch, which reads
against Helm's `createPatch`, where a key the previous manifest carried
and the new one does not is patched to null; what that comment was
guarding against is still to be verified. The KamajiControlPlane CRD
accepts only the plugin names it lists, so a plugin outside that list
that the apiserver leaves off by default can no longer be enabled; that
limit is Kamaji's, since it owns the flag and restricts the field. A
name moved out of an `extraArgs` entry is not checked against that list
by the chart, so one outside it fails when helm-controller applies the
KamajiControlPlane, with an error that names it, rather than at render;
the chart keeps no copy of the list to check it with. A repeated address
type in a moved entry is dropped, keeping the first place it appears,
because the field is a set. With `oidc.mode: None` the chart does not
read `extraArgs` for the OIDC flags at all, and two shapes regress with
the bump there: a kube feature-gate term written both bare and
`kube:`-prefixed across entries, and two `--emulated-version` entries
naming kube. The old build kept one entry per flag, the new one passes
both, and the apiserver refuses both at startup. Next to a flag and
value written as two entries the chart keeps the refusals that hold
whatever the hidden value is, but it does not read positions: a hidden
`--feature-gates` term keeps the opt-out refusal off even when it sits
before the explicit opt-out and so cannot undo it.

What the live run could not exercise: the konnectivity datapath, because
the stand had no tenant workers, and `--requestheader-uid-headers`,
which comes from the tenant Kubernetes chart and no image swap reaches.
Both are covered by render and unit tests only.

One shape the old collapse was quietly repairing, with OIDC on, needs a
guard now that it is gone. A bare `--requestheader-uid-headers` in
`extraArgs` used to be overwritten by the entry of the same name the
chart renders, because the collapse kept one value per flag name; the
same held for a bare `--feature-gates` on `v1.32`, where the chart
renders a gate entry too. A bare `--emulated-version`, or a bare
`--feature-gates` on any other version, had no entry of the chart's to
be overwritten by and was emitted bare in sorted position, where it took
the next flag as its value, so it never worked. Without it those entries
reach the apiserver as written, and the apiserver gives a valueless long
flag the next argument whatever that argument looks like. What follows
the operator's last entry is the `--authentication-config` the chart
appends, so the flag swallows it and the AuthenticationConfiguration is
never applied. On `--requestheader-uid-headers` the path is accepted as
a header name and the control plane comes up with OIDC silently off,
failing every login for that tenant with nothing in the cluster to say
why; the other two reject it and the apiserver exits. The chart refuses
to render that shape and says which entry to fix.

Two things in the image build fall out of the bump. The datastore
deletion fix carried here as a patch landed upstream verbatim, so its
patch no longer applies, and one failing patch fails the whole apply of
the directory. The Go module overlay is obsolete for the opposite
reason: this tag already carries `golang.org/x/net`,
`golang.org/x/crypto` and `go.opentelemetry.io/otel/sdk` above the
versions the overlay forced, so keeping it would downgrade all three.

What I checked: `helm unittest` passes at 307 cases for the package,
with the OIDC suite carrying every shape above, and the image builder
stage builds end to end at the new tag with the one remaining patch.
Kamaji's own support for Kubernetes 1.36 and 1.37 does not move the
tenant version enum here; that is a separate update of the `kubernetes`
package. No image reference in `packages/system/kamaji/values.yaml` is
touched, since CI stamps that digest.

### Screenshots

No UI changes.

### Downstream repositories

I walked the trigger map file by file against the diff. The chart
templates, the vendored Kamaji chart, the image build and the docs reach
nothing downstream. The one generated file that reaches downstream is
`packages/apps/kubernetes/values.schema.json`, which gains
`controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` and a changed `extraArgs`
description; both new fields are optional with defaults, so a stored
Kubernetes object needs no change, no field is removed or renamed and no
version enum moves. The new fields are the one trigger the map fires on:
the Terraform provider keeps a hand-written schema.
`packages/apps/kubernetes/README.md` feeds the website's managed-apps
reference pages, which a bot regenerates on a stable tag.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: model `controlPlane.apiServer.admissionControllers` and
`controlPlane.kubelet.preferredAddressTypes` on the kubernetes resource
(schema, model, expand/flatten).
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kamaji): update Kamaji to 26.9.4-edge. Tenant kube-apiserver arguments are now derived from a hash of the control plane's extraArgs instead of from its length, so on the first reconcile after the upgrade every tenant control plane rewrites those arguments from scratch and rolls once. The rebuilt list also orders them differently, Kamaji's own flags sorted first and the operator's kept verbatim after them, so the roll happens whether or not the tenant sets any extraArgs. A tenant with OIDC enabled takes a second roll when the chart's own flags land. Rolling the Kamaji image back to an earlier build does not restore the old behaviour: the older code reads that annotation as a number, gives up when it finds a checksum, and from then on never clears a flag you remove from extraArgs, on any TenantControlPlane, until the annotation is set back to a number by hand. The tenant Kubernetes chart reads every extraArgs entry the way the apiserver accumulates them: a --requestheader-uid-headers list that omits X-Remote-Uid now gets the chart's own element beside it instead of failing the render, a feature gate set in an earlier entry is honoured, and naming the kube feature-gates component both bare and kube:-prefixed fails the render, because the apiserver refuses that configuration outright. Writing --feature-gates or --emulated-version as a flag and a value in two separate entries now keeps the chart from rendering UID flags on its own, because an entry the chart cannot read may hide an opt-out; a --requestheader-uid-headers list of your own still gets X-Remote-Uid beside it, and write --flag=value to get the rest back. Action required for tenants that pass their own apiserver flags: Kamaji now sets a list of flags itself and drops a controlPlane.apiServer.extraArgs entry naming one, where the previous build let that entry win, and the Kubernetes application now refuses to render such an entry, except for the two it moves into a field. A tenant carrying one of the others stops reconciling after the upgrade, and no later change to that tenant applies, until the entry is removed; the entry no longer has an effect either way. The list is in the description of controlPlane.apiServer.extraArgs. An --enable-admission-plugins=value entry is moved into the new controlPlane.apiServer.admissionControllers, a list that replaces the control plane's default set as the entry did, and a --kubelet-preferred-address-types=value entry into the new controlPlane.kubelet.preferredAddressTypes, while that field is empty; nothing has to be edited for them. Before upgrading, list the affected entries with: kubectl get kuberneteses.apps.cozystack.io --all-namespaces --output json | jq --raw-output '["--advertise-address","--client-ca-file","--enable-admission-plugins","--service-cluster-ip-range","--kubelet-client-certificate","--kubelet-client-key","--kubelet-preferred-address-types","--proxy-client-cert-file","--proxy-client-key-file","--requestheader-allowed-names","--requestheader-client-ca-file","--secure-port","--service-account-key-file","--service-account-signing-key-file","--tls-cert-file","--tls-private-key-file","--etcd-servers","--etcd-cafile","--etcd-certfile","--etcd-keyfile","--etcd-prefix","--etcd-compaction-interval","--egress-selector-config-file"] as $owned | .items[] | .metadata as $m | (.spec.controlPlane.apiServer.extraArgs // [])[] | select((split("=")[0]) as $f | $owned | index($f)) | "\($m.namespace)/\($m.name): \(.)"' Entries it lists for --enable-admission-plugins or --kubelet-preferred-address-types need a change only when written as two entries, when they name nothing, or next to a non-empty field of their own.
```






<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Configure tenant control-plane admission controllers and kubelet
preferred address types.
- Choose a cert-manager issuer for webhook certificates; self-signed
certificates remain the default.
- **Bug Fixes**
- Improved OIDC argument handling for repeated, split, quoted, and
conflicting settings, with clearer render-time errors.
- DataStores are recognized as unused when no tenant control planes
reference them.
- **Validation**
- DataStore endpoints are checked for length, count, and IPv6 address
formatting.
- Chart-managed tenant control-plane arguments are rejected when
supplied through `extraArgs`.
- **Documentation**
  - Updated Kubernetes OIDC and certificate issuer guidance.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-24 22:26:41 +04:00
Aleksei Sviridkin 047c715775 chore: clean up stray console file and test fixtures
Drop a local kubeconfig that was committed by accident alongside the
console sources and ignore the filename so it cannot come back. Replace
environment-specific values in the vSphere migration fixtures with
synthetic ones: the tests only care about the shape of the inventory
record, not where it was captured.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 20:37:49 +03:00
Alexey Artamonov 70dcd9daed fix(backups): let a restored Postgres app converge its credentials
A restore sets bootstrap.enabled on the target app, which skips the
init-job, so the recovered roles keep the source's password hashes while
the target's credentials Secret holds generated ones and every
application login fails. Once recovery converges, the driver records
RecoveryConverged and then clears bootstrap.enabled, so the next
HelmRelease reconcile runs the init-job and applies the passwords. Ready
carries RestoreCompletedCredentialsPending and an Event says the
convergence is still pending. A clear that keeps failing is retried
under BootstrapDisablePending within a fixed grace window after
convergence and then fails the restore. bootstrap.enabled is serialized
without omitempty so the patch sends false instead of dropping the key.

Running the init-job on a restored cluster exposes its delete pass to a
catalog the spec does not describe: the driver sets spec.databases to
the Backup's snapshot, but recovery replays WAL past it, so a database
created after the Backup comes back still marked as managed and would
be dropped with its data. The first run after a restore, told apart by
the per-RestoreJob bootstrap.newServerName that the cluster has not yet
recorded, releases such a database and its roles from management
instead, and the run records the value once every pass has succeeded.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Alexey Artamonov 9f857f64e1 feat(mariadb)!: generate every user password, root included
users[].password is gone from the chart, its schema and the API type.
Every password, root included, is generated into <release>-credentials
and kept across reconciles by lookup, and a password left in values is
ignored. An upgrade that finds no credentials Secret fails instead of
minting new passwords, because mariadb-operator applies root only at
datadir bootstrap and would advertise a root it never set. The Secret
carries k8s.mariadb.com/watch so the operator applies a changed password
without waiting for its requeue interval.

A username or database name becomes a Secret key and a resource name, so
render-time guards reject a name outside the DNS-1123 rule or past its
limit: 80 for a username, as the User CRD allows, and 64 for a database,
as the server allows. Names that differ only in "_" and "-" collide on
one resource; that is rejected on install only, since an existing
release already applied the overwrite. A user in both admin and readonly
of a database keeps a single ALL grant under the unchanged Grant name,
because renaming a Grant makes the operator revoke the live privilege. A
role member past 128 characters, the server's account-name limit, and a
database name of 65-80 characters, which the Database CRD accepts but
the server does not, are rejected on install and on upgrade only when
their Grant or Database does not exist yet, so a release that already
applied one keeps upgrading. User CR names and keys are quoted so a
username that reads as a YAML scalar stays a string.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:10:57 +03:00
Alexey Artamonov 2b5877eff9 feat(postgres)!: generate user passwords and read them at runtime
users[].password is gone from the chart, its schema and the API type.
Every application-user password is generated into <release>-credentials
and kept across reconciles by lookup, so an upgraded release keeps the
password it already has and a password left in values is ignored. The
init-job mounts that Secret and applies each password with ALTER ROLE
and a psql variable, so no password is rendered into the init-script
Secret that Helm keeps in release history. While bootstrap is enabled
the Secret is marked credentials-pending, and NOTES say the generated
passwords apply once the init-job runs.

Names reach superuser SQL and the init-script shell, so render-time
guards reject a username, database or extension name outside
[-._a-zA-Z0-9] or past the length PostgreSQL silently truncates: 63
bytes for a role or extension, 54 for a database, whose <db>_readonly
role would otherwise truncate and fail the next run. A username over 63
bytes that the credentials Secret already carries still upgrades.
users.postgres, the CNPG superuser, is rejected in every bootstrap
state. Changing a database's key in values drops the database, so the
reject message and the README give the rename that keeps the data: a
platform administrator renames it and its roles as the CNPG superuser
before the key changes.

The init-job cannot hang its Helm hook: the readiness wait ends on
elapsed time below activeDeadlineSeconds and the Job never retries, so a
purged restore target fails with a diagnostic. The delete-users pass
reads managed users from pg_shdescription instead of parsing the
version-dependent \du+ layout, which on PostgreSQL 16 and later matched
nothing and left a removed user able to log in; it compares names as
CREATE ROLE stores them and reassigns a removed user's objects in every
database before dropping it. A database without roles renders, and
extension names are quoted for hyphenated names and folded to lowercase
so they resolve as they did unquoted. A value that carries its own
surrounding quotes, the only way a hyphenated extension rendered before,
has that pair removed and keeps its case, so such a release still
upgrades. The README reads a password with jq, since jsonpath splits a
dotted key.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:10:57 +03:00
Myasnikov Daniil 9faa18a239 test(hack): detect BATS suites without a runner
Nested monitoring suites remained unexecuted after their runner was
removed. Audit both runner inventories so missing invocations and stale
suite paths fail in the unit lane. Make-variable arguments must be
reported even when their values hide the file suffix.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-24 20:00:27 +05:00
Aleksei Sviridkin 7a0aab2bdc refactor: apply the Go 1.27 modernizers
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.

Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
Aleksei Sviridkin db6c20f841 chore(deps): raise Go modules to their latest compatible releases
Several pinned modules carry published advisories: golang.org/x/crypto,
net, text and mod, grpc, the opentelemetry SDK and exporters, velero,
the etcd client and cel-go. Rather than take only the minimum fixed
versions, every module outside the Kubernetes lockstep set moves to
its newest release whose own requirements keep k8s.io/* and
controller-runtime where they are and fit Go 1.27.1. Where the newest
release asks for more, the module stops at the last one that does not:
cert-manager v1.19.6, source-controller/api v1.8.5, source-watcher/api
v2.1.1, fluxcd meta v1.26.0; helm-controller/api, fluxcd kustomize,
gateway-api and external-snapshotter stay put.

A few modules stop below their newest release on purpose. cel-go and
the etcd modules stay on the lines k8s.io/apiserver v0.35 was built
and tested with (cel-go v0.29.2, the version Kubernetes itself ships,
and etcd v3.6.15), since the apiserver is their only consumer here.
grpc stays on v1.83.2: v1.84.0 predates the fix for GO-2026-6443, a
server panic the CSI driver's gRPC server can reach, and no v1.84
patch release carries it yet. kubevirt.io/api and the CDI API stay on
the versions upstream csi-driver builds against: from CDI v1.60
StorageSpec.Resources changes type and upstream no longer compiles.

velero v1.18 adds RestoreSpec.ResourcePolicy, which the strategy CRD
embeds, so the CRD is regenerated. The backup controller only consumes
the velero v1 types; the new Queued and ReadyToStart backup phases
fall through to its requeue path like any other in-progress phase.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
Aleksei Sviridkin ec7387109e build(go): move the toolchain and builder images to Go 1.27.1
Go 1.26.4 carries standard-library advisories in net/http, net/url,
crypto/tls, encoding/asn1 and html/template that govulncheck reports
as reachable from the first-party binaries. The shipped stdlib comes
from the builder image, so the builder tags move together with the
go directive; the digests are the multi-arch indexes that golang:1.27,
1.27-alpine and 1.27-bookworm resolve to at 1.27.1.

api/apps/v1alpha1 keeps go 1.26.4. It is a published types-only
module, no binary here links its stdlib, and raising its directive
would force every consumer onto Go 1.27.

linstor-csi v1.11.2 pins grpc v1.81.0, which no longer builds on
Go 1.27: golang.org/x/net drops http2.TrailerPrefix there. Its image
now raises grpc to v1.83.2 before building, as the kamaji image does
for its advisories; the upstream unit tests pass on it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
renovate[bot] a7dfbe6f8c chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:04 +03:00
Aleksei Sviridkin 73ac282562 chore(deps): advance kubevirt csi-driver to upstream f5de10d
The previous pin was the pre-merge head of the "check VMI status when
detaching" change; upstream main carries the same change with
identical file contents, so moving onto main keeps it. On top of it
come NodeStage rescanning udev, Unpublish using CleanupMountPoint as
other CSI drivers do, and a runc security bump.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:03 +03:00
Aleksei Sviridkin 2be5d4625c test(kubernetes): cover the kubevirt CSI driver wrapper
Only the rate-limit flags were tested. Pin what a CSI spec or client
bump could change under the wrapper: RPCs the driver does not serve
answer Unimplemented, the RWX DataVolume carries the requested size
under spec.storage.resources.requests, and the NFS export, access-mode
and endpoint-selector helpers.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:03 +03:00
Aleksei Sviridkin 09a658d43b test(dashboard): pin JWT verification in the token proxy
Only the cold-start path of verifyAndParseJWT was tested; every other
test served an empty JWKS, so no token was ever verified and a change
that weakened verification would have passed. Cover a valid token and
the ones that must be refused: expired, not yet valid, signed by
another key, unknown or missing kid, alg none, and HS256 keyed with
the RSA public key.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:02 +03:00
Aleksei Sviridkin e66f134564 fix(e2e-sandbox): install mc from its GitHub release
dl.min.io now answers 410 Gone for every community mc build, since
MinIO archived the project, so the sandbox image no longer builds.
The final mc release is still published as a GitHub release asset.
Pin it and verify the per-arch checksum, because the old install
trusted whatever the URL served.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:02 +03:00
Aleksei Sviridkin 3800f63513 fix(ingress): trust CF-Connecting-IP only from Cloudflare ranges
cloudflareProxy passed the Cloudflare ranges to the controller under
set_real_ip_from, which is an nginx directive and not an ingress-nginx
ConfigMap key, so the controller dropped it without a warning. With
use-forwarded-headers on, the realip module then trusted every source
from the default proxy-real-ip-cidr of 0.0.0.0/0, and any client that
reached the load balancer directly could set CF-Connecting-IP to an
address of its choice. That address is what whitelist-source-range
checks, so the tenant whitelist, and any per-Ingress one, could be
walked past with one header.

proxy-real-ip-cidr is the key the controller reads for the trusted
ranges.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:04:27 +03:00
Aleksei Sviridkin a92e89e969 docs(kubernetes): describe the OIDC UID header from v1.33
The tenant OIDC guide, the version field description and the e2e
helper comment still described a --feature-gates term rendered on
v1.32. Left as they were, they would tell an operator that the chart
switches that alpha gate on, which it no longer does. The guide now
says what a v1.32 tenant has to write itself to get the UID, and which
--emulated-version values fail the render, so an operator who meets
that refusal finds it explained. The breaking-changes note in the
README said the chart refuses the mixed gate spelling while OIDC is
on; with that refusal gone it would promise a render error that no
longer comes.

The version description returns to its earlier wording in values.yaml
and in the update-versions.sh template, because that template rewrites
the line on every version bump and would otherwise bring the old text
back. make generate carries it into the schema, README, API types and
ApplicationDefinition.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:10:53 +03:00
Aleksei Sviridkin 0078727620 fix(kubernetes): render the OIDC UID header only from v1.33
RemoteRequestHeaderUID is alpha and off on v1.32, so the chart had to
enable an alpha gate on v1.32 tenants with OIDC on to make
--requestheader-uid-headers start. From v1.33 the gate is beta and on
by default. No release has shipped the v1.32 path, so moving the floor
to an effective v1.33 now removes a behaviour no released chart had:
the chart renders no --feature-gates term of its own, and below the
floor it renders no UID flag and runs none of the UID checks, as it
already did on v1.31. The refusal of a list that ends in a bare flag
still applies there, because it guards the --authentication-config the
chart appends on every version.

The refusal of a kube gate spelling mixed between bare and
kube:-prefixed terms goes too. Once the chart writes no gate term, the
spelling of the operator's terms no longer concerns anything it
renders, and the chart already passes other gate values the apiserver
refuses, unknown names and unparseable values, through to it.

--emulated-version is still read, because emulation moves the
effective version below the floor. Its kube version is now read from
the numbers at the start of the value, with the element split at its
first equals sign as toVersionMap splits it, so v1.32, 1.32.0 and
1.32-rc all read as 1.32. With OIDC on, an --emulated-version entry
without quotes whose kube version has a non-zero patch component fails
the render and names the entry: the apiserver refuses such a value and
would otherwise only crash-loop.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:10:53 +03:00
Myasnikov Daniil 16d23c8a53 test(monitoring): preserve OIDC coverage without live BATS suites
Rendering the inner chart alone cannot catch a wrapper dropping tenant
OIDC settings. The audience mapper and System-mode reconcile flag also
need assertions before the unused live suites are removed.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-24 17:08:47 +05:00