Commit Graph
2787 Commits
Author SHA1 Message Date
Aleksei Sviridkin b2ae6b773a fix(tenant): fail fast on an empty tenant apex and correct gateway docs (#4538)
Closes #4536

## What this PR does

Two fixes to how tenant charts handle the tenant apex host
(`_namespace.host`), from an audit of what happens when a tenant's
`host` is changed or left empty.

**1. Fail fast on an empty tenant apex (was: silent malformed
hostname).**
Charts that publish a service compose their hostname as
`<sub>.<_namespace.host>` (for example `harbor.<apex>`, `s3.<apex>`,
`grafana.<apex>`). When the injected apex was empty and no per-app
`host` override was set, they emitted a malformed trailing-dot hostname
(`harbor.`) into Ingresses, HTTPRoutes and Certificate dnsNames with no
error. Only the gateway path guarded against this.

A new library helper `cozy-lib.ns-host-required` now sources the apex at
each hostname-emitting site and fails the render with a clear message
when the apex is empty and no override is set. It is scoped to the
injected state, so standalone renders (`helm template` / `helm lint`
with no `_namespace`, or an empty `_namespace` map), no-exposure
tenants, and conditionally emitted resources (external=false, the
kubernetes no-etcd beacon path) are unaffected. The two post-delete
cleanup hooks are left unguarded so an empty apex cannot block
uninstall.

Charts covered: harbor, postgres (external), qdrant (external),
kubernetes (control-plane endpoint), vpn, seaweedfs, bootbox, monitoring
(grafana, alerta, grafana OIDC client), and system/bucket (which also
gains the `cozy-lib` library it was missing, both the local subchart
symlink and its package-source registration).

**2. Correct the tenant `gateway` parameter documentation.**
The `gateway` description (values.yaml, values.schema.json, README, the
Go type, the tenant-rd CRD, and an inline comment in namespace.yaml)
claimed the chart auto-enables a Gateway for tenants whose apex is
derived from the parent (empty `host`) and leaves it off for custom-apex
tenants. The resolver `tenant.gatewayEffective` never inspects `host`:
an unset `gateway` key always means the tenant owns no Gateway and
inherits from the nearest ancestor that owns one. The unit tests already
pin this behavior. The description is rewritten to match and
regenerated.

**Behavior change.** A tenant with an empty injected apex and no `host`
override that enables one of the covered components now fails its render
(the HelmRelease reports NotReady) instead of deploying a broken
hostname. In practice the apex is always injected (the root tenant has
`root-host`, children inherit it), so this only trips a genuinely
misconfigured tenant.

Tests: a new `cozy-lib.ns-host-required` unit test, plus per-chart guard
tests for harbor, qdrant, kubernetes, postgres, vpn, bootbox, seaweedfs,
monitoring and system/bucket. All affected chart suites pass.

### Screenshots

Not applicable (no UI change).

### Downstream repositories

I walked the trigger map against the diff. The only touchpoint is the
tenant `README.md` gateway description, which the website release bot
regenerates automatically from `README.md` on a stable tag (no manual
follow-up). The terraform provider is not affected: this is a
description-only change to the existing `gateway` field, with no field,
default or enum change, and the tenant-rd CRD schema reaches the
provider as opaque JSON. Boxes left for a maintainer to confirm.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tenant): tenant charts now fail rendering with a clear error when a tenant's apex host is empty and no per-app host override is set, instead of silently emitting a malformed hostname; the tenant `gateway` parameter description is corrected to match actual behavior (an unset `gateway` never auto-enables a Gateway).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Host-dependent services now show a clear rendering error when neither
a tenant host nor an app host is configured, instead of producing an
invalid hostname.
* Tenants without `gateway: true` use the nearest ancestor’s Gateway,
whether their host is inherited or custom. Enable `gateway: true` for a
tenant’s own Gateway, IP, and certificate; this is required to make a
custom apex publicly reachable.
* **Documentation**
* Clarified Gateway inheritance behavior and when a tenant needs its own
Gateway.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 21:32:36 +03:00
Aleksei Sviridkin f301cb551b feat(api): add configurable resources and replicas for cozystack-engine components (#2956)
## What this PR does

Exposes standard Helm `resources` values for the three core workloads in
the `cozystack-engine` package, and makes `cozystack-controller`
replicas configurable (was hardcoded to `1`).

**Changes by component:**

| Component | `resources` | `replicas` |
|---|---|---|
| `cozystack-api` | ✅ NEW (`cozystackAPI.resources`) | already
configurable |
| `cozystack-controller` | ✅ NEW (`cozystackController.resources`) | ✅
NEW (`cozystackController.replicas`, default `1`) |
| `lineage-controller-webhook` | ✅ NEW
(`lineageControllerWebhook.resources`) | already configurable |

Each template uses `{{- with .Values.xxx.resources }}` so the
`resources:` block is only rendered when values are provided. Default
`resources: {}` produces no output, so resource requests/limits stay
opt-in.

**Leader election:** `cozystack-controller` now always runs with leader
election — the chart passes `--leader-elect` unconditionally and ships a
namespaced `Role`/`RoleBinding` for the Lease in `cozy-system` (where
controller-runtime creates it). Raising `cozystackController.replicas`
therefore adds standbys instead of a second active writer over the
reconcilers that patch HelmReleases, Gateways, Certificates,
ResourceQuotas and Secrets; this holds whatever `replicas` is. A restart
can pause reconciliation for up to the Lease duration (15s by default),
because `LeaderElectionReleaseOnCancel` is off. With `replicas: 1` the
change for an existing install is the flag plus the two RBAC objects.

**Why:** Without these values, operators cannot set QoS guarantees and
must resort to `kubectl patch`, Flux Kustomize SSA patches, or Kyverno
mutations — all fragile under Helm upgrades.

Closes #2955

### Release note

```release-note
feat(api): add configurable `resources` (requests/limits) for cozystack-api, cozystack-controller, and lineage-controller-webhook.
`cozystack-controller` replicas is now configurable via `cozystackController.replicas` (default: 1).
The `cozystack-controller` Deployment now always runs with leader election and creates a Lease in `cozy-system`, so raising `replicas` adds standbys rather than a second active controller; a restart can pause reconciliation for up to the Lease duration (15s by default).
An empty `resources: {}` renders no resources block, so requests and limits remain opt-in.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Container resource requests and limits can now be configured for the
API, controller, and webhook services; defaults remain unchanged.
* The controller replica count is configurable, and leader election is
enabled for multi-replica operation.
* Controller leader-election access is limited to the `cozy-system`
namespace.

* **Documentation**
  * Documented the webhook resource configuration option.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 20:56:43 +03:00
Aleksei Sviridkin b75507bbf1 test(velero): stop asserting the KubeVirt plugin image string
The plugin image is now built in-tree and stamped by digest, so any
assertion on its string only restates the pin and breaks on every
build or bump without testing what the chart does.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 20:04:11 +03:00
Aleksei Sviridkin 5a619c5fdc Merge remote-tracking branch 'origin/feat/velero-kubevirt-plugin-multiarch' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>

# Conflicts:
#	Makefile
2026-09-28 19:57:27 +03:00
Aleksei Sviridkin 8c1266649d Merge remote-tracking branch 'origin/feat/harbor-multiarch' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 19:49:55 +03:00
Aleksei Sviridkin 79854b16fb Merge remote-tracking branch 'origin/fix/build-list-orphan-images' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 19:49:54 +03:00
Aleksei Sviridkin 6d759cc73e feat(build): build package images for amd64 and arm64 (#4498)
## What this PR does

Package images can now be built for amd64 and arm64 in one `make image`,
and a published build is multi-arch unless `PLATFORM` says otherwise.
This is the build half of arm64 support from #1961 and #903. CI still
publishes amd64 only until it gets a native arm64 leg, which comes in a
separate PR.

Builder stages now run on the build platform and compile for the target
one, so Go never runs under emulation. Under emulation it is slow and it
crashes: the migration-controller Dockerfile already records a "bad
pointer in Go heap" from exactly that. On top of this, `PLATFORM`
defaults to `linux/amd64,linux/arm64`. A build on an arm64 workstation
can't publish an arm64-only digest any more. That happened once already,
the cilium pin was arm64-only until 3f36a1b45 rebuilt it by hand.

Two images were only right because they were emulated. On a native
builder they would ship the wrong architecture. token-proxy ran `go
build` with no GOARCH, and kube-ovn calls upstream `make build-go`,
which hardcodes `GOARCH=amd64`. Both are fixed now. The new bats test
wants every compiling stage on `$BUILDPLATFORM` and actually using
`TARGETARCH`. A bare `ARG TARGETARCH` does not count, this is how
token-proxy got through.

Some things stay amd64. talos and testing pin it with `override`,
because matchbox serves amd64 Talos assets and the sandbox runs
`qemu-system-x86_64`. `LOAD=1` builds for the host only, since the
classic docker image store can't load an index. Every CI build job pins
`PLATFORM: linux/amd64`, so CI output does not change.

This changes local builds. A `make image` on the default docker driver
with the classic image store now fails with "Multi-platform build is not
supported for the docker driver", where before it built for the host.
Use a `docker-container` builder (`BUILDER=<name>`), or pass
`PLATFORM=linux/amd64` (or `LOAD=1`) to build one architecture.

I checked it on an arm64 host with a docker-container builder.
cozystack-api, dashboard, kubeovn, platform-migrations, cilium and
capi-providers-cpprovider build and push as amd64+arm64 indexes.
Binaries pulled from each platform are x86-64 and aarch64:
cozystack-api, token-proxy, the kamaji provider manager, cilium-agent
and all four kube-ovn binaries. piraeus-server builds for both
platforms, but the push to my test registry failed on registry 500s. The
capi-providers-cpprovider inline cache works with a multi-platform push.
Not covered is running the arm64 images on an arm64 cluster, CI has
none.

This is stacked on #4485 and targets its branch, so the diff here is
only the two commits on top.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#711
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(build): package images build for linux/amd64 and linux/arm64 by default, with Go builders cross-compiling on the build platform; a local `make image` now needs a docker-container builder, or PLATFORM=linux/amd64 / LOAD=1 for a single architecture. CI still publishes amd64.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Build Improvements**
* Container image builds now target both amd64 and arm64 by default when
building without loading images locally.
* CI and release builds are pinned to amd64; Talos and testing images
remain amd64-only.
* Cross-platform builds now compile on the builder’s native platform
while targeting the requested architecture.
* **Documentation**
* Clarified platform defaults and architecture requirements for local
and CI builds.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:43:47 +03:00
Aleksei Sviridkin 597d966d7d fix(redis): run the multi-arch redis_exporter so the sidecar starts on arm64 (#4522)
## What this PR does

Redis and Valkey instances, and the Redis inside Harbor, never become
fully ready on arm64 nodes. Their failovers pin
`oliver006/redis_exporter:v1.55.0-alpine`, which upstream publishes for
amd64 only. On arm64 the exporter sidecar exits with `exec
/redis_exporter: exec format error`, and the pod stays not ready.

The same release without the suffix is a multi-arch index with amd64,
arm and arm64 builds. Its entrypoint, user and port match the `-alpine`
image, and none of these charts run a shell in the exporter container.
So this swaps the tag and pins it by digest, with no version bump.

I checked both tags with `crane`. On an arm64 test cluster a Redis and a
Valkey instance hit the exec format error before the change and came up
2/2 with it. The Harbor Redis pods went to 2/2 the same way.

On upgrade the exporter image in the pod template changes, so the
operator rolls every Redis and Valkey instance and Harbor's Redis, one
pod at a time as it does for any failover update.

Fixes #4519

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes one image reference in three templates. No entry in the
trigger map covers a sidecar image.

### Release note

```release-note
fix(redis): Redis, Valkey and Harbor's Redis now use the multi-arch redis_exporter image, so the exporter sidecar starts on arm64 nodes.
```
2026-09-28 19:36:36 +03:00
Aleksei Sviridkin 17838d46dc fix(ingress): run the default backend on arm64 (#4517)
## What this PR does

The ingress default backend can't start on arm64 nodes: it crash-loops
with `exec /server: exec format error`. `packages/system/ingress-nginx`
turns the default backend on without setting an image, so it gets the
chart default `registry.k8s.io/defaultbackend-amd64:1.5`. That image is
amd64 only, and upstream never published a multi-arch defaultbackend.

This pins `registry.k8s.io/ingress-nginx/custom-error-pages:v1.2.9` by
its multi-arch index digest. Upstream ships it as the replacement
default backend, with amd64, arm and arm64 builds. It answers 404 on
:8080 and 200 on `/healthz`, so the chart's port and probes still fit.
The host ingress, the tenant ingresses and the tenant Kubernetes
ingress-nginx addon all render this package without their own image, so
this one change covers them.

I ran the arm64 image locally under the chart's securityContext (uid
65534, read-only root filesystem, all capabilities dropped). `/healthz`
returned 200, `/` returned 404, and memory stayed around 6Mi of the 20Mi
limit. The same override runs on an arm64 test cluster. A new
helm-unittest case checks the default backend image by digest; it fails
on main and passes here.

There are two side effects. custom-error-pages logs a line for every
request, because it looks for error page templates under `/www` that we
don't ship and falls back to a plain `404 page not found`. The old
backend didn't log, and its body was `default backend - 404`. The status
code is still 404.

Also, Renovate won't update this pin. The `helm-values` manager is
disabled in `.github/renovate.json`, and no regex manager matches the
registry/image/tag/digest layout. The controller image in the same file
has the same problem, so I kept Renovate config out of this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `packages/system/ingress-nginx/values.yaml` and a
new test in that package. No entry in the trigger map covers a system
package's image values.

### Release note

```release-note
fix(ingress): the ingress default backend now uses the multi-arch custom-error-pages image instead of the amd64-only defaultbackend image, so it starts on arm64 nodes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Pinned the default error-page image to a specific version and digest,
improving consistency across deployments.
  * Added a chart test to verify the configured image.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:36:25 +03:00
Aleksei Sviridkin aa59c2f1d5 fix(foundationdb-operator)!: drop the amd64-only 7.1 init container (#4512)
## What this PR does

The FoundationDB operator never starts on arm64. Its
`foundationdb-kubernetes-init-7-1` init container runs
`foundationdb/fdb-kubernetes-monitor:7.1.67` and fails with `exec format
error`. All 67 published 7.1.x tags of that image are single amd64
manifests, so there is no multi-arch 7.1 tag to move to. The 7.3.63 and
7.4.1 tags, and the operator image itself, publish linux/arm64. Upstream
hits the same gap in its own build: the operator Dockerfile at v2.30.0
swaps its built-in 7.1 client for 7.3 on arm64.

The fix removes the 7.1 entry when the chart is vendored, instead of
overriding it in the package values. The `update` recipe deletes it with
`yq`, and the committed chart equals what the recipe produces. A `null`
override would not last. Helm 4.2 and later ignore a `null` that a
parent chart sets on a subchart key when no user-supplied values reach
that subchart (helm/helm#32132). The platform passes this package no
values, so that is exactly the case here. helm-unittest renders with
Helm 3, so the override would silently stop working after a Flux bump.
`helm template` now renders only the 7.3 and 7.4 init containers under
both Helm 4.1.1 (what helm-controller v1.5.0 uses) and Helm 4.3.0. The
test pins exactly two init containers, so it fails if a later `make
update` brings 7.1 back.

This is a breaking change. The operator loses the 7.1 `fdbcli`,
`fdbbackup`, `fdbrestore` and client library, so it can no longer manage
or upgrade a FoundationDB cluster pinned to 7.1.x. The app defaults to
7.3.63 and nothing in this repository sets 7.1, but `cluster.version` is
a free string. A user who set 7.1.x has to upgrade that cluster to 7.3
before installing this release.

Related: #4511, the app still accepts any string as the FoundationDB
version, 7.1 included.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches one system package: its update recipe, its vendored
chart values, and its test. No app schema, platform value, CRD or shared
tooling changes, and a search of the cozystack org finds no page or code
that offers FoundationDB 7.1.

### Release note

```release-note
fix(foundationdb-operator)!: the FoundationDB operator now starts on arm64. It no longer ships the 7.1 client, because no arm64 build of it exists, so it cannot manage or upgrade a FoundationDB cluster on 7.1.x. Upgrade such clusters to 7.3 before installing this release.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* The FoundationDB operator chart no longer includes the 7.1
init-container configuration. Its remaining configured init-container
versions are 7.3 and 7.4.
* The chart update process applies the same version change to copied
chart values.
* Image validation now checks for exactly two init containers, using
versions 7.3.63 and 7.4.1.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:36:19 +03:00
Aleksei Sviridkin 2e8219fb92 fix(build): pass the shared buildx flags from every image target (#4485)
## What this PR does

Five image builds skipped the shared buildx flags in
`hack/common-envs.mk` and wrote their own: flux-plunger,
kubeovn-plunger, both dashboard images, and metallb. The plungers
ignored `PLATFORM`, so on an arm64 workstation they came out as arm64
images that fail on amd64 nodes with `exec format error`. CI builds on
amd64, so it never showed. Dashboard and metallb hardcoded
`--platform=linux/amd64`; for metallb that sat next to `$(BUILDX_ARGS)`,
so setting `PLATFORM` gave buildx two platforms. All of them now build
through `$(BUILDX_ARGS)` only.

How it split: kubeovn-plunger landed two days before 9f2b98d36 moved the
other Makefiles onto `BUILDX_ARGS`, flux-plunger was copied from it, and
dashboard kept its own flags when that change touched it.

CI builds do not change. CI sets only `BUILDER`, which these targets
already honoured, and every build runner is x86-64.

One behaviour change is intentional: a local `make image` for dashboard
or metallb now follows the host architecture, like every other package.
On an arm64 machine, build for an amd64 cluster with
`PLATFORM=linux/amd64`.

A new bats test dry-runs every package image target with a probe
platform and a marker in `BUILDX_EXTRA_ARGS`. Each buildx call must
carry the marker and exactly that platform. On main it flags the five
targets above.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(build): every package image build honours PLATFORM, SBOM and BUILDX_EXTRA_ARGS; local dashboard and metallb builds now follow the host architecture unless PLATFORM is set.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* System image builds now use shared Buildx settings for platform
selection and publishing options, while retaining their image tags,
cache settings, and metadata.
* MetalLB image builds no longer explicitly target `linux/amd64`; the
selected platform follows Buildx defaults and configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:36:07 +03:00
Aleksei Sviridkin 9c94337f17 fix(mariadb): correct webhook cert-manager override key path (#4537)
## What this PR does

Fixes cozystack/cozystack#3974: the values override at
`packages/system/mariadb-operator/values.yaml` targeted
`webhook.certificate.certManager`, a field that does not exist in the
vendored mariadb-operator chart. The chart's actual key is
`webhook.cert.certManager.enabled` (see
`charts/mariadb-operator/values.yaml:139-142`), so the override was
silently ignored and the webhook always fell back to mariadb-operator's
own cert-controller instead of cert-manager. This corrects the key path
so the override takes effect.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(mariadb): correct webhook cert-manager override key path so cert-manager is actually used for the mariadb-operator webhook certificate
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the MariaDB operator’s certificate-management configuration to
use its current setting structure. The certificate manager remains
enabled, now under the updated configuration path. Installations that
use the operator will apply the enabled setting from the revised values
configuration.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 18:53:51 +03:00
Aleksei Sviridkin 26e4083d52 fix(linstor): adopt existing CR on 409 AlreadyExists in k8s backend (#4532)
## What this PR does

This adds a LINSTOR server patch for a controller that could refuse
every write of one property until it was restarted, which left a PV
stuck in `Released` with its ZFS volume still allocated. It is a
backport of LINBIT/linstor-server#531.

linstor-csi starts every volume delete by setting
`BalanceResourcesEnabled` on the resource definition. With the
Kubernetes CRD backend that write can come back as 409 AlreadyExists
when the fabric8 client retries a POST the apiserver already persisted.
A single-object transaction has no rollback entry, so the controller
drops the key from memory while the CR stays, and the caching client
never re-reads it. Every retry of the delete then fails the same way.

The patch makes create treat a 409 as success when the object exists: it
takes the live resourceVersion and replaces the object with the intended
content. The unit test that comes with the patch was red before the fix
and green after, on v1.33.3 and on upstream master, and the whole patch
series still applies to v1.33.3.

The retry being the trigger is my best explanation, not proven: the run
had no apiserver logs. Until this ships, restarting linstor-controller
clears the state.

Closes #4531

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): the LINSTOR controller no longer gets stuck refusing writes to a property after a retried Kubernetes API request, which could leave a deleted volume in Released with its storage still allocated.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved recovery when Kubernetes reports that an object already
exists during creation. The system now adopts the existing object and
continues updating it, helping prevent volume operations from becoming
stuck and requiring a controller restart.
* Other conflict and error cases continue to report their original
errors.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 18:39:43 +03:00
IvanHunters 2fa51bf587 test(charts): isolate guard coverage for every guarded template
Add per-template guard suites so each guarded hostname site is exercised
on its own: harbor harbor.yaml and httproute.yaml, bootbox machines.yaml,
seaweedfs cosi-deployment.yaml, bucket deployment.yaml and httproute.yaml,
monitoring alerta.yaml, grafana httproute.yaml and the grafanaHost helper
via oidc-keycloak.yaml. Each suite renders only its target template, so
removing that file's guard reddens it. Also add a postgres positive case
that renders the Certificate with an apex set (external=true).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 18:32:49 +03:00
Aleksei Sviridkin fa6e3ce850 docs(kubernetes): clarify node pool Helm timeout semantics (#4496)
## What this PR does

Corrects the explanation for the Kubernetes node-pool Helm timeout.

With readiness waiting disabled, nothing in this chart's install or
upgrade is bounded by the 15-minute timeout: the ordinary
`talos-reconcile` Job continues asynchronously, and the chart has no
install or upgrade hooks. The timeout takes effect only if readiness
waiting is re-enabled or such a hook is added.

The annotations and their values are unchanged.

Fixes #4406

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified that with readiness waiting disabled, the 15-minute Helm
timeout does not limit chart installs or upgrades: reconciliation runs
asynchronously, and the chart has no install or upgrade hooks.
* Noted that the timeout applies if readiness waiting is re-enabled or a
hook is added, and that the release settles while reconciliation
continues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 17:55:57 +03:00
Aleksei Sviridkin ec9843be4a docs(virtual-machine): correct persistent EFI migration documentation (#4494)
## What this PR does

Removes the outdated warning that persistent EFI state on default RWO
storage makes a virtual machine node-pinned and prevents live migration.
The warning no longer reflects the supported migration behavior and
could discourage users from persisting Secure Boot keys.

This is a documentation-only change. The chart documentation, JSON
schema, generated API comment, and ApplicationDefinition schema were
regenerated without changing defaults, validation, or runtime behavior.

Fixes #4420

### Validation

- `make -C packages/apps/vm-instance generate`
- `make generate`
- `make manifests`
- `make -C packages/apps/vm-instance test` — 59 tests passed
- `git diff --check`
- A second complete generation pass produced no additional changes

### Screenshots

N/A — documentation-only change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated EFI persistence descriptions to remove claims that using
default RWO storage pins virtual machines to a node or prevents live
migration. The persistence behavior and UEFI-only applicability remain
unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 17:53:50 +03:00
Nikita Aboltin cb8728dde1 docs: note the measured idle usage next to the empty resources defaults
The resources keys stay empty so requests and limits are opt-in, and the comment
records the measured idle usage next to them, so a limit that does get set is not
placed below what the component needs at idle.

Signed-off-by: Nikita Aboltin <aboltin.nikita@rwb.ru>
2026-09-28 16:43:44 +03:00
IvanHunters 0f4d39f1ef test(charts): add empty-apex guard tests for the remaining guarded charts
Add host_guard_test.yaml for postgres, vpn, bootbox, seaweedfs,
monitoring and bucket. Each renders its guarded template with a
non-empty injected _namespace whose host is empty and no per-app host
override, and asserts the ns-host-required failure, plus a positive
render case with the apex set. Completes guard coverage alongside the
existing harbor, qdrant and kubernetes tests.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 16:01:31 +03:00
IvanHunters eedb446e8a docs(tenant): clarify that gateway=false behaves like unset
The rewritten gateway description claimed setting false opts out of
Gateway publishing entirely, including inheritance. It does not:
tenant.gatewayEffective returns "false" for both an explicit false and
an unset key, so namespace.yaml keeps _namespace.gateway pointing at the
nearest ancestor that owns a Gateway and the tenant still publishes
through it. State that false currently behaves the same as unset, and
fix the matching stale comment in gateway_default_test.yaml.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:52:46 +03:00
Nikita Aboltin 8732dc96cf feat(api): add configurable resources and replicas for cozystack-engine components
- cozystack-api, cozystack-controller, lineage-controller-webhook: conditional
  resources block via `{{- with .Values.*.resources }}`, defaulting to `{}` so the
  default render stays unchanged
- cozystack-controller: replicas comes from values (was hardcoded 1); the chart
  now passes --leader-elect unconditionally and grants the Lease verbs through a
  namespaced Role and RoleBinding in cozy-system (the Lease lives in the manager's
  own namespace), so raising replicas adds a standby instead of a second active
  writer over the reconcilers that patch HelmReleases, Gateways, Certificates,
  ResourceQuotas and Secrets

Signed-off-by: Nikita Aboltin <aboltin.nikita@rwb.ru>
2026-09-28 15:19:45 +03:00
IvanHunters 37f59369d3 fix(charts): fail fast on empty tenant apex in host composition
Charts that publish a service compose their hostname as
<sub>.<_namespace.host>. When the injected tenant apex is empty and no
per-app host override is set, they used to emit a malformed trailing-dot
hostname (e.g. "harbor.") into Ingresses, HTTPRoutes and Certificate
dnsNames silently. Source the apex through cozy-lib.ns-host-required at
each hostname-emitting site so the render fails with a clear message
instead, matching the gateway path.

Covered: harbor, postgres (external), qdrant (external), kubernetes
(control-plane endpoint), vpn, seaweedfs, bootbox, monitoring
(grafana, alerta, oidc client), and system/bucket (adds the cozy-lib
subchart symlink it lacked). The guard is placed at the point each
resource is actually emitted, so standalone renders, no-exposure
tenants and the kubernetes no-etcd beacon path are unaffected. The
post-delete cleanup hooks are left unguarded so an empty apex cannot
block uninstall.

Behavior change: a tenant with an empty injected apex and no host
override that enables one of these components now fails its render
(HelmRelease NotReady) instead of deploying a broken hostname. In
practice the apex is always injected (root has root-host, children
inherit it), so this only trips a genuinely misconfigured tenant.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:14:45 +03:00
chathurabuddi 44a06486a9 fix(mariadb): correct webhook cert-manager override key path
The chart schema is webhook.cert.certManager.enabled, not
webhook.certificate.certManager, so the override was silently
ignored and the webhook fell back to mariadb-operator's own
cert-controller instead of cert-manager.

Assisted-by: LLM
Signed-off-by: chathurabuddi <10394110+chathurabuddi@users.noreply.github.com>
2026-09-28 17:41:20 +05:30
IvanHunters 784d290c42 docs(tenant): correct gateway param description to match behavior
The gateway parameter description claimed the chart auto-enables a
Gateway for tenants whose apex is derived from the parent (empty host)
and leaves it off for custom-apex tenants. The resolver
tenant.gatewayEffective never inspects host: an unset gateway key
always means the tenant owns no Gateway and publishes through the
nearest ancestor that owns one, regardless of host. The unit tests in
gateway_default_test.yaml already pin this real behavior.

Rewrite the description to match the code and regenerate the schema,
README, Go types and the tenant-rd CRD.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 14:07:51 +03:00
Aleksei Sviridkin 11991b9d05 feat(harbor): build the Harbor images for amd64 and arm64
Upstream publishes the Harbor v2.15 component images for amd64 only,
so every Harbor pod crash-loops with "exec format error" on an arm64
node. The multi-arch build upstream exists on main but is in no
release yet, so the package now rebuilds the pinned v2.15.1 release
itself: the Go components cross-compile on the build platform, the
registry and the Trivy adapter build from the tags upstream's Makefile
pins, the Trivy binary is fetched per architecture against its release
checksum, and the runtime images keep upstream's photon base, users
and entrypoints. The build fails when those registry and Trivy pins,
the swagger generator or the golang and node build images no longer
match upstream's Makefile at the pinned release, so a version bump
cannot pair a new core with an old registry, scanner or toolchain.

The portal compiles to static JS, so it lives in its own Dockerfile
listed as architecture-neutral in the cross-compile check, and takes
the release pin from the main one so both files build the same
commit.

Because values.yaml now pins the images, `make update` pins the chart
release that ships the same Harbor version and fails when its
appVersion differs, instead of pulling the newest chart onto old
images. Renovate stops proposing tag bumps for the rebuild's build
bases, which follow upstream's toolchain for that release.

The committed refs stay on the upstream digests until CI stamps the
rebuilt ones.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 02:11:46 +03:00
Aleksei Sviridkin f02963a2c7 fix(keycloak-operator): keep the image host inside repository
The image target stamped only `tag`, leaving the committed
`registry: ghcr.io` in place, so a build against the CI registry would
render the public host with a digest that exists only in the build
registry. Stamping the build host into `registry` instead would split
the source registry across two keys, where the nightly mirror's
substring host rewrite cannot reach it and the published tree would
keep pointing at the private registry.

Set `registry: ""` and keep the whole host in `repository`, stamped
by the image target. The upstream chart joins the two keys, so the
rendered image string is unchanged, and the ref now has the same
shape as every other stamped reference the mirror and promotion
already handle.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 00:57:24 +03:00
Aleksei Sviridkin 11ba78a2b9 fix(linstor): adopt existing CR on 409 AlreadyExists in k8s backend
The fabric8 client retries a POST after a 5xx or a dropped
connection, so a create that the apiserver already persisted can come
back as 409 AlreadyExists. A single-object transaction has no
rollback entry, so the controller drops the key from memory while the
CR stays in the apiserver, and the caching client never re-reads it:
every later write of that key fails the same way until the controller
restarts.

This left a CDI scratch PV stuck in Released, because linstor-csi
starts every volume delete by setting BalanceResourcesEnabled on the
resource definition and that write never succeeded.

Treat 409 on create as success when the object exists: take its
resourceVersion and replace it with the intended content.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 17:50:59 +03:00
frolov.so 15fee54558 docs(kubernetes): clarify node pool Helm timeout semantics
Readiness waiting is disabled for node-pool installs and upgrades, so
the ordinary reconciliation Job runs asynchronously and the configured
timeout is inert. Saying otherwise obscures the actual failure boundary.

Fixes cozystack/cozystack#4406

Assisted-by: LLM
Signed-off-by: frolov.so <opsmon.sh@yandex.com>
2026-09-27 17:18:01 +03:00
Aleksei Sviridkin dae5d5c5ae fix(build): name the repository of the keda and kuberture pins
Both packages pinned their images by writing tag@digest into the tag
field alone and left the repository to the vendored chart's defaults.
Tooling that reads refs from the package values saw a bare
"2.20.2@sha256:..." with no image to resolve, so the release gate
could not tell whether these images serve arm64. The values now
repeat the chart's registry and repository beside the pin; the
rendered manifests are unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 07:57:31 +03:00
Aleksei Sviridkin 769d3b3897 fix(build): push the kamaji provider image under the build's IMAGE_TAG
The capi-providers-cpprovider Makefile assigned IMAGE_TAG itself, which
beats the value CI puts in the environment. Every PR, main and release
line build therefore pushed the same component-version tag and
overwrote each other's, and its second tag was built from an undefined
TAG variable and ended in a bare "-".

The image now goes through image-tags like every other package: the
build-unique IMAGE_TAG always, the component version only under
PUBLISH_VERSIONED=1. The stamped refs keep the component version as
their tag, so the promote, retag and mirror tooling see the same refs
as before.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 07:57:31 +03:00
Aleksei Sviridkin 07700c74ce fix(redis): run the multi-arch redis_exporter so the sidecar starts on arm64
The redis, valkey and harbor Redis failovers pin
oliver006/redis_exporter:v1.55.0-alpine, which upstream publishes for
amd64 only. On an arm64 node the exporter sidecar exits with an exec
format error, so the Redis pod never becomes fully ready.

The unsuffixed v1.55.0 tag is a multi-arch index (amd64, arm, arm64)
of the same exporter release, with the same entrypoint, user and
port, and none of these charts run a shell in that container. Pin it
by digest, so the reference is reproducible.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 03:04:44 +03:00
Aleksei Sviridkin d4dfae656f fix(ingress): run the default backend on arm64
The ingress-nginx chart defaults the default backend to
registry.k8s.io/defaultbackend-amd64, which ships no arm64 build and
crash-loops there with "exec format error". Every ingress controller
rendered from this package, the host and tenant ingresses and the
tenant Kubernetes addon, enables that backend without naming an image.

Pin the upstream multi-arch replacement, custom-error-pages, by its
index digest. It answers 404 on :8080 and serves /healthz, so the
chart's port, probes and restricted securityContext apply unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 22:32:06 +03:00
Aleksei Sviridkin 9eb3a64dd1 fix(foundationdb-operator)!: drop the amd64-only 7.1 init container
Every foundationdb/fdb-kubernetes-monitor 7.1.x tag is a single amd64
image, so on arm64 the operator's 7.1 init container fails with exec
format error and the operator pod never starts. No multi-arch 7.1
monitor exists to bump to, and upstream already swaps its own 7.1
client for 7.3 when building the operator for arm64.

The vendoring recipe now deletes the 7.1 entry from the upstream chart
values, and the committed chart matches its output. A null override in
the package values was rejected: Helm 4.2 and later ignore a null the
parent chart sets for a subchart key when no user-supplied values reach
that subchart (helm/helm#32132), and the platform passes this package
none. helm-unittest renders with Helm 3, so the override would stop
working on a Flux bump without any test noticing.

The app defaults to 7.3.63, which keeps its init container, as does
7.4.

BREAKING CHANGE: the operator no longer carries the 7.1 fdbcli and
client library, so it can neither manage nor upgrade a FoundationDB
cluster pinned to a 7.1.x version. Upgrade such clusters to 7.3 before
installing this release.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 20:30:50 +03:00
Aleksei Sviridkin 72ec1189fd feat(velero): build the KubeVirt plugin image for amd64 and arm64
Upstream publishes quay.io/kubevirt/kubevirt-velero-plugin for amd64
only, so on an arm64 cluster the velero server never starts: its
plugin init container fails with "exec format error". Rebuild the
plugin in-tree from the v0.9.0 release tag, cross-compiled on the
build platform, into an image that behaves like upstream's: it copies
the plugin binary into the plugins volume at /target.

The image target stamps the built reference by digest into
values.yaml, and the package joins the root build list so every build
path refreshes that pin. Until the first stamp is committed the
upstream reference stays in place, which the chart test accepts
alongside the digest-pinned in-tree one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 20:30:40 +03:00
Aleksei Sviridkin 101d1d9861 chore(seaweedfs): bump chart to 4.45 (#4079)
## What this PR does

Refreshes the vendored SeaweedFS chart from 4.31 to 4.45 through `make
update`. This supersedes #2918, which stopped at 4.34: both things that
bump was after (SIGHUP hot-reload of the JWT signing keys, and native
`expires_after_seconds` under
`global.seaweedfs.securityConfig.jwtSigning`) are in 4.45 too, along
with three months of chart fixes.

What changes in a rendered cozystack install:

- Every SeaweedFS server moves from `chrislusf/seaweedfs:4.31` to
`4.45`. The chart takes its image tag from `appVersion`, so the
`master`, `volume` and `filer` StatefulSets and the `s3` Deployment all
get a new image and roll. None of them sets an `updateStrategy`, so they
fall back to RollingUpdate and restart one pod at a time, in every
tenant that runs SeaweedFS. Selectors, `serviceName` and
`volumeClaimTemplates` are byte-identical across the bump, so no
immutable field is touched and no volume is recreated.
- The mysql Secret and the `WEED_MYSQL_*` env block now render only for
the mysql filer store, so the dummy `seaweedfs-db-secret` and its env
vars drop out of a postgres2 install. That Secret is a `pre-install`
hook resource and Helm never deletes hook resources on upgrade, so an
existing one stays behind unused; the tenant cleanup hook still removes
it when the tenant goes away.
- The filer ServiceMonitor selects on a `monitoring: "true"` label that
upstream moved from the filer-client Service onto the filer Service, so
metrics come from one Service instead of two. See the upgrade note
below.
- The filer db-init ConfigMap volume is marked optional.
- S3 gained a Lance catalog port (9101), rendered on the deployment and
published on the Service. This package pins it off, see below.
- The new per-component NetworkPolicy templates are opt-in and off by
default.

The rendered s3 port set is therefore unchanged from 4.31: `swfs-s3` and
`metrics`, same as before the bump.

## Catalog listeners

`weed s3` ships two table-catalog listeners that default to on: Lance on
9101, new in this bump and published on the s3 Service, and Iceberg on
8181, which predates it and stays on the pod IP. Both are served over
plain HTTP with no way to supply a certificate, while every other port
on this gateway is TLS by way of `s3-tls-main-port.patch`, and both
expose writable namespace and table routes. Nothing in cozystack
consumes either.

- Lance is pinned off in `values.yaml`, which drops the container port
and the Service port with it.
- Iceberg could not be pinned off at all: the chart wraps
`-port.iceberg` in an `if` and the value defaults to null, so the flag
was dropped and weed served its own 8181. A new carried patch renders
the flag the way 4.45 already renders `-port.lance`, so null reaches
weed as 0. An explicitly configured port still wins in both cases.

## Volume resize hook

The hook decides twice: one gate emits the `--cascade=orphan` delete of
the volume StatefulSet, the other emits the PVC patches. Both were
comparing sizes as strings, in different ways, and the two disagreed
about what a size change is.

The PVC gate reached its operands through an `include`, which returns a
string, so byte counts were compared lexicographically. A grow that
crossed a decimal exponent and lowered the leading digit (5Gi to 10Gi,
8Gi to 16Gi, 20Gi to 100Gi) read as a shrink: the patches were skipped
while the delete above them still ran, leaving the StatefulSet recreated
and the claims at the old size, hook exit 0 and nothing logged. The same
comparison read a real shrink across an exponent as a grow and emitted a
patch the API server rejects, failing the pre-upgrade hook after the
delete had landed.

The delete gate compared the live `volumeClaimTemplates` size against
the desired one raw. Since the API server re-emits a quantity in
canonical form, that never converges for a non-canonical spelling:
values saying `10240Mi` come back from the cluster as `10Gi`, so the
volume StatefulSet is deleted and recreated on every upgrade for a size
that never moved, and no patch follows because the numeric gate
correctly sees no growth.

A third defect sits in the patch command itself, which formatted the
desired size with `%s`. Helm decodes values through JSON, so a size
written without quotes arrives as a float64 and the rendered command
carried a Go format error where the quantity belongs. The hook fails on
it, and it fails after the orphan delete has already run. It is `%v`
now: `21474836480` renders as `2.147483648e+10`, which
`resource.ParseQuantity` accepts and the API server stores as
`21474836480`. The tenant app always quotes `size`, so no tenant could
reach this.

Carried as one patch normalising both gates through
`seaweedfs.resource-quantity`, with unit tests on the exponent-crossing
grow, a grow that always worked, a shrink, a respelling, and a size set
as a number.

## Upgrade notes

Two things to know before upgrading.

`filer.ingress` is gone from the chart's values. Upstream replaced it
with `filer.ingresses.http.*` and added `filer.ingresses.grpc.*`; those
six removed keys are the only contract break in the whole values file
across the bump. Nothing in tree sets the old key and the tenant-facing
SeaweedFS resource has no filer-ingress field, so a default install is
unaffected. An operator who set `filer.ingress.enabled: true` through a
platform-level `spec.components.seaweedfs-system.values` override is
not: Helm accepts the unknown key without complaint and the Ingress
simply stops rendering, with nothing in an event or a log to read.

Filer metrics stop being double-counted. Before this bump both
`seaweedfs-filer` and `seaweedfs-filer-client` matched the filer
ServiceMonitor, so Prometheus scraped two targets per filer pod and
every filer series was duplicated. One target is the correct number, but
panels built on `rate(SeaweedFS_filer_request_total{...})` and
`rate(SeaweedFS_filerStore_request_total{...})` in the shipped dashboard
step down by half at upgrade time. That step is the duplicate going
away, not a throughput loss. Quantile panels are ratios and are
unaffected.

## Patches

Ten carried patches, all applying cleanly from a fresh download.
`resize-api-server-annotation.diff` needed a rebase: upstream now
renders pod-template labels on the resize hook, so the patch appends the
cozystack apiserver label to them instead of creating the block. Two are
new and are described above: `s3-iceberg-port-explicit.patch` and
`volume-resize-quantity.diff`. Both belong upstream and are carried only
until they land there.

## Verification

Checked locally: a fresh download of 4.45 with all ten patches applied
in Makefile order reproduces the committed tree byte for byte; `helm
unittest` passes 73 in `packages/system/seaweedfs` and 35 in
`packages/extra/seaweedfs`; the naming-guard canary still refuses a
blind upgrade; and a rendered diff against 4.31 comes back identical for
object names, StatefulSet selectors, `serviceName` and
`volumeClaimTemplates`, so a running install keeps its workloads and
volumes across the upgrade. Each fix above has a unit test that is red
without it.

Not checked locally, worth an eye in review or e2e: the live upgrade
path, since `helm template` cannot exercise the `lookup` that preserves
the security.toml JWT keys, and server-side apply convergence when the
`monitoring` label moves between the two filer Services.

### Screenshots

N/A, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(seaweedfs): bump the vendored SeaweedFS chart to 4.45, with the Lance and Iceberg table catalogs kept off and both gates of the volume resize hook comparing sizes numerically. Every SeaweedFS server moves to the 4.45 image, so the master, volume and filer StatefulSets and the s3 Deployment roll one pod at a time on upgrade. Two operator-visible notes: filer metrics stop being double-counted, so filer rate panels step down by half at upgrade time, and the chart's `filer.ingress` values key is replaced by `filer.ingresses.http`, so an override still using the old key is silently inert.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Updated SeaweedFS to version 4.45, with expanded chart configuration
for enterprise licensing, NetworkPolicies, custom scheduling,
certificate-based SFTP authentication, and existing-secret credentials.
* Added separate HTTP and gRPC filer ingress options, S3 Lance ingress
and listener settings, and Lance worker support.
  * Added a Grafana dashboard view for filer object-size distributions.
* **Improvements**
* Added configurable node ports and load-balancer settings across
services, plus custom certificate DNS names.
* Improved volume-resize handling for equivalent storage quantities and
added support for resizing PVCs.
* **Documentation**
* Expanded chart guidance for persistence, credentials, networking,
licensing, and MySQL configuration.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 18:51:48 +04:00
Aleksei Sviridkin e62f510faf chore(deps): move the Kubernetes libraries to v0.37.1
The root module's k8s.io/* move from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, its release for Kubernetes
1.37. Two things held them back: the apimachinery fork, which had to be
rebased onto every minor, and the watch handler that from
k8s.io/apiserver v0.36 panicked on the initial-events-end bookmark the
aggregated storages sent to plain watches. Both are gone.

The rest of the upgrades are what these releases require: cel-go
v0.29.2, the etcd client v3.7.0, kube-openapi, structured-merge-diff and
the x/ modules. code-generator moves with the libraries.

Regenerating the strategy CRDs that embed core types adds the
Kubernetes 1.37 fields and follows one removal: the alpha PodSpec field
workloadRef became schedulingGroup upstream. Once the new CRD is
applied, the apiserver prunes workloadRef from every stored strategy it
serves, and the object's next write erases it from etcd. It only ever
took effect behind an alpha feature gate.

The gateway README and the tenantgateway CRD tests cite the
apiextensions-apiserver list-type ratchet at the pinned version; it is
byte-identical in v0.37.1, so only the version they name changes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 16:08:25 +03:00
Aleksei Sviridkin dab250d210 fix(kamaji): resolve Gateway API availability once at startup
Kamaji checked for the Gateway API on every TenantControlPlane
reconcile. That discovery call ignores the reconcile context, so it is
bounded only by the client's 32s default timeout, which is longer than
the 30s reconcile budget. A slow discovery endpoint could therefore use
up the whole reconcile. The result could also disagree with the
watches, which were set up once at startup.

The patch carries the fix proposed upstream in clastix/kamaji#1327,
not yet merged there, onto the Kamaji tag cozystack ships.
Availability is resolved once when the manager starts, and the same
value drives the watches, the indexer, the reconciled resources and the
admission webhook's Gateway API check. A discovery error at startup now
stops the manager, so it is no longer read as "Gateway API not
installed". A Gateway API installed after Kamaji starts needs a Kamaji
restart. On this platform Kamaji depends on cert-manager, which depends
on gateway-api-crds, so the CRDs are already present when Kamaji
starts.

The patch keeps the upstream tests so that it matches the upstream
commit. The docs hunk is left out because it is not part of the image.
Drop this patch once Kamaji ships the change.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:44:26 +03:00
Aleksei Sviridkin d9cbbec5ff fix(build): cross-compile image builders on the build platform
The remaining Go builder stages, plus the kube-ovn, console and
piraeus-server builders, ran on the target platform. A multi-platform
build therefore ran the whole compiler under QEMU: slow, and for Go
unreliable, as the migration-controller Dockerfile already records a
"found bad pointer in Go heap" crash from exactly that. The builders
now run on $BUILDPLATFORM and produce the target architecture from
TARGETARCH.

Two builders only produced the right architecture because they were
emulated. token-proxy ran `go build` with no GOOS/GOARCH, and kube-ovn
calls upstream `make build-go`, which hardcodes GOARCH=amd64 even on an
arm64 build. On a native builder both would have shipped host binaries
inside the other architecture's image, and nothing fails until a node
of that architecture starts the container. token-proxy now passes the
target explicitly; kube-ovn picks build-go or its arm64 twin
build-go-arm from TARGETARCH.

The console builder emits static JS and linstor-server's debian/control
declares every package Architecture: all, so both run natively without
a target.

A bats test requires every stage that compiles to sit on a BUILDPLATFORM
FROM and to read TARGETARCH or TARGETPLATFORM on a RUN or ENV line. A
bare ARG declaration does not count, which is what let token-proxy
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:05:06 +03:00
IvanHunters 6e0c998b97 fix(seaweedfs): compare volume sizes as numbers in the resize hook
Both sides of the size comparison reach it through an include, which
returns a string, so the hook compared byte counts lexicographically. A
grow that crossed a decimal exponent and lowered the leading digit (5Gi
to 10Gi, 8Gi to 16Gi, 20Gi to 100Gi) read as a shrink: the PVC patches
were skipped while the --cascade=orphan delete above them still ran, so
the volume StatefulSet was recreated and the claims stayed at the old
size, hook exit 0 and nothing logged. The same comparison read a real
shrink across an exponent as a grow and emitted a patch the API server
rejects, failing the pre-upgrade hook after the delete had landed.

The gate that emits the --cascade=orphan delete compared the live
volumeClaimTemplates size against the desired one as raw strings, so it
disagreed with the numeric gate below it about what a size change is. The
API server re-emits a quantity in canonical form, which makes this
permanent rather than one-off: values spelling 10Gi as 10240Mi come back
from the cluster as 10Gi, the strings never converge, and the volume
StatefulSet is deleted and recreated on every upgrade for a size that
never moved. Normalise both sides through seaweedfs.resource-quantity.

The hook also formatted the desired size with %s. Helm decodes values
through JSON, so a size written without quotes reaches the template as a
float64 and the patch command carried a Go format error in place of the
quantity; it now renders with %v.

Carry the comparison as a patch until upstream takes it, and cover both
directions, the respelling and the unquoted size.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 14:56:07 +03:00
Aleksei Sviridkin 623113f920 fix(build): pass the shared buildx flags from every image target
Three package Makefiles spelled out their docker buildx flags instead
of using $(BUILDX_ARGS) from hack/common-envs.mk: flux-plunger,
kubeovn-plunger and both dashboard images. kubeovn-plunger landed just
before the change that moved the other Makefiles onto BUILDX_ARGS,
flux-plunger was modelled on it, and dashboard kept its own flags when
that change touched it.

Those targets ignored PLATFORM, SBOM and BUILDX_EXTRA_ARGS. On an arm64
workstation the plungers came out as arm64 images, which fail on amd64
nodes with "exec format error"; CI builds on amd64 and never saw it.
Dashboard and metallb hardcoded --platform=linux/amd64 instead; for
metallb that sat next to $(BUILDX_ARGS), so setting PLATFORM gave buildx
two platforms and a multi-platform build.

All of them now build through $(BUILDX_ARGS) alone. CI sets only
BUILDER, which these targets already honoured, and runs on amd64, so
its builds are unchanged. Local builds of dashboard and metallb now
follow the host architecture like every other package: on an arm64
machine build them for an amd64 cluster with PLATFORM=linux/amd64.

A bats test dry-runs every package image target with a probe platform
and a marker in BUILDX_EXTRA_ARGS, and requires each buildx call to
carry the marker, so $(BUILDX_ARGS) is really passed, and that
platform and no other.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 14:32:44 +03:00
frolov.so db97e375bd docs(virtual-machine): correct persistent EFI migration documentation
Assisted-by: LLM
Signed-off-by: frolov.so <opsmon.sh@yandex.com>
2026-09-26 11:40:08 +03:00
Aleksei Sviridkin 2c2c0301f7 chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447)
## What this PR does

This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.

Before the bumps:

- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.

Held back on purpose, with the reasons in the commit messages:

- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.

govulncheck reports nothing reachable in any of the five modules,
standard library included.

Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.

Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.

Closes #4487.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added a restore resource policy option for Velero strategies.
  * Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
  * Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 10:17:29 +04:00
Aleksei Sviridkin 4a2dea2d08 feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages (#4087)
## What this PR does

**Series behind #3481** — three separable pull requests:

1. **#4087 — Cluster API providers** (capmox + in-cluster IPAM)
2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on #4088.

---

This is the provisioning layer #3481 asks to specify first: the part
that
creates the VMs. Neither of the other two is needed to review it.

### What it adds

Two Cluster API providers, packaged the way the existing KubeVirt
provider is —
vendored components, `gzip -9 -n` for a reproducible artifact, a `make
components` target that regenerates it from the upstream release and
checks
the downloads against the SHA-256 sums the Makefile pins:

| Package | Version | Contract |
|---|---|---|
| `capi-providers-infraprovider-proxmox` | capmox v0.7.7 | v1beta1 |
| `capi-providers-ipam-in-cluster` | CAIP v1.0.3 | v1beta1 |

capmox v0.7.7 rather than the current v0.9.x on purpose: from v0.8 it
declares
contract `v1beta2`, which the CAPI core shipped here (v1.10.1) does not
serve.

Both join the `iaas` bundle as **opt-in** packages, where the KubeVirt
infrastructure provider is unconditional. The asymmetry is deliberate:
capmox
drives a hypervisor outside the cluster and needs a Secret holding a
Proxmox API
token, which capi-operator reads before it installs anything — with no
Secret the
provider is never deployed at all. Enabling it by default would ask
every
Cozystack install that has no Proxmox to supply hypervisor credentials.

```yaml
bundles:
  enabledPackages:
    - cozystack.capi-provider-infra-proxmox     # brings IPAM with it
    - cozystack.capi-provider-ipam-in-cluster   # also enableable on its own
```

Enabling capmox emits the IPAM provider automatically — capmox allocates
worker
addresses from an `InClusterIPPool` and has no DHCP mode, so on its own
it never
gets a `ProxmoxCluster` to `Ready`. IPAM stays independently enableable,
since
assigning addresses is a generic Cluster API concern.

Naming capmox in `enabledPackages` while disabling IPAM stops the
render. The
cluster would catch that combination on its own — the capmox
`PackageSource`
depends on the IPAM `Package`, so with that `Package` absent capmox goes
`Ready=False` with `DependenciesNotReady` and no `HelmRelease` is
created — but
it would not say why. Failing the render names the two conflicting list
entries
at the moment someone writes them, instead of leaving a condition to
decode.

### Why the earlier attempt did not produce a VM

#1927's blocker — "ProxmoxMachine does not produce a VM" — was not a
capmox
incompatibility. It was four independent causes, each of which only
became
visible once the previous one was cleared:

1. **`PROXMOX_URL` included `/api2/json`.** capmox appends that itself,
so every
   call went to `/api2/json/api2/json/…` and returned 501.
2. **No IPAM provider.** `no matches for kind "InClusterIPPool"`; the
`ProxmoxCluster` never reached `Ready` and every `ProxmoxMachine` sat in
   `WaitingForClusterInfrastructure`.
3. **`templateSelector.matchTags` is set equality, not containment**
(`slices.Equal` after sort). A template carrying four tags is not
selected by
   naming one of them: `found 0 VM templates with tags`.
4. **capmox waits for the qemu-guest-agent, which Talos does not ship.**
   `error waiting for agent: timed out` — cleared with
   `checks.skipQemuGuestAgent` and `skipCloudInitStatus`.

None of these are in the provider. They are packaging and configuration,
which
is why the earlier work could look complete and still never boot a
machine.

### Evidence

Run on Proxmox VE 9.2.11 with Talos v1.13.6 and Cozystack's CAPI stack
(core v1.10.1, CABPT v0.6.12, Kamaji control-plane provider v0.19.0):

```
$ kubectl get proxmoxmachine,machine -n tenant-proxmox
proxmoxmachine/kubernetes-px-md0-…   Ready=True   VMProvisioned=True
machine/kubernetes-px-md0-…          Running      proxmox://0ef88cc1-6aea-4563-af64-344d8060d5c0

$ qm list
106 kubernetes-px-md0-jmzds-h9npx  running
```

The machine boots Talos from the nocloud ISO, reads the machineconfig
capmox
writes, and joins. That is the layer #69 called a hard blocker.

### What this PR does not claim

- One Proxmox node, one tenant cluster, manual runs — there is no e2e
coverage
  for Proxmox in CI, and this PR does not add any.
- It does not make Proxmox a supported platform. It makes the
provisioning layer
reviewable, which #3481 names as the precondition for that conversation.
- It does not touch `capi-providers-core`. The same drift check, pointed
at that
package, shows that the `startupProbe` from #2946 never reached the
shipped
blob; that fix changes what every new install runs and is #4483, with
its
  own release note.
- Nothing here exposes `ProxmoxCluster.spec.credentialsRef` to a tenant,
and
nothing should: `capmox-manager-role` grants get/list/watch/patch on
Secrets
  cluster-wide and that field takes a namespace. Whatever renders a
`ProxmoxCluster` for a tenant in the rest of the series has to keep the
field
  out of tenant hands.

### Testing

`make unit-tests` — exit 0. New suites:
`packages/core/platform/tests/sources_capi_provider_proxmox_test.yaml`
and
`bundles_proxmox_wiring_test.yaml` (both bundle paths, the
enable/disable
precedence, and the refusal above).

Every negated `containsDocument` in the wiring suite carries `any:
true`;
without it the negation passes whether or not the document is in the
render.
Three mutations, each reverted afterwards — emitting the capmox
`Package`
unconditionally, emitting the IPAM `Package` unconditionally, and
dropping the
`disabledPackages` check from `$proxmoxEnabled` — each turn red the
tests named
after the property they break, and the unmutated chart stays green (44
suites,
198 tests).

`hack/capi-provider-components_test.bats` covers both new provider
packages:
the blob and the YAML are the same bytes; the Makefile version, the
ConfigMap
name and `spec.version` are one value — `applied-spec-hash` covers the
provider
spec and the config Secret but not the ConfigMap, so a bump that
regenerates the
files while `spec.version` stays behind installs the old provider
silently; and
the vendored files match the SHA-256 sums the Makefile pins, so the pin
holds
even when nobody runs `make components`.

The render was compared against `main` for five configurations
(defaults,
`isp-full`, `isp-full-generic` with paas, an unrelated
`enabledPackages`, and
`disabledPackages` naming IPAM). No line is removed in any of them and
the
`Package` count is unchanged; the only additions are the two
`PackageSource`
objects, which the chart renders for every package, optional or not.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/website — matches, no follow-up yet.**

- `packages/core/platform/values.yaml` changed.
`content/en/docs/next/operations/configuration/platform-package.md` is a
hand-written table of the `spec.components.platform.values.*` keys; this
PR does not add a key, but it documents two new values that
`bundles.enabledPackages` now accepts, and the table is where a reader
would look for them.
- Two platform components are added
(`capi-providers-infraprovider-proxmox`,
`capi-providers-ipam-in-cluster`) → `guides/platform-stack/_index.md`,
`operations/configuration/licenses.md`.
- The `iaas` bundle gains two optional packages →
`operations/configuration/variants.md`.

**Everything else: no match.** Nothing under `packages/apps/` or
`packages/extra/` (terraform-provider, website app lists), no
`packages/core/installer/values.yaml` or `networking.*`/`publishing.*`
key the Ansible role sets, no `hack/` file or package-layout change
(ccp, external-apps-example), no `ApplicationDefinition` semantics, no
telemetry metric, no node prerequisites (talm, ansible), no
commit-convention change (community).

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(cluster-api): add optional Cluster API packages for Proxmox — the capmox infrastructure provider and the in-cluster IPAM provider it requires. Neither is deployed unless named in `bundles.enabledPackages`, since capmox drives a hypervisor outside the cluster and does nothing without Proxmox API credentials. Enabling the Proxmox provider brings the IPAM provider with it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added optional Proxmox infrastructure provider support for IaaS
clusters.
- Added in-cluster IP address management, usable independently or
alongside Proxmox.
- Proxmox deployments require a pre-created credentials Secret and
automatically include in-cluster IPAM.
  - Added validation for incompatible Proxmox and IPAM settings.

- **Tests**
- Added coverage for provider dependencies, package enablement,
disabling behavior, invalid configurations, and packaged provider
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 01:48:39 +04:00
Aleksei Sviridkin e883f6f7fc fix(capi): ship the core provider manifests the repository actually holds (#4483)
## What this PR does

`capi-providers-core` ships the Cluster API core manifests twice: as
`files/core-components.yaml`, which a reviewer reads, and as
`files/components.gz`, which the chart mounts into the cluster. #2946
added a `startupProbe` to `capi-controller-manager` and edited only the
YAML; the blob was never regenerated, so the probe never reached a
cluster. This regenerates the blob from the YAML (`gzip -9 -n -c
files/core-components.yaml > files/components.gz`), so the decompressed
blob matches the YAML byte for byte, and the only change it carries is
the six lines of #2946.

### What this reaches, and what it does not

New installs only. capi-operator re-applies a provider when the hash of
its spec and its config Secret changes (`applied-spec-hash`); the
ConfigMap holding the components is in neither, so a cluster that
already runs `capi-providers-core` keeps its manager without the probe
until the provider's version moves. Whether existing clusters should get
the probe, and how, is left open on purpose: moving `spec.version` and
the ConfigMap name together would force the re-apply, but that is a
version decision for the package, not a side effect of regenerating a
blob.

### Also in this diff

- `.helmignore`: `files/.*-components.yaml` is a glob, so it matched
only names beginning with a dot, and `helm package` shipped the readable
YAML alongside the blob that duplicates it. Now
`files/*-components.yaml`.
- `hack/capi-core-components_test.bats` holds the two files to the same
bytes from now on; it goes red on a YAML edited without regenerating the
blob.

Found by the drift check written for the Proxmox provider packages in
#4087, pointed at the package that was already here; split out because
it changes what every new install runs and deserves its own release
note.

### Testing

`bats hack/capi-core-components_test.bats`: 1/1; red with a line
appended to the YAML, green again after `gzip -9 -n -c`. `helm package`
of the chart now contains `files/components.gz` and
`files/metadata.yaml` only.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(cluster-api): ship the `startupProbe` from #2946 in the packaged core provider components. New installs get it; a cluster that already runs the core provider keeps its manager without the probe until the provider's version moves, because capi-operator re-applies components only when the provider spec or its config Secret changes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Added a check that the compressed core components match the YAML
source, with a diff provided when they differ.
* **Chores**
* Updated chart packaging exclusions to cover component YAML files
regardless of whether their names begin with a dot.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 22:43:55 +04:00
Aleksei Sviridkin a285a5ef36 feat(kubernetes): add a proxmox substrate for managed clusters and worker pools (#4088)
## What this PR does

**Series behind #3481** — three separable pull requests:

1. #4087 — the Cluster API providers (capmox + in-cluster IPAM)
2. **#4088 — the `substrate` switch** in `kubernetes` and
`kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on this one.

---

#4087 adds the Cluster API providers; this PR teaches the two
managed-cluster
charts to point at them.

### The shape

A single value, `substrate: kubevirt | proxmox`, on both charts — not a
parallel
set of charts. The control plane is Kamaji either way; what changes is
the
infrastructure half:

| | `kubevirt` (default) | `proxmox` |
|---|---|---|
| `Cluster.infrastructureRef` | `KubevirtCluster` | `ProxmoxCluster` |
| worker template | `KubevirtMachineTemplate` | `ProxmoxMachineTemplate`
|
| apiserver Service | `ClusterIP` | `LoadBalancer` |
| tenant CCM/CSI | in-cluster (`kccm`, kubevirt-csi) | rendered off; the
Proxmox pair is #4089 |

The KubeVirt path renders byte-for-byte what it rendered before. Checked
by
rendering `kubernetes-nodes` with default values at `main` and at this
head and
diffing the output, not only by the snapshots — the snapshots cover
`nodegroup.yaml` alone, and an earlier revision of this branch changed
the
reconcile Job while they stayed green. The parent chart's default render
differs
only in the random fields of `talos-secrets`, which differ between two
renders of
`main` as well.

`osImage` (#4171) is refused on the proxmox substrate. Only half of it
could be
honoured there — `factory.version` reaches the machineconfig's installer
image
while the worker disk keeps coming from the Proxmox template — and a
field that
half-works is worse than one that says no. `talos.version` and
`talos.schematicID` still set the installer target.

### Three things that only show up once workers are off-cluster

**The apiserver endpoint.** A KubeVirt worker runs inside this cluster
and
reaches the tenant apiserver by ClusterIP. A Proxmox worker does not: it
lives on
the hypervisor's L2 and cannot route to `10.96.0.0/16`. The machine
provisions
fine, `VMProvisioned=True`, `providerID` set — and then never joins,
with no CSR
to show for it. On the proxmox substrate the Kamaji Service is therefore
a
`LoadBalancer`, and the reconcile Job reads
`.status.loadBalancer.ingress[0].ip` instead of `.spec.clusterIP`.

**And DNS, for the same reason.** The worker machineconfig carried
`nameservers: ${COREDNS_IP}` — the management cluster's CoreDNS
ClusterIP,
equally unreachable. The proxmox path takes its own resolvers
(`proxmox.dnsServers`) and the chart refuses to render without them,
rather than
writing an address that cannot answer.

**`externalManagedControlPlane` and the managed-by annotation.**
capmox's
webhook nil-derefs on a `ProxmoxCluster` whose control-plane endpoint is
empty
unless `spec.externalManagedControlPlane` is set, which is correct here
— Kamaji
owns the endpoint. Separately, the `cluster.x-k8s.io/managed-by: kamaji`
annotation that the KubeVirt path carries must **not** be set on the
Proxmox
object: with it, CAPI never sets an ownerRef and capmox stops at
`Waiting for Cluster Controller to set OwnerRef`, leaving the address
pool
uncreated and every worker waiting on infrastructure.

### Linked clones by default

`proxmox.full: false` — worker disks are linked clones of the template
rather
than full copies. Measured on a ZFS-backed store: **8 KiB per worker
instead of
10.2 GiB**. Operators who want independent disks set `full: true`; the
value is
plumbed straight through to `ProxmoxMachineTemplate`.

`proxmox.storage` is a full-clone parameter and is refused with a linked
clone:
a linked clone always lives on the template's storage, and both Proxmox
(`parameter 'storage' not allowed for linked clones`) and capmox's own
CRD
(`Must set full=true when specifying storage`) reject the pair. The
chart says
so at render time, naming both values, instead of failing at apply time
with an
error about a `ProxmoxMachineTemplate` nobody wrote.

### Sizing

`ProxmoxMachineTemplate` takes `numCores` and `memoryMiB`, not
Kubernetes
quantities, so the pool's `resources.cpu`/`memory` are converted
(millicores →
cores, bytes → MiB, both rounded up). The rounding is pinned in
`tests/substrate_branches_test.yaml` with `1500m` and `8G`, where
rounding up and
down disagree; `tests/substrate_proxmox_sizing_test.yaml` covers the
refusal to
size a pool that gives neither.

### Known upstream behaviour this surfaces

`TalosConfigTemplate.spec` is immutable by schema, so the reconcile Job
cannot
rewrite an endpoint it has already published — `kubectl apply` fails
with
`TalosConfigTemplate.Spec is immutable`, and the template has to be
deleted for
the Job to recreate it. A fresh install never meets it; it appears when
the
apiserver address changes on an existing cluster. Handling it changes
the
KubeVirt path too, so it is not in this PR: it is a separate change on
`proxmox/tct-replace`, with its own upgrade note.

### Evidence

```
$ KUBECONFIG=<tenant> kubectl get nodes -o wide
kubernetes-px-md0-jmzds-ndnl6  Ready  <none>  75s  v1.35.6  10.0.0.160  Talos (v1.13.6)

$ KUBECONFIG=<tenant> kubectl get pods -A
cozy-cilium  cilium-…            Running
kube-system  konnectivity-agent  Running
kube-system  coredns-…           Running
```

No management-cluster address remains in the rendered
`TalosConfigTemplate`.

### Testing

`make unit-tests` — exit 0. `apps/kubernetes` 26 suites / 314 tests;
`apps/kubernetes-nodes` 29 suites / 160 tests / 12 snapshots, all
unchanged on
the default path. Each of the ten commits passes both suites on its own.

Every substrate branch is asserted on both sides, and every assert goes
red on
the mutation it exists for: inverting the `SVC_IP` condition, inverting
the
`nameservers` condition, `ceil`→`floor` in the sizing, removing the
proxmox
gate from each of the kccm and csi templates (the two RBAC bindings
included),
pinning the retention lookup back to `KubevirtMachineTemplate`, widening
the
IPv6 class of the `dnsServers` validation, dropping the parent chart's
`dnsServers` validation, the storage-with-linked-clone refusal, and the
`osImage` refusal. Negated `containsDocument` asserts carry `any: true`.

`hack/talos-reconcile-heredoc_test.bats` additionally runs the rendered
heredoc
through a real shell: a `dnsServers` entry with a substitution is
refused and
named, an IPv6-shaped one with a substitution is refused too, and a real
address survives the shell unchanged.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/terraform-provider-cozystack — matches, no follow-up yet.**

- `packages/apps/kubernetes/values.schema.json` and
`packages/apps/kubernetes-nodes/values.schema.json` both gain fields:
`substrate`, and a `proxmox` object (`allowedNodes`, `dnsServers`,
`ipv4Config`, and on the pool side `templateTags`, `network`, `storage`,
`full`). The provider hand-maintains a schema, a model and an
expand/flatten pair per Kind, with no codegen from these files.
- `values.yaml` gains defaults for those fields (`substrate: kubevirt`,
`proxmox.full: false`, `prefix: 24`). The provider restates defaults it
models and sends its own, so a stale one wins silently rather than
showing as a diff.

**Everything else: no match.** No app added, renamed or removed (website
app lists); no version enum, `kind`, `plural`, `release.prefix` or
output Secret/Service name changed; `packages/core/platform/values.yaml`
untouched; the `*-rd` files are regenerated, not renamed (ccp); no
`hack/` change; no node prerequisites (talm, ansible); no telemetry
metric; no commit-convention change (community).

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): add a `substrate` value to the `kubernetes` and `kubernetes-nodes` applications, selecting whether worker VMs run on KubeVirt in this cluster (`kubevirt`, the default and unchanged) or on an external Proxmox VE cluster through capmox (`proxmox`). On the Proxmox substrate the tenant apiserver is published through a LoadBalancer and worker disks default to linked clones.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for choosing KubeVirt or Proxmox for Kubernetes clusters
and node pools.
* Added Proxmox configuration for nodes, networking, storage, DNS,
templates, cloning, and IPv4 allocation.
* Proxmox deployments now use the appropriate cluster, machine,
networking, and control-plane resources.

* **Validation**
* Added checks for supported substrates, required Proxmox settings,
sizing, and incompatible options.

* **Documentation & Tests**
* Documented the new settings and expanded coverage for rendering,
validation, cloning, sizing, and template replacement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 22:09:23 +04:00
Aleksei Sviridkin b16edec3f3 fix(tcp-balancer): rename backends and pin HAProxy so the chart starts on 3.3+ (#4366)
## What this PR does

`haproxy:latest` now resolves to 3.3 and newer. Since 3.3 a backend
named like a frontend is a fatal config error, so every tcp-balancer pod
fails at config parse (#1966). We hit it on Cozystack 1.6.1 with
`haproxy:3.4.4` behind the tag.

- backends get a `_backend` suffix and every `default_backend` follows;
the rename is the one from #2321 by @officialasishkumar
- the image is `haproxy:{{ .Chart.AppVersion }}` with `appVersion:
3.4.4`, the current 3.4 LTS tag
- the whitelist `acl`, `accept` and `reject` lines of each frontend
render inside one guard
- `tests/` holds helm-unittest suites, and the package `Makefile` gets
the `test` target the sweep looks for
- the `whitelistHTTP` description says it needs a non-empty `whitelist`

Two inputs change behaviour on upgrade. On `main` both fail at config
parse with `no such ACL : 'whitelist'` on every HAProxy version, because
the `acl` line sits inside the `with` and `accept if whitelist` does
not:

- `whitelistHTTP: true` with an empty `whitelist` now fails the render
with `whitelistHTTP is enabled but whitelist is empty`. Its HelmRelease
upgrade fails and the release stays on its old revision, which is
already crash-looping.
- `kubernetesAndTalos` with an empty `whitelist` now starts, and ports
6443 and 50000 accept any source. With `external: true` the Service is a
LoadBalancer and carries both ports. `whitelist` is the only switch
those frontends have, so an empty list means no filtering.

Every other input renders the same config apart from the backend names.
The image and the `checksum/config` annotation change, so the Deployment
rolls, and for releases already crash-looping on `latest` the rollout is
the fix.

Validation: renders of `main` and of this branch for the default,
whitelist and empty-whitelist inputs were checked with `haproxy -c` on
2.9.7 and 3.4.4. `helm unittest .` passes 9/9, and removing the `reject`
or `accept` line from any single frontend turns a test red.

Fixes #1966

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: chart templates, `Chart.yaml`,
the package `Makefile`, `tests/`, and a description-only change in
`values.yaml` with its generated carriers. No field is added, removed or
renamed and no default changes, so nothing in the map fires.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tcp-balancer): rename HAProxy backends and pin the image to the chart appVersion so the balancer starts on HAProxy 3.3+. whitelistHTTP with an empty whitelist now fails the render; kubernetesAndTalos with an empty whitelist leaves ports 6443 and 50000 unfiltered.
```
2026-09-25 19:29:39 +04:00
Marian Koreniuk 8722462066 fix(kubernetes-nodes): refuse osImage on proxmox instead of honouring half of it
#4171 gave a worker pool an osImage, choosing where its system disk comes from:
a CDI clone of a shared golden image, or an HTTP import from an Image Factory.
A Proxmox worker's disk comes from neither. capmox clones a VM template that
already carries Talos, and nothing in this chart can change what that template
holds.

Half of the field would still appear to work, which is why it is refused rather
than ignored. osImage.factory.version reaches the machineconfig's installer
image and changes the reconcile Job, so a pool could set it, watch the Job
roll, and end up running one Talos while booting another: v1.12.6 moves the
installer to .../installer/<schematic>:v1.12.6 and leaves the
ProxmoxMachineTemplate untouched. osImage.builtin has no meaning here at all.

Nothing is lost by refusing it: the installer target on this substrate is
settable through talos.version and talos.schematicID, which do reach it.

The branches suite gains the refusal case. The substrate parameter's description said changing the value "rolls the
pool". The chart refuses the change on a live pool, and the description says
so; it reaches the README, the values schema and the CRD.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 517baf945d fix(kubernetes,kubernetes-nodes): validate dnsServers, refuse a substrate switch, drop the Pod threshold on proxmox
Three faults in the proxmox substrate, none of which the render refused.

1. proxmox.dnsServers reached the reconcile Job's shell unescaped.

The Job applies the worker TalosConfigTemplate through an unquoted heredoc, so
every byte of its body is subject to parameter expansion and command
substitution at Job runtime. The INVARIANT comment above that block says each
tenant-controlled value must be shell-escaped or pattern-validated; the new
nameserver list was neither. `quote` there is YAML quoting.

Rendered with `$(id -u)` and a backtick pair in dnsServers, extracted the way
hack/talos-reconcile-heredoc_test.bats does and run under sh, the shell executed
both and put their output in the worker's resolver list. The Job's pod holds the
parent cluster's Talos token, cluster id, cluster secret and bootstrap token in
its environment, and its Role can read the Talos CA and patch the parent
KamajiControlPlane.

Each entry is now validated as an IPv4 or IPv6 address at render time, in both
charts. Validation rather than escaping because Talos takes addresses here: an
address cannot carry a metacharacter, so the property holds by construction, and
the ordinary mistake of writing a hostname is caught in the same move. IPv6 is
matched on its character set rather than its full grammar — what matters is which
bytes may reach the shell, and the apiserver rejects a malformed address anyway.

2. Nothing stopped a live cluster or pool from being switched between substrates.

`substrate` had no immutability of any kind, and the `## @immutable` marker this
adds is dashboard-enforced only, so both charts now read the live object.

Switching a cluster removes the KubevirtCluster, the cloud-controller-manager,
the kubevirt-csi controller and the in-tenant -csi HelmRelease with all their
RBAC from the render. Flux then uninstalls the CSI driver behind every PVC the
tenant has. CAPI does not prevent it — its Cluster webhook forbids removing
spec.infrastructureRef but admits a change of kind.

Switching a pool rolls every worker to the other hypervisor and drops the
machine template the draining MachineSet still references, because the retention
loop looks up only the kind the new substrate uses. A pool whose substrate
differs from its parent's was not rejected either, though the README says the
two must match.

Both guards are inert offline, like assertNoForeignPool and assertParentVersion
beside them: with no apiserver the lookup is empty and a first install renders.

3. The worker WorkloadMonitor reported a proxmox pool as broken.

The controller lists Pods matching the monitor's selector and sets
Operational=false while fewer than minReplicas are ready. On kubevirt those Pods
are the virt-launcher Pods. A proxmox worker is a VM on another machine with no
Pod here, so the count is always zero and any threshold pinned the pool — and
through the lineage labels its whole application — to WorkloadsReady=False while
every node was healthy. The monitor still renders for the dashboard and the
lineage; what it no longer carries on proxmox is a threshold counted in Pods.

Two smaller things. The wait-loop log line is written per substrate, so a
default kubevirt render of kubernetes-nodes is byte-identical to main and no
existing pool gets a new reconcile Job on upgrade. And the comment claiming a
rename of $kubevirtmachinetemplate would roll every live worker VM is replaced:
the hash is taken over the rendered template, not the variable name, so renaming
it leaves the render byte-identical.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk c5996c1d0f fix(kubernetes,kubernetes-nodes): give Proxmox workers an endpoint they can reach
A worker on the proxmox substrate is a VM on the hypervisor network, not a Pod
in this cluster. It was handed two addresses only an in-cluster consumer can
route to: the apiserver as a ClusterIP, and the management cluster's CoreDNS
ClusterIP as its resolver. The VM booted, the Machine reported Ready with a
providerID, the disk was where it should be — and no CSR ever arrived. Nothing
in that picture points at an address.

The control-plane chart now publishes the tenant apiserver as a LoadBalancer on
the proxmox path, putting the endpoint on the network the workers share, and
the pool's reconcile Job reads the Service's LoadBalancer address instead of its
ClusterIP for the host entry the workers dial. The nameservers follow the same
reasoning: proxmox pools carry their own resolvers rather than pointing at a
CoreDNS they cannot reach, and the chart refuses to render a pool that has not
named any.

KubeVirt keeps the ClusterIP on both counts. Its workers are in this cluster,
where a ClusterIP is the right answer and a LoadBalancer address would be an
unnecessary one.

Verified end to end on a real tenant cluster: the node joins, its CSR is
approved, and cilium, konnectivity and coredns come up in the tenant.

One operational note this surfaced, not caused by it: TalosConfigTemplate.spec
is immutable, so the reconcile Job cannot rewrite an endpoint that has already
been published — the template has to be deleted for the Job to recreate it.
That matters only when an existing cluster's apiserver address changes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk d78af928fc feat(kubernetes-nodes): default Proxmox worker disks to linked clones
A full clone on a ZFS pool configured with `sparse 0` reserves the whole
volsize the moment the VM is created: 10.2G for a worker whose Talos image
occupies 337M. A linked clone writes only its own increment — measured at 8K
for the same worker on the same host — because it shares the template's base
snapshot until it diverges.

The cost is that the clone holds that snapshot, so the template cannot be
rotated while any linked clone still references it. That is bounded rather
than permanent: consolidation detaches each worker 24 hours after creation, so
template rotation is blocked for at most a day.

Consolidation is not a chart concern, but two of its constraints are worth
recording where the flag is set, because both were discovered by testing on a
live host rather than from the docs. A linked clone can only be detached by
moving its disks to a *different* storage — Proxmox refuses a same-storage
move outright. And both disks must move: leaving efidisk0 behind keeps the
template pinned just as effectively as the system disk would.

Set full: true for a pool that must be independent of the template from the
first second.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:14 +02:00
Marian Koreniuk db839d1198 feat(kubernetes): add a proxmox substrate for managed clusters
Companion to the same switch in kubernetes-nodes. The control plane is Kamaji
on either substrate, so this changes exactly one object — the one
Cluster.infrastructureRef points at — plus the two tenant addons that exist
only because the workers are KubeVirt VMs living in this cluster.

A KubevirtCluster needs no spec: its workers are in the same cluster and reach
the apiserver over the pod network. A ProxmoxCluster addresses an external
hypervisor, so it carries what capmox cannot infer — which Proxmox nodes may
host VMs, which addresses they may take, and who resolves names for them.
controlPlaneEndpoint stays absent on both: the cluster.x-k8s.io/managed-by
annotation hands that to Kamaji.

The address pool is required rather than defaulted. capmox has no DHCP mode; it
allocates from an InClusterIPPool built out of ProxmoxCluster.spec.ipv4Config,
so an unset pool is not a smaller configuration but a ProxmoxCluster that never
reaches Ready and worker Machines that wait on it forever. Refusing at render
time turns that into a message instead of a silence.

The KubeVirt cloud-controller-manager, its cloud-config and the kubevirt-csi
driver are gated off on the proxmox path. They reach the workers'
VirtualMachines and DataVolumes in this cluster; on Proxmox those objects do
not exist, and leaving them rendered would put controllers in the tenant
reconciling nothing. Their Proxmox equivalents are separate packages and land
separately.

The default path is unchanged: rendering it before and after this commit
differs only in the per-render random bootstrap tokens, and all 19 existing
suites pass untouched.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:29:09 +02:00