Commit Graph
7777 Commits
Author SHA1 Message Date
Aleksei Sviridkin b6a30bd504 docs(e2e): describe the per-suite runner and the pod-leak guard
The agent guide, the suite README and a workflow comment said the E2E
step is a single chainsaw invocation; it is now one per suite, and a
suite whose pods outlive its cleanup fails. Say how a pod is charged
to a suite, what the budget is, and how the allowlist is kept honest.
The workflows' failure line now points at LEAKED-PODS as well, since a
run that failed only on a leak has no assertion diff to point at.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:19:49 +03:00
Aleksei Sviridkin 084ac47237 feat(build): build published images for amd64 and arm64 by default
PLATFORM defaulted to empty, so buildx built for the host alone. On an
arm64 workstation `make image` published an arm64-only digest, which
amd64 nodes then fail to pull; the cilium pin shipped exactly that
until it was rebuilt by hand. A build now targets linux/amd64 and
linux/arm64 unless PLATFORM is set, and the digest a package stamps is
the index digest.

LOAD=1 keeps building for the host only. The classic docker image store
cannot load a multi-platform index, and loading is the documented local
workflow.

This changes local pushes: a `make image` on the default docker driver
without the containerd image store now fails with "Multi-platform build
is not supported for the docker driver", where it used to build for
the host. Such a build needs a docker-container builder (BUILDER), or
LOAD=1 or an explicit PLATFORM for a single architecture.

talos and testing pin linux/amd64 with `override`, placed before the
include because common-envs.mk expands BUILDX_ARGS immediately. The
Talos assets are amd64: matchbox bakes the kernel and initramfs into
its image, and the e2e sandbox boots the disk under qemu-system-x86_64.
An arm64 variant of either would still serve or boot amd64 files.

CI pins linux/amd64 on every image build. Nothing there sets up QEMU,
fork builds export a single-platform archive, and tags.yaml builds on
the default docker driver, so the default would break or emulate those
jobs. Native arm64 builds belong in a leg of their own.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:14:12 +03:00
Aleksei Sviridkin c5cd21af96 test(e2e): fail the suite whose pods outlive its cleanup
A Chainsaw suite could pass while its workloads kept running after its
own cleanup: brokers of Kafka releases whose application was deleted,
a TenantControlPlane stuck on its finalizer holding its apiserver pods.
Nothing noticed until a later suite failed on Insufficient memory,
hours on and under the wrong name.

Chainsaw v0.2.15 has no hook after a test's cleanup, so the suite now
runs one `chainsaw test` per suite directory and checks between them.
A pod is charged to the suite when it is not terminal, did not exist
before the suite, and the root of its ownerReferences chain was created
at or after the suite started on the apiserver's clock. That leaves
alone a platform pod a rollout, an eviction or a crash replaces
mid-suite, though not one whose platform object is itself recreated
mid-suite. Charged pods get the default cleanup timeout of the chainsaw
configuration, five minutes, counted from the moment chainsaw returns;
past that the run names the suite, each pod and its root, and fails.
A timeout or a spent budget takes the verdict from the last good pod
list; any other read failure before then stops the wait and is
reported as inconclusive rather than as a leak; a failed first reading
after the suite leaves it unchecked, and it fails.

The runner hands chainsaw the tests a single invocation would discover
and merges the per-suite JUnit reports into the chainsaw-report.xml
the workflows already collect. Suites with a filed leak are exempt by
an allowlist that must name existing suite directories, so a rename
cannot keep an exemption alive.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:06:42 +03:00
Aleksei Sviridkin d9cbbec5ff fix(build): cross-compile image builders on the build platform
The remaining Go builder stages, plus the kube-ovn, console and
piraeus-server builders, ran on the target platform. A multi-platform
build therefore ran the whole compiler under QEMU: slow, and for Go
unreliable, as the migration-controller Dockerfile already records a
"found bad pointer in Go heap" crash from exactly that. The builders
now run on $BUILDPLATFORM and produce the target architecture from
TARGETARCH.

Two builders only produced the right architecture because they were
emulated. token-proxy ran `go build` with no GOOS/GOARCH, and kube-ovn
calls upstream `make build-go`, which hardcodes GOARCH=amd64 even on an
arm64 build. On a native builder both would have shipped host binaries
inside the other architecture's image, and nothing fails until a node
of that architecture starts the container. token-proxy now passes the
target explicitly; kube-ovn picks build-go or its arm64 twin
build-go-arm from TARGETARCH.

The console builder emits static JS and linstor-server's debian/control
declares every package Architecture: all, so both run natively without
a target.

A bats test requires every stage that compiles to sit on a BUILDPLATFORM
FROM and to read TARGETARCH or TARGETPLATFORM on a RUN or ENV line. A
bare ARG declaration does not count, which is what let token-proxy
through.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:05:06 +03:00
IvanHunters 6e0c998b97 fix(seaweedfs): compare volume sizes as numbers in the resize hook
Both sides of the size comparison reach it through an include, which
returns a string, so the hook compared byte counts lexicographically. A
grow that crossed a decimal exponent and lowered the leading digit (5Gi
to 10Gi, 8Gi to 16Gi, 20Gi to 100Gi) read as a shrink: the PVC patches
were skipped while the --cascade=orphan delete above them still ran, so
the volume StatefulSet was recreated and the claims stayed at the old
size, hook exit 0 and nothing logged. The same comparison read a real
shrink across an exponent as a grow and emitted a patch the API server
rejects, failing the pre-upgrade hook after the delete had landed.

The gate that emits the --cascade=orphan delete compared the live
volumeClaimTemplates size against the desired one as raw strings, so it
disagreed with the numeric gate below it about what a size change is. The
API server re-emits a quantity in canonical form, which makes this
permanent rather than one-off: values spelling 10Gi as 10240Mi come back
from the cluster as 10Gi, the strings never converge, and the volume
StatefulSet is deleted and recreated on every upgrade for a size that
never moved. Normalise both sides through seaweedfs.resource-quantity.

The hook also formatted the desired size with %s. Helm decodes values
through JSON, so a size written without quotes reaches the template as a
float64 and the patch command carried a Go format error in place of the
quantity; it now renders with %v.

Carry the comparison as a patch until upstream takes it, and cover both
directions, the respelling and the unquoted size.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 14:56:07 +03:00
IvanHunters ba46b9fc60 ci(testing): download the flux installer with --fail in the e2e-sandbox image
The flux install piped `curl -sSL` into bash, so on a 4xx/5xx the error
page was piped into the shell and curl's exit code was lost. Download the
installer to a file with `curl -fsSL` first, then run it, so an HTTP error
fails the build step instead of feeding bash a bad script.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
IvanHunters 95e68bad86 ci(workflows): use a literal match in the flux version guards
The guard ran `grep -qx "flux: v${FLUX_VERSION}"`, where the dots in the
version are regex metacharacters, so a client printing `flux: v2X8X6`
would satisfy it. Use `grep -Fqx` for a fixed-string match in the build
toolchain guards this change added.

Also correct the step comments that still read "install if absent
(idempotent)": the step now installs whenever the pinned version is not
the one present, not only when flux is missing.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
IvanHunters 03c6c13443 ci(testing): pin flux version in the e2e-sandbox image
The e2e-sandbox image fetched flux through `install.sh` with no version,
so the build resolved the installer's floating "latest". Add an
ARG FLUX_VERSION, matching the neighbouring COZYHR_VERSION and
CRUST_GATHER_VERSION pins, and pass it to install.sh so the sandbox's
flux CLI is fixed and reproducible like the rest of the image's tools.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
IvanHunters 8c0f91ff00 ci(workflows): pin flux version in remaining install-latest steps
The nightly toolchain steps still fetched flux through the installer's
floating "latest". Pin them to FLUX_VERSION 2.8.6 with the same
version-gated guard so nightly's flux push artifact step uses a fixed,
reproducible CLI rather than whatever latest resolves to.

Also switch e2e-fork's already-pinned step from a presence check to the
same version gate, so a runner with a pre-cached older flux is corrected
rather than left as is, and fix that step's comment to point at the flux
distro cozystack actually ships
(internal/fluxinstall/manifests/fluxcd.yaml, v2.8.0).

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:47 +03:00
IvanHunters edbc4f1b3e ci(workflows): pin flux version in build toolchain steps
The "Set up build toolchain" steps installed flux with a bare
`curl https://fluxcd.io/install.sh | sudo bash`, letting the installer
resolve its floating "latest". latest had advanced to v2.9.5, and a
matrix job on PR #4429 failed when that release's checksums transiently
4xx'd during install.

Pin FLUX_VERSION to 2.8.6 and guard on the installed client version, the
same shape the "Set up promotion toolchain" steps already use, so the
flux CLI is fixed and reproducible across CI instead of following latest.
This does not remove the checksum host's transient errors; it removes the
uncontrolled version bump. Related to the yq floating-latest pin in #3640.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:47 +03:00
Aleksei Sviridkin b113cc77d5 fix(kafka): delete the release's node pools on uninstall (#4490)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Deleting a Kafka application left its brokers running. The migration
hook creates the KafkaNodePools with `kubectl apply`, so they are not in
the release manifest and `helm uninstall` does not remove them, and each
pool owns the StrimziPodSet that runs its brokers. The pre-delete hook
now deletes the release's pools, selected by
`strimzi.io/cluster=<release>`, which is the label the migration hook
stamps. Its Role gains `list` and `delete` on `kafkanodepools`, which is
what a label-selected delete issues. Fixes #4488.

The pools are deleted after the KafkaTopics, also when the topic wait
timed out. The topic operator clears topic finalizers through the
brokers, so the brokers have to outlive it.

Deleting the pools before Helm deletes the Kafka CR does not bring
brokers back. For the few seconds in between, Strimzi 0.45.1-rc1 (the
version the operator chart pins) fails the reconcile with
`KafkaNodePools are enabled, but no KafkaNodePools found` before it
builds any StrimziPodSet. A regular pool is the ownerReference of its
StrimziPodSet (`NodePoolUtils.createKafkaPools` at that tag), so the
pods go away with the pool by garbage collection. If the Kafka CR is
already gone when the hook runs, the pools are still found by label.

An absent CRD exits 0, and so does a release with no pools, because a
label-selected delete that matches nothing succeeds. Any other failure
exits 1. The call uses `--wait=false` and no request timeout, so the
#4276 invariants and the 270s deadline arithmetic in `delete.yaml` stay
as they were.

The cleanup lives in the pre-delete hook because only there a failure
gets retried. In Helm v4.1.1, which helm-controller v1.5.0 vendors, a
pre-delete hook error returns from `Uninstall.Run` before anything is
deleted, so the next reconcile runs the hook again. A post-delete error
is only collected: the release is still purged and Helm never runs its
hooks again, so a failure there would leave the pools for good.

An ownerReference from each pool to the Kafka CR would also work. I did
not take it because it changes what the migration hook applies on every
install and upgrade, including the ZooKeeper migration path that adopts
a namespace-wide `kafka` pool.

PVCs are out of scope. The pools set `deleteClaim: false`, so Strimzi
leaves the broker and controller volumes in place when a pool is
deleted, before and after this change.

Not covered by this change:

- Pools leaked by releases deleted before this fix stay where they are.
Find them with `kubectl get kafkanodepools.kafka.strimzi.io -A -L
strimzi.io/cluster`; a pool whose cluster has no Kafka CR in its
namespace (`kubectl get kafkas.kafka.strimzi.io -A`) is leaked, and
`kubectl -n <namespace> delete kafkanodepools.kafka.strimzi.io -l
strimzi.io/cluster=<cluster>` removes it and its brokers.
- `examples/backups/kafka-metadata/cleanup.sh` returns without waiting
for the Kafka uninstalls, unlike `examples/backups/kafka/cleanup.sh`,
which is why the metadata suite's new checks carry a 12m budget. The new
step already holds the suite until the pools and pods are gone, so the
next suite no longer starts into a namespace that is still uninstalling.
Adding the same `wait_deleted` calls to the script would let those
checks drop to the default timeout; left as a follow-up because the
example is outside this change.
- The "Deletion and PVC retention" section of
`packages/apps/kafka/README.md` does not yet say that the pre-delete
hook now removes the node pools while `deleteClaim: false` keeps the
PVCs. That one sentence is left as a follow-up.
- Helm runs the hooks stored in the release's latest revision, so a
release uninstalled before it has been upgraded onto this chart still
runs the old hook and still leaks its pools.

Tests:

- `packages/apps/kafka/tests/delete_test.yaml`: a new case pins, in one
pattern, that the pool delete follows the topic delete, is scoped to
this release, and does not wait, and pins the exact Role rule.
- `hack/kafka-pre-delete-hook.bats` runs the rendered hook against a
fake kubectl. Its fake now answers the pool call separately, and four
new cases pin the exit codes and ordering: pools are deleted after the
topics, still deleted after a topic-wait timeout, an absent pool CRD
exits 0, and a failed pool delete exits non-zero.
- Each new pin was checked by a mutation that turned it red: widening
the selector, moving the pool delete before the topics, waiting on it,
exiting on an absent CRD, not exiting on a failure, skipping the pools
on the timeout branch, and removing or widening the Role rule.
- E2E: the `kafka`, `kafka-2-backup-roundtrip` and
`kafka-3-metadata-roundtrip` tests each gain a step,
`verify-uninstall-removes-node-pools`, that checks, after the apps are
deleted, that no KafkaNodePool and no pod labelled
`strimzi.io/cluster=<cluster>` remains. The pod check covers every
Strimzi pod of the cluster, not only the brokers, on purpose. The
`kafka` test now deletes its app itself so it can look afterwards.

E2E success criterion, registered before the run: in this PR's E2E run,
every `verify-uninstall-removes-node-pools` step (one in each of the
tests `kafka`, `kafka-2-backup-roundtrip` and
`kafka-3-metadata-roundtrip`) passes on the first attempt.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches one chart template and its unit tests, one hack unit
test, and two e2e suites. No values, schema, README, package list, hack
tooling, CRD or node requirement changes, so no entry of the trigger map
applies.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kafka): deleting a Kafka application now also deletes its KafkaNodePools, so its brokers stop instead of running on after the application is gone. Volumes are kept, as before. Node pools left behind by applications deleted before this fix are not removed automatically: delete them with `kubectl -n <namespace> delete kafkanodepools.kafka.strimzi.io -l strimzi.io/cluster=<cluster>`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Kafka app uninstall now removes associated Kafka node pools after
attempting to remove Kafka topics. If topic cleanup times out, uninstall
continues; node pool cleanup failures are reported.
* End-to-end checks now verify that Kafka node pools and pods are
removed after uninstall and backup cleanup. If resources remain,
diagnostic details are provided.
* **Tests**
* Added coverage for node pool cleanup outcomes, including missing
resource definitions and deletion failures.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 15:38:27 +04:00
Aleksei Sviridkin 623113f920 fix(build): pass the shared buildx flags from every image target
Three package Makefiles spelled out their docker buildx flags instead
of using $(BUILDX_ARGS) from hack/common-envs.mk: flux-plunger,
kubeovn-plunger and both dashboard images. kubeovn-plunger landed just
before the change that moved the other Makefiles onto BUILDX_ARGS,
flux-plunger was modelled on it, and dashboard kept its own flags when
that change touched it.

Those targets ignored PLATFORM, SBOM and BUILDX_EXTRA_ARGS. On an arm64
workstation the plungers came out as arm64 images, which fail on amd64
nodes with "exec format error"; CI builds on amd64 and never saw it.
Dashboard and metallb hardcoded --platform=linux/amd64 instead; for
metallb that sat next to $(BUILDX_ARGS), so setting PLATFORM gave buildx
two platforms and a multi-platform build.

All of them now build through $(BUILDX_ARGS) alone. CI sets only
BUILDER, which these targets already honoured, and runs on amd64, so
its builds are unchanged. Local builds of dashboard and metallb now
follow the host architecture like every other package: on an arm64
machine build them for an amd64 cluster with PLATFORM=linux/amd64.

A bats test dry-runs every package image target with a probe platform
and a marker in BUILDX_EXTRA_ARGS, and requires each buildx call to
carry the marker, so $(BUILDX_ARGS) is really passed, and that
platform and no other.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 14:32:44 +03:00
frolov.so db97e375bd docs(virtual-machine): correct persistent EFI migration documentation
Assisted-by: LLM
Signed-off-by: frolov.so <opsmon.sh@yandex.com>
2026-09-26 11:40:08 +03:00
Aleksei Sviridkin 2c2c0301f7 chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447)
## What this PR does

This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.

Before the bumps:

- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.

Held back on purpose, with the reasons in the commit messages:

- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.

govulncheck reports nothing reachable in any of the five modules,
standard library included.

Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.

Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.

Closes #4487.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added a restore resource policy option for Velero strategies.
  * Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
  * Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 10:17:29 +04:00
Aleksei Sviridkin 7002397e40 test(kafka): assert that uninstall leaves no node pool or broker behind
The Kafka suites passed while the node pools of the apps they had
deleted, and the brokers those pools own, kept running into the suites
that came after. Each suite now checks, after its apps are gone, that
no KafkaNodePool and no pod labelled strimzi.io/cluster=<cluster>
remains in the namespace.

The kafka suite deletes its app explicitly so it can look afterwards.
The metadata roundtrip's cleanup does not wait for the uninstalls, so
its checks carry a budget sized from the pre-delete Job's deadline.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 07:35:32 +03:00
Aleksei Sviridkin c64fec66ff fix(kafka): delete the release's node pools on uninstall
The migration hook creates the KafkaNodePools with kubectl apply, so
they are not in the release manifest and helm uninstall leaves them.
Each pool owns the StrimziPodSet that runs its brokers, so after the
application is deleted every broker keeps running, and keeps its memory
request, with nothing left to remove it.

The pre-delete hook now deletes the pools selected by
strimzi.io/cluster=<release> once the KafkaTopic delete has finished,
including when that wait timed out: the topic operator clears topic
finalizers through the brokers, so the brokers have to outlive it. The
call does not wait, so the Job's deadline arithmetic is unchanged. An
absent CRD is not a failure. Any other failure exits non-zero: once the
uninstall completes Helm never runs the hook again, so a retry is the
chart's last chance to remove the pools.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 07:35:28 +03:00
Aleksei Sviridkin 4a2dea2d08 feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages (#4087)
## What this PR does

**Series behind #3481** — three separable pull requests:

1. **#4087 — Cluster API providers** (capmox + in-cluster IPAM)
2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on #4088.

---

This is the provisioning layer #3481 asks to specify first: the part
that
creates the VMs. Neither of the other two is needed to review it.

### What it adds

Two Cluster API providers, packaged the way the existing KubeVirt
provider is —
vendored components, `gzip -9 -n` for a reproducible artifact, a `make
components` target that regenerates it from the upstream release and
checks
the downloads against the SHA-256 sums the Makefile pins:

| Package | Version | Contract |
|---|---|---|
| `capi-providers-infraprovider-proxmox` | capmox v0.7.7 | v1beta1 |
| `capi-providers-ipam-in-cluster` | CAIP v1.0.3 | v1beta1 |

capmox v0.7.7 rather than the current v0.9.x on purpose: from v0.8 it
declares
contract `v1beta2`, which the CAPI core shipped here (v1.10.1) does not
serve.

Both join the `iaas` bundle as **opt-in** packages, where the KubeVirt
infrastructure provider is unconditional. The asymmetry is deliberate:
capmox
drives a hypervisor outside the cluster and needs a Secret holding a
Proxmox API
token, which capi-operator reads before it installs anything — with no
Secret the
provider is never deployed at all. Enabling it by default would ask
every
Cozystack install that has no Proxmox to supply hypervisor credentials.

```yaml
bundles:
  enabledPackages:
    - cozystack.capi-provider-infra-proxmox     # brings IPAM with it
    - cozystack.capi-provider-ipam-in-cluster   # also enableable on its own
```

Enabling capmox emits the IPAM provider automatically — capmox allocates
worker
addresses from an `InClusterIPPool` and has no DHCP mode, so on its own
it never
gets a `ProxmoxCluster` to `Ready`. IPAM stays independently enableable,
since
assigning addresses is a generic Cluster API concern.

Naming capmox in `enabledPackages` while disabling IPAM stops the
render. The
cluster would catch that combination on its own — the capmox
`PackageSource`
depends on the IPAM `Package`, so with that `Package` absent capmox goes
`Ready=False` with `DependenciesNotReady` and no `HelmRelease` is
created — but
it would not say why. Failing the render names the two conflicting list
entries
at the moment someone writes them, instead of leaving a condition to
decode.

### Why the earlier attempt did not produce a VM

#1927's blocker — "ProxmoxMachine does not produce a VM" — was not a
capmox
incompatibility. It was four independent causes, each of which only
became
visible once the previous one was cleared:

1. **`PROXMOX_URL` included `/api2/json`.** capmox appends that itself,
so every
   call went to `/api2/json/api2/json/…` and returned 501.
2. **No IPAM provider.** `no matches for kind "InClusterIPPool"`; the
`ProxmoxCluster` never reached `Ready` and every `ProxmoxMachine` sat in
   `WaitingForClusterInfrastructure`.
3. **`templateSelector.matchTags` is set equality, not containment**
(`slices.Equal` after sort). A template carrying four tags is not
selected by
   naming one of them: `found 0 VM templates with tags`.
4. **capmox waits for the qemu-guest-agent, which Talos does not ship.**
   `error waiting for agent: timed out` — cleared with
   `checks.skipQemuGuestAgent` and `skipCloudInitStatus`.

None of these are in the provider. They are packaging and configuration,
which
is why the earlier work could look complete and still never boot a
machine.

### Evidence

Run on Proxmox VE 9.2.11 with Talos v1.13.6 and Cozystack's CAPI stack
(core v1.10.1, CABPT v0.6.12, Kamaji control-plane provider v0.19.0):

```
$ kubectl get proxmoxmachine,machine -n tenant-proxmox
proxmoxmachine/kubernetes-px-md0-…   Ready=True   VMProvisioned=True
machine/kubernetes-px-md0-…          Running      proxmox://0ef88cc1-6aea-4563-af64-344d8060d5c0

$ qm list
106 kubernetes-px-md0-jmzds-h9npx  running
```

The machine boots Talos from the nocloud ISO, reads the machineconfig
capmox
writes, and joins. That is the layer #69 called a hard blocker.

### What this PR does not claim

- One Proxmox node, one tenant cluster, manual runs — there is no e2e
coverage
  for Proxmox in CI, and this PR does not add any.
- It does not make Proxmox a supported platform. It makes the
provisioning layer
reviewable, which #3481 names as the precondition for that conversation.
- It does not touch `capi-providers-core`. The same drift check, pointed
at that
package, shows that the `startupProbe` from #2946 never reached the
shipped
blob; that fix changes what every new install runs and is #4483, with
its
  own release note.
- Nothing here exposes `ProxmoxCluster.spec.credentialsRef` to a tenant,
and
nothing should: `capmox-manager-role` grants get/list/watch/patch on
Secrets
  cluster-wide and that field takes a namespace. Whatever renders a
`ProxmoxCluster` for a tenant in the rest of the series has to keep the
field
  out of tenant hands.

### Testing

`make unit-tests` — exit 0. New suites:
`packages/core/platform/tests/sources_capi_provider_proxmox_test.yaml`
and
`bundles_proxmox_wiring_test.yaml` (both bundle paths, the
enable/disable
precedence, and the refusal above).

Every negated `containsDocument` in the wiring suite carries `any:
true`;
without it the negation passes whether or not the document is in the
render.
Three mutations, each reverted afterwards — emitting the capmox
`Package`
unconditionally, emitting the IPAM `Package` unconditionally, and
dropping the
`disabledPackages` check from `$proxmoxEnabled` — each turn red the
tests named
after the property they break, and the unmutated chart stays green (44
suites,
198 tests).

`hack/capi-provider-components_test.bats` covers both new provider
packages:
the blob and the YAML are the same bytes; the Makefile version, the
ConfigMap
name and `spec.version` are one value — `applied-spec-hash` covers the
provider
spec and the config Secret but not the ConfigMap, so a bump that
regenerates the
files while `spec.version` stays behind installs the old provider
silently; and
the vendored files match the SHA-256 sums the Makefile pins, so the pin
holds
even when nobody runs `make components`.

The render was compared against `main` for five configurations
(defaults,
`isp-full`, `isp-full-generic` with paas, an unrelated
`enabledPackages`, and
`disabledPackages` naming IPAM). No line is removed in any of them and
the
`Package` count is unchanged; the only additions are the two
`PackageSource`
objects, which the chart renders for every package, optional or not.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/website — matches, no follow-up yet.**

- `packages/core/platform/values.yaml` changed.
`content/en/docs/next/operations/configuration/platform-package.md` is a
hand-written table of the `spec.components.platform.values.*` keys; this
PR does not add a key, but it documents two new values that
`bundles.enabledPackages` now accepts, and the table is where a reader
would look for them.
- Two platform components are added
(`capi-providers-infraprovider-proxmox`,
`capi-providers-ipam-in-cluster`) → `guides/platform-stack/_index.md`,
`operations/configuration/licenses.md`.
- The `iaas` bundle gains two optional packages →
`operations/configuration/variants.md`.

**Everything else: no match.** Nothing under `packages/apps/` or
`packages/extra/` (terraform-provider, website app lists), no
`packages/core/installer/values.yaml` or `networking.*`/`publishing.*`
key the Ansible role sets, no `hack/` file or package-layout change
(ccp, external-apps-example), no `ApplicationDefinition` semantics, no
telemetry metric, no node prerequisites (talm, ansible), no
commit-convention change (community).

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(cluster-api): add optional Cluster API packages for Proxmox — the capmox infrastructure provider and the in-cluster IPAM provider it requires. Neither is deployed unless named in `bundles.enabledPackages`, since capmox drives a hypervisor outside the cluster and does nothing without Proxmox API credentials. Enabling the Proxmox provider brings the IPAM provider with it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added optional Proxmox infrastructure provider support for IaaS
clusters.
- Added in-cluster IP address management, usable independently or
alongside Proxmox.
- Proxmox deployments require a pre-created credentials Secret and
automatically include in-cluster IPAM.
  - Added validation for incompatible Proxmox and IPAM settings.

- **Tests**
- Added coverage for provider dependencies, package enablement,
disabling behavior, invalid configurations, and packaged provider
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 01:48:39 +04:00
Aleksei Sviridkin e883f6f7fc fix(capi): ship the core provider manifests the repository actually holds (#4483)
## What this PR does

`capi-providers-core` ships the Cluster API core manifests twice: as
`files/core-components.yaml`, which a reviewer reads, and as
`files/components.gz`, which the chart mounts into the cluster. #2946
added a `startupProbe` to `capi-controller-manager` and edited only the
YAML; the blob was never regenerated, so the probe never reached a
cluster. This regenerates the blob from the YAML (`gzip -9 -n -c
files/core-components.yaml > files/components.gz`), so the decompressed
blob matches the YAML byte for byte, and the only change it carries is
the six lines of #2946.

### What this reaches, and what it does not

New installs only. capi-operator re-applies a provider when the hash of
its spec and its config Secret changes (`applied-spec-hash`); the
ConfigMap holding the components is in neither, so a cluster that
already runs `capi-providers-core` keeps its manager without the probe
until the provider's version moves. Whether existing clusters should get
the probe, and how, is left open on purpose: moving `spec.version` and
the ConfigMap name together would force the re-apply, but that is a
version decision for the package, not a side effect of regenerating a
blob.

### Also in this diff

- `.helmignore`: `files/.*-components.yaml` is a glob, so it matched
only names beginning with a dot, and `helm package` shipped the readable
YAML alongside the blob that duplicates it. Now
`files/*-components.yaml`.
- `hack/capi-core-components_test.bats` holds the two files to the same
bytes from now on; it goes red on a YAML edited without regenerating the
blob.

Found by the drift check written for the Proxmox provider packages in
#4087, pointed at the package that was already here; split out because
it changes what every new install runs and deserves its own release
note.

### Testing

`bats hack/capi-core-components_test.bats`: 1/1; red with a line
appended to the YAML, green again after `gzip -9 -n -c`. `helm package`
of the chart now contains `files/components.gz` and
`files/metadata.yaml` only.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(cluster-api): ship the `startupProbe` from #2946 in the packaged core provider components. New installs get it; a cluster that already runs the core provider keeps its manager without the probe until the provider's version moves, because capi-operator re-applies components only when the provider spec or its config Secret changes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Added a check that the compressed core components match the YAML
source, with a diff provided when they differ.
* **Chores**
* Updated chart packaging exclusions to cover component YAML files
regardless of whether their names begin with a dot.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 22:43:55 +04:00
Aleksei Sviridkin a285a5ef36 feat(kubernetes): add a proxmox substrate for managed clusters and worker pools (#4088)
## What this PR does

**Series behind #3481** — three separable pull requests:

1. #4087 — the Cluster API providers (capmox + in-cluster IPAM)
2. **#4088 — the `substrate` switch** in `kubernetes` and
`kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager

#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on this one.

---

#4087 adds the Cluster API providers; this PR teaches the two
managed-cluster
charts to point at them.

### The shape

A single value, `substrate: kubevirt | proxmox`, on both charts — not a
parallel
set of charts. The control plane is Kamaji either way; what changes is
the
infrastructure half:

| | `kubevirt` (default) | `proxmox` |
|---|---|---|
| `Cluster.infrastructureRef` | `KubevirtCluster` | `ProxmoxCluster` |
| worker template | `KubevirtMachineTemplate` | `ProxmoxMachineTemplate`
|
| apiserver Service | `ClusterIP` | `LoadBalancer` |
| tenant CCM/CSI | in-cluster (`kccm`, kubevirt-csi) | rendered off; the
Proxmox pair is #4089 |

The KubeVirt path renders byte-for-byte what it rendered before. Checked
by
rendering `kubernetes-nodes` with default values at `main` and at this
head and
diffing the output, not only by the snapshots — the snapshots cover
`nodegroup.yaml` alone, and an earlier revision of this branch changed
the
reconcile Job while they stayed green. The parent chart's default render
differs
only in the random fields of `talos-secrets`, which differ between two
renders of
`main` as well.

`osImage` (#4171) is refused on the proxmox substrate. Only half of it
could be
honoured there — `factory.version` reaches the machineconfig's installer
image
while the worker disk keeps coming from the Proxmox template — and a
field that
half-works is worse than one that says no. `talos.version` and
`talos.schematicID` still set the installer target.

### Three things that only show up once workers are off-cluster

**The apiserver endpoint.** A KubeVirt worker runs inside this cluster
and
reaches the tenant apiserver by ClusterIP. A Proxmox worker does not: it
lives on
the hypervisor's L2 and cannot route to `10.96.0.0/16`. The machine
provisions
fine, `VMProvisioned=True`, `providerID` set — and then never joins,
with no CSR
to show for it. On the proxmox substrate the Kamaji Service is therefore
a
`LoadBalancer`, and the reconcile Job reads
`.status.loadBalancer.ingress[0].ip` instead of `.spec.clusterIP`.

**And DNS, for the same reason.** The worker machineconfig carried
`nameservers: ${COREDNS_IP}` — the management cluster's CoreDNS
ClusterIP,
equally unreachable. The proxmox path takes its own resolvers
(`proxmox.dnsServers`) and the chart refuses to render without them,
rather than
writing an address that cannot answer.

**`externalManagedControlPlane` and the managed-by annotation.**
capmox's
webhook nil-derefs on a `ProxmoxCluster` whose control-plane endpoint is
empty
unless `spec.externalManagedControlPlane` is set, which is correct here
— Kamaji
owns the endpoint. Separately, the `cluster.x-k8s.io/managed-by: kamaji`
annotation that the KubeVirt path carries must **not** be set on the
Proxmox
object: with it, CAPI never sets an ownerRef and capmox stops at
`Waiting for Cluster Controller to set OwnerRef`, leaving the address
pool
uncreated and every worker waiting on infrastructure.

### Linked clones by default

`proxmox.full: false` — worker disks are linked clones of the template
rather
than full copies. Measured on a ZFS-backed store: **8 KiB per worker
instead of
10.2 GiB**. Operators who want independent disks set `full: true`; the
value is
plumbed straight through to `ProxmoxMachineTemplate`.

`proxmox.storage` is a full-clone parameter and is refused with a linked
clone:
a linked clone always lives on the template's storage, and both Proxmox
(`parameter 'storage' not allowed for linked clones`) and capmox's own
CRD
(`Must set full=true when specifying storage`) reject the pair. The
chart says
so at render time, naming both values, instead of failing at apply time
with an
error about a `ProxmoxMachineTemplate` nobody wrote.

### Sizing

`ProxmoxMachineTemplate` takes `numCores` and `memoryMiB`, not
Kubernetes
quantities, so the pool's `resources.cpu`/`memory` are converted
(millicores →
cores, bytes → MiB, both rounded up). The rounding is pinned in
`tests/substrate_branches_test.yaml` with `1500m` and `8G`, where
rounding up and
down disagree; `tests/substrate_proxmox_sizing_test.yaml` covers the
refusal to
size a pool that gives neither.

### Known upstream behaviour this surfaces

`TalosConfigTemplate.spec` is immutable by schema, so the reconcile Job
cannot
rewrite an endpoint it has already published — `kubectl apply` fails
with
`TalosConfigTemplate.Spec is immutable`, and the template has to be
deleted for
the Job to recreate it. A fresh install never meets it; it appears when
the
apiserver address changes on an existing cluster. Handling it changes
the
KubeVirt path too, so it is not in this PR: it is a separate change on
`proxmox/tct-replace`, with its own upgrade note.

### Evidence

```
$ KUBECONFIG=<tenant> kubectl get nodes -o wide
kubernetes-px-md0-jmzds-ndnl6  Ready  <none>  75s  v1.35.6  10.0.0.160  Talos (v1.13.6)

$ KUBECONFIG=<tenant> kubectl get pods -A
cozy-cilium  cilium-…            Running
kube-system  konnectivity-agent  Running
kube-system  coredns-…           Running
```

No management-cluster address remains in the rendered
`TalosConfigTemplate`.

### Testing

`make unit-tests` — exit 0. `apps/kubernetes` 26 suites / 314 tests;
`apps/kubernetes-nodes` 29 suites / 160 tests / 12 snapshots, all
unchanged on
the default path. Each of the ten commits passes both suites on its own.

Every substrate branch is asserted on both sides, and every assert goes
red on
the mutation it exists for: inverting the `SVC_IP` condition, inverting
the
`nameservers` condition, `ceil`→`floor` in the sizing, removing the
proxmox
gate from each of the kccm and csi templates (the two RBAC bindings
included),
pinning the retention lookup back to `KubevirtMachineTemplate`, widening
the
IPv6 class of the `dnsServers` validation, dropping the parent chart's
`dnsServers` validation, the storage-with-linked-clone refusal, and the
`osImage` refusal. Negated `containsDocument` asserts carry `any: true`.

`hack/talos-reconcile-heredoc_test.bats` additionally runs the rendered
heredoc
through a real shell: a `dnsServers` entry with a substitution is
refused and
named, an IPv6-shaped one with a substitution is refused too, and a real
address survives the shell unchanged.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.

**cozystack/terraform-provider-cozystack — matches, no follow-up yet.**

- `packages/apps/kubernetes/values.schema.json` and
`packages/apps/kubernetes-nodes/values.schema.json` both gain fields:
`substrate`, and a `proxmox` object (`allowedNodes`, `dnsServers`,
`ipv4Config`, and on the pool side `templateTags`, `network`, `storage`,
`full`). The provider hand-maintains a schema, a model and an
expand/flatten pair per Kind, with no codegen from these files.
- `values.yaml` gains defaults for those fields (`substrate: kubevirt`,
`proxmox.full: false`, `prefix: 24`). The provider restates defaults it
models and sends its own, so a stale one wins silently rather than
showing as a diff.

**Everything else: no match.** No app added, renamed or removed (website
app lists); no version enum, `kind`, `plural`, `release.prefix` or
output Secret/Service name changed; `packages/core/platform/values.yaml`
untouched; the `*-rd` files are regenerated, not renamed (ccp); no
`hack/` change; no node prerequisites (talm, ansible); no telemetry
metric; no commit-convention change (community).

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubernetes): add a `substrate` value to the `kubernetes` and `kubernetes-nodes` applications, selecting whether worker VMs run on KubeVirt in this cluster (`kubevirt`, the default and unchanged) or on an external Proxmox VE cluster through capmox (`proxmox`). On the Proxmox substrate the tenant apiserver is published through a LoadBalancer and worker disks default to linked clones.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for choosing KubeVirt or Proxmox for Kubernetes clusters
and node pools.
* Added Proxmox configuration for nodes, networking, storage, DNS,
templates, cloning, and IPv4 allocation.
* Proxmox deployments now use the appropriate cluster, machine,
networking, and control-plane resources.

* **Validation**
* Added checks for supported substrates, required Proxmox settings,
sizing, and incompatible options.

* **Documentation & Tests**
* Documented the new settings and expanded coverage for rendering,
validation, cloning, sizing, and template replacement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 22:09:23 +04:00
Aleksei Sviridkin 09d02fdc6c docs(apps): correct the shared resources and preset paragraphs (#4394)
## What this PR does

Corrects two sentences of the boilerplate shared by the app READMEs,
both raised in the review of #3936.

- "When left empty, the preset defined in `resourcesPreset` is applied"
describes an all-or-nothing rule, but
`cozy-lib.resources.defaultingSanitize` merges per key: every resource
`resources` leaves unset is taken from the preset. Twelve READMEs
carried the sentence.
- The legacy flat names were called aliases of their "1:1 instance-type
equivalents", which reads as the `c1` series listed just before it.
`nano` through `small` equal the `t1` sizes, and `medium` equals
`c1.small` rather than `c1.medium`, the trap
`docs/operations/resource-presets.md` already warns about. Ten READMEs
carried the sentence.

Only the hand-written section changes. `make generate` for all twelve
packages leaves the diff at exactly those 22 lines.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: README prose only, no values,
schema, default or package change. The website regenerates the app
reference pages from these READMEs on release, so nothing to open there.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified how resource settings work across several app READMEs: any
unset resource field now inherits its value from the preset, rather than
requiring the entire resource block to be empty.
* Updated legacy preset name guidance to explain that deprecated flat
names keep their original sizes, including `nano`–`small` matching `t1`
sizes and `medium` matching `c1.small`.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 20:17:41 +04:00
Aleksei Sviridkin b16edec3f3 fix(tcp-balancer): rename backends and pin HAProxy so the chart starts on 3.3+ (#4366)
## What this PR does

`haproxy:latest` now resolves to 3.3 and newer. Since 3.3 a backend
named like a frontend is a fatal config error, so every tcp-balancer pod
fails at config parse (#1966). We hit it on Cozystack 1.6.1 with
`haproxy:3.4.4` behind the tag.

- backends get a `_backend` suffix and every `default_backend` follows;
the rename is the one from #2321 by @officialasishkumar
- the image is `haproxy:{{ .Chart.AppVersion }}` with `appVersion:
3.4.4`, the current 3.4 LTS tag
- the whitelist `acl`, `accept` and `reject` lines of each frontend
render inside one guard
- `tests/` holds helm-unittest suites, and the package `Makefile` gets
the `test` target the sweep looks for
- the `whitelistHTTP` description says it needs a non-empty `whitelist`

Two inputs change behaviour on upgrade. On `main` both fail at config
parse with `no such ACL : 'whitelist'` on every HAProxy version, because
the `acl` line sits inside the `with` and `accept if whitelist` does
not:

- `whitelistHTTP: true` with an empty `whitelist` now fails the render
with `whitelistHTTP is enabled but whitelist is empty`. Its HelmRelease
upgrade fails and the release stays on its old revision, which is
already crash-looping.
- `kubernetesAndTalos` with an empty `whitelist` now starts, and ports
6443 and 50000 accept any source. With `external: true` the Service is a
LoadBalancer and carries both ports. `whitelist` is the only switch
those frontends have, so an empty list means no filtering.

Every other input renders the same config apart from the backend names.
The image and the `checksum/config` annotation change, so the Deployment
rolls, and for releases already crash-looping on `latest` the rollout is
the fix.

Validation: renders of `main` and of this branch for the default,
whitelist and empty-whitelist inputs were checked with `haproxy -c` on
2.9.7 and 3.4.4. `helm unittest .` passes 9/9, and removing the `reject`
or `accept` line from any single frontend turns a test red.

Fixes #1966

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: chart templates, `Chart.yaml`,
the package `Makefile`, `tests/`, and a description-only change in
`values.yaml` with its generated carriers. No field is added, removed or
renamed and no default changes, so nothing in the map fires.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tcp-balancer): rename HAProxy backends and pin the image to the chart appVersion so the balancer starts on HAProxy 3.3+. whitelistHTTP with an empty whitelist now fails the render; kubernetesAndTalos with an empty whitelist leaves ports 6443 and 50000 unfiltered.
```
2026-09-25 19:29:39 +04:00
Marian Koreniuk 8722462066 fix(kubernetes-nodes): refuse osImage on proxmox instead of honouring half of it
#4171 gave a worker pool an osImage, choosing where its system disk comes from:
a CDI clone of a shared golden image, or an HTTP import from an Image Factory.
A Proxmox worker's disk comes from neither. capmox clones a VM template that
already carries Talos, and nothing in this chart can change what that template
holds.

Half of the field would still appear to work, which is why it is refused rather
than ignored. osImage.factory.version reaches the machineconfig's installer
image and changes the reconcile Job, so a pool could set it, watch the Job
roll, and end up running one Talos while booting another: v1.12.6 moves the
installer to .../installer/<schematic>:v1.12.6 and leaves the
ProxmoxMachineTemplate untouched. osImage.builtin has no meaning here at all.

Nothing is lost by refusing it: the installer target on this substrate is
settable through talos.version and talos.schematicID, which do reach it.

The branches suite gains the refusal case. The substrate parameter's description said changing the value "rolls the
pool". The chart refuses the change on a live pool, and the description says
so; it reaches the README, the values schema and the CRD.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 4bc6621a5c test(kubernetes,kubernetes-nodes): assert both sides of every substrate branch
Every `if substrate` in both charts is asserted on both sides, so an inverted
condition cannot pass:

- the reconcile Job reads SVC_IP from the LoadBalancer status on proxmox and
  from the ClusterIP on kubevirt;
- the machineconfig carries the pool's nameservers on proxmox and none on
  kubevirt;
- the proxmox gate on each kubevirt-only template in the parent chart, the two
  RBAC bindings included, so a proxmox cluster renders no binding to a Role,
  ClusterRole or ServiceAccount it does not render;
- the sizing conversion at 1500m and 8G, where ceil and floor disagree
  (2 cores and 7630 MiB against 1 and 7629);
- the old-template retention loop keeps a ProxmoxMachineTemplate a MachineSet
  still references;
- the proxmox worker monitor carries no Pod threshold while the kubevirt one
  does;
- a live cluster and a live pool refuse a substrate switch while a first install
  still renders;
- both charts refuse a dnsServers entry that is not an address, the parent
  chart included, where the value reaches ProxmoxCluster.spec.dnsServers.

The negations carry `any: true`: without it a negated containsDocument passes
whether or not the document is in the render.

hack/talos-reconcile-heredoc_test.bats gains the dnsServers cases: the render
refuses a command substitution and names the entry; an entry with a colon in it
is refused too, which pins the IPv6 branch of the validation and not only the
IPv4 one; and a real address survives the Job's shell unchanged. The heredoc is
extracted by shape — `cat <<EOF` at the start of a line or right after `=$(` /
`="$(` — so the same case keeps working whichever way the Job writes it.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 517baf945d fix(kubernetes,kubernetes-nodes): validate dnsServers, refuse a substrate switch, drop the Pod threshold on proxmox
Three faults in the proxmox substrate, none of which the render refused.

1. proxmox.dnsServers reached the reconcile Job's shell unescaped.

The Job applies the worker TalosConfigTemplate through an unquoted heredoc, so
every byte of its body is subject to parameter expansion and command
substitution at Job runtime. The INVARIANT comment above that block says each
tenant-controlled value must be shell-escaped or pattern-validated; the new
nameserver list was neither. `quote` there is YAML quoting.

Rendered with `$(id -u)` and a backtick pair in dnsServers, extracted the way
hack/talos-reconcile-heredoc_test.bats does and run under sh, the shell executed
both and put their output in the worker's resolver list. The Job's pod holds the
parent cluster's Talos token, cluster id, cluster secret and bootstrap token in
its environment, and its Role can read the Talos CA and patch the parent
KamajiControlPlane.

Each entry is now validated as an IPv4 or IPv6 address at render time, in both
charts. Validation rather than escaping because Talos takes addresses here: an
address cannot carry a metacharacter, so the property holds by construction, and
the ordinary mistake of writing a hostname is caught in the same move. IPv6 is
matched on its character set rather than its full grammar — what matters is which
bytes may reach the shell, and the apiserver rejects a malformed address anyway.

2. Nothing stopped a live cluster or pool from being switched between substrates.

`substrate` had no immutability of any kind, and the `## @immutable` marker this
adds is dashboard-enforced only, so both charts now read the live object.

Switching a cluster removes the KubevirtCluster, the cloud-controller-manager,
the kubevirt-csi controller and the in-tenant -csi HelmRelease with all their
RBAC from the render. Flux then uninstalls the CSI driver behind every PVC the
tenant has. CAPI does not prevent it — its Cluster webhook forbids removing
spec.infrastructureRef but admits a change of kind.

Switching a pool rolls every worker to the other hypervisor and drops the
machine template the draining MachineSet still references, because the retention
loop looks up only the kind the new substrate uses. A pool whose substrate
differs from its parent's was not rejected either, though the README says the
two must match.

Both guards are inert offline, like assertNoForeignPool and assertParentVersion
beside them: with no apiserver the lookup is empty and a first install renders.

3. The worker WorkloadMonitor reported a proxmox pool as broken.

The controller lists Pods matching the monitor's selector and sets
Operational=false while fewer than minReplicas are ready. On kubevirt those Pods
are the virt-launcher Pods. A proxmox worker is a VM on another machine with no
Pod here, so the count is always zero and any threshold pinned the pool — and
through the lineage labels its whole application — to WorkloadsReady=False while
every node was healthy. The monitor still renders for the dashboard and the
lineage; what it no longer carries on proxmox is a threshold counted in Pods.

Two smaller things. The wait-loop log line is written per substrate, so a
default kubevirt render of kubernetes-nodes is byte-identical to main and no
existing pool gets a new reconcile Job on upgrade. And the comment claiming a
rename of $kubevirtmachinetemplate would roll every live worker VM is replaced:
the hash is taken over the rendered template, not the variable name, so renaming
it leaves the render byte-identical.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk 33e1de52d2 test(kubernetes-nodes): pin the linked-clone default and three silent guards
Nothing failed when `full` was wrong: the pool came up, the workers joined, and
each one quietly cost the whole template disk instead of the increment — 10.2
GiB against 8 KiB on a ZFS store. A default whose only symptom is disk usage is
exactly the kind that drifts, so both it and its override are now asserted.

The bridge and dnsServers guards were two of the guards on this path with no
coverage. Both fire on absence, so the suite omits each and the tests supply the
other one back; declaring either at suite level would make it impossible to take
away, which is why the sizing guard already lives alone.

The storage guard is the third: `storage` with a linked clone is refused and the
message names both values, while `storage` with `full: true` reaches the
ProxmoxMachineTemplate.

Each assert goes red on the edit it exists for: flipping the default reds the
clone suite, deleting a guard reds the guard suite.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk ceb7119a1b fix(api): regenerate deepcopy for the proxmox value structs
The proxmox blocks added to the Kubernetes and KubernetesNodes values carry
string slices, and the generated deepcopy never learned about them: a copy
shared allowedNodes, dnsServers and templateTags with its original instead of
owning them.

Cause was procedural rather than conceptual — the per-package `make generate`
under packages/apps regenerates the chart schema and the Go types, but the
deepcopy functions come from the repository-root target.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk c5996c1d0f fix(kubernetes,kubernetes-nodes): give Proxmox workers an endpoint they can reach
A worker on the proxmox substrate is a VM on the hypervisor network, not a Pod
in this cluster. It was handed two addresses only an in-cluster consumer can
route to: the apiserver as a ClusterIP, and the management cluster's CoreDNS
ClusterIP as its resolver. The VM booted, the Machine reported Ready with a
providerID, the disk was where it should be — and no CSR ever arrived. Nothing
in that picture points at an address.

The control-plane chart now publishes the tenant apiserver as a LoadBalancer on
the proxmox path, putting the endpoint on the network the workers share, and
the pool's reconcile Job reads the Service's LoadBalancer address instead of its
ClusterIP for the host entry the workers dial. The nameservers follow the same
reasoning: proxmox pools carry their own resolvers rather than pointing at a
CoreDNS they cannot reach, and the chart refuses to render a pool that has not
named any.

KubeVirt keeps the ClusterIP on both counts. Its workers are in this cluster,
where a ClusterIP is the right answer and a LoadBalancer address would be an
unnecessary one.

Verified end to end on a real tenant cluster: the node joins, its CSR is
approved, and cilium, konnectivity and coredns come up in the tenant.

One operational note this surfaced, not caused by it: TalosConfigTemplate.spec
is immutable, so the reconcile Job cannot rewrite an endpoint that has already
been published — the template has to be deleted for the Job to recreate it.
That matters only when an existing cluster's apiserver address changes.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:15 +02:00
Marian Koreniuk d78af928fc feat(kubernetes-nodes): default Proxmox worker disks to linked clones
A full clone on a ZFS pool configured with `sparse 0` reserves the whole
volsize the moment the VM is created: 10.2G for a worker whose Talos image
occupies 337M. A linked clone writes only its own increment — measured at 8K
for the same worker on the same host — because it shares the template's base
snapshot until it diverges.

The cost is that the clone holds that snapshot, so the template cannot be
rotated while any linked clone still references it. That is bounded rather
than permanent: consolidation detaches each worker 24 hours after creation, so
template rotation is blocked for at most a day.

Consolidation is not a chart concern, but two of its constraints are worth
recording where the flag is set, because both were discovered by testing on a
live host rather than from the docs. A linked clone can only be detached by
moving its disks to a *different* storage — Proxmox refuses a same-storage
move outright. And both disks must move: leaving efidisk0 behind keeps the
template pinned just as effectively as the system disk would.

Set full: true for a pool that must be independent of the template from the
first second.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:14 +02:00
Marian Koreniuk 135eb6bac4 fix(kubernetes): make the Proxmox cluster object reconcilable
Two corrections found by building a real tenant cluster on Proxmox rather than
by rendering the chart. Each produced a symptom that pointed away from its
cause.

capmox skips control-plane endpoint validation only when
spec.externalManagedControlPlane is set. Kamaji owns the endpoint here, so the
field was semantically true all along and merely absent — and on an absent
controlPlaneEndpoint the webhook does not return a validation error, it panics
(internal/webhook/proxmoxcluster_webhook.go:105). The install failed with a
nil-pointer stack trace out of an admission webhook, which reads like a broken
provider rather than a missing field.

The cluster.x-k8s.io/managed-by annotation now renders only on the KubeVirt
path. It tells Cluster API the infrastructure object is owned by something
outside CAPI, so the Cluster controller never sets an ownerRef on it — and
capmox's reconciler exits at "Waiting for Cluster Controller to set OwnerRef on
ProxmoxCluster" without ever creating the InClusterIPPool its workers draw
addresses from. Harmless on KubevirtCluster, which nothing reconciles;
load-bearing on ProxmoxCluster, which capmox must.

With both in place the object reaches Ready=True, the pool is created, and the
Cluster reaches Provisioned.

The proxmox suite's ProxmoxCluster case asserts both: the field is set and
the annotation is absent.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:46:14 +02:00
Marian Koreniuk db839d1198 feat(kubernetes): add a proxmox substrate for managed clusters
Companion to the same switch in kubernetes-nodes. The control plane is Kamaji
on either substrate, so this changes exactly one object — the one
Cluster.infrastructureRef points at — plus the two tenant addons that exist
only because the workers are KubeVirt VMs living in this cluster.

A KubevirtCluster needs no spec: its workers are in the same cluster and reach
the apiserver over the pod network. A ProxmoxCluster addresses an external
hypervisor, so it carries what capmox cannot infer — which Proxmox nodes may
host VMs, which addresses they may take, and who resolves names for them.
controlPlaneEndpoint stays absent on both: the cluster.x-k8s.io/managed-by
annotation hands that to Kamaji.

The address pool is required rather than defaulted. capmox has no DHCP mode; it
allocates from an InClusterIPPool built out of ProxmoxCluster.spec.ipv4Config,
so an unset pool is not a smaller configuration but a ProxmoxCluster that never
reaches Ready and worker Machines that wait on it forever. Refusing at render
time turns that into a message instead of a silence.

The KubeVirt cloud-controller-manager, its cloud-config and the kubevirt-csi
driver are gated off on the proxmox path. They reach the workers'
VirtualMachines and DataVolumes in this cluster; on Proxmox those objects do
not exist, and leaving them rendered would put controllers in the tenant
reconciling nothing. Their Proxmox equivalents are separate packages and land
separately.

The default path is unchanged: rendering it before and after this commit
differs only in the per-render random bootstrap tokens, and all 19 existing
suites pass untouched.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:29:09 +02:00
Marian Koreniuk 85f576756b feat(kubernetes-nodes): add a proxmox substrate for worker pools
Worker pools are already Talos machines created by Cluster API; only the
substrate underneath them is fixed to KubeVirt. This adds `substrate:
kubevirt|proxmox` and swaps exactly two things: the machine template object
and every reference to it. Autoscaler capacity annotations, kubelet
reservations, MachineDeployment, MachineHealthCheck, WorkloadMonitor and the
talos-reconcile Job are substrate-agnostic and rendered once, which is why
this is a branch in the existing chart rather than a second copy of it.

The default path is unchanged down to the bytes: the render snapshot and all
15 existing suites pass untouched, and the hash-named template keeps its
variable name so no live worker VM rolls on upgrade.

Five things the Proxmox path must do differently, each learned from a
failure that pointed somewhere else:

- Skip capmox's guest-agent and cloud-init-status probes. Talos ships
  neither, and left on they leave the VM booted and the Machine stuck on
  "error waiting for agent: the operation has timed out". This is the same
  class of problem `virtualMachineBootstrapCheck.checkStrategy: none` solves
  for CAPK's SSH probe; capmox's CRD names TalOS in the skipCloudInitStatus
  description.

- Clone a template carrying the nocloud image, not openstack. capmox injects
  bootstrap data as a nocloud ISO, and an openstack image ignores it exactly
  as silently as a nocloud image ignores the configdrive the KubeVirt path
  depends on.

- Pass templateTags through verbatim. capmox compares the template's tag set
  with slices.Equal after sorting, so a selector listing a subset matches
  nothing and reports "found 0 VM templates with tags ..." while the tags are
  plainly present.

- Require resources.cpu and resources.memory. Proxmox has no
  VirtualMachineClusterInstancetype, so the lookup that normally supplies
  effective CPU and memory has nothing to read — and those two numbers feed
  the kubelet reservations and the autoscaler capacity annotations. A pool
  that cannot state them would be scheduled against invented capacity, so it
  is refused at render time. KubeVirt-only knobs (gpus, logSerialConsole,
  podCpuLimit/Request) are refused too rather than accepted and ignored.

- Refuse proxmox.storage unless full is true. A target storage is a full-clone
  parameter: a linked clone always lives on the template's storage, and both
  Proxmox ("parameter 'storage' not allowed for linked clones") and capmox's
  own CRD ("Must set full=true when specifying storage") reject the pair.
  Left to them, a pool with storage set and a linked clone fails at apply time
  with an error about a ProxmoxMachineTemplate the operator never wrote;
  failing the render names the two values instead.

The Talos image is not a chart input on this path: capmox clones a template
that already exists on the hypervisor, so building and rotating that template
is an operator responsibility with no equivalent on the KubeVirt side.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:29:08 +02:00
Marian Koreniuk 9934014908 fix(capi): ship the core provider manifests the repository actually holds
#2946 added a startupProbe to capi-controller-manager so it survives the
certificate provisioning delay. It edited
capi-providers-core/files/core-components.yaml, which is the copy a reviewer
reads. The chart mounts files/components.gz, and that blob was never
regenerated, so the probe never reached a cluster.

The blob is regenerated from the YAML it is supposed to carry:

  gzip -9 -n -c files/core-components.yaml > files/components.gz

The decompressed blob now matches the YAML byte for byte, and the only change
it carries is the six lines of #2946.

This reaches new installs only. capi-operator re-applies a provider when the
hash of its spec and its config Secret changes; the ConfigMap holding the
components is in neither, so a cluster that already runs capi-providers-core
keeps its manager without the probe until the provider's version moves.
Whether existing clusters should get the probe, and how, is left open here on
purpose: moving spec.version and the ConfigMap name together would force the
re-apply, but that is a version decision for the package, not a side effect of
regenerating a blob.

.helmignore is corrected in the same change: `files/.*-components.yaml` is a
glob, so it matched only names beginning with a dot, and `helm package`
shipped the readable YAML alongside the blob that duplicates it.

hack/capi-core-components_test.bats holds the two files to the same bytes from
now on, so the next edit of one without the other fails in CI.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:26:13 +02:00
Marian Koreniuk 7843ec98b5 test(capi): catch a vendored provider whose blob and YAML have drifted
Each provider package ships the manifests twice: the YAML a reviewer reads and
the gzip blob the chart mounts. `make components` writes both from one download
and nothing keeps them together afterwards, so editing the YAML alone produces a
diff that reviews cleanly while the cluster runs the version nobody read.

Both new provider packages are covered. A second pair of tests holds the
version together: capi-operator hashes the provider spec and the config Secret
into applied-spec-hash, and the ConfigMap is in neither, so a bump that
regenerates the vendored files while the ConfigMap name and spec.version stay
behind installs the old provider and reports the old version, with nothing
reporting an error. The Makefile variable, the ConfigMap name and spec.version
have to be one value. A third pair checks the vendored files against the
SHA-256 sums the Makefile pins, so the pin holds even when nobody runs
`make components`.

Written for dash rather than bash, since the repo's runner executes these under
sh — process substitution is not available there.

Each pair goes red on the edit it exists for: a YAML edited without
regenerating the blob, a Makefile version bumped on its own, a ConfigMap
renamed without spec.version, and a vendored file that no longer matches its
pinned sum.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:26:05 +02:00
Marian Koreniuk 18b83a72b3 feat(platform): make the Proxmox substrate installable from a bundle
Until now the two Cluster API packages behind the Proxmox substrate existed but
no bundle emitted them, so a Cozystack install shipped neither and the substrate
was reachable only by applying Package CRs by hand.

They join the iaas bundle as opt-in packages, where the KubeVirt infrastructure
provider is unconditional. The asymmetry is the point: capmox drives a
hypervisor outside this cluster and needs a Secret holding a Proxmox API token,
which capi-operator reads before it installs anything — with no Secret the
provider is never deployed at all. Enabling it by default would ask every
Cozystack install that has no Proxmox to supply hypervisor credentials.

Enabling capmox brings the in-cluster IPAM provider with it — capmox allocates
worker addresses from an InClusterIPPool and has no DHCP mode, so on its own it
never gets a ProxmoxCluster to Ready. IPAM stays independently enableable,
since assigning addresses is a generic Cluster API concern rather than a Proxmox
one.

Naming capmox in enabledPackages while disabling IPAM stops the render. The
cluster would catch that combination on its own: the capmox PackageSource
depends on the IPAM Package, so with that Package absent capmox goes
Ready=False with DependenciesNotReady and no HelmRelease is ever created. What
it would not do is say why. Failing the render names the two conflicting list
entries at the moment someone writes them, instead of leaving a condition to
decode later.

The negated containsDocument asserts in the wiring suite carry `any: true`.
Without it a negated containsDocument passes whether the document is in the
render or not, so the two properties the suite exists for — off by default,
and disabledPackages winning — could not fail.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:25:00 +02:00
Marian Koreniuk 67db200530 feat(cluster-api): add Proxmox infrastructure provider package
Packages ionos-cloud/cluster-api-provider-proxmox (capmox) the same way
capi-providers-infraprovider packages CAPK: vendored components gzipped into
a ConfigMap, an InfrastructureProvider that fetches them by label.

Two details differ from the KubeVirt package on purpose.

The fetchConfig label is `infraprovider-provider: proxmox`, not the shared
`infraprovider-components: cozy`. A shared label would put both providers in
one match. capi-operator would still install the right one — it keys every
matched ConfigMap by its name as the version and takes the one equal to
spec.version — but it builds that repository in a single pass and returns an
error for the whole list as soon as one match is malformed: a name that does
not parse as semver, or missing metadata. Sharing the label would let a broken
ConfigMap belonging to one provider stop the other from being installed. The
bootstrap packages already separate on a `bootstrap-provider` key; this
follows that.

components.gz is produced with `gzip -n`. Without it the header carries a
timestamp, so regenerating from identical input yields a different blob and
the committed artifact appears to change when nothing has. `make components`
records the version pin, the source URLs and the flag, so refreshing the
vendored manifests is reproducible rather than remembered. It also checks
both downloads against the SHA-256 sums recorded in the Makefile before
packaging them: a release tag is mutable, so the pin is on the bytes, and
bumping CAPMOX_VERSION means recording the new sums next to it.

Pinned to the v0.7 series: from v0.8 capmox declares the v1beta2 Cluster API
contract, while capi-providers-core here is v1.10.1, which serves v1beta1.

The package source depends on the in-cluster IPAM provider and deliberately
not on cozystack.kubevirt: the KubeVirt provider needs it because its VMs run
inside this cluster, whereas a Proxmox hypervisor is external and reached
over its API with the credentials named by `credentialsSecretName`.

That Secret is a precondition rather than an optional extra, and the chart
does not create it — a hypervisor token belongs to the platform's secret
management, not to a Helm release. capi-operator reads the Secret before it
installs the provider, so while it is missing nothing is deployed at all.
values.yaml carries what an operator needs at that point: that PROXMOX_URL is
the bare https://<host>:8006 — capmox appends /api2/json itself, and a URL
that already carries it sends every call to /api2/json/api2/json/... and gets
501 back — the full user@realm!tokenid form of PROXMOX_TOKEN, and the fact
that a rotated token reaches the manager only after the next provider
reconcile and a restart of its pod.

Both edges are asserted in tests/sources_capi_provider_proxmox_test.yaml;
losing the IPAM edge reproduces a failure mode whose symptom points nowhere
near its cause.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:23:45 +02:00
Marian Koreniuk 86834d30a9 feat(cluster-api): add in-cluster IPAM provider package
Cluster API ships address management as a separate provider, and Cozystack
has never needed one: KubeVirt workers lease their addresses from the
cluster network's DHCP. An infrastructure provider that allocates addresses
instead of leasing them cannot work without it.

capmox is such a provider. It builds an InClusterIPPool out of
ProxmoxCluster.spec.ipv4Config and hands every VM a static address from it.
With no IPAM provider installed, the pool kind does not resolve:

  Reconciler error: no matches for kind "InClusterIPPool"
                    in version "ipam.cluster.x-k8s.io/v1alpha2"

ProxmoxCluster then never reaches Ready and every ProxmoxMachine sits in
WaitingForClusterInfrastructure indefinitely — indistinguishable, from the
outside, from a provider that is simply broken.

Pinned to the v1.0 series rather than v1.1.0: both serve InClusterIPPool
v1alpha2, but v1.0 declares the v1beta1 contract that matches
capi-providers-core v1.10.1.

`make components` checks both downloads against the SHA-256 sums recorded
in the Makefile before packaging them. A release tag is mutable, so the pin
is on the bytes rather than on the tag; bumping CAIP_VERSION means recording
the new sums next to it.

The package carries no edge to any infrastructure provider. The dependency
belongs on the consumer's side, which keeps this reusable for any future
provider with the same requirement.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-25 16:23:45 +02:00
scooby87 cffe509dcf feat(monitoring): per-tenant OTLP collector (tracing phase 2) (#3763)
## What this PR does

Phase 2 of distributed tracing (design cozystack/community#38, epic
#3761), built on the phase-1 backend (#3762, merged): a per-tenant
**OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It
lives in the tenant namespace (like a per-tenant vmagent), so
app→collector traffic is intra-namespace and never crosses the tenant
boundary. No OTel operator exists in the platform, so this is a plain
Deployment + ConfigMap + Service (the fluent-bit pattern).

## Changes

- **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`,
HTTP `:4318`) → `memory_limiter` + head sampling
(`probabilistic_sampler`, default 10%) + optional `attributes/redact`
(PII/secret control point) + `batch` → `otlphttp` exporter to the
`tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428`
single, endpoints per upstream docs). PSS-restricted securityContext,
digest-pinned image via `cozy-lib.image`, config-checksum roll.
- **app-facing Service `otel-traces`** (4317/4318).
- **`tracingCollector` values** — `enabled` / `replicas` /
`samplingPercentage` (schema-bounded 0-100) / `redactAttributes` /
`resources`. A **fail-guard** rejects an enabled collector fronting more
than one `tracingStorages` backend (it exports to a single backend by
design).
- **`otel-traces` WorkloadMonitor** and dashboard resource-map entry,
rendered under the collector's own gate. The same resource-map Role also
gets the pre-existing `alermanager` typo corrected to `alertmanager`, so
the alertmanager WorkloadMonitor is readable from the dashboard.
- helm-unittest coverage for the collector and its WorkloadMonitor;
regenerated `values.schema.json` / `README.md` / `monitoring-rd`
ApplicationDefinition.

## Verification

- `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all
pass.
- `helm template` renders cleanly; `make generate` idempotent.
- Live check on a test cluster (single-mode backend): the collector
starts with `redactAttributes` containing a numeric, a colon-bearing and
an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted
and is queryable from the backend through the Jaeger API, with the
redacted attributes removed and the others kept.
- Local `/branch-review` — **LGTM** (one blocker + four recommendations
fixed, each with a test: multi-backend guard, memory_limiter cgroup
limit, configurable replicas, sampling bounds, trust-boundary wording).

## Scope / deferred (epic #3761)

`AccountID`/`ProjectID` header injection for the shared-central topology
(phase 4) and engine integrations (phase 3) are follow-up PRs.

### Screenshots

N/A, no UI change.

### Downstream repositories

The diff touches only `packages/extra/monitoring`,
`packages/system/monitoring` and `packages/system/monitoring-rd`. The
website's monitoring reference page is regenerated from the package
README by the release bot, and the Terraform provider models only the
tenant's `monitoring` flag, not these values, so no follow-up is needed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage in cluster or single-instance modes,
with retention settings and Grafana datasources.
* Added an enabled-by-default OpenTelemetry collector with configurable
replicas, sampling, attribute redaction, and resource limits.
* Added workload monitoring and automatic resource recommendations for
configured trace components.
  * Added readiness health checks for both tracing backend modes.

* **Bug Fixes**
* Corrected dashboard permissions for alert monitoring and trace
collector resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 16:27:45 +03:00
Aleksei Sviridkin 368d00b5d4 test(e2e): remove unused monitoring suites and audit BATS runners (#4019)
## What this PR does

I removed the two monitoring OIDC BATS suites that had no runner after
the Chainsaw migration, and brought the runner audit and wrapper OIDC
test from [#3778](https://github.com/cozystack/cozystack/pull/3778).

Also added checks for audience mapper and `OIDC_MODE=System`. The named
`KeycloakRealmGroup` test documents coverage the document-count and kind
assertions already provided.

Runner audit reports unused BATS suites, missing files in runner paths,
and unresolved Make variables (including `$(E2E_SUITE)`).

Live Monitoring API admission, Grafana operator reconciliation and mode
transitions are not tested here.

### Validation

- `make -j4 unit-tests SHELL=/usr/bin/dash` passed.
- Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart
tests passed.
- Runner audit: 12 cases passed under dash and Bats 1.14.0. The full
shell syntax and BATS sweep passed under dash too.
- Three chart mutations and seven runner mutations failed as expected.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```
2026-09-25 16:55:05 +04:00
scooby87 03e2da5805 feat(postgres,mariadb)!: drop plaintext user passwords (#4078)
## What this PR does

Removes the plaintext `users[].password` field from the `apps/postgres`
and `apps/mariadb` charts (and the mariadb `root` account), so
managed-database passwords can no longer be pinned in tenant values or
committed to git. Passwords are always chart-generated into the
`<release>-credentials` Secret and preserved across reconciles via
`lookup`, matching the model `redis`/`valkey` already use. This closes
the "User passwords" item of the portal upstream epic
(aenix-org/cozyportal#1060).

The generated password never lands in the init-script Secret or the SQL
it runs: the postgres init-job reads each user's password from the
mounted `<release>-credentials` Secret at run time and applies it
through a psql variable (`ALTER ROLE … WITH PASSWORD :'pw'`), so it does
not linger in the init-script manifest a tenant can read. It does live
in cleartext `stringData` in the `<release>-credentials` Secret itself,
which is a chart manifest and therefore kept in Helm release history for
`MaxHistory` revisions like any chart-generated secret — that residue is
exactly why rotation cannot be built on the chart (see below). A
`password` or a `passwordRotation` key left over in a tenant's values is
accepted-but-inert and draws an admission warning, so an operator
editing it is told it has no effect rather than getting a silent 200.

Backup/restore into a copy keeps the target's chart-managed credentials
working: the CNPG driver clears `bootstrap.enabled` once recovery is
healthy so the init-job re-applies the generated passwords onto the
recovered roles, and the MariaDB driver excludes `mysql.global_priv`
from the dump (`ignoreGlobalPriv`) for backups captured from this
release onward, so restoring one of those into a copy does not clobber
the target's chart-managed users or `root`. Excluding the grant table
has a flip side, and it applies to every MariaDB restore, not only a
to-copy one: only accounts declared through `User` CRs (the chart's
`users` map) are captured, so an account created out of band directly in
MySQL is in no backup and a restore brings back its data and `mysql.db`
grants but not the account itself. This is also a backup-side fix only:
a restore from a backup taken **before** this change still replays the
source's `mysql.global_priv` and overwrites the target's users and
`root`, and `root` then needs an out-of-band reset (the operator applies
`rootPasswordSecretKeyRef` only at datadir bootstrap, so there is no
in-band path). See `docs/operations/backup-classes.md` for the
convergence window, the out-of-band-account limitation, and this
old-backup caveat.

Password **rotation** is deliberately **not** in this PR. A
chart-rendered credentials Secret keeps its cleartext in Helm release
history for `MaxHistory` revisions, so a rotation built on it cannot
revoke a leaked password; rotation is being reworked as a controller
that owns the Secret — design proposal in cozystack/community#72.

Migration is seamless: existing releases keep their current passwords on
upgrade (the value already lives in `<release>-credentials`, which
`lookup` re-reads and preserves); a `password` still present in a
tenant's values is ignored, so no existing HelmRelease fails validation.

Verification: `helm unittest` passes for both charts (postgres 87,
mariadb 126, including render-time username guards that reject any name
k8s would later reject as an invalid Secret key or User resource name);
`go build ./...`, `go vet`, the backup-controller tests and the
apps-apiserver tests are green; `make generate` was run with the
CI-pinned `cozyvalues-gen` v1.6.0 so `values.schema.json`, the READMEs,
the API types, and the `cozyrds` ApplicationDefinitions are all in sync
(the branch was rebased onto `main`, which reconciled cleanly with the
read-replica autoscaling feature that landed there).

### Screenshots

N/A — no UI change.

### Downstream repositories

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: affected — `content/en/docs/next/cozystack-api/go-types.md`
is hand-written and still shows `Password:` in the Go examples, and the
managed-app reference pages regenerate from these chart READMEs, so both
document `users[name].password` until refreshed (not yet filed).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: affected — the provider hand-maintains the postgres/mariadb
schemas and per-field defaults; this PR removes `users[].password`. A
follow-up issue/PR is needed there (not yet filed).

`terraform-provider-cozystack` and `website` are reached by this diff;
both boxes are left unticked because the follow-ups are not yet filed.
Same-repo future work this change suggests: the controller-owned
password rotation (design in cozystack/community#72), and extending the
"passwords only in the Secret" model to the other engines that still
accept plaintext (`clickhouse`, `mongodb`, `nats`, `opensearch`,
`rabbitmq`, `vpn`).

### Release note

```release-note
feat(postgres,mariadb)!: passwords for managed PostgreSQL and MariaDB users can no longer be set via `users[].password` — they are always auto-generated and stored only in the `<release>-credentials` Secret, never rendered into the init-script or the SQL it runs. Existing releases keep their current passwords on upgrade; a `password` (or a `passwordRotation`) left in values is ignored with an admission warning. Password rotation is deferred to a follow-up controller design (cozystack/community#72). MariaDB backups now exclude the `mysql.global_priv` grant table (`ignoreGlobalPriv`), so a restore recreates only accounts declared through the chart's `users` map — an account created out of band directly in MySQL is not captured and must be recreated after a restore.
```
2026-09-25 15:49:53 +03:00
myasnikovdaniil 0cb37dba9e docs(release): add changelog for v1.6.3 (#4077)
This PR adds the changelog for release `v1.6.3`.

✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.3.md`.
2026-09-25 15:41:55 +05:00
Alexey Artamonov 9d6fea7017 fix(monitoring): correct the alertmanager name in the dashboard Role
The Role listed the WorkloadMonitor as 'alermanager', so the real
'alertmanager' monitor was never granted: a tenant dashboard is
forbidden from reading its status and the alertmanager health tile
breaks. The test pins the correct name and the absence of the typo.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 13:35:34 +03:00
Alexey Artamonov 60b1b05434 feat(monitoring): add per-tenant OTLP collector for distributed tracing
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.

The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.

redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.

The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-25 13:35:34 +03:00
Aleksei Sviridkin 2d2186d1f0 feat(monitoring): VictoriaTraces backend + Grafana datasource (tracing phase 1) (#3762)
## What this PR does

Phase 1 of distributed tracing (design: cozystack/community#38; epic:
#3761): a **VictoriaTraces backend and Grafana traces datasource** in
the monitoring stack, mirroring the logs (`VLCluster`) implementation.
No new operator: `VTCluster`/`VTSingle` already ship in
victoria-metrics-operator.

- **`tracingStorages`** values in `extra/monitoring` (annotated,
schema-generating) and `system/monitoring` (defaults), per-entry `mode:
cluster|single`. **Empty by default, so tracing is off out of the box**:
nothing produces spans until the collector ships, so no backend is
provisioned until an operator adds an entry.
- **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or
`VTSingle` (edge/dev/small); `retentionPeriod` plus optional
`retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup
label**, reused by the existing post-delete hook, so no hook change is
needed.
- **Render-time validation** (`_helpers.tpl`): the installed
VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise
reach the operator and fail after admission. Each `tracingStorages`
entry is checked before anything renders: `name` must be an RFC 1123
label of at most 42 characters (the `vtstorage-<name>-<hash>` pod
revision label is capped at 63) and unique across the list, `mode` must
be `cluster` or `single`, `storage` must be a positive quantity (a zero
size makes the operator mount an EmptyDir in single mode, and a negative
one fails when the PVC is created), and `retentionDiskUsageBytes` must
be a VictoriaMetrics `BytesString` (the operator rejects the
Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage`
field is also typed as a quantity in the schema, and the rendered name
is quoted so a numeric name stays a string.
- **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces
datasource per entry (built-in Grafana core datasource, no plugin).
- **VPA**: a `VerticalPodAutoscaler` for each trace component
(`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle`
Deployment), matching the metrics/logs sizing, with `updateMode:
Initial`.
- **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on
`status.updateStatus`, alongside the existing `VLCluster` gate.
- **VictoriaLogs datasource fix**
(`templates/vlogs/grafana-datasource.yaml`): the range over
`logsStorages` emitted no `---` between documents, so two or more
entries collapsed into one document and only the last datasource
survived. Each entry is now its own document.
- **Tests and schema**: helm-unittest suites for off-by-default, cluster
vs single, datasource type and both query URLs, retention, PVC label,
VPA targets, every validation guard with its boundaries, and per-entry
vlogs datasource documents, plus regenerated `values.schema.json`,
`README.md` and the `monitoring-rd` ApplicationDefinition.

#### Upgrade impact

With default values the only rendered object that changes is the
`monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle`
`healthCheckExprs` and `tracingStorages: []`. Flux evaluates those
expressions only against objects in the release inventory, so they are
inert while tracing is off. No VictoriaTraces object is created on
upgrade. Installs with two or more `logsStorages` entries additionally
gain the `GrafanaDatasource` objects that were previously dropped (see
the VictoriaLogs fix above); nothing else under VictoriaLogs or
VictoriaMetrics changes.

#### Verification

- `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring`
(11 tests) pass. Each validation guard and each branch of the
zero-storage pattern was mutation-checked: reverting it turns its own
test red.
- `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly.
`tracingStorages: []` (the default) renders nothing. An invalid,
duplicate or over-long `name`, a zero or negative `storage`, or a bad
`retentionDiskUsageBytes` (`8Gi`) is refused at render with an
actionable message; a numeric `name` renders as a string. `make
generate` is idempotent.
- **Datasource URLs verified against upstream docs** ([VictoriaTraces
and
Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)):
cluster `vtselect:10471/select/jaeger`, single
`vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana
datasource; the Tempo Query API (`/select/tempo`, `type: tempo`,
VictoriaTraces v0.9.4+) has a richer TraceQL UX but is
upstream-experimental, so Jaeger is the default. Both URLs are pinned by
unit tests.
- The `retentionDiskUsageBytes` pattern, the zero-storage check and the
name limit were checked against victoria-metrics-operator v0.68.4
(`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>`
StatefulSet name) and the Kubernetes `ControllerRevisionName` naming.

#### Scope / deferred (tracked in #3761)

Per-tenant OTLP Collector, engine integrations, shared-central topology,
and docs are follow-up PRs. This PR intentionally does **not** enable
the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and
the gRPC listener lands with the collector.

Builds on the now-merged design proposal cozystack/community#38.

### Screenshots

Not a UI change.

### Downstream repositories

Walked against the diff, no box ticked yet:

- cozystack/website: `monitoring` is in the website `Makefile`
`SERVICES` list, so its reference page is regenerated from
`packages/extra/monitoring/README.md` automatically on the next stable
tag. A user guide for tracing belongs to the docs follow-up in #3761; no
website issue or PR exists for it yet.
- cozystack/terraform-provider-cozystack: the provider has no
`Monitoring` resource (only the tenant's `monitoring` flag), so the new
schema field does not reach it.
- The remaining repositories in the trigger map are not reached by this
diff: no platform values, variants, installer values, `hack/` tooling,
node prerequisites, ApplicationDefinition semantics or telemetry names
change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable trace storage with cluster or single-node
deployment modes.
* Supports retention periods, persistent storage sizing, storage
classes, and optional disk-usage limits.
* Automatically configures Grafana tracing data sources for each enabled
trace backend.
* Monitoring readiness reflects the health of configured trace storage
services.
  * Invalid trace storage settings are rejected during configuration.
* **Bug Fixes**
* Every VictoriaLogs storage now gets its own Grafana datasource; with
several log storages, all but the last were previously dropped.
* **Tests**
* Added coverage for storage rendering, retention settings, validation
guards, Grafana endpoints, health checks, and cleanup behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 14:21:26 +04:00
Mattia Eleuteri 325feec805 fix(migration): restore the class reclaim policy on imported volumes
The handoff sets the transferred PersistentVolume to Retain so the data
survives the window in which the Forklift claim is deleted and claimRef
is re-pointed at the VMDisk's claim. It never gave the policy back, so
every imported disk kept Retain for good while its StorageClass says
Delete. Deleting such a VMDisk removes its DataVolume and claim through
the normal path and strands the volume Released with its backing
storage, billed and out of any tenant's reach.

Retain is now held only while it protects something. Once the new
claim and the volume both report Bound to each other, the transferred
claim is gone, and CDI has adopted the claim into the VMDisk
DataVolume as its only owner, the volume gets its class's policy back
(Delete when the class names none or no longer exists). The VMInstance
is created and the VM reported Succeeded only after that, so a task
that fails later still leaves disks that clean up like any other.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-25 12:07:23 +02:00
Aleksei Sviridkin 47df27de57 fix(api): let one ApplicationDefinition own each application kind (#4424)
## What this PR does

Nothing enforced that `spec.application.kind` is unique across
ApplicationDefinitions, and every consumer that maps a kind back to a
definition resolved a duplicate its own way. The chartRef reconciler
selected HelmReleases by the kind label alone and let every definition
rewrite every release of its kind; the lineage webhook and the CA
projection reconciler took whichever definition came first in an
unordered list; cozystack-api registered one resource per definition. A
second definition declaring a kind already in use could therefore
repoint that kind's tenant releases at its own artifact, whatever the
definition was named. Fixes #4361.

This adds one ownership rule, in `internal/shared/appdefowner`: the
definition created first owns the kind, with ties broken by name. Every
Go consumer goes through it:

- the chartRef reconciler rewrites releases only for the owner of their
kind, emits a `KindClaimed` Warning event on a definition that is not
the owner, and re-reconciles the next owner when the current one is
deleted or changes kind;
- cozystack-api registers the owner only and logs the definitions it
leaves out; its restart hash now moves when ownership moves, while a
cluster without duplicate kinds hashes exactly as before, so the upgrade
itself does not restart the API;
- the lineage webhook and the CA projection reconciler resolve the
owner, and the CA projection watch now also reacts to a kind change;
- the lineage webhook gains a validating endpoint that refuses a new
ApplicationDefinition whose kind another definition already declares. An
update that keeps its kind is allowed, so a duplicate that predates the
check keeps upgrading, and so is a second definition from the same Helm
release, because a release that renames its definition object creates
the new one before it deletes the old one. It is registered with
`failurePolicy: Ignore`, and a failure to read the existing definitions
lets the request through with a warning instead of an error answer,
which Ignore would not cover: ApplicationDefinitions ship with the
platform, so the check must never block an install or upgrade, and the
ownership rule holds whatever it lets through.

Two decisions differ from the wording of the issue, deliberately:

- a duplicate kind is not a fail-stop at cozystack-api start-up. One
malformed or duplicate definition taking the API down for every tenant
is the problem #4359's design comment calls out, so the API serves the
owner and logs the rest instead;
- "created first owns" rather than "the platform's definition owns". It
needs no notion of provenance, and a definition that arrives later can
never take a kind over. The consequence worth knowing: deleting the
owner hands its releases to the next definition declaring that kind.

Known gaps left for follow-ups, both pre-existing:

- the dashboard console (`packages/system/dashboard/images/console`)
still picks the first definition matching a kind or plural in the list
it receives. It switches no release, but with a duplicate present it can
show the duplicate's form;
- `plural` collisions are not checked: a definition with a new kind but
an existing plural still takes the plural's endpoint at registration.
Existing releases are not reachable through it, since List and Get
filter by the kind label.

Tests cover the rule itself, each consumer, the webhook (create, update
keeping and changing kind) and the chart; every guard was
mutation-checked term by term.

### Screenshots

Not a UI change.

### Downstream repositories

The change alters `ApplicationDefinition` semantics (kind uniqueness and
ownership), which the trigger map routes to `cozystack/website`
(`content/en/docs/next/cozystack-api/application-definitions.md`). The
follow-up there is not opened yet; it needs the maintainers to settle
the behaviour first. No other repository in the map is reached: no CRD,
kind, plural, prefix or values change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(api): an application kind is now owned by one ApplicationDefinition, the one created first. A second definition declaring a kind already in use is refused at admission, and one that predates the check no longer manages that kind's releases or its API registration.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application kinds now have a consistent owner: the earliest-created
definition takes precedence, with name used to break timestamp ties.
* New definitions that claim a kind already owned by another definition
are rejected. Existing duplicates can keep their current kind, and
definitions from the same Helm release can be renamed.
* **Bug Fixes**
* Runtime configuration and Helm releases now use the owning definition,
preventing duplicates from replacing its configuration.
  * Ownership changes trigger affected definitions to be reconciled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-25 13:31:24 +04:00
IvanHunters 0687124b00 fix(seaweedfs): pass the Iceberg catalog port explicitly so null means off
The chart wraps -port.iceberg in an if, and icebergPort defaults to null,
so the flag is dropped and weed falls back to its own 8181 default. The
chart therefore cannot express "no Iceberg catalog": the value that reads
as off is the one that leaves an unmanaged plaintext catalog listening on
the pod IP, with the same writable routes and the same
admin-when-no-identities fallback as the Lance one. Render the flag the
way 4.45 already renders -port.lance, so null reaches weed as 0 and an
explicit port still wins.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:53:18 +03:00
IvanHunters ea04e671a7 fix(seaweedfs): keep the Lance catalog listener off by default
Chart 4.45 ships upstream's Lance Namespace catalog enabled (lancePort:
9101) and publishes it on the s3 Service in every tenant namespace. weed
serves that catalog over plain HTTP with no way to supply a certificate,
while every other port on this gateway is TLS by way of the
s3-tls-main-port patch, and its namespace and table routes are writable.
Nothing in Cozystack consumes the catalog, so pin the port off and keep
the s3 port set the bump inherited.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:53:18 +03:00
IvanHunters 46b4b35270 chore(seaweedfs): bump chart to 4.45
Refreshes the vendored chart from 4.31 through `make update`. All carried
patches still apply; the volume-resize-hook one is rebased onto the pod
template labels upstream now renders, so it appends the cozystack apiserver
label instead of creating the block.

Rendered names, StatefulSet selectors and volumeClaimTemplates are identical
to 4.31, so running workloads keep their volumes across the upgrade.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:53:18 +03:00
IvanHunters 0bae19fccc feat(lineage-webhook): refuse a second ApplicationDefinition for a kind
A definition declaring an application kind that another definition already
declares now fails when it is applied, instead of sitting next to the first
one where it has no effect. An update that keeps its kind is allowed, so a
duplicate that predates the check keeps upgrading, and so is a second
definition from the same Helm release, since a release that renames its
definition object creates the new one before it deletes the old one.

The webhook is registered with failurePolicy Ignore: ApplicationDefinitions
ship with the platform itself, so an outage of the webhook must never block
a bootstrap or an upgrade, and a failure to read the existing definitions
lets the request through with a warning rather than answering with an error,
which Ignore would not cover. It is the feedback layer; which definition
manages a kind is decided by the ownership rule whatever it lets through.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:39:35 +03:00
IvanHunters 0b7dd23b99 fix(appdef): let one ApplicationDefinition own each application kind
Nothing enforced that spec.application.kind is unique, and each consumer that
maps a kind back to a definition resolved a duplicate its own way: the
chartRef reconciler let every definition rewrite every release of its kind,
the lineage webhook and the CA projection reconciler took whichever came
first in an unordered list, and cozystack-api registered one resource per
definition. A second definition declaring a kind already in use could
therefore repoint that kind's tenant releases at its own artifact.

Add one ownership rule, the definition created first owns the kind with ties
broken by name, and route every consumer through it. A definition that
arrives later with a kind already in use no longer manages that kind's
releases whatever it is named: the chartRef reconciler skips it with a
Warning event and re-reconciles the next owner when the current one goes,
cozystack-api registers the owner only and logs the rest, its restart hash
moves when ownership moves, and the lineage and CA projection lookups
resolve the owner.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:39:35 +03:00