The CAPI webhook validates nodeStartupTimeout with a 30s floor but does
not validate unhealthyConditions timeouts at all, so a zero or negative
readyUnknownTimeout or readyFalseTimeout applied and remediated a
Machine the moment its Ready condition flipped. Route both fields
through the same render-time duration guard as maxNodeProvisionTime,
extracted into a kubernetes.positiveDuration helper in each chart.
The guard now also accepts the canonical form a duration serializes
back to ("20m0s", "1h0m0s"): metav1.Duration prints that shape, so a
value copied from kubectl output previously failed the render. The
first segment must still be positive and later segments may be zero,
which keeps every accepted string a positive duration.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
nodeStartupTimeout is the window a Machine has to produce a Node at
all, measured from the later of the Machine's creation and its
InfrastructureReady transition. What has to fit inside it is the guest
boot and the kubelet registering with the tenant apiserver, and on a
host under load neither is quick. At 10m a worker that was still
booting was deleted before it ever joined, and its replacement started
the same race from the beginning.
Also record what a percentage maxUnhealthy does at small group sizes,
because the interaction is not what the field name suggests: the value
is scaled against the machines the check covers and rounded down, and
remediation proceeds only while the unhealthy count is at or below the
result. At the default "50%" a group of one never remediates, and a
group of two remediates one unhealthy machine but not two.
The golden-parity gate pinned the child chart's MachineHealthCheck
values to the parent's defaults as inputs, which made a default moved
on both sides fail there for the wrong reason. Drop the pins so the
comparison is between the two charts' defaults, which is the contract
it exists to hold, and pin the defaults themselves in each chart's own
unit tests instead.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The two unhealthyConditions timeouts were fixed in the template at 30s
for `Ready=Unknown` and 300s for `Ready=False`, and both are too tight
for a worker joining on a loaded host.
`Ready=Unknown` is what the node lifecycle controller writes when a
kubelet stops reporting, on a grace period of its own, so 30s on top
of that deleted the VM of a guest that had merely stopped being
scheduled for a stretch, before it had a chance to answer again. And
`Ready=False` is the condition a registered Node holds until its CNI
is up, so 300s was the entire budget a first boot got for the CNI
rollout to reach it -- a worker deleted mid-rollout takes the rollout
with it and its replacement starts over.
Move both to values, defaulting to 5m and 10m. The remediation of a
worker that is genuinely dead is correspondingly later, which costs
less than it appears to: Pods leave an unreachable Node on the node
lifecycle controller's taint-based eviction and the toleration that
carries, not on either of these timers.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The cluster-autoscaler treats a machine as long-unregistered once its
own max-node-provision-time elapses, and a worker that has not
registered by then stops counting towards its node group's upcoming
capacity. On a host slow enough that the Talos image import, the guest
boot and the CNI rollout do not fit inside that window, the timer
fires on machines that were still joining: the group reads as short of
capacity, so the autoscaler asks for replacements, and the extra
provisioning load makes the join it was waiting for slower still.
Render the per-node-group override the autoscaler reads off the
MachineDeployment, sourced from a new maxNodeProvisionTime value that
defaults to 30m. The duration is checked at render time because the
autoscaler discards one it cannot parse and falls back to its own
default without failing, and because the values schema types the field
as a string and so accepts every malformed duration.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Both podCpuLimit and podCpuRequest were rendered under `with`, which
skips a zero as readily as an absent field, and their guards were keyed
the same way. The generated schema behind them admits a leading minus
and a zero as readily as a sane value, so rather than patch whichever
edge turned up, the whole value domain is settled in one place.
Presence, not truth, is the predicate everywhere either field is read.
A negative value is refused: it never reaches a running Pod. A zero
request on its own is allowed and reserves nothing. A zero request
beside a non-zero limit is refused: KubeVirt's defaulter tests the
request with IsZero, which an explicit zero satisfies exactly as an
absent key does, so it overwrites the zero with the limit and the
worker asks the scheduler for its whole ceiling. That is the outcome
podCpuRequest exists to prevent, reached silently and reported nowhere,
which is why it is refused rather than written down.
A zero limit removes the ceiling rather than being the tightest one --
the kubelet turns a zero CPU limit into no CFS quota -- but it reaches
a Pod only paired with a zero request. KubeVirt keeps deriving a
request from the vCPU count when the VM declares none, the defaulter
that would have copied the limit across does not fire because the limit
is itself zero, and API server validation refuses a request above a
limit that exists. Zero is a limit that exists. So a lone zero limit is
refused too.
Both fields document the same rules and each class has a test on both
charts. A group that sets neither field renders the bytes it rendered
before, so the content-hash name of the worker VM template does not
move.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Adds nodeGroups.<name>.podCpuRequest, rendering into
domain.resources.requests.cpu on the worker VM, and the same field on
the kubernetes-nodes chart, which is the other half of a byte-identity
contract rather than a separate feature surface.
podCpuLimit on its own cannot raise the ceiling of a worker without
raising what that worker asks the scheduler for, because KubeVirt
copies a declared CPU limit into the request when the VM declares none.
The request is the half that has to hold still, so the pair can now be
moved apart.
Both fields conflict with instancetype sizing for the same reason and
in the same upstream check, so the request gets the guard the limit
already had. A request above its own limit fails the render too, rather
than reaching API server validation on a Pod two objects away from the
CR that named the values; the quantities are compared as numbers, so
500m under a ceiling of 1 is accepted where a lexical guard would
refuse it.
A node group that sets neither field renders the bytes it rendered
before, which the content-hash name of the KubevirtMachineTemplate
depends on, and the render-parity gate gains a case that compares the
pair across the two charts.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The field was documented as moving the ceiling alone and leaving the CPU
request at the value KubeVirt derives from the guest vCPU count and the
cluster CPU allocation ratio. KubeVirt does the opposite: its VMI
defaulter copies a declared CPU limit into the CPU request whenever the
VM declares no request of its own, so a node group that sets only the
limit asks the scheduler for its whole ceiling, and a ceiling the nodes
cannot spare leaves the workers Pending on Insufficient cpu.
The same paragraph promised a floor that does not exist either. A limit
below the derived request was said to leave the container with a request
above its limit; the copy makes the two equal instead, whatever the
limit is.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The split-out pool chart renders the same worker VM as the monolithic
one, and tests/render-parity.sh exists to prove it renders it
byte-identically, because the KubevirtMachineTemplate is content-hash
named and any divergence rolls every live worker VM. A field that only
one of the two charts knows breaks that quietly rather than loudly: a
pool adopted into a kubernetes-nodes release would render without the
CPU limit it was given, the hash would change, and the ceiling would
fall back to the vCPU count with nothing reporting it.
So the field, its render-time guard against combining it with an
instancetype, and its tests arrive here in the shape the parent chart
has them, and the parity gate gains a case that compares the two
renders with the limit set.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The chart had no way to give a worker VM a CPU ceiling different from
its vCPU count. `resources.cpu` sizes the guest, and the pod side is
filled in by KubeVirt: with a ResourceQuota limiting CPU in the
namespace, which every tenant carrying resourceQuotas has, it derives
limits.cpu from the guest vCPU count. The QEMU emulator thread and the
IO threads then come out of the same allowance as the vCPU threads,
which is tight for a nested guest that pays emulation overhead.
Declaring the limit on the VM is what stops KubeVirt deriving one, and
it is the only way out that does not make things worse: dropping cpu
from the tenant resourceQuotas would let the LimitRange apply its 250m
default instead, four times tighter than what it replaced.
The limit is usable only on a group sized by resources, because
KubeVirt's instancetype applier reports domain.resources.limits.cpu as
a conflict rather than resolving it, and a VM carrying both is refused
by the admission webhook with nothing on the CR to say why. Sizing by
resources is what drops the instancetype from the VM, so the chart
fails the render for a group that kept its instancetype, beside the
existing guard for the same conflict class one field over.
The field renders no CPU request, so the request stays derived from the
vCPU count and the cluster CPU allocation ratio, and the scheduler
weight of a worker does not move with its ceiling. It renders nothing
at all when unset, byte for byte: the enclosing KubevirtMachineTemplate
is named by a hash of its content, so anything else there would roll
every live worker VM on upgrade.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes#3545).
The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:
- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
status.state, and persists the S3 destination + storage descriptor
(by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
from the source backup status, snapshot fallback) so the same artifact
restores in-place or into a differently-named instance; PITR via
spec.options.pitr (date/latest).
Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.
Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.
Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
Gives the ClickHouse `clickhouse-backup` sidecar a way to trust an S3
endpoint whose certificate is signed by a private CA, and uses that to
make the ClickHouse backup/restore e2e round-trip actually run in CI.
This actualizes #2600, whose feature code and example scripts have since
landed on `main` independently, and whose remaining contribution — the
e2e test — was still in the retired BATS format. #2600 can be closed
once this merges.
### 1. `backup.endpointCA` on the ClickHouse chart
The sidecar had **no TLS/CA surface at all**: bucket coordinates and
credentials arrive as `S3_*` env, and nothing let it verify an endpoint
signed by a private CA. Cozystack's own in-cluster SeaweedFS is exactly
that case — it serves `:8333` behind the self-signed "SeaweedFS CA" — so
a ClickHouse release could only ever back up to a publicly-trusted
endpoint. MariaDB (`storage.s3.tls.caSecretKeyRef`) and Postgres (barman
`endpointCA`) have had a CA field all along; this closes the gap for
ClickHouse.
`backup.endpointCA` takes `{name, key}`. When set, the chart mounts that
Secret into the sidecar and points `SSL_CERT_DIR` at it. Empty — the
default — renders no env, no mount and no volume, so existing releases
are byte-for-byte unchanged.
**Why `SSL_CERT_DIR` and not `AWS_CA_BUNDLE`.** Both are honoured by
`altinity/clickhouse-backup:2.7.4`; I measured them against a
self-signed endpoint and public AWS S3 in turn. Go reads `SSL_CERT_DIR`
*in addition to* the system bundle, so both endpoints verify.
`AWS_CA_BUNDLE` *replaces* the SDK's trust pool: the private endpoint
verifies, and public S3 then fails `x509: certificate signed by unknown
authority` — it would silently break a release that also talks to a
public endpoint. `S3_DISABLE_CERT_VERIFICATION` was rejected outright,
since it drops verification rather than trusting one more CA.
### 2. The e2e suite, and what it was really testing
Two Tests added to the existing `hack/e2e-chainsaw/clickhouse/` suite
(in name order, mirroring the `etcd`/`mariadb`/`postgres` shape — see
the layout rationale below for why they live here rather than in a new
directory):
- **`clickhouse-1-backup-contracts`** (CI, `tenant-test`, ~98s): a
backup-enabled ClickHouse asserted against the *running* cluster — the
`clickhouse-backup` sidecar on the `ch-backup-api` `:7171` port, and the
`<release>-backup-api-auth` Secret whose keys the sidecar's
`API_USERNAME`/`API_PASSWORD` actually resolve `valueFrom`. Each is a
silent break if a chart bump renames it. Needs no object store.
- **`clickhouse-2-backup-roundtrip`** (CI, `tenant-root`): `Bucket` →
`Altinity` strategy + `BackupClass` → source ClickHouse with a sentinel
row → `BackupJob` → in-place `RestoreJob` → to-copy `RestoreJob` into a
second instance, driving `examples/backups/clickhouse/run-all.sh` so
test and docs cannot drift.
The round-trip was originally gated behind `CLICKHOUSE_E2E_S3_ROUNDTRIP`
and **skipped in CI** — its `PASS (1.01s)` meant the gate fired, not
that a backup happened. `backup.endpointCA` removes the reason for the
gate, so the suite now adopts the same three preconditions
mariadb/postgres rely on: run in `tenant-root` (an isolated tenant's
Cilium egress allowlist blocks its Pods from reaching `tenant-root`'s
SeaweedFS), target the in-cluster endpoint instead of the `Bucket`'s
external ingress URL (an unroutable placeholder in CI), and trust its
certificate via the copied CA. It also pre-cleans with `cleanup.sh`,
because `tenant-root` is shared and a leftover `Succeeded` `BackupJob`
would satisfy the harness's wait without a backup ever running.
Both backup Tests live in the existing `hack/e2e-chainsaw/clickhouse/`
suite rather than a new suite directory, so no selector mapping is
added: `hack/select-e2e.sh` already derives `clickhouse` both from
`packages/apps/clickhouse/**` (the `*-application` rule) and from
`examples/backups/clickhouse/**` (the harness path), and both paths
therefore select the suite that actually runs the harness. That
alignment is the reason for the layout — a separate `clickhouse-backup/`
directory would have made an edit to the harness select a suite that
cannot exercise it. `hack/select-e2e.sh` and `hack/select-install.sh`
are otherwise untouched (the only change is a comment recording that
invariant), and the existing `backup example harness edit selects its
app suite` case in `hack/select-e2e_test.bats` covers the mapping.
### Verification
- `helm unittest` — 17 pass, including 5 new `endpointCA` cases: the
unset default renders nothing, the env/mount/volume wiring, a custom key
still projected as `ca.crt`, composition with `useSystemBucket`, and
`backup.enabled=false` renders nothing.
- The CA-trust behaviour was measured against the real image, not
assumed: without a CA knob the sidecar reports `x509: certificate signed
by unknown authority`; with `SSL_CERT_DIR` the handshake succeeds.
- `hack/select-e2e_test.bats` (16) and `hack/select-install_test.bats`
(17) pass; `select-install.sh --validate` reports graph OK.
- `make generate` is idempotent, so the pre-commit job's `git diff
--exit-code` stays clean.
- The contracts Test passed in CI on earlier runs of this branch. The
round-trip only became CI-live with this work, so it still needs a green
E2E run before merge — the branch has since been rebased onto current
`main` to get one.
### Screenshots
Not applicable — no UI changes.
### Downstream repositories
Walked the trigger map against the diff, file by file. The load-bearing
change is the new `backup.endpointCA` field in
`packages/apps/clickhouse/values.yaml` / `values.schema.json`;
everything else is `hack/e2e-chainsaw/`, `examples/backups/`, one docs
paragraph, and regenerated artifacts, none of which appear in the map.
- [ ] No downstream repository is affected by this change
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/18 —
"add, remove or rename a field in an app's `values.schema.json`" reaches
the provider's hand-maintained schema, model and expand/flatten pair, so
the new nested object is not surfaced until it is added there. Opened as
an issue rather than a PR because the natural moment is alongside that
repo's next `api/apps/v1alpha1` pin bump, which is a maintainer call.
Not ticked, with reasons rather than silence: **cozystack/website** —
the trigger is *adding, renaming or removing a package*, and this adds a
field to an existing one. `clickhouse` is already in the website
`Makefile`'s app list and its reference page is generated from this
package's `README.md`, so the regenerated `backup.endpointCA` rows are
carried by the existing release-tag automation with no manual PR.
**cozystack/external-apps-example** — vendors `hack/package.mk` and
`update-crd.sh`, neither of which this PR touches.
**cozystack/ansible-cozystack**, **ccp**, **talm**, **cozyhr**,
**cozy-proxy**, **cozystack-telemetry-server**, **examples** — no
installer values, `hack/` layout, namespace, variant, node-prerequisite,
annotation, label or metric in their trigger rows is touched here.
### Release note
```release-note
feat(clickhouse): add `backup.endpointCA` so the clickhouse-backup sidecar can verify an S3 endpoint whose certificate is signed by a private CA, such as Cozystack's in-cluster SeaweedFS. Leave it unset for publicly-trusted endpoints; existing releases are unaffected.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional private S3 endpoint CA support for ClickHouse
backup/restore sidecars via `backup.endpointCA`, including configurable
Secret name and CA bundle key (default `ca.crt`).
* Sidecars mount the CA and extend trust while leaving public endpoints
unchanged.
* **Documentation**
* Updated ClickHouse chart docs, operational backup guidance, and
backup/restore examples to cover CA setup, overrides, and cleanup.
* **Tests**
* Added helm-unittest and end-to-end coverage validating CA
rendering/mounting behavior and ClickHouse backup/restore workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
State in the README that in-guest registry pulls are routable again via
talos.registryMirrors, which knob covers which artifact (the mirror
routes registry pulls; the worker OS disk image arrives as a raw HTTP
artifact under talos.imageFactoryURL and no registry mirror touches
it), and the limits: skipFallback is needed for air-gap,
machine.registries.config is not passed through, and existing nodes
keep their boot-time config until replaced. Trim the field doc comment.
The regenerated schema, README tables and cozyrds openAPISchema follow
from those sources.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a talos.registryMirrors passthrough to the kubernetes and kubernetes-nodes
charts that renders into the worker TalosConfigTemplate machine.registries.mirrors,
so an operator can point a registry such as ghcr.io at an in-cluster pull-through
mirror and a worker boot no longer depends on live public egress. Default {}
keeps the rendered worker reconcile Job byte-identical to before, content-hash
Job name included. Free-form mirror values are shell-escaped for the unquoted
apply heredoc, covered by an execution-level bats test.
Also add the e2e ghcr-mirror (registry:2 pull-through plus a Cilium egress
policy) and wire it into the kubernetes chainsaw suites, so CI exercises the
mirror path and takes the flaky public ghcr.io kubelet-image pull off live
egress.
Refs #3513 (addresses the egress variant of the tenant-worker node-join
failures; the dominant variant is CI-runner CPU starvation, tracked in the
issue).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
A worker VM that stalls during boot before Talos apid answers is
invisible to every diagnostic the platform has: no Node registers,
no certificate request arrives, and talosctl needs the apid that
never came up. The guest serial console does show that window, but
KubeVirt attaches its guest-console-log container only when the VMI
asks for it, and the platform turns console logging off cluster-wide.
Add an opt-in per-node-group logSerialConsole that sets the KubeVirt
field, which takes precedence over the cluster-wide setting. The key
renders nothing unless true -- including when set to false -- because
the KubevirtMachineTemplate is content-hash named, so a byte emitted
for a group that did not ask for it renames the template and rolls
every live worker VM.
The description carries the two costs the field name does not show.
Changing the value rolls the group it is set on, so the stuck VM
whose console was wanted is replaced along with the rest and the
setting buys visibility into the next boot rather than the current
one. And the cluster-wide default it overrides exists because that
same container has been seen holding virt-launcher in
PodInitializing, which would take a group down rather than explain
it.
Worker VMs are emitted by two charts. Both carry the field, and the
golden-parity check gains a case that keeps their output identical.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings).
1. The backup-s3-ca volume carries no `optional: true` while the adjacent
useSystemBucket credential refs deliberately do, and the asymmetry was
undocumented — an operator naming a Secret that does not exist gets the whole
ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the
fail-closed behaviour and wrote down why the two cases differ: credentials are
projected by the platform after install, so a strict ref would wedge every
fresh release until the first BackupJob, whereas this CA is named by the
tenant, must already exist, and is trust material — optional would mount an
empty dir, the Pod would look healthy, and the first backup would fail deep
inside clickhouse-backup with an x509 error naming neither the Secret nor the
chart. The comment also states the blast radius plainly.
2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read
was reported as "CA not found" followed by advice (set S3_CA_SECRET) that
cannot fix a permissions problem. It now separates present / absent /
not-allowed-to-look and says so, and the README no longer labels step 03
simply "tenant": copying the CA reads a Secret in tenant-root, which is a
platform capability. Added a section spelling out the three ways to satisfy
that (admin credentials, a one-off admin copy, or skipping the CA entirely on
a publicly-trusted endpoint).
3. Caveat, and a usability regression this branch introduced: on a cluster with
a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default
the user never chose. It now distinguishes an explicitly named Secret (still
an error when missing — a typo should not be guessed past) from the unset
default, where it warns and continues with backup.endpointCA unset. Verified
all four paths against a fake kubectl: Forbidden, absent+default,
absent+explicit, and present.
4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy
`schedule` CronJob writes to S3 through restic and does not consume it, so a
private-CA endpoint combined with `schedule` still fails TLS. Documented on
the field itself, so it lands in values.schema.json, the README and the
cozyrds openAPISchema.
The PR body's claim about select-e2e.sh/select-install.sh mappings was stale
after the restructure that removed them; corrected there rather than in code.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The clickhouse-backup sidecar had no TLS/CA surface at all: bucket coordinates
and credentials arrive as S3_* env, and nothing let it verify an endpoint whose
certificate is signed by a private CA. Cozystack's own in-cluster SeaweedFS is
exactly that case — it serves :8333 behind the self-signed "SeaweedFS CA" — so a
ClickHouse release could only ever back up to a publicly-trusted endpoint, while
MariaDB (storage.s3.tls.caSecretKeyRef) and Postgres (barman endpointCA) have had
a CA field all along.
Add backup.endpointCA {name, key}. When set, the chart mounts that Secret into
the sidecar and points SSL_CERT_DIR at it. Empty (the default) renders no env, no
mount and no volume, so existing releases are untouched.
SSL_CERT_DIR rather than AWS_CA_BUNDLE, measured against
altinity/clickhouse-backup:2.7.4 with a self-signed endpoint and public AWS S3 in
turn: Go reads SSL_CERT_DIR in addition to the system bundle, so both verify;
AWS_CA_BUNDLE replaces the SDK trust pool, so the private endpoint verifies and
public S3 starts failing `x509: certificate signed by unknown authority`. That
would silently break a release that also talks to a public endpoint.
S3_DISABLE_CERT_VERIFICATION was rejected outright — it drops verification
instead of trusting one more CA.
Covered by helm-unittest: the unset default, the mount/env/volume wiring, a
custom key still projected as ca.crt, composition with useSystemBucket, and that
backup.enabled=false renders nothing.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Gate the Proxied ingress Service on _namespace.etcd like the addon
HelmRelease it fronts, so an etcd-less tenant with the addon pinned on does
not get a selector-less Service (+ test). Correct the secrets framing: the
tenant-visible surface is an allowlist (include: [] withholds everything),
so drop the never-load-bearing exclude entry, the bats cross-check and the
rename-coupling comments, and state the real mechanism. Fix the etcd
prerequisite docs (an ancestor's etcd satisfies the wrapped chart; the
awaiting-etcd beacon is the diagnostic), reword the tenant toggle to stop
advertising the not-yet-shipped placement routing, require maxReplicas >= 1
on the ingress-role node group, and fix comment drift (Service name after
the release rename, placement-hook label, package-guard lookup semantics).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Wire the computeplane module into the tenant chart as a bool toggle, matching
the other tenant modules. The module's own Helm release takes the -module
suffix so the canonical release name computeplane (and with it the
computeplane-admin-kubeconfig Secret) stays reserved for the cluster
HelmRelease the module renders.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The cluster field referenced x-cozystack-options source 'kubernetes',
which has no provider (cozystack-api returns NotFound), so the dashboard
dropdown 404s. Drop the annotation so the field is a plain text input;
a real provider listing Kubernetes CRs is a follow-up.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The pre-commit run-make-generate hook regenerates the Go types from the
values annotations; the serverName default note landed in values.yaml,
README, schema and postgres-rd but missed types.go.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300): the apps/postgres user-facing docs still
described the removed native spec.backup.barmanObjectStore / archive_command
model. Update the values.yaml @field/@typedef/@param annotations (useSystemBucket,
schedule, S3CredentialsSecret, EndpointCA, endpointCA, the @param backup note,
bootstrap.serverName) and the hand-written README Operations prose to the
barman-cloud plugin + ObjectStore model, then regenerate values.schema.json, the
README parameters table, the api types, and the postgres-rd ApplicationDefinition
schema (which feed the dashboard form help + cozyrd schema). Field names,
defaults, and the storageClass immutability marker are unchanged; only backup/
bootstrap description text changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Add optional nodeGroups[name].maxUnhealthy and nodeGroups[name].nodeStartupTimeout
fields that override the cluster-wide nodeHealthCheck defaults per worker node
group. When unset, a group inherits nodeHealthCheck.{maxUnhealthy,nodeStartupTimeout},
so existing values are unaffected. The MachineHealthCheck render resolves the
per-group value first, reuses the existing int-or-string validation, and names
the offending group in the fail() message when an override is invalid.
Delivers the per-group granularity of the standalone maxUnhealthy proposal on
top of the cluster-wide nodeHealthCheck from the Talos worker bootstrap PR.
Co-authored-by: mattia-eleuteri <mattia@hidora.io>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24.
That kernel predates both fixes for a pair of related KVM x86 shadow-paging
use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359,
fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables
nested virtualization inside the VM, and either bug lets a nested guest
escape into the worker.
Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so
it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0
ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries
only CVE-2026-46113. Nested virtualization stays available to tenants — the
kernel bump is the fix.
The new pin changes the rendered worker spec, so the content hash in the
KubevirtMachineTemplate and talos-reconcile Job names rotates and each
worker pool rolls once on upgrade. Pinned hashes in the chart tests are
recomputed accordingly.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Tenant worker VMs boot from a Talos raw disk image that CDI streams over
HTTP from the hardcoded public Talos Image Factory, and the in-guest
talos-reconcile Job pulls its installer image from the same hardcoded host.
Neither was overridable, so air-gapped / rate-limited / flaky-egress
environments could not point workers at a mirror (the README flagged this
as a Phase-2 gap), and CI had no way to avoid bulk-pulling the OS image
from the public internet per worker.
Add two chart values, both defaulting to the public factory so rendered
output is byte-identical and behaviour is unchanged:
- talos.imageFactoryURL (default https://factory.talos.dev) - base URL for
the worker OS disk image streamed in by CDI.
- talos.installerRepository (default factory.talos.dev/installer) - OCI
repository prefix for the in-guest installer image.
Regenerated values.schema.json, the aggregated-API Go types, the
kubernetes-rd cozyrds schema and the README via `make generate`; added
unittests for the override and trailing-slash trimming.
Refs #3231
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Follow-ups from the self-review of #3233:
- `values.yaml` / helper docstring / regenerated README + schema +
cozyrds openAPISchema + Go type doc: the previous "one
ingress-nginx worker" contract was already misleading (helper
default carried `minReplicas: 0` but the chart hardcoded
`spec.replicas: 2`), and the replicas fix makes the docstring
strictly false — an empty-nodeGroups install now provisions zero
workers until either the cluster-autoscaler responds to an
unschedulable Pod or the operator explicitly scales the group.
Update the docstrings to describe the actual behaviour and to
spell out the ingress-nginx-addon implication so operators are
not surprised at addon-install time.
- `tests/nodegroups_default_test.yaml`: two regression cases pinning
`MachineDeployment.spec.replicas == $group.minReplicas` for
`minReplicas: 0` (the render-side OIDC bats path — zero worker
DataVolumes provisioned) and for `minReplicas: 2` (the
kubernetes-latest / kubernetes-previous bats path — byte-
identical to pre-PR behaviour). Without these a future refactor
can silently re-hardcode `replicas: 2` and turn no tests red.
helm-unittest suite grows from 171 to 173, both new cases pass.
Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Gemini flagged that the `values.yaml` @field docstring on
`oidc.users[].email` still described the claim mapping as
`preferred_username`, misleading operators after the claim was
switched to `email` for real (see the earlier fix and its
end-to-end verification on dev3). Rewrite the description to
match the actual mechanic: `email` claim, requested via
`--oidc-extra-scope=email` in the chart-generated kubeconfig,
emitted by every conformant OIDC provider.
`make generate` re-propagates the update through README.md,
types.go and the cozyrds embedded schema.
Note on the other finding in the same review — Gemini also
flagged `audiences` as being incorrectly nested under `issuer`
in the AuthenticationConfiguration Secret. That is a
false positive: per the kube-apiserver v1beta1 spec
(apiserver.config.k8s.io/v1beta1, `JWTAuthenticator.Issuer`),
`audiences` IS a child of `issuer`, not a sibling of `jwt[i]`.
The current shape is correct and verified end-to-end on dev3
(alice/bob tokens carry `aud: [tenant-test-kubernetes-test3,
kubernetes]` and apiserver validates them successfully with
CRB-driven RBAC).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Reviewer flagged a real customer-upgrade regression: the previous
values.yaml docstring on `controlPlane.apiServer.extraArgs` documented
`--oidc-issuer-url`/`--oidc-client-id` as a valid usage. An existing
customer who wired OIDC by hand through that passthrough (following
the doc) would see their tenant kube-apiserver BOOT-FAIL the moment
they toggle `spec.oidc.mode` to `System` or `CustomConfig`:
kube-apiserver refuses to boot with both `--authentication-config`
(injected by the new spec.oidc) and any legacy `--oidc-*` flag —
`--authentication-config cannot be used with any of --oidc-*`.
Add a render-time guard: when `spec.oidc.mode != None`, scan the
user-supplied `controlPlane.apiServer.extraArgs` for any flag with
prefix `--oidc-` and fail with a clear message pointing at the
`spec.oidc.mode: CustomConfig` migration path (they can carry the
same issuer/audience inside `customConfig.config`).
When `spec.oidc.mode: None` the passthrough still works — customers
who manage OIDC entirely by hand keep working until they opt into
the new selector explicitly.
Also update the values.yaml docstring on `extraArgs` to steer OIDC
users to `spec.oidc.mode` and warn against the collision.
Two new helm-unittest cases: the collision fails, and mode=None with
legacy flags renders normally (backward compat).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Introduce a flat selector — `mode: System | CustomConfig | None` —
plus a `users[]` list and a `customConfig` block for tenant-supplied
AuthenticationConfiguration. Default `None` preserves today's
behavior.
The `System` mode trusts the platform `cozy` realm via a per-cluster
public client; `CustomConfig` accepts an inline `AuthenticationConfiguration`
or a Secret reference (mutually exclusive). The `users[]` list drives
per-user ClusterRoleBindings inside the tenant cluster, with two
roles: `admin` and `view`.
Regenerates deepcopy, JSON schema, README, and the kubernetes-rd
cozyrds embedded schema.
Implements Phase 1 API of the per-tenant OIDC design proposal
(cozystack/community#24).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Worker node VMs set evictionStrategy: External, so they live-migrate.
Live migration requires the system DataVolume on a ReadWriteMany volume,
and linstor-csi only grants RWX on a DRBD-backed StorageClass. When a node
group leaves storageClass empty the template previously omitted the field
and fell through to the management cluster default StorageClass, which may
be a non-DRBD class the CSI driver rejects for RWX — breaking migration.
Fall back to the application-level storageClass (the replicated/DRBD class)
when the node group storageClass is empty, instead of the cluster default.
When both are empty the field is still omitted, preserving the explicit
cluster-default opt-out.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A node group is sized either by instanceType or by explicit resources, but the
generated values schema required resources on every node group. A node group
with instanceType and no resources — or an empty resources dropped during
typed-CR to HelmRelease translation — failed schema validation with "missing
property 'resources'" even though sizing was fully specified.
Mark resources optional in the node group definition and regenerate the schema,
API types, README, and ApplicationDefinition. The chart template emits
domain.cpu/memory only when resources is present, so a node group sized purely
by instanceType now passes validation and installs.
When a node group sets both instanceType and explicit resources, explicit
resources take precedence: the instancetype is omitted from the rendered VM and
the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit
domain.cpu/memory with no instancetype to conflict with. instanceType stays
required and defaulted (no schema change), so the console's required
instanceType dropdown keeps working — sending the default instanceType plus
explicit resources simply sizes the VM by resources. This replaces an earlier
render-time hard failure on the both-set case, which was unreachable to clear
from the dashboard (the required dropdown cannot be emptied).
Explicit resources are all-or-nothing alongside an instanceType: a partial spec
(only cpu, or only memory) on a group that carries a non-empty instanceType
would emit one domain override next to the instancetype — which KubeVirt rejects
— so it now fails at render time with a clear error. Partial resources on a
group without an instanceType (used to tune kubelet reservations) are unaffected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Tenant Kubernetes clusters had no way to configure the kube-apiserver
authentication through the app values: no knob for OIDC, the structured
AuthenticationConfiguration, extra apiserver flags, or extra volumes.
The only workaround was patching the underlying KamajiControlPlane
out-of-band, which is invisible to GitOps and lost on recreation.
Add three opt-in fields under controlPlane.apiServer, all empty by
default so existing clusters render unchanged:
- extraArgs -> KamajiControlPlane spec.apiServer.extraArgs
- extraVolumeMounts -> spec.apiServer.extraVolumeMounts
- extraVolumes -> appended to spec.deployment.extraVolumes
This unblocks OIDC SSO via the --oidc-* flags and, more importantly,
structured authentication: --authentication-config mounted from a
ConfigMap/Secret plus --requestheader-uid-headers, which is the only
way to map a user UID for aggregated/extension apiservers.
Because the control-plane Deployment runs on the management cluster,
the passthrough is validated to preserve tenant<->management isolation:
- extraVolumes allow only configMap and secret sources; host-reaching
sources (hostPath, csi, nfs, persistentVolumeClaim) and token
projection (projected) are rejected, so a tenant with self-service
access to the Kubernetes CR cannot mount management-node paths or
project arbitrary tokens into the apiserver pod.
- each extraVolume must have a unique, non-empty name and exactly one
source; the reserved names talos-ca and talos-tls-cert are rejected
to avoid a duplicate volume name in the rendered podspec.
- each extraVolumeMount must reference a user-declared extraVolume; the
chart-managed talos secret volumes cannot be mounted.
Regenerate values.schema.json, README, the Go types and deepcopy, and
the ApplicationDefinition. Add helm unittest coverage for the default
(no-op) render, each field's passthrough, the name/source guards, and
the mount-reference guard (valid and rejected inputs).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Tenant Kubernetes clusters previously exposed a single hardcoded
"kubevirt" StorageClass mapped to one infra class. Discover all
remote-accessible LINSTOR infra StorageClasses (provisioner
linstor.csi.linbit.com, allowRemoteVolumeAccess != false) via Helm
lookup and propagate each to the tenant under the same name.
- csi.yaml builds a storageClasses map filtered to remote LINSTOR
classes; node-local classes (e.g. "local") are excluded because the
backing volume cannot follow a live-migrated worker VM.
- The class named by .Values.storageClass (default "replicated") is
marked the tenant default; the legacy "kubevirt" StorageClass is kept
as a non-default alias so existing tenant PVCs keep binding.
- A non-destructive fallback preserves the legacy mapping when the
lookup returns empty (dry-run), so tenant StorageClasses are never
wiped on a transient reconcile.
- infraStorageClassEnforcement is intentionally left disabled (AllowAll)
to avoid breaking tenants that already self-served custom
StorageClasses under the current default.
Refs: #2094, #1424
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Switch tenant worker bootstrap from Ubuntu+kubeadm to Talos. MachineDeployment.bootstrap moves from KubeadmConfigTemplate to a TalosConfigTemplate produced by a post-install/upgrade reconcile hook, which also materialises the kubeadm-format bootstrap-token Secret in the tenant kube-system. Workers boot from the Talos openstack image via a CDI DataVolume; the system disk is virtio-blk with logical=512/physical=4096 so 4Ki-native backends (LINSTOR/DRBD) and SeaBIOS both work; the separate kubelet disk is dropped. Pod/service CIDRs, dnsDomain, controlPlane endpoint and cilium k8sServiceHost are pinned to the tenant ranges / Kamaji apiserver ClusterIP to survive the DNS chicken-and-egg at bootstrap. Adds a Talos<->Kubernetes support-matrix fail() guard and drops v1.30. Removes the dead ubuntu-container-disk build chain and the orphaned copy-patch-containerd template.
Co-authored-by: IvanHunters <xorokhotnikov@gmail.com>
Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The submodule still pulled golang.org/x/net v0.53.0, affected by GO-2026-5026
(failure to reject ASCII-only Punycode-encoded labels in golang.org/x/net/idna,
which the submodule imports). The root module was already on v0.55.0; align the
submodule so the advisory is closed across both go.mod files.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Clears the OpenSSF Scorecard Vulnerabilities check (was 0/10). An
osv-scanner pass over both go.mod files reported 12 advisories:
- 10 stdlib advisories against the Go toolchain pinned at 1.26.2
(GO-2026-4918/4971/4977/4980/4982/4986 fixed in 1.26.3;
GO-2026-5037/5038/5039 fixed in 1.26.4) — closed by raising the go
directive to 1.26.4 in both the root module and api/apps/v1alpha1.
- golang.org/x/net 0.52.0 (GO-2026-4918, GO-2026-5026) — bumped to
v0.55.0; go mod tidy carried the matching x/crypto, x/sys, x/term,
x/text and x/tools minor bumps via MVS.
After the bump osv-scanner reports 0 advisories. Both modules build,
and 'make generate' produces no drift (codegen-drift stays green).
Versions were changed with 'go mod edit'/'go get'/'go mod tidy', not by
hand-editing the generated go.mod.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
## What this PR does
Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and
the embedded management-cluster manifests, and folds in the chart fixes
that become hard errors under the new strict helm-controller v1.5.
Flux v2.8's helm-controller v1.5.0 ships:
- **Server-Side Apply with `--force-conflicts`** — strict CRD schema
enforcement; misplaced fields (silently dropped on v2.7) now fail the
apply.
- **kstatus-based health checking** by default — parent HR waits for
every applied resource (including child HRs) to be Ready before flipping
its own Ready, surfacing latent ordering deadlocks.
- **Helm v4 embedded** — `upgrade.force: true` is deprecated and now
conflicts with SSA.
- **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper
readiness gating (used in PR #2601 split).
Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same
Flux upgrade triggers it.
## Commits
**Flux upgrade itself:**
- `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts
to v0.48.0; web UI opt-in.
- `feat(flux)`: regenerate embedded management-cluster manifests via
`make update` in `packages/core/flux-aio` (timoni bundle build).
**Chart fixes for strict SSA — fields the chart sent that v2.7 silently
dropped, v2.8 rejects:**
- `fix(kubevirt-instancetypes)`: drop persistent strip that produced
null `preferredTPM` (folds #2612).
- `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`,
`minimumUptimeSecondsForBounce` from inside `automationOptions` to
direct children of `spec`.
- `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a
phantom `template.spec`.
- `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is
set.
- `fix(platform)`: drop deprecated `upgrade.force: true` from
HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths.
**Ordering / deadlock fixes under v2.8 kstatus:**
- `fix(vpa)`: break circular wait between parent install and nested
`vpa-for-vpa` HR.
- `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant
addon child HRs (parent waits on child via kstatus, child waited on
parent — deadlock).
**E2E waits for v2.8 kstatus timing:**
- `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's
faster dispatch).
- `test(e2e)`: wait for parent HR Ready before downstream asserts in
`run-kubernetes.sh` and `vminstance.bats`.
## Scope discipline
This PR is part of the split of #2619 (the consolidated CI fixes branch)
into review-friendly pieces. Companion PRs:
- **PR #2601** (seaweedfs split) — folded into this PR (commits
`29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the
seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption
migration 43 (targetVersion 44), and the configurable db resources all
land here, because the strict-SSA `upgrade.force` removal and the
kstatus parent-HR timeout bump only make sense together with the split.
#2601 is superseded.
- **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) —
independent, lands separately.
- Several smaller standalone fixes lifted out of #2619 (startup probes,
cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC
test improvements) — opened as separate PRs.
## Verification
- `helm template` renders cleanly for both `fluxcd` and
`fluxcd-operator` packages with `web.enabled=false` (default) and
`web.enabled=true`.
- Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0
/ v2.1.0 controller image strings.
- No references to the v0.39-removed `--disable-wait-interruption` flag
anywhere in `packages/` or `internal/`.
### Release note
```release-note
Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking).
When upgrading existing clusters:
- Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too.
- HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it.
- KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass.
- KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained.
- FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Flux Status web UI: optional web server, config secret, service port,
HTTPRoute/Ingress, network policy, and RBAC roles.
* New SeaweedFS DB Helm chart and optional managed DB release.
* **Improvements**
* CRD/schema enhancements: new provider kinds, validations, variant
option, and external checksum refs.
* Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for
reliability.
* **Chores**
* Bumped Flux Operator and Flux versions; documentation links updated to
fluxoperator.dev.
* **Bug Fixes**
* Removed aggressive HelmRelease force-upgrade/install flags.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The imageType default was switched to split in values.yaml/values.schema.json/
RD schema but make generate was not run, leaving the +kubebuilder:default marker
in api/apps/v1alpha1/foundationdb/types.go at "unified" and the README param
table sized for the old value — failing the pre-commit generate check (exit 123).
Regenerated with cozyvalues-gen v1.5.0.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds TLS support to the Postgres managed service via the CNPG operator's
built-in cert management.
- Introduces a nullable `tls.enabled` field with tri-state semantics:
when unset, defaults to the value of `external` (auto-on for externally
published services, off for cluster-internal). Explicit `tls.enabled`
always wins.
- Uses CNPG's operator-managed TLS chain — the chart no longer renders
its own cert-manager chain. The operator auto-generates a self-signed CA
and signs server, client, and replication certs from it. The operator's
default SAN list already covers the three built-in services (rw/r/ro) in
four DNS forms each (bare, `.ns`, `.ns.svc`,
`.ns.svc.<cluster-domain>`).
- When `external: true` and TLS is on, the chart sets
`spec.certificates.serverAltDNSNames` so the operator appends the
external hostname `<release>.<tenant-host>` to the auto-generated leaf
cert. In all other cases the chart leaves `spec.certificates` absent and
the operator uses its default SAN set.
- Trust anchor for clients: the operator-managed `<release>-ca` Opaque
Secret (key `ca.crt`). Mount that key into client pods to verify TLS.
Why operator-managed instead of a chart-side cert-manager chain: an
earlier iteration that supplied `serverCASecret` and `serverTLSSecret`
from a chart-rendered cert-manager chain failed the CNPG admission
webhook with `missing ca.key secret data`. CNPG expects a `ca.key` field
on `serverCASecret`, while cert-manager writes the private key under
`tls.key`. Operator-managed mode avoids the schema fight entirely.
Verified end-to-end on a sandbox cluster: cert chain is created by the
operator within ~60 seconds, leaf cert's SAN list includes both the
standard service forms and the external hostname, TLSv1.3 handshake via
`openssl s_client -starttls postgres` against the rw service with the
operator CA returns verify code 0.
### Release note
```release-note
feat(postgres): add TLS support via CNPG operator-managed certificates with tri-state tls.enabled
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* TLS configuration added for PostgreSQL with a tri-state tls.enabled to
control operator-managed certificate behavior and optional injection of
the external hostname into server certificate SANs.
* **Documentation**
* Added TLS configuration guide, client CA retrieval and psql
verification examples, and updated parameters reference.
* **Tests**
* Added integration tests covering tri-state TLS behavior and
external-hostname injection scenarios.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2686?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds TLS support to the Qdrant managed service via cert-manager.
- Introduces a nullable `tls.enabled` field with tri-state semantics:
when unset, defaults to the value of `external` (auto-on for externally
published services, off for cluster-internal). Explicit `tls.enabled`
always wins.
- Renders a self-contained cert-manager chain in the tenant namespace:
self-signed Issuer, CA Certificate, CA Issuer, leaf Certificate.
- SAN policy covers the three DNS forms per service (bare, `.ns.svc`,
`.ns.svc.<cluster-domain>`) plus the external hostname when `external:
true`.
- Wires the leaf cert into the Qdrant HelmRelease so both the REST and
gRPC endpoints use the chart-managed CA chain.
Verified end-to-end on a sandbox cluster: cert chain reaches Ready, TLS
handshake against the tenant service succeeds with the chart CA bundle.
### Release note
```release-note
feat(qdrant): add TLS support via cert-manager (tri-state tls.enabled)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* TLS config for Qdrant with tri-state behavior (inherit from external,
or explicitly enable/disable); optional cert-manager provisioning and
automatic TLS secret mounting.
* **Documentation**
* README and chart values/schema updated to document TLS parameters,
defaults, and precedence.
* **Tests**
* Expanded Helm tests covering TLS rendering, certificates (SANs/ECDSA),
reload annotations, and volume/secret injection.
* **Chores**
* Added a Helm test target to the Makefile.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2685?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds TLS support to the NATS managed service via cert-manager.
- Introduces a nullable `tls.enabled` field with tri-state semantics:
when unset, defaults to the value of `external` (auto-on for externally
published services, off for cluster-internal). Explicit `tls.enabled`
always wins.
- Renders a self-contained cert-manager chain in the tenant namespace:
self-signed Issuer, CA Certificate, CA Issuer, leaf Certificate.
- SAN policy covers the three DNS forms per service (bare, `.ns.svc`,
`.ns.svc.<cluster-domain>`) plus the external hostname when `external:
true`.
- Wires the leaf cert into the NATS HelmRelease so client connections
and cluster routes both use the chart-managed CA chain.
Verified end-to-end on a sandbox cluster: cert chain reaches Ready, TLS
handshake against the tenant service succeeds with the chart CA bundle.
### Release note
```release-note
feat(nats): add TLS support via cert-manager (tri-state tls.enabled)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional TLS support for NATS with a nullable/tri-state
tls.enabled (true/false/null) that defaults to external access when
unset
* Conditional cert-manager integration to provision CA and TLS
certificates and enable cluster TLS, and toggle cluster routing to FQDN
when TLS is enabled
* **Documentation**
* Updated configuration docs and schema defaults to include new TLS
settings and fallback behavior
* **Tests**
* Added tests validating tri-state TLS behavior, cert-chain generation,
SANs, and conditional rendering
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2684?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds TLS support to the Kafka managed service via Strimzi listener
configuration.
- Introduces a `tls.enabled` field that controls the external listener
(port 9094). When unset, defaults to the value of `external` (auto-on
when external access is enabled). Explicit `tls.enabled` always wins.
- Strimzi manages the cluster PKI automatically; the operator publishes
`<release>-cluster-ca-cert` and `<release>-clients-ca-cert` for client
trust setup. No cert-manager chain is rendered by this chart — the
operator owns the chain end-to-end.
- The internal TLS listener on 9093 is always on (Strimzi default); this
field only governs the external listener on 9094.
- External LoadBalancer listener is decoupled from `tls.enabled`: it is
gated on `external: true` only. Previously, an explicit `tls.enabled:
true` with `external: false` would force the external listener to be
rendered.
Verified end-to-end on a sandbox cluster: external listener exposes a
TLS endpoint signed by the Strimzi-managed cluster CA, client
connections with the published trust bundle succeed.
Note: this chart does not configure listener authentication. With TLS on
the external listener, clients can connect without credentials unless
`KafkaUser` resources with `tls`/`scram-sha-512`/`oauth` authentication
are configured outside this chart.
### Release note
```release-note
feat(kafka): add TLS support on the external listener via Strimzi (tri-state tls.enabled)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tri-state TLS for the external Kafka listener: explicit on, explicit
off, or inherit from the external toggle; chart values now include a
top-level tls object.
* **Documentation**
* Expanded chart docs/README clarifying tls.enabled inheritance,
plaintext risk when disabled with external access, and authentication
guidance.
* **Tests**
* Added/updated schema, rendering, RBAC/secret-naming, helm unit, and
end-to-end tests covering TLS tri-state and external-listener behavior.
* **Bug Fixes**
* RBAC and secret-name expectations updated to use the `-cert` suffixed
secret names.
* **Chores**
* Added a Makefile test target to run helm unit tests.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2681?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolve the conflict in the generated packages/system/postgres-rd/cozyrds/postgres.yaml by regenerating openAPISchema and keysOrder from the merged values. The CRD now carries both the chart's tls.enabled field and main's backup useSystemBucket / BackupClass-driven fields.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
version is a general deployment parameter, not a TLS setting; relocate
its @param/@enum block back into Common parameters so the generated
parameter reference groups it correctly instead of under TLS.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The generated JSON schema types tls.enabled as a scalar boolean, which
rejects an explicit null value; describe the tri-state unset case as
"omitted" so the wording matches what the schema actually accepts.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The README still described the chart-side cert-manager chain that was
replaced when the chart switched to CNPG operator-managed TLS via
spec.certificates.serverAltDNSNames. Rewrite the section to explain the
actual cert ownership boundary (CNPG owns the chain, chart only injects
the external hostname), the trust anchor location (operator-managed
<release>-ca Secret, key ca.crt), and the chart-side semantics of the
tri-state tls.enabled flag. The values.yaml @field comment is updated
in the same direction, with regenerated schema/types.go/cozyrds.
Clarifies that tls.enabled=false only skips the chart-side SAN
injection - CNPG retains its built-in TLS on the wire regardless of
this flag. To force PostgreSQL TLS off entirely, the caller would set
postgresql.parameters.ssl="off" at the CNPG layer.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Aleksei Sviridkin
