Commit Graph
7777 Commits
Author SHA1 Message Date
Andrey Kolkov 2d82c1ea26 test(backups): keep main's redis-tls e2e Tests; fast-fail redis backup waits
The branch was cut before main gained the redis-tls and
redis-tls-flip-rejected Chainsaw Tests, so its copy of the redis
chainsaw-test.yaml carried only the plaintext Test plus the new
redis-2-backup-roundtrip. Carry main's two TLS Tests here too, so a conflict
resolution at merge time cannot drop the TLS end-to-end coverage regardless
of which side it takes. The file still needs a rebase onto current main to
merge cleanly; this only removes the risk of losing the Tests.

Also pass the terminal Failed value to the BackupJob/RestoreJob phase waits
so a failed run aborts immediately with a legible cause instead of polling to
the 600s timeout.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-08 17:57:06 +04:00
Andrey Kolkov 919efa044f fix(backups): branch the Redis Job-existence check on IsNotFound
The readiness re-gating guard read `r.Get(...) == nil` to mean "the Job does
not exist yet", folding any non-NotFound error (an apiserver hiccup, an RBAC
denial) into that branch. On such a transient error the driver would re-enter
the readiness precondition even though the Job may already exist and have
completed, reintroducing the spurious terminal failure this guard removes.

Switch on the Get error: nil means the Job exists (observe it), NotFound
means run the pre-launch readiness gate, and any other error requeues without
deciding. Same on the backup and restore paths.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-08 17:57:06 +04:00
Andrey Kolkov ac79c3757b test(backups): self-contained Redis backup round-trip e2e
The Redis driver shipped no round-trip backup/restore e2e, unlike its
ClickHouse, MariaDB and RabbitMQ siblings, so the backup to S3 and restore
data path had no automated coverage. The existing demo backed up through the
platform cozy-default BackupClass, whose endpoint is the shared system
bucket; in the e2e environment that bucket's advertised external address is
not routable, so a cozy-default round-trip cannot run there.

Rework examples/backups/redis to provision its OWN Bucket + Redis strategy +
BackupClass (mirroring the MariaDB and RabbitMQ demos), projecting the
bucket's keys and the seaweedfs CA into the Secrets the strategy references
and honouring an S3_ENDPOINT override so the backup Pod targets the
in-cluster seaweedfs. Add a redis-2-backup-roundtrip Chainsaw test that
drives run-all.sh in tenant-root and asserts the sentinel round-trips
through object storage in place and into a copy.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-08 17:57:06 +04:00
Andrey Kolkov ff061a8eea fix(backups): stop re-gating Redis readiness after the backup Job exists
The Redis backup/restore reconcilers checked the application's Ready
condition on every pass, ahead of the Job-state switch. Once the Job has
been created that re-check is wrong: if the app's Ready flips False after
launch (an unrelated HelmRelease blip) and stays so past the deadline, the
driver marks a run Failed whose Job may already have completed, a spurious
terminal failure of a successful backup or restore.

Gate the readiness precondition on the Job not yet existing; once it does,
observe the Job directly. The deadline still bounds the pre-launch wait.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-08 17:56:20 +04:00
Andrey Kolkov 632f7b31a0 test(backups): cover Redis backup-path unmappable kind
The restore path already asserts a terminal failure for an applicationRef
that passes the Redis gate but has no REST mapping; add the symmetric
backup-path case so both dispatch sides are covered.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Andrey Kolkov 5c603586e1 docs(backups): document Redis in backup-classes
The Redis route was added to the cozy-default BackupClass but
backup-classes.md still described the pre-Redis bound set. Add the Redis
row to the out-of-the-box table, the endpoint-format table (S3_ENDPOINT
bare host, https:// prepended when unscheme'd), and the projected-secret
key table (AWS_* plus endpoint/bucketName/region). Document the new
endpointCASecretName knob and its https/TLS interaction.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Andrey Kolkov b0eb104276 feat(backups): add Redis backup/restore strategy driver
The spotahome RedisFailover operator ships no backup CRD, so add a
dedicated strategy.backups.cozystack.io/Redis driver that runs a one-shot
Job: it dumps an RDB snapshot from the current master (discovered via the
operator's sentinel Service) to the system bucket over S3, and on restore
replays it (RDB -> RESP via redis-cli --pipe) into the target's master.

Structurally a specialisation of the Job/Altinity script-Job pattern with
a Redis applicationRef gate and an app-readiness precondition. Backup
parameters round-trip through the Backup's DriverMetadata (no
underlyingResources snapshot); restore drives from .Backup.ApplicationRef
so a to-copy restore reads the source's object-storage prefix while
writing into a differently-named target. Both in-place and to-copy are
supported; PITR is not (RDB is a discrete snapshot).

- api: Redis/RedisList strategy type + generated CRD and deepcopy
- driver: reconcileRedis / reconcileRedisRestore + pure helpers, wired
  into both dispatch switches, supportedBackupStrategyKinds and the no-op
  restore cleanup case
- default: gated cozy-default-redis strategy (system-bucket coords via
  cozy-backups-creds env secretKeyRef, https:// scheme + optional CA) and
  the Redis route in the cozy-default BackupClass
- image: images/redis-backup bundling redis-cli + curl + an RDB->RESP
  converter, wired via backupStrategyController.redisBackupClientImage
- tests: driver unit tests (gate, precondition, render, param round-trip,
  both reconcile paths) and the helm rendering-gate coverage
- example: examples/backups/redis honest marker round-trip for both
  restore flows

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-09-08 17:56:20 +04:00
Alexey Artamonov 49e8aa58f9 fix(keda): pin job=kube-state-metrics in the ScaleStuck Pending join
The DatabaseAutoscalerScaleStuck Pending-intersection term selected
kube_pod_status_phase and kube_pod_labels without job="kube-state-metrics",
unlike every other selector in this file (the rule's own HPA terms, all of
ScaleBlocked, AtMax). Each VMRule runs against a tenant's own vmselect; if that
vmselect ever carries a second series for either metric under a different job
(a tenant-run KSM alongside federated/remote-written series), the on(namespace,pod)
match becomes ambiguous and the whole rule errors out silently while its siblings
keep working. Pin the job label on both selectors so the join stays one-to-one.
The alerts unittest now asserts the pin on both terms so a revert goes red.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 16:35:57 +03:00
Alexey Artamonov 6fa79c74f4 fix(cozystack-basics): strip the autoscaler RBAC rationale from rendered output
The 8-line rationale for the tenant read-on-ScaledObject/HPA grant was a `#`
YAML comment, which Helm copies verbatim into the rendered manifest in every
tenant and which then drifts out of date the moment the grant changes. Convert
it to a `{{/* */}}` Go-template comment so the WHY stays for maintainers but is
stripped at render. No RBAC rule changes; the 58 cozystack-basics unittests
still pass and `helm template` no longer emits the prose.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 16:35:57 +03:00
Aleksei Sviridkin 4007db7a00 docs(gpu): drop two stale references in the vGPU page (#4096)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

## What this PR does

Two stale references in `docs/gpu-vgpu.md`.

The `Last verified` line says "this PR's `vgpu` variant". The variant
ships, so there is no PR left for a reader to resolve that against.

The link to `packages/system/gpu-operator/examples/README.md` calls it
the flow "that predates the `container` variant". That README now opens
with the container variant and sends non-Talos hosts to it. The link is
there because it is the Talos path.

The rest is gone. #4101 rewrote the same three passages in main, and got
right the one thing this PR had wrong: `bundles.iaas.gpuOperatorVariant`
takes `container` now, so the `disabledPackages` plus hand-written
`Package` route described here is not the way in. The rebase takes
main's text for all three and keeps only the two references #4101 did
not touch.

### Screenshots

Docs only, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Checked cozystack/website: neither line exists in
`content/en/docs/next/virtualization/gpu.md` or `vgpu.md`. The `v1.2`
snapshot carries its own PR-relative `Last verified` line, pointing at
cozystack/cozystack#2323, but that is a released version's record and
out of scope here.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Updated GPU/vGPU verification details to reference the released vGPU
variant.
- Documented the available container variant, including deployment
guidance and its scope compared with passthrough.
- Updated known limitations to reflect that the container variant is
shipped and does not configure host-device wiring through KubeVirt.
- Clarified when KubeVirt GPU device wiring is applied based on GPU
configuration settings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 12:06:22 +04:00
Alexey Artamonov 1979bd3f5c fix(cozystack-basics): let tenants read their own autoscaler objects
The autoscaling chart renders a ScaledObject into the tenant namespace and KEDA
creates a keda-hpa-<release> HorizontalPodAutoscaler beside it, but the
cozy:tenant:view aggregation granted no read on keda.sh or autoscaling. So the exact
failure the feature documents - a monitoring: true tenant whose derived vmselect
carries no kube_pod_labels, whose trigger never resolves - lands as a TriggerError
ScaledObject and a FailedGetExternalMetric HPA the tenant cannot get, surfaced only
in a root-scoped alert they cannot reach. A tenant who flips the knob and sees
nothing happen has no in-band way to find out why.

Grant get/list/watch on keda.sh/scaledobjects and autoscaling/horizontalpodautoscalers
in cozy:tenant:view:base. Read verbs only: the objects are managed by the chart and
KEDA, not edited by the tenant. Pinned by clusterroles-options unittest.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 10:48:16 +03:00
Alexey Artamonov a43e3e2709 fix(keda): digest-pin the three KEDA images
The shim rendered the operator, metrics-apiserver and admission-webhooks images
tag-only, and the values comment claimed a @sha256 pin was impossible without
patching the subchart. That is wrong: the vendored template renders
repository:{{ tag | default appVersion }}, and repository:tag@sha256:<digest> is a
valid OCI reference where the digest wins at resolution, so the pin goes straight
into the tag field of these shim values with no subchart change.

Pin all three to the multi-arch image-index digests for the 2.20.2 tag on ghcr.io,
matching how every other cozystack system image is pinned, and replace the
incorrect note. Guard the operator image keeps a digest with a helm-unittest so a
re-vendor that drops the override goes red rather than silently shipping a floating
tag. Bump the digests in lockstep with KEDA_CHART_VERSION on re-vendor.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 10:48:06 +03:00
Alexey Artamonov e89c39cb96 test(cozystack-api): cover the annotation-to-ReleaseConfig wiring
Complete() reads release.cozystack.io/* annotations into ReleaseConfig once per
ApplicationDefinition, but nothing exercised the assignments that join them. A
mutation dropping release.HelmServerSideApply = serverSideApply left the suite green
while shipping a silent nil: every Postgres HelmRelease would keep helm-controller's
server-side default, Helm would force-own the rendered spec.instances seed, and the
next reconcile would revert KEDA's live /scale count - the highest-blast-radius line
in the feature, untested.

Extract the per-CRD assembly into buildResourceFromCRD so the wiring is unit-testable
without a live discovery client, and add TestBuildResourceFromCRD covering the
apply-strategy annotation (present -> false, absent -> nil), the disable-wait sibling
on the same loop, and the healthCheckExprs pass-through. Reverting either the SSA or
the healthCheckExprs assignment now turns the test red.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 10:47:57 +03:00
Alexey Artamonov e54d728087 fix(postgres): stop an unready ScaledObject from wedging the release
A Postgres app with autoscaling.enabled but no resolvable metrics endpoint (a
monitoring: true tenant who did not point serverAddress at a vmselect carrying
kube_pod_labels) renders a ScaledObject that stays Ready=False reason=TriggerError
forever. helm-controller's default wait gates the whole HelmRelease on every
rendered resource's kstatus, so that one unready ScaledObject drives the release to
UpgradeFailed and it retries forever - wedging an otherwise-healthy database and
blocking every later chart change to it, while the tenant cannot even read the
ScaledObject to see why (observed on dev3).

Autoscaling health is a degraded-but-alive state surfaced by the DatabaseAutoscaler*
alerts, not a release-readiness gate. Add a spec.release.healthCheckExprs entry on
the postgres ApplicationDefinition that treats the ScaledObject as always-ready for
release readiness (current: "true"); the CNPG Cluster and every other resource keep
their built-in kstatus wait, so the database's own health still gates the release.
Setting any healthCheckExprs couples WaitStrategy to poller
(config.ResolveWaitStrategy); that the poller wait stays equivalent to the default
for the kstatus-checked Cluster is verified on cozystack-pr-test, not here
(render-blind). The cozyrd unittest pins the expr so a regeneration cannot drop it.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 10:47:48 +03:00
Aleksei Sviridkin 5db773892b fix(cdi): omit the CDI clone strategy override when unset
templates/cdi-cr.yaml rendered cloneStrategyOverride unconditionally,
so the chart's values offered no way to leave the choice to CDI's own
per-StorageProfile selection. The CRD enum takes copy, snapshot and
csi-clone and carries no empty member, so an empty value rendered
cloneStrategyOverride: "" which that enum does not accept, and a null
one rendered a bare key. Guard the field on a non-empty value, the
way uploadProxyURLOverride is already guarded just below it.

The default stays csi-clone, so a release that overrides nothing
renders what it rendered before. Whether csi-clone is the right
default is a separate question, untouched here.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-08 03:13:35 +03:00
Aleksei Sviridkin 648238f886 docs(gpu): drop two stale references in the vGPU page
The "Last verified" line credited the run to "this PR's vgpu variant".
The variant ships and is selectable from platform values, so a reader
on main has no PR to resolve that phrase against.

The link to the gpu-operator examples README described it as the flow
"that predates the container variant". That README now opens by naming
the container variant and sending non-Talos hosts to it, so its age is
no longer what sets it apart. What earns it the link is that it is the
Talos path.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-08 02:36:16 +03:00
Aleksei Sviridkin 4f337d62e5 fix(kubevirt): verify the vendored operator manifest and add a test target (#4099)
## What this PR does

`packages/system/kubevirt-operator` had only an `update:` target, so
`hack/helm-unit-tests.sh` skipped it: that script decides what to run by
probing `make -C <dir> -n test`. Nothing rendered the chart after a
refresh. The same recipe pulled the upstream manifest with `wget` and
ran two transforms over whatever came back, with no check on the
download.

`update` now fetches outside `templates/`, checks the download against a
pinned sha256, transforms it there, and moves it in last. The vendored
file gets written once per refresh with finished content, and a failure
at any step leaves it alone. It has to stay out of `templates/` because
Helm renders every file in there, and that directory also holds
`prometheusrule.yaml`, which is ours and not upstream's.

Upstream ships no checksum for `kubevirt-operator.yaml`.
`v1.9.0.tar.gz.asc` signs the source tarball, not the generated
manifest. So I measured the pin: downloaded the v1.9.0 asset, hashed it,
got `f11307caafc3c23ffedf9887d8beb5a4419e2694da242fa68f63d1ec820de2e0`,
and confirmed the recipe's two transforms turn that download into the
committed `templates/kubevirt-operator.yaml` byte for byte. It catches
an asset that moved under its tag. It is not authenticity verification,
and the comment says that.

The digest command is resolved at parse time: `sha256sum` when it is on
PATH, `shasum -a 256` otherwise. Worth saying which one actually ran,
because the usual claim about macOS is out of date: this machine picked
`sha256sum`, and it is not GNU coreutils but `/sbin/sha256sum`, a native
Darwin binary hardlinked to `/sbin/md5` that reports `sha256sum (Darwin)
1.0` and prints the same `<digest> <path>` format. I checked the
fallback separately with `make update SHA256SUM='shasum -a 256'`, which
finished and left the tree clean. The fallback still earns its place for
older macOS, but it does not make `update` portable on its own: `sed -i`
and `awk -i inplace` in the same recipe need GNU sed and gawk, and the
comment now says so.

`test:` runs `helm unittest`, then greps the vendored manifest for
`RELEASE`, same shape as `packages/system/reloader`. helm-unittest
cannot see `RELEASE`, so without the grep a pin bumped without
re-running `update` leaves every assertion green against the old
release. The suite pins that the `PrometheusRule` still renders, and
that the `virt-operator` Deployment carries the pinned release in its
image, `VIRT_OPERATOR_IMAGE` and `KUBEVIRT_VERSION`, plus both
transforms.

Every assertion is mutation-tested: removing `prometheusrule.yaml`,
drifting each of the three version strings, reverting the namespace
rewrite, adding a `Namespace` document back, renaming the Deployment,
and bumping `RELEASE` without re-vendoring all go red. `any: true` on
the negated `containsDocument` is doing real work, drop it and the suite
stays green with a `Namespace` document present. The sweep now runs 84
package suites, one of them new, all green.

Closes #4031.

### Screenshots

No UI changes.

### Downstream repositories

Walked the trigger map against the diff. Two entries look close and
neither fires. The website triggers on developer tooling and package
Makefiles, but `content/en/docs/next/development.md` documents `make
update` only as "Update Helm chart and versions from the upstream
source", which still holds, and its `make test` line refers to the e2e
sandbox in `packages/core/testing`, not per-package unit tests. `ccp`
triggers on `hack/` layout and what a make target does, and its skills
anchor on `hack/package.mk` and `hack/common-envs.mk`; nothing under
`hack/` is touched and its `package-bump` skill handles vendored charts,
in-repo image builds and version enums, none of which is this package's
raw manifest download. Everything else in the map needs an app, a
schema, an installer value, a variant or a release asset, and this PR
changes none of them.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubevirt): kubevirt-operator now verifies the upstream manifest against a pinned sha256 before vendoring it, and the package has a unit-test target so hack/helm-unit-tests.sh no longer skips it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved manifest updates so incomplete downloads or integrity-check
failures no longer overwrite the existing working manifest.
  - Temporary download files are cleaned up after failed updates.

- **Tests**
- Added validation for the bundled KubeVirt operator manifest, including
the expected version, image configuration, namespace, monitoring alert,
and removal of an unwanted namespace resource.

- **Chores**
- Pinned the KubeVirt operator release and verified downloaded manifests
against their expected checksum.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:53:03 +04:00
Aleksei Sviridkin a6caad7c57 fix(platform): correct what disabledPackages does to ouroboros (#4098)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

`bundles.disabledPackages` does not uninstall anything. Two comments in
the platform chart said it does, offered next to `kubectl delete
package.cozystack.io cozystack.ouroboros` as if the two were
interchangeable.

A name in that list only stops `_helpers.tpl` emitting the Package
document, which is the same suppression you already get from leaving
`publishing.proxyProtocol` false. Every emitted Package carries
`helm.sh/resource-policy: keep`, and that annotation is there precisely
so disabling a package stops deleting it, so Helm leaves the Package CR
alone. The CR stays, the HelmRelease it owns stays, ouroboros keeps
running, and the pre-delete hook never fires because nothing uninstalls
the release. Deleting the Package CR is what gets there: GC takes the
HelmRelease, Flux uninstalls, the hook runs on the way out.

There was a second cost. The guard in the system bundle looks up the
live Package CR and never reads `disabledPackages`, so an operator who
followed the old advice hit the guard anyway with no idea why. New test
case pins that.

Comments only in the templates, so rendered output is unchanged. Chart
suite green at 156 tests.

### Screenshots

Not a UI change.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff. One repo is reached, and I left
its box empty rather than tick it without a link.

`cozystack/website` has the same wrong instruction verbatim in three
files: `content/en/docs/next/networking/hairpin-proxy-protocol.md:122`,
and the identical line under `v1.4/` and `v1.5/`. It tells the reader
that removing the Package CR "(or add it to `bundles.disabledPackages`)"
triggers helm uninstall and fires the pre-delete hook. It also
contradicts `operations/configuration/components.md:72` on the same
site, which says `disabledPackages` will not remove components that are
already installed. The site fix is cozystack/website#683. It turned out
to be four files rather than three: `origin/main` there also carries a
`v1.6` tree, and all four copies were byte-identical.

Nothing else is reached: no package added or renamed, no
`values.schema.json`, no version enum, no default, no key the Ansible
role sets, nothing under `hack/`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/683
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(platform): the platform chart no longer says `bundles.disabledPackages` uninstalls a package. A name in that list only stops the Package document being rendered. `helm.sh/resource-policy: keep` leaves the existing Package CR on the cluster and the component keeps running, so deleting the Package CR is what uninstalls it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Clarified proxy protocol safety guidance, including the effects of
disabling the Ouroboros package and the limitations of using
`bundles.disabledPackages`.
- Documented that disabling package rendering does not remove an
existing live package or stop its associated resources.

- **Tests**
- Added regression coverage confirming that disabling the package does
not bypass the proxy protocol safety guard when the live package remains
active.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:52:00 +04:00
Aleksei Sviridkin 1fc990dd7f fix(cdi): update CDI to v1.66.1 (#3920)
## What this PR does

CDI v1.64.0 -> v1.66.1, operator manifest refetched with `make update`.

The bump was stuck since v1.65.0: its bundled qemu-img broke HTTP
imports to block volumes on 4Kn devices (DRBD/LINSTOR). This PR first
targeted v1.66.0 on the assumption that
kubevirt/containerized-data-importer#4179 had closed it, and that was
wrong: the qemu-img in the v1.66.0 importer is 10.1.0-21.el9, which does
not carry the first-sector fix, and the E2E run on that head failed
exactly there. The fix is in 10.1.0-24.el9, which reached the release
branch through kubevirt/containerized-data-importer#4274 and shipped in
v1.66.1. Verified by running the released image: `cdi-importer:v1.66.1`
reports `qemu-img version 10.1.0 (qemu-kvm-10.1.0-24.el9)`.

The other v1.64.0 symptom is a resourceVersion storm on the prime PVC
during WFFC imports, which makes the scheduler retry FailedScheduling
dozens of times per import; it was reported by a production user
importing golden images on a local StorageClass. That one is fixed by
commit e547586e9, a one-line guard in `updatePvcFromPod` that clears the
bound-condition annotations only when they came from scratch-space
creation. It is in v1.66.0 and later. The commit is squashed under a
title about kubevirtci lanes, so it appears in no release note and
searching for it by symptom finds nothing.

Two local edits to the vendored manifest survive refetches now. The
startupProbe patch gets the correct -p4 strip level, as written it never
applied from the package directory. The control-plane tolerations were
hand-edited into the manifest with no patch behind them, so any refetch
silently dropped them; they live in patches/tolerations.diff now. `make
update` round-trips to a byte-identical file.

Behavior changes checked against this repo: /metrics on CDI components
now requires a bearer token, nothing here scrapes it. Scratch PVCs
inherit the target StorageClass instead of the cluster default (since
v1.65.0). The PVC-rendering webhook is always registered now, but only
matches PVCs labeled cdi.kubevirt.io/applyStorageProfile=true, which
nothing here sets.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(cdi): update CDI to v1.66.1. Fixes HTTP imports to block volumes on 4Kn devices (DRBD/LINSTOR) with the repaired qemu-img, and the resourceVersion storm on prime PVCs during WaitForFirstConsumer imports.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Kubernetes tolerations for critical add-on and control-plane
nodes.
* Added resource claim requests and webhook PVC rendering configuration
to CDI custom resources.
* Added permissions for token reviews, metrics, and profiling endpoints.
  * Updated affinity feature-gate documentation to beta status.

* **Improvements**
  * Updated CDI operator and operand images to version 1.66.1.
  * Refreshed Kubernetes schema metadata and deployment configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:51:10 +04:00
Aleksei Sviridkin d0fce925cf fix(platform): declare the injected _cluster values in system charts (#4102)
## What this PR does

Ten charts under `packages/system` read `_cluster`, and `bucket` and
`monitoring` also read `_namespace`. The platform injects both through
the `cozystack-values` Secret, so installs never notice. Rendering a
chart on its own does: `helm template packages/system/kubevirt` dies on
`index of untyped nil` before it emits anything, and the `_namespace`
readers die on `nil pointer evaluating interface {}.host`.

Each of the ten now declares the keys it reads, with an empty default.
Injected values merge over chart defaults, so an installed release
renders exactly what it rendered before. Checked per chart against a
realistic injected `_cluster`/`_namespace`: six came out byte-identical,
and the other four differ only in `randAlphaNum` secrets, which also
differ between two renders of the same tree.

The declaration on its own is not worth much, so every chart also gets a
suite that renders with nothing injected. The existing suites all set
`_cluster` themselves and cannot see whether `values.yaml` still
declares it. Drop a declaration and exactly that chart's suite goes red,
with the error its comment quotes; I ran that for all twelve. None of
the new suites carries a suite-level `templates:` list, on purpose: that
list narrows what helm-unittest renders, and would hide the same
breakage in a template added later.

`kubevirt-cdi` had no `tests/` and no `test:` target, so it gets both.
`etcd-operator` reads `_cluster` too and stays as it is, since its one
call site already guards with `| default dict`. `cozystack-basics`
already declared the key.

Closes #4030.

### Not in this PR

No CI job runs `helm lint` or a standalone `helm template`, which is why
none of this was red. A gate is worth proposing separately. Its value is
narrow though: `_cluster` is always populated on a real install, and
eight of these ten already had a suite rendering the affected template
with `_cluster` supplied. The two with nothing were `kubevirt`, whose
`vm-exportproxy-*` templates no test touched, and `kubevirt-cdi`, which
had no tests at all.

`helm lint` still exits 1 on `keycloak` and `monitoring` with `chart
metadata is missing these dependencies: cozy-lib`. Both carry a
`charts/cozy-lib` symlink that `Chart.yaml` never declares. That one is
not a two-chart item: 82 charts across
`packages/{system,apps,extra,core}` fail lint the same way, which is
invisible because no job runs `helm lint`. Separate defect, untouched
here.

Fifteen charts outside `packages/system` break the same way:
`apps/{bucket,foundationdb,harbor,kubernetes,mongodb,nats,opensearch,postgres,vpn}`
and `extra/{bootbox,etcd,gateway,info,ingress,seaweedfs}`. `cozy-lib`'s
own helpers already guard every access with `| default dict`, so routing
those reads through the library would close the class instead of
patching fifteen more values files.

`kubevirt`'s suite only pins that the routes stay absent with nothing
injected. Nothing pins that they show up once `_cluster` selects the
service.

### Screenshots

No UI change: the diff is ten `values.yaml` files, ten helm-unittest
suites, and one package `Makefile`.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file by file
against this diff. No package is added, renamed or removed under
`packages/apps` or `packages/extra`;
`packages/core/platform/values.yaml` is untouched; no variant, bundle,
component, release asset, namespace or `ApplicationDefinition` semantic
changes; no `values.schema.json` and no version enum moves, so the
provider's hand-written schemas and defaults are unaffected; nothing
under `hack/` moves and no existing make target changes behaviour. The
one `Makefile` edit adds a `test:` target to
`packages/system/kubevirt-cdi`, matching the target the other nine
already have, so `make generate` and the ccp skills that drive it see
nothing new.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): declare the platform-injected `_cluster` and `_namespace` values in the ten system charts that read them, so each chart renders with `helm template` outside a cluster instead of failing on a nil map. No change to what an installed release renders.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved standalone Helm chart rendering when platform-provided values
are unavailable.
- Prevented rendering failures across system charts, including bucket,
dashboard, Keycloak, monitoring, KubeVirt, and related components.

- **Tests**
  - Added comprehensive standalone rendering coverage for system charts.
- Added validation for expected ingress, gateway, TLS route, and
resource output under different configuration scenarios.
  - Added a chart test command for KubeVirt CDI.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:50:43 +04:00
Aleksei Sviridkin 675163399d fix(platform): accept the container gpu-operator variant in iaas (#4101)
## What this PR does

`bundles.iaas.gpuOperatorVariant: container` failed the whole platform
render. The `cozystack.gpu-operator` PackageSource ships that variant,
`values-container.yaml` configures it deliberately, and
`hack/check-gpu-operator-variants.bats` covers it, but the `iaas` bundle
guard only accepted `default` and `vgpu`.

The exclusion made sense when it was written and stopped making sense a
few minutes later. The variant and the guard came from parallel
branches, neither an ancestor of the other, variant merged first. Before
the guard the bundle hardcoded `default`, so `container` was selected by
hand-writing a Package CR, and leaving it out of a bundle-side list cost
nothing. The guard made the bundle the variant owner and closed that
route without noticing it was one. Its comment calls itself typo
protection and names `passthrough` as the example typo.

Allowing the value is not the whole fix. The same file puts
`extraFeatureGates: [HostDevices]` on the KubeVirt CR whenever
gpu-operator is enabled, and builds a `permittedHostDevices` table under
it. Both are VM-facing. The container variant keeps the host driver
bound and pins `vfioManager` off, so no GPU reaches a VM: the gate would
advertise something the node cannot serve, and the table would name
resources no sandbox device plugin registers there. Both now render for
`default` and `vgpu` only, and `container` gets the gpu-operator Package
with the kubevirt component untouched.

The helm-unittest case that pinned the rejection is gone. Two replace
it: the Package renders at the container variant with no component
overrides, and the KubeVirt wiring stays absent even with
`gpu.permittedHostDevices` and `gpu.mediatedDevicesConfiguration` set.
Comments and docs that called the mirroring unconditional on variant, or
the container variant unmerged, are fixed in the same pass.

No issue for this one, so the history is in the body.

### Screenshots

No UI change.

### Downstream repositories

Walked the trigger map against the diff. The only file it touches that
appears there is `packages/core/platform/values.yaml`, and the website
hit is not the hand-written key table (that page has no GPU section): it
is the documented workaround this change makes obsolete.
`content/en/docs/next/operations/gpu-container-workloads.md` currently
tells readers the bundle rejects `container` and routes them to a
hand-crafted Package CR, and `virtualization/gpu.md` / `vgpu.md`
describe the KubeVirt mirroring as unconditional on variant.

Filed as an issue rather than a PR because the docs must not land ahead
of this change, and the `v1.6` copies describe released behaviour and
stay correct as they are.

Nothing else matches: no package added, renamed or removed, no
`values.schema.json` or version enum touched, no `ApplicationDefinition`
semantics, no installer values key, no `hack/` layout change, no
platform variant or bundle recomposed. `gpuOperatorVariant` appears in
none of `ccp`, `ansible-cozystack` or `terraform-provider-cozystack`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/685
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): accept `container` as a value for `bundles.iaas.gpuOperatorVariant`, so the gpu-operator variant the platform already ships can be selected from platform values instead of a hand-crafted Package CR. The KubeVirt `HostDevices` feature gate and `permittedHostDevices` table are now emitted for the `default` and `vgpu` variants only; `container` serves GPUs to pods and gets neither.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `container` GPU operator variant for workloads that do not
require VM passthrough.
* Container workloads no longer receive KubeVirt host-device or
mediated-device GPU wiring.
* Clarified GPU configuration behavior for passthrough, vGPU, SR-IOV,
and container workloads.

* **Documentation**
* Updated passthrough guidance, configuration references, examples, and
KubeVirt wiring instructions for all supported variants.
* Added details for manually configured package resources and
platform-specific container setup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:50:33 +04:00
Aleksei Sviridkin b7e60855e9 test(tests): wire the gpu-operator suite into the unit sweep (#4110)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

`hack/helm-unit-tests.sh` decides whether to enter a package by asking
`make -C <pkg> -n test`. `packages/system/gpu-operator` declared only
`update:`, so the sweep skipped it silently and the three test cases in
`tests/kernel_module_config_test.yaml` had never run. They pass, so
nothing was hiding behind them. Still worth fixing: the directory reads
as coverage to whoever opens it, and a suite nothing runs is where a
stale assertion sits.

The target is phony. `hack/package.mk`'s `.PHONY` line lists only the
targets that file defines, so a path named `test` next to the Makefile
would make the target up to date and skip the recipe.

Rather than fix the one package I enumerated the rest of the tree. 84
packages hold a `tests/` directory, 83 ship at least one
`tests/*_test.yaml`, and gpu-operator was the only one of those without
the target. The 84th is `packages/system/nfs-driver`, whose `tests/`
holds a PersistentVolumeClaim and a StorageClass to apply by hand. No
suite there, so a `test:` target would only make the sweep fail it for
reporting zero suites. Left alone.

`hack/package-test-target-coverage.bats` pins the class. Every other
guard in the sweep runs inside a directory it already reached, so a
package with no target escapes all of them at once and the run still
reports success. The new case probes with `make -n` instead of grepping
for `test:`, which is the question the sweep asks, and it mirrors the
sweep's other precondition too: the sweep requires `$dir/Makefile` and
returns before probing, where make would also accept `GNUmakefile` or no
makefile at all. The verdict comes from the sweep's command unchanged,
and a failing package is probed a second time with `LC_ALL=C
--no-print-directory` purely to read the diagnostic, since both of those
are visible to a Makefile through `LC_ALL` and `MAKEFLAGS`.

Mutations, because a guard that has never been red proves nothing:

```text
target removed from gpu-operator  -> red, names packages/system/gpu-operator, quotes make's error
Makefile renamed to GNUmakefile   -> red; make still answers 0 here, so this leg is load-bearing
glob pointed at a missing dir     -> red via the blindness leg, instead of passing on an empty set
unmodified tree                   -> green, 83 packages scanned
```

The full `hack/helm-unit-tests.sh` sweep now enters gpu-operator and
finishes green.

One detail that is invisible on a Mac: the probe runs under
`--no-print-directory`. On modern GNU make `-C` turns on `-w`, and the
fatal path prints `Leaving directory` after the error, so `tail -n 1`
picks up the directory trace and drops the diagnostic. Measured on 4.4.1
where that happens, and on 3.81 (still `/usr/bin/make` on macOS) where
`-C` prints no trace at all.

### Screenshots

No UI changes.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against both changed files. Two entries come
close and both miss. `ccp` gates on `hack/package.mk`,
`hack/common-envs.mk` and the `packages/{apps,system,extra,core}`
layout; nothing here is moved or renamed and no existing target changes
what it does, so a new `test:` target sitting beside 80 identical ones
is not something its skills read. The `website` entry for developer
tooling points at `content/en/docs/next/development.md`, whose Testing
section covers the e2e sandbox under `packages/core/testing` and says
nothing about the per-package helm-unittest sweep.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(tests): run the gpu-operator helm-unittest suite, which the sweep had been skipping because the package declared no `test:` target
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Tests**
- Added coverage checks to verify that all Helm test suites are included
in the package test sweep.
  - Added diagnostics for packages missing a runnable test target.

- **Bug Fixes**
- Added a test target to the GPU operator package, enabling its Helm
tests to run through the standard package test workflow.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 02:49:10 +04:00
Aleksei Sviridkin a3ddba441f fix(opensearch-operator): set vm.max_map_count from a node DaemonSet
OpenSearch refuses to start when vm.max_map_count is below 262144 and
the store may use mmap. The kernel default is 65530 and nothing in the
platform's node configuration raises it: the Talos profiles declare no
sysctls, and neither does any documented node-preparation path.

The value cannot come from the OpenSearch pod. The kernel does not
namespace vm.*, so kubelet refuses it in a pod's securityContext.sysctls
and in --allowed-unsafe-sysctls alike, both of which admit namespaced
sysctls only. The operator's own answer, spec.general.setVMMaxMapCount,
appends a privileged init container to every OpenSearch pod. Tenant
namespaces carry no PodSecurity label of their own and take the cluster
default, which Talos ships as baseline through the apiserver's
PodSecurityConfiguration; that default rejects the container, so the
StatefulSet is created and never produces a pod.

Set the value once per node instead, from a DaemonSet in the operator's
own namespace. The init container raises the limit to a floor rather
than assigning it: the tunable is node-global, so a node already tuned
higher for another mmap-heavy workload keeps its own value. The pod runs
at system-node-critical: it is a precondition for every OpenSearch pod on
the node, so it must not lose a scheduling race to the workload it
enables, though priority is not start ordering: a pod that starts first
just fails its own bootstrap check and retries. The idle container runs
the sleep under a shell with a TERM handler, because PID 1 of a
namespace receives only the signals it installed a handler for and a
bare sleep would be SIGKILLed after the full grace period on every
rollout and drain. The write
goes to /proc/sys only, with no hostPath and no edit to the node's
on-disk configuration; the pod restarts with the node and reapplies the
value. The component is declared privileged so the platform labels that
namespace enforce=privileged, which is what this DaemonSet needs and
what tenant namespaces deliberately do not get.

The file returns to a path bb660b57c emptied. That earlier revision
bind-mounted the host's /etc to append to sysctl.conf and ran an
unpinned image; this one writes no host file and pins the image by
digest.

This removes the reason for an OpenSearch pod to ask the operator for
the sysctl, but the app chart still asks. Dropping that request is a
separate change and is what stops the pods being rejected.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-08 00:35:06 +03:00
Aleksei Sviridkin 59731429a8 fix(vm-instance): let complete resources replace the instance type
KubeVirt refuses a VirtualMachine that carries an instancetype matcher and
explicit domain sizing at once. validateCPU in pkg/instancetype/apply
conflicts on domain.cpu.cores and domain.cpu.sockets when they are
non-zero, and validateMemory conflicts on
domain.resources.requests.memory whenever the key is present. The chart
rendered both, because the matcher keyed off instanceType while the
domain sizing keyed off resources, and neither side looked at the other.

values.yaml ships instanceType "u1.medium", so clearing that field in the
dashboard only falls back to the default. A user who then fills in cpu,
sockets and memory gets a VM the webhook rejects, with an error naming
fields the form gives them no way to clear.

Resources that size the VM completely now replace the matcher, the same
way the kubernetes-nodes chart sizes its worker VMs. Beside an instance
type, sizing that is only partial stops the render, because there is then
no safe reading: dropping the matcher would size the VM from fields the
user never finished, and keeping it either loses to the webhook or
silently discards what was typed. With no instance type in play there is
nothing to conflict with, so that half is left as it renders today.

Completeness is decided from what domainResources emitted, not from which
keys the user set, and the two readings come apart. resources.cpu on its
own emits nothing, so it conflicts with nothing and goes on rendering a
plain instancetype-sized VM. cpu and sockets pass through int64, which
yields 0 for a quantity the schema accepts, so a cpu of 500m beside a
whole sockets and a memory would read as complete from the values, drop
the matcher, and hand KubeVirt a cores of 0 the user never asked for. And
a resources block nulled outright leaves .Values.resources nil, where a
raw field read is a render-time nil pointer and the emitted object is
merely empty.

The existence check and the update hook read the same effective value.
Keying the lookup on the raw one would fail a resource-sized render over
an instance type it never references, which is the same confusing error
this fixes. Reading .Values in the hook would see the leftover
instanceType, skip the removal patch, and leave
spec.instancetype.revisionName on a VM Helm had just given explicit
domain resources.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-08 00:26:21 +03:00
Aleksei Sviridkin d949099f5e fix(ci): map the recurring app scopes to their area labels
Scopes that PR titles use for apps under packages/apps were missing
from the labeler's scopeToArea table. A PR whose title carries only
such a scope falls through to the area/uncategorized fallback, and the
area label has to be set by hand afterwards.

Each scope added here clears the bar the table's own comment sets for
a new entry, counted over the whole PR history in both title forms the
labeler parses, and has an unambiguous home among the labels that
already exist. A scope that clears the bar but has no such home is
left out rather than pushed into the nearest label: AGENTS.md asks for
a new area/* in that case, which is a separate decision.

The homes come from the label descriptions. area/virtualization names
cdi, and vm-disk renders a DataVolume while vm-instance renders a
VirtualMachine. area/kubernetes is the tenant Kubernetes app, and
kubernetes-nodes is the worker half split out of it. area/database is
described as managed databases and already lists kafka, so rabbitmq,
nats, qdrant and foundationdb sit there without stretching it. vpc
renders Vpc and Subnet, and area/networking already names kube-ovn.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-07 23:43:38 +03:00
Alexey Artamonov 3a4bade677 docs(postgres): document the kind-wide scope of the client-side apply flip
The release.cozystack.io/helm-server-side-apply annotation moves every Postgres
HelmRelease to client-side apply, including releases that never enable
autoscaling. IvanHunters flagged that this platform-wide side effect was
unexplained and that non-autoscaled releases silently lose SSA drift
re-assertion (client-side diffs previous-vs-new render only, never live state;
driftDetection is configured nowhere in the repo).

Record why the scope is deliberate rather than narrowing it. helm-controller
v1.5.0 resolves an unset Upgrade.ServerSideApply through "auto" from the last
release's apply method, so gating client-side on "autoscaling active" would
force an SSA->client-side ownership handoff on the live spec.instances the
moment a tenant enables autoscaling - the exact live-field handoff the seed
design was chosen to avoid. Keeping the apply method stable for the kind's whole
lifecycle keeps enable/dry-run/transition/disable clean two-way merges with no
apply-method transition. The drift-re-assertion cost on never-autoscaled
releases is called out as an accepted trade-off, with a per-instance apply
strategy as the follow-up if it is ever needed.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 22:06:21 +03:00
Aleksei Sviridkin e104c39f55 docs(rabbitmq): say that deleting the application leaves the TLS Secrets behind
The README covered only the disable path. Deleting the application hits
the same cert-manager behaviour: the chart renders Certificate and Issuer
objects and never the Secrets, and cert-manager does not own what it
issued, so the CA and server private keys stay in the tenant namespace
with nothing referencing them. Name the two Secrets and the cleanup.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-07 21:59:35 +03:00
Alexey Artamonov 678d22f209 docs(postgres): move the serverAddress namespace-pin note out of a template comment
Closes IvanHunters' MINOR on scaledobject.yaml:97. The API-owner-gate marker was a Go
template comment, stripped at render, so the ownership question and the pin's residual
surface were recorded where nobody reads them again. State the constraint on the
serverAddress field doc (propagated into values.schema.json, README and the cozyrds
openAPISchema) including the accepted residual: tenant-root is always in the allow list
because the derived default points there, so a tenant can still aim the operator at a root
Service as a blind in-cluster probe (no response body reaches the tenant), which the egress
NetworkPolicy does not narrow. The API-owner sign-off is tracked on the PR, not in a
stripped comment.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 21:03:15 +03:00
Alexey Artamonov 1192922a88 docs(keda): document the teardown order for the ScaledObject finalizer
Closes IvanHunters' MINOR on keda/values.yaml:19. The resource-policy: keep comment
reasons about an admin removing the keda package but covers only the CRDs, not the
finalizer.keda.sh KEDA stamps on every ScaledObject: with the operator gone a tenant
deleting their Postgres app gets a ScaledObject stuck Terminating and a HelmRelease
uninstall that hangs, recoverable only by an admin finalizer patch. Note that autoscaling
must be disabled on every tenant Postgres before the package is removed, so the operator
clears the finalizers while it is still up.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 21:03:15 +03:00
Alexey Artamonov 07f1daa22d docs(postgres): correct the replicas "restored on disable" claim
Closes IvanHunters' MINOR on values.yaml:58. The replicas description (propagated into
values.schema.json, README and the cozyrds openAPISchema) said the count is "restored on
disable". Under client-side apply it is not: the two-way merge writes instances only when
the rendered value changes, so a cluster KEDA grew past replicas stays there after
autoscaling.enabled: false with no autoscaler left to reduce it — the sibling dryRun doc
already states the true rule, so the two contradicted each other. Reword both the @param
and the @typedef to say the live count is not auto-restored and that the operator lowers
replicas to reclaim the grown capacity; regenerated the derived artifacts.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 21:01:30 +03:00
Alexey Artamonov 7e664d269b test(postgres-rd): pin the client-side apply annotation on the cozyrds AppDefinition
Closes IvanHunters' MAJOR on cozyrds/postgres.yaml:12 (coverage half). Dropping the
release.cozystack.io/helm-server-side-apply annotation left the whole suite green: the
Go builder test pins the plumbing that consumes it but nothing pinned the annotation at
the point it is set, so the single load-bearing invariant of the feature — client-side
apply, without which server-side apply reverts KEDA's live /scale on every reconcile —
was unguarded. Add a helm-unittest that renders the ApplicationDefinition and asserts the
annotation, plus a test target so hack/helm-unit-tests.sh runs it; removing the annotation
now goes red.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 20:58:21 +03:00
Alexey Artamonov 568ccf4835 fix(keda): gate the operator CiliumNetworkPolicy on the cilium.io/v2 API
Closes IvanHunters' MAJOR on keda/values.yaml:60. The shim hardcodes
networkPolicy.flavor: cilium and the vendored templates gate only on
networkPolicy.enabled + flavor==cilium, never on the cilium.io/v2 CRD being served.
keda is offered on every variant from the shared system bundle, so on a variant that
ships no Cilium the CiliumNetworkPolicy is unmappable and the whole HelmRelease fails
to install. Add a .Capabilities.APIVersions.Has "cilium.io/v2" check to the three
CiliumNetworkPolicy templates so the package installs on any CNI; where Cilium is not
served the egress backstop is simply absent and the chart's serverAddress namespace-pin
remains the control. The vendored edit is captured as patches/ so make update re-applies
it instead of silently reverting.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 20:56:18 +03:00
Alexey Artamonov 3a09c3e9dc fix(postgres): guard dry-run/transition against the paused HPA min-clamp
Closes IvanHunters' MAJOR on scaledobject.yaml:39. The render guard rejected
replicas > maxReplicas but not the symmetric low side. Under dry-run/transition the
chart renders the static instances=replicas while the ScaledObject carries
minReplicaCount=effectiveMin with both scale directions paused. The pause disables
the HPA's scaling policies, not its min-replica clamp, so when replicas < effectiveMin
the HPA raises the live count to the floor on its first reconcile — a scale-up in a
mode documented as "actuates neither way" — and because the rendered instances value
never changes, the client-side two-way patch never pulls it back, leaving permanent
extra standbys and PVCs. Fail closed and tell the operator to stage replicas to the
floor first. Under dry-run/transition this floor guard is strictly broader than the
db.yaml quorum wedge (any replicas <= maxSyncReplicas is also < effectiveMin), so it
supersedes it there; the wedge stays exercised on the disabled path.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 20:48:11 +03:00
Andrei Kvapil d3559d48ed feat(migration): tenant self-service VM import from VMware vSphere (#3931)
## What this PR does

Adds tenant self-service import of virtual machines from VMware vSphere,
as a pair of CRDs in a new `migration.cozystack.io` group reconciled by
a controller in core. Konveyor Forklift is the transfer engine
underneath; tenants never see it.

A `VMImportSource` registers a long-lived connection to a provider —
type, endpoint, credentials entered on the spec and materialized into a
Secret by the controller, because tenants cannot create Secrets in
Cozystack. A `VMImportTask` runs the one-shot operation: reference a
source, name the machines, get `VMDisk`s and `VMInstance`s. The outputs
carry no owner reference back to the task, so deleting a finished import
removes the migration machinery and nothing else.

This supersedes the chart-based tenant API in #1982 while keeping its
Forklift packages unchanged, carried over here with attribution. The
design is written up in cozystack/community#62, which records why a Helm
chart lost to a controller: `apps.cozystack.io` is a projection over
HelmReleases, so "the outputs survive deletion" would need
`helm.sh/resource-policy: keep`; a completed migration is terminal while
Flux reconciles toward a steady state forever; and the number and size
of the disks an import produces is only known after the source VM has
been inspected, which a template cannot express.

Two things are worth calling out for review.

**The controller ships no vSphere client.** Forklift already
authenticates, validates VM references and reports unmapped networks and
datastores, and the KubeVirt VirtualMachine it builds carries the source
VM's CPU, memory and disk order — so the inventory is read from
Forklift's own objects rather than from a second client that could
disagree with the one doing the work. A tenant sees Forklift's verdicts
verbatim rather than a paraphrase.

**The volume handoff copies nothing.** The transferred PersistentVolume
is retained and re-bound into the claim a `VMDisk` expects, with
`cdi.kubevirt.io/storage.populatedFor` telling CDI the claim is already
populated — the same primitive the Velero VM-restore path already
depends on in production. That closes both the raw-PVC gap raised in
review on #1982 and the duplicate-copy waste tracked in #3924.

Everything is opt-in: three packages added to `bundles.enabledPackages`,
and one platform value (`vmImport.vddkImage`) naming the VDDK image,
which Cozystack cannot ship because it is built from a proprietary
VMware SDK. Empty is a supported state and makes a vSphere source report
itself unavailable at registration rather than mid-transfer.

### Validation

Run on a live Cozystack v1.6.2 cluster and rolled back afterwards.
Confirmed working end to end: CRD install and printer columns, the VDDK
policy (including that no credentials Secret is projected for a source
that cannot work), credential projection with the ownership guard, the
Provider pair with the operator's image stamped on it, the local-cluster
destination Provider reaching Ready/Connected/Inventory, Forklift's
connection verdict mirrored onto the source verbatim, the diagnostic
chain from Forklift through the source to the task, that nothing is
created while a source is unready, and that deleting a source
garbage-collects both Providers and the Secret.

That testing found four defects, none of which were visible from reading
code, and all are fixed here: the task controller could not start at all
when Forklift is absent (a watch on an absent CRD never syncs, so
controller-runtime refused to start it and the pod sat at 0/1); a
vSphere secret needs a CA certificate rather than a thumbprint, so as
first modelled every source would have sat at `SecretNotValid` forever
with a reason that reads like a credentials problem; Forklift's absence
was returned as a reconcile error, so a source kept a stale condition
indefinitely while the controller hot-looped; and the controller's own
Deployment violated the restricted Pod Security Standard, which is
indefensible for a feature whose design argument is that an import stays
inside it.

A further four assumptions were corrected against operational experience
from the original implementation's work on a real vSphere environment,
which this session had no access to. Each failed silently: the
VirtualMachine Forklift creates is labelled with the Plan's **UID**, not
its name, so the disk would never have been adopted; the transfer
arrives as a CDI-populated claim with no DataVolume, carrying an owner
reference to that VirtualMachine, and clearing it was conditional on a
DataVolume existing — on the original implementation garbage collection
removed such a claim 357ms after the VM went, destroying a successfully
migrated disk; the terminal signal is the phase the engine settles on,
not only a completion timestamp; and progress read from a name-matched
pipeline step pegs at zero if a release renames it.

**What could not be tested:** everything past authenticating to vCenter
— inventory resolution, the loop that learns network and datastore
mappings from Forklift's validation conditions, the transfer itself, and
this controller's own code path through the handoff. No vCenter was
available. The handoff mechanics were separately proven by hand on the
same cluster in earlier work; what remains unproven is this
implementation of them. The mapping loop is the least-supported part of
the change and is the thing to look at hardest.

### Screenshots


https://github.com/user-attachments/assets/f313f6ba-8e72-42d9-b65a-033866ad39d5

### Downstream repositories

Walked the trigger map against the diff. `cozystack/website` is reached
on two counts and the follow-up is open as
[cozystack/website#673](https://github.com/cozystack/website/pull/673),
a migration guide covering the pre-flight checks, the administrator's
VDDK build, and the driver caveat the lab run surfaced:

- `packages/core/platform/values.yaml` gains `vmImport.vddkImage`, and
`content/en/docs/next/operations/configuration/platform-package.md` is a
hand-written table of those keys.
- Three platform components are added, which reaches
`guides/platform-stack/_index.md` and
`operations/configuration/licenses.md` — the latter is where the
proprietary VDDK image belongs, since Cozystack references it without
shipping it.

The others were checked and are not affected: no package is added under
`packages/apps/` or `packages/extra/` (terraform-provider, website app
lists), no key the Ansible role sets changes, nothing under `hack/` is
moved or renamed and no make target changes behaviour (ccp,
external-apps-example), and no node prerequisite changes (talm).

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/673
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(migration): import virtual machines from VMware vSphere into a tenant. A VMImportSource registers a connection to vCenter and a VMImportTask imports named machines, producing ordinary VMDisks and VMInstances that survive deletion of the import. Opt-in: add cozystack.forklift-operator, cozystack.forklift and cozystack.migration-controller to bundles.enabledPackages, and set vmImport.vddkImage to a VMware VDDK image you build yourself — Cozystack cannot redistribute it. Guest conversion and warm migration are not included in this version.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added vSphere VM migration support, including source connections,
import tasks, inventory discovery, progress tracking, and conversion
into Cozystack VMs and disks.
* Added dashboard pages for managing migration sources and imports, with
source-dependent VM selection and progress details.
* Added optional VM firmware settings for BIOS/UEFI, Secure Boot, and
persistent EFI storage.
* Added opt-in platform packages for migration services and Forklift
integration.
* **Security**
* Restricted access to the ingress admission webhook port with a network
policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-07 19:27:07 +02:00
Andrei Kvapil 6384c8f4d1 feat(ingress-nginx): restrict the admission webhook to the API server
The admission webhook only ever needs to be reachable by the API server,
and leaving it open to every pod turns any flaw in it into something any
workload in any namespace can reach -- which is what made CVE-2025-1974
critical rather than merely serious.

The policy is deny-only and explicitly disables default-deny: a Cilium
rule carrying an ingress section would otherwise put the controller
behind one, meaning every port it serves would have to be enumerated and
the first one forgotten would break silently.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-07 17:03:44 +02:00
Andrei Kvapil 54621bc046 feat(migration): tenant self-service VM import from VMware vSphere
Hidora's customers are leaving VMware now, and a tenant has to be able to
import their own machines without a platform administrator. The chart
approach could not express that: a tenant can neither create the Secret
holding vCenter credentials nor name the VDDK image, so both had to come
from somewhere they cannot reach.

Two CRDs in a new `forklift.cozystack.io` group, split by lifecycle. A
VMImportSource is a long-lived connection -- type, endpoint, credentials,
readiness -- and a VMImportTask is the one-shot operation that names
machines and produces VMDisks and VMInstances. The outputs carry no owner
reference back to the task, so deleting a finished import leaves the
machines alone. Konveyor Forklift does the transfer underneath; the
controller renders its objects, mirrors its verdicts, and hands each
finished volume into a VMDisk without copying it.

What a live vCenter migration forced out, and no amount of source reading
would have:

- A guest copied verbatim keeps the drivers it had under VMware and no
  virtio, so on a virtio disk it does not fail the import -- it succeeds
  and then does not boot. Imported disks go on SATA.
- VDDK connects straight to the ESXi host at the address vCenter
  advertises, which is routinely unreachable, and on the first cluster
  collided with the Service CIDR. `spec.hosts` redirects per host.
- Firmware must be carried across or a UEFI guest imports as BIOS and
  never boots (stacked on #3002).
- Deleting a task mid-transfer orphaned its DataVolume; a finalizer now
  removes the engine's volumes and leaves the outputs.

The console gains a Migration section: connections are editable, and the
VM field is a picker built from a machine list the controller publishes,
since the aggregated API cannot reach the inventory and should not hold a
source's credentials to try. Two general mechanisms fall out of that --
an option source may take an argument, and one named in a CRD annotation
may reference a sibling field.

Design: cozystack/community#62. Docs: cozystack/website#673.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-07 17:03:44 +02:00
Aleksei Sviridkin d000d02b99 feat(rabbitmq): add TLS support for AMQPS and management HTTPS (#2683)
## What this PR does

Adds TLS for AMQP and the management API, and hands the tenant a
key-free trust anchor instead of a private key.

- A chart-authored cert-manager chain issues the broker certificate. The
chart names its CA Secret in a `TenantProjection` sentinel; the
CA-extraction controller reads `ca.crt` from that Secret and writes a
key-free projection, stamping `internal.cozystack.io/tenant-ca` on the
projection it owns. That label is what the ResourceDefinition selects.
Nothing labels the CA Secret itself, which holds the CA private key and
is excluded by name.
- The leaf `<release>-tls` carries the broker's private key and reaches
no tenant. Neither the dashboard Role nor the ResourceDefinition grants
it, and the ResourceDefinition excludes it by name as well, so a label
mis-stamped onto it later still does not get through.
- SAN coverage now lists every resolvable form explicitly, including the
bare `*.<release>-nodes` wildcard that the operator uses for
`RABBITMQ_NODENAME`.

## TLS is opt-in here, unlike the sibling charts

`tls.enabled` defaults to `false` and is not inferred from `external`.
That diverges from nats, qdrant and kafka, which infer it, and the
divergence is deliberate for a reason specific to this operator.

Setting `spec.tls.secretName` makes the cluster operator rewrite the
`port` key in the `<release>-default-user` Secret from 5672 to 5671, on
the mere fact of TLS and regardless of whether the plaintext listener is
still open. That Secret is the Service Binding the chart hands the
tenant. A client with a hardcoded `:5672` keeps working. A client that
reads the endpoint from the binding, which is the documented way, gets
moved to the TLS port and must then speak TLS and be given a CA to
verify against, neither of which it was configured for. Inferring TLS
from `external` would fire all of that on an ordinary upgrade, with no
configuration change, against exactly the clients that followed the
documentation.

Whether the other engines should follow is an epic-level question rather
than this PR's to answer; for nats it is raised in #3348.

## The plaintext ports are still published, and this PR says so

There is no operator-supported way to publish the TLS ports alone.
`spec.override.service.spec.ports` is a strategic merge keyed on `port`
against defaults the operator rebuilds every reconcile, so it adds and
mutates entries but never subtracts one, and a TLS-only port list there
would read as a guarantee while changing nothing. The chart renders no
service override at all and says why at the callsite.

`tls.disableNonTLSListeners` is offered as an explicit opt-in, and it
refuses to render rather than failing silently in three situations.

- Together with `users` or `vhosts`. The flag forces
messaging-topology-operator onto TLS regardless of its own
configuration, after which it verifies the broker against the system
certificate pool, where a per-release CA cannot be. Every topology
object bound to the release would stop reconciling, silently:
already-created users keep working while password rotation and new users
simply never apply. There is no working route left for it, so the chart
refuses instead. Managing those objects as hand-written CRs is not a way
out, because the operator takes TLS from the `RabbitmqCluster` and not
from whoever created the CR, so manual objects break identically and
without the render-time refusal. `rabbitmqctl` or a definitions import
are.
- While topology objects are still live in the namespace. See the next
section.
- With TLS off, where no listener of either kind would remain.

Erlang distribution (25672) and epmd (4369) stay plaintext under every
setting; the flag does not touch them, so `disableNonTLSListeners: true`
is not "TLS everywhere".

A third route, closing AMQP at the broker through
`spec.rabbitmq.additionalConfig` while leaving the operator on plaintext
management, has been suggested and is plausible, but the operator's
sources are not vendored here and it was not verified. It is recorded as
an open question rather than promised.

## Closing the listeners takes two upgrades, and the chart enforces it

Clearing `users` and `vhosts` and setting `tls.disableNonTLSListeners:
true` in one edit is the natural thing to try, and it wedges the objects
being removed. The values-level refusal cannot catch it, because by then
both maps are empty. Helm applies every object in the target release
before pruning the ones the release dropped, so the broker closes
plaintext and loses the `rabbitmq.com/operator-connection-uri`
annotation while the topology objects still hold
`deletion.finalizers.<plural>.rabbitmq.com`. Their operator then has to
reach a management API it can only get to over a certificate it verifies
against the system trust pool. The delete never completes, and the
upgrade reports success over objects left in `Terminating`, with the
users still live in the broker and `kubectl delete namespace` on that
tenant hanging.

The chart now reads the namespace rather than the values. When the flag
is being introduced and any topology object is still there, the render
refuses and names them, gives the two-upgrade order and the `kubectl
get` that confirms nothing bound to this release is left, and says how
to unwedge a release that is already stuck. The check keys on the live
`RabbitmqCluster` as well, so it applies only to the transition: once
the listeners are closed, an object a tenant creates afterwards is
theirs to clean up rather than something that stops the release
rendering. Eleven unit cases cover the firing and the silent paths.

The set is wider than the three kinds this chart creates. All twelve
kinds `messaging-topology-operator` drives through its shared
`TopologyReconciler` carry that finalizer and share one deletion path,
so a `Queue` or a `Shovel` written by hand against the release wedges
exactly the same way, and a runbook step that checks only users, vhosts
and permissions says the namespace is clear when it is not. The list is
derived in one place and read from both the lookup and the `kubectl get`
the refusal prints, and a unit test holds it against the operator's own
CRDs so a bump cannot introduce a kind the guard has never heard of.
`SuperStream` is the one CRD left out: it runs on its own reconciler
with no finalizer, and the objects it owns are already on the list. An
object that binds through `connectionSecret` instead of naming the
cluster stays outside the guard, which the comment says.

## Breaking for metrics scrapers

Enabling TLS moves the metrics endpoint from 15692 to 15691. A
ServiceMonitor selecting the port by the name `prometheus` silently
stops matching and the instance's metrics disappear with no error
anywhere. Nothing in this repository scrapes RabbitMQ, so only
user-defined monitoring is affected, silently, which is why it is stated
here.

## Known limitations

1. Turning TLS back off leaves the raw CA material behind. The anchor
itself is collected: the projection carries an owner reference to the
`TenantProjection` sentinel, and Helm deletes the sentinel along with
the rest of the chain. What survives is `<release>-ca` and
`<release>-tls`, because cert-manager runs with
`enableCertificateOwnerRef: false` and does not delete the Secrets it
issued, so the README says to remove them by hand. #3349 asks the same
question of the label-driven leg of the contract, which this chart does
not use.
2. No e2e coverage for the TLS path. The one-upgrade wedge above is the
case that most deserves it: install with TLS and users, upgrade to no
users plus the flag, and assert the `User` is actually gone rather than
merely marked for deletion. A template test cannot reach that.

On certificate renewal, the leaf carries `duration: 8760h` and
`renewBefore: 720h`, so cert-manager rewrites the Secret around day 335.
The cluster operator mounts it as a projected volume with no `subPath`,
so the kubelet refreshes the files in place, and the operator does not
watch a user-supplied TLS Secret, so nothing rolls the pods. The last
hop, the broker picking the new file up, rests on Erlang's
`ssl_pem_cache` re-reading by mtime and has only been read from sources
rather than exercised against a live broker. The merged qdrant chart
ships the same duration and `rotationPolicy`, so this is a platform-wide
property rather than something this PR introduces.

### Downstream repositories

`packages/apps/rabbitmq/values.schema.json` gains the `tls` object, so
`cozystack/terraform-provider-cozystack` needs the field modelled for
`cozystack_rabbitmq`; tracked in
cozystack/terraform-provider-cozystack#42.

### Release note

```release-note
feat(rabbitmq): opt-in TLS for AMQP and the management API, with the CA published to tenants as a key-free trust anchor. Enabling TLS moves the metrics port from 15692 to 15691 (ServiceMonitors selecting the port named `prometheus` must be updated) and moves the `port` value in the `<release>-default-user` binding Secret from 5672 to 5671. Plaintext AMQP and management ports remain published; `tls.disableNonTLSListeners` closes them but also drops in-cluster plaintext clients, and must be enabled in a separate upgrade after the topology objects are gone.
```
2026-09-07 18:51:34 +04:00
Aleksei Sviridkin d685fb30ec fix(gateway)!: require tenant routes to declare their hostnames (#3475)
## What this PR does

The tenant route hostname policy skipped the shape it most needed to
check. Its CEL opened with `!has(object.spec.hostnames) ||`, so an
HTTPRoute or TLSRoute declaring no hostnames satisfied the first operand
and got admitted with nothing compared against the tenant apex. An empty
list passed too: `has()` is true on `[]`, and `all()` over an empty list
is vacuously true.

Such a route is not inert. Under Gateway API it inherits the hostnames
of every listener it attaches to, and `parentRefs` can name a Gateway in
another namespace, so it covers whatever that listener covers. Reported
in #3467.

The predicate now requires the field to be present and non-empty before
validating it. Both policies rendered from that template share one CEL
string, so HTTPRoute and TLSRoute get the change together, and both
match `CREATE` and `UPDATE`, so an existing route that drops its
hostnames is denied exactly as a new one is.

The issue is titled after TLSRoute, but the reachable shape is mostly
HTTPRoute. TLSRoute allows a hostname-less route only on the deprecated
`v1alpha2`; at `v1` and `v1alpha3` the bundle this repo vendors marks
`spec.hostnames` required and rejects an empty list on `minItems: 1`,
while `v1alpha2` carries neither constraint. HTTPRoute `v1` carries
neither either, so it accepts both shapes. All four are checkable in
`packages/system/gateway-api-crds/templates/crds-experimental.yaml`.

Checking that turned up a stale claim in the template header, which said
TLSRoute had no `v1` in Gateway API 1.4 to 1.5 and used it to justify
the rule naming only `v1alpha2`. Gateway API 1.5.1, which this repo
vendors, serves TLSRoute at `v1` as the storage version. The rule still
covers every served version because `matchConstraints.matchPolicy`
defaults to `Equivalent`, and that default is now written down, because
setting `matchPolicy: Exact` reads like a tightening and would silently
stop the policy seeing `v1` and `v1alpha3` TLSRoutes. I corrected the
text and left the coverage alone: the wrong sentence predates this
branch, but my change is what makes the file contradict itself, so the
text is mine to fix while widening version coverage is a separate job.

A `ValidatingAdmissionPolicy` has no cross-object lookup, so it cannot
resolve the parent Gateway to see which hostnames a route would inherit.
Requiring the route to name its own is what keeps every admitted
hostname comparable against the apex. That has a cost: a route relying
on listener inheritance now has to spell its hostnames out, and a tenant
that hand-writes one gets a denial where it used to get an admission.

### The controller change belongs in the same PR

`renderHTTPRedirect` built the http-to-https redirect HTTPRoute with no
hostnames on purpose, to catch every host on port 80. It lands in a
`tenant-*` namespace, so the stricter policy denies it,
`reconcileHTTPToHTTPSRedirect` returns that error, and since it runs
before the status update the TenantGateway never reaches Ready. That
redirect is what keeps app routes attaching by hostname from serving
plaintext on port 80. Neither half of this change is safe alone.

The redirect now names the tenant apex and its wildcard. Both entries
are needed: a wildcard hostname is a suffix match over any number of
labels, so `*.<apex>` covers subdomains at every depth but not the bare
`<apex>`. `renderWildcardCertificate` already builds the same pair.

Coverage narrows in one direction. A request on port 80 with a Host
outside the tenant apex now falls through to a 404 instead of a 301.
That narrowing costs no covered host: the port-80 listener admits routes
only from the Gateway's own namespace and cert-manager's, and the
tightened policy requires every route in the tenant namespace to declare
its hostnames inside the apex, so any host published from this namespace
already falls under the apex or its wildcard.

An inheriting child tenant does not widen that set either. Its hostnames
would reach port 443 through a listener on this same Gateway, and
`cozystack-gateway-hostname-policy` compares every listener against the
host label of the Gateway's own namespace, so a child apex outside that
label never gets its listener: the Gateway write carrying it is refused
at admission whole, in `reconcileGateway` and before the redirect is
rendered. A child apex under this one passes that check and the wildcard
already covers it. That holds at merge base as well, so it is not
something this branch changes.

Deriving hostnames from `spec.apex` makes that field load-bearing for
admission, so it gains `MinLength=1`. An empty apex used to reconcile to
Ready, since the redirect declared no hostnames at all; it would now
render `""` and `"*."` and be rejected on the HTTPRoute hostname schema,
surfacing as a validation error naming the route rather than the field.
The renderer checks it too, because the CRD and the controller binary
roll out separately and an older CRD in the cluster will not carry the
bound.

That schema bound is a breaking change to an existing field and the API
review gate flags it as one, so here is what it actually invalidates.
The chart that renders these CRs already fails on an empty
`_namespace.host`, and in DNS-01 or existingSecret mode an empty apex
already broke earlier, in `renderGateway`, on a listener with `hostname:
"*."`. Nothing that works today becomes invalid. What the bound rejects
is a TenantGateway written by hand, past the chart, with an empty apex,
and I would rather that be refused at admission naming the field than at
reconcile time in controller logs, which is why the bound sits in the
CRD as well as the renderer.

Naming those hostnames also meant excluding the redirect from
`collectHostnameClaims`. A route that attaches to the Gateway and
declares hostnames otherwise reads as an app publishing them, and each
claimed hostname becomes an HTTPS listener named after its first label.
The first label of `*.<apex>` is `*`, which the Gateway CRD's
listener-name pattern rejects, so the Gateway write fails and the
reconcile stops before anything downstream. The exclusion matches on
namespace, name and controller ownership rather than on the labels the
renderer stamps: those labels are writable by anyone who can create a
route, and the claim set also drives conflict resolution, so a
label-keyed exception would let any route opt out of being marked
`Accepted=False` when it loses a hostname race. Name and ownership are
each required and each is pinned by its own test, since ownership alone
would exclude any other controller-owned route and name alone would
extend the exception to a route planted under that name by someone else.

### Radius

Only namespaces starting with `tenant-` are matched. The three
first-party TLSRoutes the platform ships, `kubernetes-api` in `default`
plus `cdi-uploadproxy` and `vm-exportproxy`, set hostnames explicitly
and none sits in a `tenant-*` namespace, so nothing changes for them.

I went through every producer of routes in the tree, in three places
rather than one. No Helm template emits an absent or empty hostname list
in any configuration a tenant can reach: the vendored openbao, harbor,
opencost and flux-operator route templates can render one, but each is
behind a flag that defaults off with no values path to turn it on, and
openbao would fail CRD validation first because its default route
apiVersion requires hostnames. Exactly one Go site builds a route, the
redirect fixed here. And one e2e fixture did rely on the old behaviour:
the Gateway suite's parentRef probe declared no hostnames and is applied
into `tenant-test`, which the policy matches, so it now reads the apex
from the namespace label and names a hostname under it, the same way the
neighbouring cases in that suite already did.

cert-manager also writes routes into tenant namespaces, since it creates
the Challenge in the Certificate's own namespace and the HTTP-01 solver
route beside it. That route declares the hostname it is validating, so
the policy admits it.

### Related issues

Two open issues have a text half that this PR happens to settle, and
neither of them closes here. #3788 reports that the port-80 allow-list
contains the namespace its own comment says it excludes; the comment and
the README paragraph now describe what that filter actually does, while
the behavioural exit it names stays open. #3789 reports that the
TLSRoute rule names only the deprecated API version; that premise is now
written down in the template header, including what has to change before
a bundle drops `v1alpha2`, but the rule still names `v1alpha2` alone.
#3787 covers the wildcard-hostname listener name that the apiserver
rejects, which is what makes the claim-set exclusion here load-bearing
rather than tidy.

### Screenshots

N/A (no UI changes).

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. No package is added, renamed or removed; no
`values.yaml` key, `values.schema.json` field, version enum or default
moves; no `kind` or `plural` changes; no namespace or release-asset name
changes; nothing under `hack/` moves, is renamed, or changes what a make
target does.

One row deserves a sentence rather than a blanket denial, because the
diff does change a CRD schema: `spec.apex` on
`gateway.cozystack.io/v1alpha1 TenantGateway` gains a `minLength` bound,
which lands in the generated `gateway.cozystack.io_tenantgateways.yaml`.
The provider hand-types `Package` under `cozystack.io` and
`Plan`/`RestoreJob` under `backups.cozystack.io`; `gateway.cozystack.io`
is not among them, and a tightening on an existing field adds no new
field for it to offer typed access to. The
`packages/extra/gateway/README.md` edit sits in the `## Security model`
section, below the `## Parameters` block that `cozyvalues-gen`
regenerates, so it survives `make generate`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(gateway)!: the tenant route hostname policy now requires an HTTPRoute or TLSRoute in a `tenant-*` namespace to declare a non-empty `spec.hostnames`. A route that omitted the field, or set it to an empty list, was previously admitted without any hostname being checked against the tenant apex, even though such a route inherits the hostnames of the listener it attaches to rather than matching nothing. A route in a `tenant-*` namespace relying on that inheritance is now rejected and has to name its hostnames. The controller-owned http-to-https redirect route names the tenant apex and its wildcard accordingly, so a request reaching port 80 with a Host outside the tenant apex now gets a 404 instead of a 301. `TenantGateway.spec.apex` additionally gains a `minLength: 1` bound, because the redirect derives its hostnames from that field; a TenantGateway written past the chart with an empty apex is now rejected at admission instead of failing later in the controller.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added edge-terminated gateway support with plain-HTTP listeners and
TLS passthrough handling.
- HTTP-to-HTTPS redirects now explicitly cover the tenant apex and
subdomains.
  - Gateway configurations require a non-empty apex hostname.
  - HTTPRoute, TLSRoute, and GRPCRoute resources must declare hostnames.

- **Bug Fixes**
  - Invalid hostname-less routes are now rejected consistently.
- Controller-managed redirects no longer trigger unnecessary hostname
claims, listeners, or certificates.
- Switching certificate modes now cleans up obsolete ACME resources and
route status.

- **Documentation**
- Updated security and gateway guidance for hostname validation, edge
termination, route compatibility, and certificate handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-07 18:49:04 +04:00
myasnikovdaniil 7d7eb1c23e chore(codeowners): rebuild ownership by area, five owners minimum (#4129)
## What this PR does

CODEOWNERS applies only the last matching pattern and it replaces every
broader rule including `*`, it does not merge them. Half of what that
costs is already fixed: `bd7b76691` added @kvaps and @lllamnyp to the
rules that were missing them, and `hack/codeowners-invariant.bats` keeps
them there. That invariant reads "every rule with owners is a superset
of the `*` owners", and the other half of the problem satisfies it
without a complaint:

```
/dashboards/  @myasnikovdaniil @IvanHunters @kvaps @lllamnyp   <- widened by bd7b76691, 4 owners
/api/         @kvaps @lllamnyp                                 <- unchanged, nothing to add, still 2
```

`/api/` is exactly equal to the catch-all, so it passes the invariant
perfectly while gating on two people. Eleven rules on main have that
shape: `*`, `/api/`, the four `/pkg/` ones, both api-gate paths, the
api-review-gate workflow, `.github/CODEOWNERS` and `packages/apps/vpc`.
That is 182 files.

The catch-all is those same two handles, so the invariant makes them the
floor for the whole repository and every path without a rule of its own
sits on that floor too. Raising the floor means raising `*`, after which
the invariant carries the new set into every rule by itself. That is the
shape of this PR: `*` goes from two owners to five, and all fifty rules
carry those five.

Widening the owner sets is not a security relaxation, and it is worth
saying why explicitly. Write access does not change for anyone, all
twelve collaborators already have it, and CODEOWNERS never decided who
may change code, only whose approval unblocks a merge. Owner approval is
OR and always was, so one @kvaps approval already closed `/api/` on its
own and two pairs of eyes were never required there, what changes is the
size of the pool and not the number of approvals needed. What actually
holds those paths is the required checks, which are AND and which this
PR does not touch: DCO, pre-commit, codegen-drift, zizmor, scorecard,
e2e, and `api-review-gate.yaml` keeps its own `API_OWNERS: "lllamnyp
kvaps"`, so a sizeable API change still needs an approval from one of
them specifically. The AND gate is untouched, only the OR gate got
wider. And 111 and 56 reviews against 712 merged PRs over the six months
to 2026-09-06 is not strict control, it is a queue.

Three of the nine maintainers in MAINTAINERS.md own no path on main, and
the reason differs per person. @kingdonb was listed: `c13c6ee2c` gave
him `packages/system/fluxcd` and `packages/system/fluxcd-operator`, then
`7cca3d75e` deleted both packages with the FluxCD tenant addon, so the
rules went with them, correctly. Nobody moved him onto the flux paths
that stayed, `packages/core/flux-aio`, `packages/system/flux-plunger`,
`packages/system/flux-shard-operator` and the matching `cmd` and
`internal` dirs, so that one is a loose end from a breaking change and
not a decision about him. @matthieu-robin and @mattia-eleuteri were
never in the file, in any revision. Either way CONTRIBUTOR_LADDER.md
says a maintainer may approve changes to any area of the project, and
the file says otherwise.

Per package scoping also didn't match who actually reviews. In their own
areas the named reviewers turn out to be the authors: over the same six
months @androndo submitted 3 reviews across 85 storage PRs, @scooby87 1
across 22 virtualization ones, @sircthulhu 1 across 20 keycloak ones.
The reviews in all three areas came from the same few generalists that
the narrow rules were excluding.

So rules now group by area instead of by package (77 rules and 19
distinct owner sets become 50 and 9), every rule names at least five
owners, and an area whose owner set doesn't differ from its tier is left
to the tier rule. Networking, storage, databases and cluster-api lose
their per package rules for that reason, the tier rules already name the
people MAINTAINERS.md records for them. Nobody loses coverage, every
handle owns at least as much as before. MAINTAINERS.md needs no change.

One thing I loosened on purpose and want you to look at: the two owner
rule on `cmd/api-gate`, `internal/apigate` and `api-review-gate.yaml` is
gone. The rationale in the file was that a PR could otherwise neuter the
detector because it is compiled from head checkout during bootstrap.
Those paths still require code owner review under the go source and CI
rules, and any of the twelve collaborators with write access can push to
a branch anyway, so what the narrow rule bought was delay and not
protection. If we want that property it needs a required check naming
reviewers, not an owner list.

`hack/codeowners-invariant.bats` passes on the new file. Its exemption
for a rule owned by exactly @tym83 @kvaps is dead code once the
governance block is gone, and a dead exemption there is a hole, any
future two owner rule with that set would be waived, so it goes too. I
checked by mutation that the test still fails on `path @kvaps @tym83`.

Measurements above are deliberately not in the file, they date. Comments
there are down to the two mechanics that fail silently (last match
replaces, and a handle without write access ignored without error) plus
why generated mirrors are listed with no owner.

I checked the result against `git ls-files` with last-match semantics:
no pattern matches nothing, no pattern shadowed by a later one, no rule
under five owners, every handle is a current collaborator with write
access. Glob expansions (`capi-*`, `kubevirt*`, `keycloak*`,
`backup*-controller`, `flux-*`) match the dirs they are meant to.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(codeowners): code ownership is grouped by area instead of by package and every rule names at least five owners, so no path requires a review from one of two people
```
2026-09-07 19:47:11 +05:00
myasnikovdaniil 94a202bc5a test(e2e): run the merge-gating lanes on Talos containers (#4020)
Replaces QEMU substrate with Talos containers on both lanes that gate a
merge. srv1-srv3 run as containers, so tenant worker sits at L2 instead
of L3.

Measured in CI on commits where both substrates ran the same tree:

| | qemu | container |
|---|---|---|
| `kubernetes-latest` / `-previous` | 1143-2040s, red on every
vgif-absent runner | 466-809s, green in both strata |
| whole job | | 24-49 min faster |

Almost all of it is in Chainsaw. Install takes about half an hour either
way.

Node-join soft-red gate goes with it. It was a prosthetic for nested
virt: a worker that registers in minutes elsewhere could miss any
deadline this test can afford, and #3513 established the discriminator
is host `kvm_amd` `vgif`, not anything about the product. Remove the
nesting level and there is nothing left to tolerate, so
`hack/e2e-node-join-soft-red.sh`, the marker and the `soft_red` outputs
are gone from all four workflows.

### What moves off the per-PR path

DRBD, and with it `replicated` StorageClass, its `Immediate` binding
mode, tenant StorageClass propagation (applies only to
remotely-accessible classes), and cozystack talos node image with its
extensions. None of it is lost - `nightly.yaml` and `e2e-tag.yaml` still
run the suite on QEMU, so all four keep nightly and release-candidate
coverage. Live migration is not on that list because nothing tests it
today anyway.

One more belongs on the list and was missing from it: the nocloud disk
build. `build-talos` no longer runs `make -C packages/core/talos
talos-nocloud`, so `images/talos/profiles/nocloud.yaml` is compiled by
`nightly.yaml` and by the tag build's `make assets` rather than by
anything a PR runs, and `promote-rc.yaml` still hard-fails a promotion
whose rc release has no `nocloud-amd64.raw.xz`. Nothing in that chain
breaks, because the tag build supplies the asset as before. What a PR no
longer catches is narrow: it still compiles `installer.yaml` through the
same imager, and the two profiles are generated by the same script and
differ only in `platform`, `kind`, `imageOptions` and `outFormat`.

### Product fixes

Four, all found by measurement while building the lane, each on its own
commit:

The linstor `drbd.enabled` gate. Without it drbd-logger sidecar exits,
satellite is never Ready, piraeus registers zero nodes and every PVC
hangs Pending behind a cluster that looks healthy.

linstor `--strict-topology`. Without it external-provisioner passes
every topology segment as `requisite`, so a node-pinned volume can be
provisioned away from its pod and the pod is then unschedulable forever
with nothing erroring anywhere.

kubevirt-cdi importer resources are configurable now. Default restates
CDI's own values exactly so production behaviour does not move, and the
lane raises the ceiling through its own Package.

vm-disk binds standalone disks immediately. A VMDisk on `local` silently
never populated, while the same disk on `replicated` did.

### Where the run stands

The lane has gone green twice, most recently on `cc70cef7b` where `E2E
(in-tree)` took 2h3m against its 215m cap with 45 Chainsaw tests
passing. The measurements quoted elsewhere in this description come from
the first of those runs, on a commit that has since been rebased away:
the two tenant-Kubernetes suites came in at 569.72s
(`kubernetes-latest`) and 470.19s (`kubernetes-previous`) against a 67m
operation. Both runs answer the four earlier reds, all of which were the
lua-protobuf segfault from #4012 once `vminstance`/`vmdisk` were fixed
on the branch.

ghcr.io mirror removal is carried here too and is the same change as
#4007, whichever lands first makes the other a no-op.

### What review changed

`drbd.enabled=false` now refuses to render together with
`talos.enabled=false`. It removed the logger sidecar only, while the two
DRBD initContainers piraeus contributes unconditionally are deleted by
the Talos configuration alone, so that combination produced a satellite
that can never reach Ready. `isp-full-generic` sets
`talos.enabled=false`, so it was reachable and not theoretical.

vm-disk keeps unconditional immediate binding, a standalone disk has no
consumer to wait for. What it costs on a node-pinned class is in the
chart README now, and the annotation is pinned per source type and on
the existing-DataVolume path where `helm upgrade` retrofits it.

The rest is the harness holding its own promises: HelmRelease gate reads
are bounded and its minimum-count half is pinned by a test that fails
without it, two Chainsaw ops sit above the deadlines they contain, the
fallback-StorageClass render restores `replicated` when it fails,
container lane compose project and pool files are per-checkout, and
three comments that said the opposite of the code now say what is true
(QEMU is still wired into nightly and e2e-tag, the lane replaces two
Packages and not one, the argument-count assertion cannot see upstream
drift).

### Release note

```release-note
fix(linstor): CSI provisioning is constrained to the node the scheduler chose (`--strict-topology`), and the DRBD sidecar can be switched off with `drbd.enabled` on substrates whose kernel cannot run DRBD. On an existing cluster the new default re-templates `spec.csiController.podTemplate`, so piraeus rolls `linstor-csi-controller` once during the upgrade. `drbd.enabled=false` requires `talos.enabled=true` and the chart now refuses the other combination at render time, because only the Talos satellite configuration removes the DRBD initContainers.
fix(kubevirt-cdi): CDI worker pod resources are configurable through `importerResources`, which reaches the importer, the uploader and the host-assisted cloner. The default restates CDI's own built-in values exactly, so an install that sets nothing behaves as before.
fix(vm-disk): every disk requests immediate binding, not only `upload` sources, so a standalone disk on a WaitForFirstConsumer StorageClass is populated instead of waiting for a consumer that may never arrive. On a node-pinned class such as `local` the volume then binds where CDI's worker pod was scheduled rather than where the VM will run; see the chart README before choosing such a class for VM disks. The default `replicated` is unaffected.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added container-based end-to-end testing with local storage support.
* Added optional backup-access preflight checks for database backup
examples.
* Added configurable CDI worker resources and LINSTOR storage settings.
  * Added configurable immediate storage binding for VM disks.
* Added stronger readiness checks, diagnostics, and parallel test
execution.

* **Bug Fixes**
* Improved cleanup failure reporting, capacity validation, and
missing-image detection.
* Improved FoundationDB health verification and storage placement
behavior.

* **Removed**
  * Removed GHCR mirror support and soft-red node-join tolerance.
  * Removed standalone OIDC render test suites.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-07 19:33:20 +05:00
Alexey Artamonov ccf6165645 fix(keda): scope autoscaler alerts per-cluster and fix ScaleBlocked false-fire
ScaleStuck and ScaleBlocked joined on namespace alone, assuming one autoscaled
database per namespace — but a tenant may run several Postgres apps in one
namespace. A Pending pod of one CNPG cluster then falsely tripped a healthy
sibling's ScaleStuck, and ScaleBlocked's namespace-wide pod count masked a real
block on one cluster. Derive a `cluster` label from the keda-hpa-<cluster> HPA
name with label_replace and join the pod terms on (namespace, cluster).

ScaleBlocked also cried wolf on a healthy multi-step scale-up of a single large
database: CNPG materialises replicas one basebackup at a time (each often >15m),
so the instance count legitimately lags a desired jump of +2 with no Pending pod,
and the old rule fired with the actively-harmful "lower maxReplicas" advice. Count
only instance pods (a transient join Job pod otherwise inflates the count and
masks a block) and suppress via `unless` while a join Job is provisioning a
replica; a genuine admission block creates neither. Make the desired > current
comparison an explicit on(namespace,horizontalpodautoscaler) join.

Note in the egress NetworkPolicy values that its `cluster` entity does not
restrict cross-namespace egress (the chart's namespace-pin does) and that
off-cluster / link-local IMDS denial depends on CNI enforcement.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 15:29:06 +03:00
Alexey Artamonov 3516bbf314 docs(postgres): correct lag-brake floor and floor-raise shed comments
The lag-brake frozen summand's `or vector(0)` comment claimed it prevents a
silent pin-to-floor when the pod-label count is momentarily empty under load.
That state is unreachable: the frozen count selects the same kube_pod_labels
series the base load term joins on, so base and frozen are empty or non-empty
together and the floor never changes the result. State that it is a no-op marker.

The enablement note framed raising the quorum floor / minReplicas as purely
upward, but the merge rebases to the newly-rendered seed vs the previous seed,
never to KEDA's live count — so raising the floor (or lowering replicas) to a seed
still below a load-scaled live count sheds the surplus replicas and PVCs in one
step before the HPA re-clones. Add the HAZARD note.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 15:29:06 +03:00
Alexey Artamonov 187f18a167 fix(postgres): harden autoscaler serverAddress guard and cap maxReplicas
The serverAddress SSRF guard had two holes beyond the external-domain suffix
already closed. It extracted the host by truncating at the first `:` or `/`,
which a `http://<svc>.<ns>.svc:port@evil.example.com/` userinfo payload defeats:
the truncated text passes the `.svc` check while net/url dials the host after the
`@`. And it validated only the host shape, never the namespace, so a tenant could
point the cluster-privileged KEDA operator at a sibling tenant's or a system
namespace's Service (the egress NetworkPolicy's `cluster` entity does not stop
in-cluster egress). Parse the URL authority properly (scheme, path/query/fragment,
userinfo to the last `@`, then port) and pin the host's namespace to the tenant's
own subtree via cozy-lib.rbac.allParentTenantsAndThis. The namespace pin narrows
what a tenant may set and needs API-owner sign-off (noted at the guard).

Separately, nothing bounded maxReplicas: the schema carries no maximum and the
aggregated apiserver does not range-validate the CR, so a tenant could set
maxReplicas: 100000 and drive KEDA to scale CNPG toward that count. Add a render
guard rejecting a maxReplicas above a sane read-replica ceiling.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-07 15:29:06 +03:00
IvanHunters 67a86746a6 feat(marketplace): community marketplace for External-Apps repositories (#3940)
## What this PR does

Implements the community marketplace for Cozystack External-Apps
repositories, per the approved design proposals **community#18** and
**community#23**, building on the **community#12** authoring workflow.
Anyone can publish an External-Apps repository over OCI, discover it,
connect it to a cluster, and install its apps the same way official
packages are installed, with a publication gate that keeps the index
safe.

Additive by construction: the existing PackageSource / External-Apps
pipeline is untouched, no CRD is added or migrated, and existing
installs and upgrades are unaffected. Per-package version pinning stays
out of scope, siding with #18.

Components:

- **cozypkg CLI** — new `validate`, `init`, `push`, `tap`/`untap`,
`search`. Scaffold a repository, publish it as an OCI artifact, register
it on a cluster in one command (optional pull secret for private
repositories), and discover packages through a metadata-only community
index. `validate` lints a repository offline (schema, chart paths, `helm
lint`, dependsOn, ApplicationDefinition wiring, privileged components);
with `--require-signature` it verifies a keyless cosign signature.
- **Marketplace backend** — a read/write, cluster-scoped `Tap` resource
in the existing `core.cozystack.io` group (a computed virtual resource:
no new group, no CRD, no etcd). Lists connected repositories and the
packages they expose, connects a repository (recording intent as a
labeled Flux `OCIRepository`), and disconnects one. Catalog metadata
only, never a pull credential.
- **Tap materializer** — a controller in cozystack-controller that, once
source-controller has pulled a connected repository's artifact, fetches
it over the Flux artifact URL (digest-verified, size-bounded,
path-traversal-guarded) and materializes the PackageSource(s) it carries
under the `community.` prefix, so a dashboard-connected tap becomes
installable without the apiserver pulling artifacts.
- **Publication gate** — a template for a `cozystack/packages-index`
repository: a metadata-only entry format and a GitHub Actions gate that
validates every changed entry with `cozypkg validate` (mandatory cosign
against the entry's recorded identity) under a two-lane merge policy
(owner version bump auto-merges on signed validation; a new entry or a
security-relevant edit goes to maintainer review).
- **Dashboard** — a Repositories view in the console marketplace, backed
by `Tap`, to browse and connect/disconnect repositories.

Access control: reading the catalog is open to authenticated users
(metadata only); connecting and disconnecting require cluster-admin via
the delegated apiserver RBAC.

Verification: `go build` / `go vet` / `go test`, dashboard `pnpm
typecheck` + eslint, and the gate's `shellcheck` are all green.
Server-side authz denial and the end-to-end
connect→materialize→disconnect round-trip are not exercised here
(hermetic change) and are the main follow-up.

### Screenshots

The dashboard Repositories view is included, but screenshots are not
attached: rendering it needs a live cluster running the new `Tap`
apiserver, which this static change cannot stand up. Tracked as a
follow-up (a cozystack-pr-test run on a disposable cluster); this is one
reason the PR is a draft.

### Downstream repositories

Walked the trigger map against the diff. This is additive tooling plus a
new virtual resource; it does not force a change in any listed
repository, and `external-apps-example` keeps working on the existing
chart model. Left every box empty (rather than ticking "No downstream
repository is affected") so a maintainer can confirm — in particular
whether `external-apps-example` should later gain a PackageSource-based
example that `cozypkg init` now scaffolds.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(marketplace): add a community marketplace for External-Apps repositories — cozypkg validate/init/push/tap/untap/search, a cluster-scoped Tap API in core.cozystack.io with an operator-side artifact materializer, keyless cosign verification, a two-lane publication gate, and a dashboard Repositories view
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added Marketplace repository connections (“Taps”) with dashboard
management, readiness status, package catalogs, and optional
credentials.
  - Added repository search by name, source, and package metadata.
- Added a searchable community package index with version-pinned
resolution.
  - Added `cozypkg init`, `push`, `tap`, and `untap` commands.
- Added offline repository and artifact validation with optional chart
linting and signature checks.
- Added safeguards requiring confirmation before installing packages
with privileged components.
  - Added automatic synchronization of tapped package catalogs.

- **Documentation**
- Added community package index guidance, validation workflow, and an
example entry.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-07 14:55:12 +03:00
Myasnikov Daniil 62f2234aea fix(e2e): stop a failed teardown posting a failed test verdict
The teardown ends both merge-gating lanes under `if: always()`, and this
branch made it `exit "$cleanup_rc"`. So a `docker compose down` hiccup, a
busy zpool or a failed rmdir after a fully passing suite posts the
required "E2E Tests" status as a failure, and nothing downstream can tell
that apart from a test that actually broke. The runner is discarded when
the job ends, so there is no next run for the strictness to protect; a
reused host is a developer machine, and an annotation serves it. Loud,
not fatal.

Two comments described mechanisms that are gone. The breakpoint step
still explained reading a soft-red output that this branch removed, so a
maintainer would go looking for it; the node-join deadline now fails its
suite like any other, which is what makes `failure()` sufficient. The
build-talos header still described building the nocloud disk and e2e
needing it, when the job builds neither and the container lane downloads
no disk -- the profile is compiled by nightly and by the tag build's
`make assets` instead, and what a PR still compiles is installer.yaml,
which shares every input with it bar the platform and output format.

And the platform-package override now states its consequence for both
packages it replaces, not only for CDI: the merge-gating lane exercises
neither the shipped CDI default nor `drbd.enabled: true`.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
2026-09-07 16:33:27 +05:00
Myasnikov Daniil 8456facc79 test(e2e): pin the stability-window reset and keep the fan-out test portable
The readiness gate resets its stability window when a list read fails,
and nothing held that: the existing "transient list error" case fails on
the first read, when there is no window to invalidate, so removing the
reset left the suite green. The new case breaks a read mid-window, where
a gate that kept counting through the gap would accept a release set it
never observed stable across it.

The parallel fan-out contract asked make for --output-sync, which arrived
in GNU Make 4.0 while macOS still ships 3.81 -- so on a contributor's
machine the suite died on an unknown option instead of reporting the
contract. Guarded by version, with the reason in place. CI runs 4.x, so
the contract is still enforced where it gates a merge.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
2026-09-07 16:33:27 +05:00
Myasnikov Daniil b30c7efaf6 fix(e2e): make the OIDC teardown assertions fail closed
The three checks that System-mode artifacts are gone after the mode
switch were `if kubectl get ... >/dev/null 2>&1`, which reads every
non-zero exit as "the object is gone". An RBAC denial, a timeout or a
missing CRD produces the same exit, and stderr went to /dev/null, so the
suite reported a clean System teardown for never having observed one.

--ignore-not-found is what separates absent from unaskable: absent is
exit 0 with empty output, everything else stays non-zero. Preferred over
matching NotFound in the message, which depends on wording and, under a
runner with tracing on, captures the trace instead of the error.

Each probe is pinned individually rather than as a group: the new case
breaks one query per iteration while the other two answer absent, so
reverting any single probe to the bare form reddens the suite. Verified
by doing exactly that three times. A test that broke all three at once
would have passed on any one surviving probe, which is how the first
attempt at this test fooled me.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
2026-09-07 16:31:08 +05:00
Myasnikov Daniil a115745c69 feat(vm-disk): let the operator defer binding to the consumer
Immediate binding for every source fixed a real bug: a standalone disk
on a WaitForFirstConsumer class has no consumer to wait for, so it never
populated. It also took away the path that is correct for the other case.
For a disk a VMInstance consumes, KubeVirt renders a temporary launcher
pod carrying the VM's own affinity and nodeSelector, so the volume binds
where the VM can actually run; with the annotation always present the CDI
worker becomes the first consumer instead and the volume lands where that
pod happened to be scheduled. On a node-pinned class the VM is then
unschedulable against a node-affinity conflict, and storageClass is
immutable, so the only exit is deleting the disk.

The two cases want opposite defaults and only the operator knows which
one a given disk is. bindImmediately defaults to true, so nothing moves
for existing disks or for anyone on `replicated`, which binds Immediate
either way. Both branches are pinned in tests, including the
existing-DataVolume path, which renders its own annotation block.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
2026-09-07 16:27:24 +05:00
Myasnikov Daniil 16a49740fe chore(codeowners): rebuild ownership by area, five owners minimum
CODEOWNERS applies only the last matching pattern and it replaces every
broader rule including the catch-all, it does not merge them. Half of
what that costs is already fixed: bd7b76691 added the architects to the
rules that were missing them, and hack/codeowners-invariant.bats keeps
them there. That invariant reads "every rule with owners is a superset
of the catch-all owners", and a rule naming nothing but the architects
satisfies it perfectly while gating on two people. Eleven rules on main
have that shape, 182 files between them.

The catch-all is those same two handles, so the invariant makes them
the floor for the whole repository and every path without a rule of its
own sits on that floor. Raising the floor means raising the catch-all,
after which the invariant carries the new set into every rule by
itself. Hence the shape here: the catch-all goes from two owners to
five, and all fifty rules carry those five.

This widens who may approve, not who may change code. Write access is
unchanged for all twelve collaborators, owner approval is OR and always
was, and the required checks that actually hold these paths are
untouched, api-review-gate.yaml included, so a sizeable API change
still needs an approval from one of its own two owners.

Three of the nine maintainers in MAINTAINERS.md own no path, for two
different reasons. c13c6ee2c gave kingdonb the two FluxCD tenant addon
packages and 7cca3d75e deleted them along with the addon, so his rules
went with the packages while the flux paths that stayed were never
handed to him. matthieu-robin and mattia-eleuteri were never in the
file at all. CONTRIBUTOR_LADDER.md says a maintainer may approve
changes to any area of the project.

Rules now group by area rather than by package, every rule names at
least five owners, and an area whose owner set does not differ from its
tier is left to the tier rule. Reviewers keep the areas MAINTAINERS.md
records for them, so that file needs no change.

The two-owner rule on /cmd/api-gate/, /internal/apigate/ and the
api-review-gate workflow goes with the rest. Those paths still require
an owner review, now under the Go source and CI rules.

The invariant test's exemption for a rule owned by exactly tym83 and
kvaps is dead code once the governance block is gone, and dead code in
that position is a hole: any future two-owner rule matching that set
would be waived. Dropping it puts such a rule back under the invariant,
where it belongs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
2026-09-07 13:20:15 +05:00
IvanHunters 109ee1dd9e refactor(cozypkg): share one reserved-prefix list between init and validate
init carried the same two prefixes inline that validate declares as
reservedNamePrefixes, so a third prefix added for validate would have left
init letting it through. Both files are package cmd, so this is one
identifier. Drive the init test off the shared list for the same reason.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-07 10:58:40 +03:00