admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-03 18:47:04 +09:00
Code Issues Packages Projects Releases Wiki Activity
6,613 Commits 544 Branches 294 Tags
11bac54f931682dd94de4081bb9e8a7eef60bef1
Commit Graph
836 Commits
Author SHA1 Message Date
Aleksei Sviridkin 08bbed3fb5 feat(talos-log-collector): add talos-log-collector package (#3260) 
## What this PR does

Adds `talos-log-collector`, an optional system package: a node-local
Vector
DaemonSet that receives Talos machine and kernel logs and forwards them
to the
tenant VictoriaLogs (vlinsert), alongside the container/audit/event logs
already collected by monitoring-agents.

Talos keeps system and kernel (kmsg) logs inside the OS and does not
write them
to a host file the Fluent Bit `tail` input could read, so these logs are
currently never collected. Vector receives them over the node loopback
(Talos pushes to 127.0.0.1:5170). Talos exposes the two streams through
independent config paths, and both must point at the loopback socket:
`machine.logging.destinations` forwards the JSON-lines service logs, and
a
`KmsgLogConfig` document forwards the kernel (kmsg) stream (including
DRBD
kmsg). The node-side setup for both is in the README.

Design notes:
- The pod stays on the pod network so cluster DNS resolves vlinsert with
no
  extra config. Talos pushes to 127.0.0.1:5170; a `hostPort` bound to
`hostIP: 127.0.0.1` publishes that loopback socket into the pod via
portmap,
  keeping the receiver off the host network (no node-wide pod exposure).
- The receiver binds inside the pod netns, so a NetworkPolicy is
load-bearing:
a CiliumNetworkPolicy restricts ingress to the host entity when Cilium
is
present, otherwise a default-deny-ingress `networking.k8s.io/v1`
fallback
  keeps the socket protected on non-Cilium variants.
- Runs non-root, read-only root filesystem, all capabilities dropped.
- Optional package (opt-in via `bundles.enabledPackages`): inert until
each
Talos node is configured to push logs. Node-side setup is in the README.

Verified end-to-end on a Talos v1.12 dev cluster: service and kernel
logs
(including DRBD kmsg) reach VictoriaLogs; pods run non-root with 0
restarts.

### Release note

```release-note
feat(talos-log-collector): add talos-log-collector package to collect Talos machine and kernel logs into VictoriaLogs
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an optional node-local Talos log collector (via system bundle
packages) that forwards Talos service and kernel logs over loopback to
the platform log ingestion endpoint.
* Introduced the collector Helm chart (Vector-based) with configurable
log level, loopback TCP receive port, digest-pinned image, default
resources, and routing target.
* Added conditional Cilium network policy support with a safe
deny-by-default fallback.
* **Security**
* Hardened the collector to run non-root with reduced privileges,
read-only filesystem, and dropped Linux capabilities (and disabled
service account token automount).
* **Documentation**
* Documented configuration requirements (including kernel log
enablement), operational behavior, and teardown steps.
* **Build & Release**
  * Extended the build to include building the collector’s image.
* **Tests**
* Added Helm rendering/contract tests covering daemonset, configmap,
service account, and network policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-22 23:49:33 +04:00
Aleksei Sviridkin ad4ea5c937 feat(keycloak): publish only the named realms on the login host (#4363) 
## What this PR does

`ingress.adminHost` moves the admin console and Administration REST API
onto
their own hostname and narrows the public login route to `/realms`,
`/resources` and `/.well-known`. That `/realms` prefix publishes every
realm
the deployment has, `master` among them, so the super-admin login page
and
token endpoint stay reachable from the internet on the login hostname —
a
brute-force target the admin split was meant to remove. The Keycloak
reverse
proxy guide marks `/realms/master/` as internal-only.

The login route now enumerates the realms that are meant to be public:
the
platform realm (`authentication.oidc.realmName`, default `cozy`), which
every
component's issuer URL points at and which is therefore always
published, plus
whatever a deployment adds in the new `ingress.exposedRealms`.

Each realm is published as an exact path plus a prefix carrying the
trailing
slash. ingress-nginx renders `pathType: Prefix` as a plain nginx prefix
location rather than the element-wise match the Ingress API specifies,
so a
lone `/realms/cozy` would also serve `/realms/cozy-admin` there; the
pair
bounds the match on the segment and is equivalent to element-wise
matching on
a conformant controller and on Gateway API.

`master` is refused outright — both in `ingress.exposedRealms` and as
`authentication.oidc.realmName`, where the guard sits next to the
existing
realm-name checks so it applies whether or not a separate admin hostname
is
configured. Naming that realm puts the platform clients, the
cluster-admin
roles and every end user inside the realm that holds the Keycloak
super-admin.

This is a path boundary, not a network one. `KC_HOSTNAME_ADMIN` never
was a
security boundary, `expose-ingress-admin` defaults to the public
Gateway/ingressClass, and with `adminHost` unset the login route still
serves
`/`. Which hostnames the edge accepts and where the admin route attaches
decide the rest.

**Behavior change**, confined to deployments that already set
`ingress.adminHost`: a realm published on the login hostname beyond the
platform one now answers 404 until it is listed in
`ingress.exposedRealms`.
The `master` guard on `authentication.oidc.realmName` breaks nobody,
since
that parameter has not shipped in a release yet.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked the trigger map against the diff: it touches
`packages/system/keycloak`
and the realm-name guard in
`packages/core/platform/templates/apps.yaml`. No
app package, no `packages/core/platform/values.yaml` key the Ansible
role
sets, no platform variant or component, no `ApplicationDefinition`, no
`hack/`
layout, no telemetry metric, no vendored chart contract.

### Release note

```release-note
feat(keycloak): with `ingress.adminHost` set, the login hostname publishes only
the platform realm and any realm listed in the new `ingress.exposedRealms`,
keeping the `master` realm login page and token endpoint off the public route.
A deployment that serves additional realms on that hostname must list them.
`authentication.oidc.realmName` no longer accepts `master`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added `ingress.exposedRealms` to control which additional identity
realms are reachable through the login hostname.
- Login routes now target the platform realm and configured exposed
realms with precise realm-specific paths.
  - Duplicate realm names are handled automatically.

- **Bug Fixes**
  - Prevented publishing the privileged `master` realm.
  - Added validation for invalid realm names and configurations.
- Enforced the gateway route limit, supporting up to 31 published
realms.

- **Documentation**
- Updated configuration guidance for realm exposure and security
restrictions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-22 19:47:53 +04:00
Aleksei Sviridkin 1c88014661 fix(installer): keep node_modules out of the packages artifact (#4290) 
## What this PR does

`flux push artifact` walks the whole `packages/` tree and reads neither
`.gitignore` nor `.sourceignore`, so a working copy where somebody ran
`pnpm install` under the vendored console ships the console's dependency
tree inside the bundle. Helm then refuses the dashboard chart on every
cluster that pulls it, because one dependency file sits above the 5MB
per-file limit, and the build itself reports nothing:

```text
Could not load chart: decompressed chart file "dashboard/images/console/node_modules/.pnpm/@rolldown+binding-darwin-arm64@1.0.0-rc.15/node_modules/@rolldown/binding-darwin-arm64/rolldown-binding.darwin-arm64.node" is larger than the maximum file size 5242880
```

Passing `--ignore-paths` replaces flux's built-in list rather than
extending it, so the defaults are restated alongside `node_modules/`.
`hack/check-packages-artifact-ignores.bats` holds both halves: dropping
`node_modules/`, dropping one of the defaults, or declaring the variable
without passing it to the recipe each turn the test red. Its second case
re-reads the defaults out of the installed flux, so an upstream addition
surfaces as a failure rather than as a pattern quietly missing from the
artifact.

Measured on a tree with the console's dependencies installed: the
artifact goes from 80MB carrying 32397 `node_modules` entries to 6.7MB
carrying none, with the dashboard chart still in it.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

The two workflows that push the same artifact themselves
(`e2e-fork.yaml`, `nightly.yaml`) run from a clean CI checkout, where
the tree has no `node_modules` to pick up.

### Release note

```release-note
fix(installer): the packages OCI artifact no longer picks up node_modules from a developer's working copy, which made the dashboard chart unloadable on every cluster that pulled the bundle.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Package artifacts now exclude unnecessary files, including
version-control metadata, common media and archive formats, and
dependency directories.
* This helps prevent oversized files from exceeding Helm’s per-file size
limit.

* **Tests**
* Added validation to ensure package artifact ignore patterns remain
complete and aligned with the packaging tool’s defaults.
* Optional checks report when the packaging tool is unavailable without
failing the test suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-22 16:17:52 +04:00
Kirill Ilin 938595afe3 fix(keycloak): validate the realm list wherever it is set 
The list was read only inside the adminHost branch, so a value the chart
would refuse — master, a scalar, a name that splits the path — sat in the
values unnoticed until someone turned the admin hostname on, and the refusal
then landed on the change of authentication topology rather than on the edit
that caused it. Build the list on every render and let the branches use the
result.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
 2026-09-22 10:54:28 +05:00
Kirill Ilin c96fe3d3ef fix(platform): refuse master as the OIDC realm name 
The realm named here is where keycloak-configure creates every platform
client and the cluster-admin role, and where dashboard users sign in. Naming
master puts all of that inside the realm that holds the Keycloak super-admin,
and publishes its login page wherever the platform realm is published — and
the guard belongs where the value is set, since the Keycloak chart only reads
it once a separate admin hostname is configured.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
 2026-09-22 10:35:31 +05:00
Alexey Artamonov 142a14cb69 fix(platform): reject an empty realm name instead of coercing it to cozy 
realmName resolved through `| default "cozy"`, which fires on both null
and "". Unset (null) legitimately means the default, but an explicit ""
is a meaningless value and coercing it hid the mistake. Split the two:
null falls back to cozy, an empty string reaches the charset guard and is
rejected — the same null-vs-"" distinction keycloakInternalUrl already
draws a few lines below.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-21 12:15:10 +03:00
Alexey Artamonov ff637ed9a2 fix(platform): validate the realm-name charset and regenerate kubernetes artifacts 
The realm-name guard checked only the type and echoed the parsed value
back, so its hint told an operator who wrote `on` to set realmName:
"true" — the token YAML had already discarded. Drop the echo and add a
slug-charset check: the name is spliced verbatim into URL path segments
and a shell token, so a "/" or a space must be rejected at the source
rather than split a URL or the hook script downstream.

The description edit in the previous commit was hand-applied to the
generated kubernetes README and never reached the Go types, leaving
pre-commit red; regenerate both with cozyvalues-gen. Finish the
oidc-tenant System-mode prose that still read cozy as a fixed name.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-21 12:15:10 +03:00
Alexey Artamonov 949096694b fix(platform): reject non-string realm names and shell-quote the hook realm 
A realmName that YAML resolves to an int or a bool (a bare 123, or a
reserved word like on/off) was coerced with toString, silently renaming
the realm to something the operator never typed (on -> "true"). Fail the
render with a quoting hint at the source instead of surfacing it later as
a broken issuer URL.

The update-email hook Job interpolated the realm into its shell script
unquoted, so a name carrying a space or a shell metacharacter could break
or inject into the script; single-quote it. The Grafana and managed-cluster
issuer helpers build the URL with printf %s straight off the channel value,
so guard them with toString for the same non-string reason.

Name the configured realm in the tenant kube-apiserver oidc.mode
description and the Grafana identity-mode comment, which still read cozy as
if it were fixed.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-21 12:15:10 +03:00
Alexey Artamonov 80d32b3ab7 fix(keycloak): complete realm-name coverage and guard non-string names 
The enable-update-email hook Job in keycloak-configure still pinned
REALM=cozy, so on a custom realmName it waited on a realm the chart never
provisions and the cold install timed out — the one seventh consumer left
behind, and the only value nobody could set without breaking the install.

A bare-integer or YAML-reserved realm name reached the printf %s that
builds the derived keycloakInternalUrl as a Go int, rendering a
"%!s(...)" format error into the URL; coerce the name to a string once
at the source.

Document that renaming the realm on a running cluster provisions a fresh
empty realm and orphans the accounts in the old one, and note in the OIDC
tenant and grafana guides that the realm name is now the configurable
default. info/Makefile carried NAME=etcd, so cozyhr drove the wrong
release from that package; correct it while touching the file.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-21 12:15:10 +03:00
Alexey Artamonov 44a9a8e735 feat(platform): make the Keycloak realm name configurable 
The realm name "cozy" was hardcoded in keycloak-configure (which owns
the realm, its clients and the cozystack-cluster-admin roles), in the
dashboard and linstor-gui gatekeepers, in the info kubeconfig, and in the
Grafana and managed-Kubernetes apiserver OIDC issuers. Operators could
not rename the realm without patching vendored templates, which
`make update` reverts. The apiserver OIDC issuer is already configurable
via talm oidcIssuerUrl, so pointing it at a custom realm left every SSO
component on "cozy" and broke login with an issuer mismatch.

authentication.oidc.realmName (default "cozy") now rides the _cluster
channel so every consumer resolves the same realm, and the default
keycloakInternalUrl is derived from it so the realm is named in one place.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-21 12:15:10 +03:00
Timofei Larkin c398e0e116 Merge branch 'main' into chore/drop-http-cache 
main took migration 57 and moved migrations.targetVersion to 58 while this
branch was open, so that line is the one conflict: 57 at the merge base, 58
on main, 59 here. Resolved as 59, which is what this branch's migration 58
requires, since run-migrations.sh loops up to targetVersion - 1.

Merged rather than rebased because the branch carries commits that are not
mine to rewrite.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-09-18 13:20:03 +03:00
Timofei Larkin 2cbc02bd10 fix(platform): freeze HTTPCache on the artifact the cluster runs 
Pinning the frozen source to a digest written into the migration picks one
release for every cluster, and v1.6.0 is not even the last one that ships
http-cache: v1.6.1 through v1.6.3 and v1.7.0-alpha.1 all do, and 1.6 patch
releases may keep doing so. A cluster on any of those would be rolled back
to an older chart by the upgrade that was meant to leave its instances
alone, and an air-gapped mirror resolves the constant only if someone
mirrored that exact manifest.

The hook runs before the upgrade re-points cozystack-packages, so the live
object still names the release the instances render from, in whatever
registry the operator serves it from and with whatever pull secret and
verification they configured. Reading the digest off it makes the freeze a
no-op by construction and needs no knowledge of which tag was last.
spec.ref.digest is used when the installer pinned one; a tag or semver ref
goes through status.artifact.revision, whose digest half is the manifest
digest. status.artifact.digest is the checksum of the downloaded tarball
and cannot stand in. A source with neither has nothing to freeze on and the
migration stops there. The copy also carries the no-delete label the
platform chart puts on the original, because it is the only chart source
those instances have left.

The tenant grant travels with the freeze as well. cozy:tenant:admin:base
enumerates the apps.cozystack.io resources a tenant owner may write, and an
application that has left the catalog has no business in that list: the
next sweep over it takes the entry as dead config, and the failure is
silent and remote, a Forbidden for the one person who still needs to remove
what they have. The chart drops httpcaches, and a cluster that freezes gets
a ClusterRole under the tenant admin aggregation label instead, applied
before the upgrade drops the rule so the access never lapses and carrying
the verb set the chart had so nothing changes for the owner. A cluster with
no instance is granted nothing.

The Package delete on the no-instance path has to block. The operator
reconciles off an informer cache, so a reconcile in flight against the
pre-delete entry re-creates the release it owns, and with it the
ApplicationDefinition and the Helm storage the following lines remove.
Returning immediately made the ordering a property of the command log
rather than of the cluster.

The preset check's SIGPIPE note named http-cache-rd as its example of a
large enum, and that chart goes away here; kubernetes-rd carries the same
enums.

Numbered 58 rather than 57: main already carries a migration 57, the
SeaweedFS volume-size-limit one, and its targetVersion is already 58.

The whole shape of this retirement was worked out here rather than
anywhere reusable, so docs/deprecating-applications.md writes it down as a
procedure: how to pick between migrating onto a successor and freezing,
what the prune takes with it, and what the migration owes a cluster on
each branch. The next application to go should not have to rediscover it.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-09-18 13:12:56 +03:00
Yan Bondarenko 5ed076cff8 feat(kubevirt): expose migration configuration through platform values 
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
 2026-09-18 13:24:36 +04:00
IvanHunters 617cba1621 fix(installer): keep node_modules out of the packages artifact 
`flux push artifact` walks the whole packages tree and reads neither .gitignore nor .sourceignore, so a working copy where somebody ran `pnpm install` under the vendored console ships its dependency tree inside the bundle. Helm then refuses the dashboard chart on every cluster that pulls it, because one dependency file sits above the 5MB per-file limit, and the build itself reports nothing. Passing --ignore-paths replaces the built-in list rather than extending it, so the defaults are restated alongside node_modules/.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-09-17 09:34:18 +03:00
Aleksei Sviridkin a2db2aeb8c fix(seaweedfs): raise the number of buckets a volume server can serve 
A volume server started with -max 0 takes a slot count derived from disk
rather than a fixed one, replication 001 spends a slot on each of the
two servers per logical volume, and a collection claims COPY_2 volumes
whenever it grows. So the bucket ceiling is set by the product of the
size limit and COPY_2, and moving either number alone moves it. The
mechanism is upstream in MaybeAdjustVolumeMax (weed/storage/store.go)
and findVolumeCount (weed/topology/volume_growth.go).

Do not divide these out of the nominal PVC size, because the count
follows the space the filesystem makes available and not the volume's
advertised size: the shipped 10Gi volume PVC was seen to give nine slots
per server at the upstream 1000 and 19 at 500, which puts 400 at 23 to
24 by the same division, or eight collections against two. The span is
the observation's own, since the free space behind 19 at 500 fits more
than one filesystem geometry and those divide differently at 400; eight
either way, because a collection that cannot get a whole batch of COPY_2
keeps the volumes it did get and writes into them. The division itself
is measured: on a 1 GB image the master reports max as the filesystem's
available space over the limit in every run, 25 slots and nine
collections at the 40 and 3 that scale to this pair. Two is what the
platform's own cozy-backups bucket and a single tenant bucket already
use. Past that every S3 PUT to a new bucket times out while the disk
still reads as nearly empty, and no COSI object reports anything wrong,
because the claim and the access are both granted.

400 and 3 make the budget 1200 MB per collection. Splitting the
reduction across both numbers rather than taking it out of the size
limit alone keeps the cost off the curve that grows: open descriptors
track stored bytes divided by the limit, so a 10 TiB server keeps
roughly 52k of them at this limit against 21k upstream, where moving the
size limit alone would have put it at 105k. A .dat and an .idx stay open
per volume; the .vif beside them is written and closed. The ceiling
reached is the same either way.

Both figures describe the shipped replicationFactor of 2. The master
picks its growth count from the replication the filer asks for, so at 1
and 3 copies it reads COPY_1 and COPY_3 instead and this COPY_2 does not
apply; lowering the size limit still raises the ceiling there.

The size limit was out of an operator's reach: it lived in the
data-plane chart, which is pulled by ExternalArtifact from a component
that declares no install section, so a Package-level values override for
it is dropped before any merge. The app chart now declares
master.volumeSizeLimitMB and sends it on every release, and 400 is set
there. The data-plane chart keeps upstream's 1000 as its fallback on
purpose. A platform upgrade can re-render that release from its artifact
before the parent has sent anything new, and a lower fallback would then
reach the master first and drop the limit ahead of an operator's own
setting. COPY_2 is sent by nothing and lives in the data-plane chart, so
it does land in that window, where a grow costs 3000 MB against the 6000
MB it cost before.

The app chart also writes maxVolumes into dataDirs on every release, and
because dataDirs is a list Helm replaces the data-plane chart's copy
wholesale, so the zero that makes the slot count auto-derived at all
comes from there. Losing it drops the server to the subchart's fixed
seven without failing any render.

The field is bounded at both ends, because either end takes an instance
down. Zero is not a smaller limit but an off switch: the volume server
leaves its slot calculation untouched at zero, keeps reporting no
capacity, and nothing is ever grown. Past 30000 the master does not
start at all. The bound sits in the values schema rather than in a
template guard, so Helm refuses the value before any template runs. The
apps API does not: it reads that schema only for defaults, so it accepts
a value out of range, and the release then fails to render. An explicit
null gets past both bounds, because Helm drops the key before it
validates, and would render an empty -volumeSizeLimitMB= that the master
exits on at start, measured on the shipped image, so the template
requires the value.

No bound against the disk goes with it, though the field invites one. A
limit at or above the volume's own size does not break an instance:
MaybeAdjustVolumeMax floors the slot count at one, measured on 4.31 with
a 255 MiB disk against a 1000 MB limit, which reported Max 1 and served
a write it then read back. Such a server is capped at a single slot
rather than dead, so refusing it at render would be a capacity policy
that stops a working release on every reconcile, not only on the
upgrade. The field's description carries the relation instead, where the
value is chosen.

The growth count is deliberately not exposed beside the limit. The
master picks which of its four growth counts to apply from the
replication the filer requests, and the filer takes that from
replicationFactor, so a field wired to COPY_2 would do nothing at 1 or 3
copies while still appearing to be a setting. A knob that is silently
inert for two of the three supported values is worse than no knob.

Two further prices. A collection absorbs the budget before it needs
another grow, so growth requests arrive five times as often as at the
upstream pair. A collection also starts with half as many writable
volumes, and a volume server takes the writes into each volume one at a
time (syncWrite holds that volume's dataFileAccessLock), so a heavily
parallel writer spreads over fewer volumes until the collection grows
again; that holds for instances kept at 1000 too. And lowering the limit
on an instance that already holds data carries the other one: its
volumes between the new limit and the old go oversized on the next
heartbeat and leave the master's writable set, and each affected
collection needs one successful grow before its next write lands.
Nothing is lost and the volumes stay readable; the room left inside them
for further writes is given up and does not return as capacity.

Those grows compete for one pool, and the pool is smaller than it looks.
An oversized volume is charged no headroom, while one below the limit is
charged all of its unfilled room; a grow then claims a whole COPY_2
batch and takes as much of it as is there, and the volumes it creates
are empty and charge their headroom straight back. Measured on a local
SeaweedFS 4.31 setup: two oversized collections with two free slots
between them, the first to write took both, and the second stayed
refused with the disk still reporting a quarter of itself free.
Collections that need no grow are not free either, since their volumes
below the new limit were created six at a time and each can hold back a
whole limit. Also measured there, with the limit and batch scaled down:
a server with free space for one batch per collection refused the grow
its one affected collection needed, because a second, nearly empty
collection held the room, and the same free space without it let the
grow through. The minFreeSpacePercent reserve only comes into it on a
server whose writable volumes have all passed the limit, where nothing
is charged headroom: from 8000 MiB of filesystem upward, a 10Gi PVC
measuring 9979, five percent is at least 400 MB, and such a server then
refuses a grow only once its disk is down to the reserve. Everywhere
else the headroom charge decides, and a refusal can arrive with the disk
far from full whatever the reserve. Deleting a collection gives the room
back a slot per volume, however little it stored, since the headroom
charged to those volumes leaves with them.

So, on a variant that runs platform migrations, the upgrade does not
change the limit on an existing instance unless its operator already set
one, which the previous chart accepted and ignored. Platform migration
57 writes master.volumeSizeLimitMB: 1000 into every existing SeaweedFS
instance that has not chosen a limit, and 400 applies only to instances
created from here on. It leaves alone only an instance whose master is
not an object, which no v1 chart renders, and names that one in its log.
The default variant runs no platform migrations, so a cluster there that
installed SeaweedFS by hand gets no pin, and the document below gives
the same patch to apply before upgrading. A chosen limit is a whole
number the new chart accepts; anything else counts as not chosen and is
replaced with 1000, which is what the previous chart ran in its place,
since it ignored the key and the new chart would refuse to render it. A
valid value already set is kept, and since the previous chart ignored
it, the upgrade is when it starts to apply; the document below lists the
ones that change the limit ahead of it. Raised values are listed as
well: existing buckets keep writing, but every volume below the new
limit is charged up to it, and on a local SeaweedFS 4.31 setup seven
nearly empty volumes left no free slot once the limit went up tenfold,
so a new bucket was refused. The value goes into the application's
HelmRelease, whose spec.values is the application spec, for an instance
created through the API and for one the tenant module renders alike. The
tenant module never renders spec.values on that release, and a Helm
upgrade leaves a key alone that neither manifest carries: measured with
Helm 4.3.0 on client-side, server-side and forced server-side apply, and
read in createPatch at the v4.1.1 that helm-controller v1.5.0 runs.
Where the migration runs, it is a pre-upgrade hook of the platform
release, and the cozystack-packages source every application chart comes
from is a manifest of that same release, applied only after its hooks;
so no instance meets the new chart unstamped, and a failed migration
stops the platform upgrade before that source moves. That holds while
the platform source is pinned by digest, as it is out of the box; one
that follows a tag, a semver range or a branch lets source-controller
move cozystack-packages on its own schedule. Until cozystack-packages
moves, each patched release keeps its old chart, which ignores the key,
since no v1 release reads it or closes the master object in its schema.
An instance kept at 1000 still gains, because at a replication factor of
2 a grow now claims three volumes rather than six, so it has room for
roughly twice as many new buckets.

Lowering an existing instance is left to its operator, and
docs/operations/seaweedfs-lowering-the-volume-size-limit.md is the check
for it. It lists each server's volume files and adds one limit for every
volume below the limit and a batch for every collection with no volume
at or below 90 percent of it, since the master grows such a collection
on its own schedule without a write; measured the same way, a collection
with every volume in that band and no writes got two new volumes per
server within one pass. Where each disk type is served by as many volume
servers as the replication factor, as the shipped two at a factor of 2
are, each collection's volumes sit on every server of that type, and
that bounds the claim from above without assuming how the collections
grew. The master places a new volume on any server of the requested disk
type with room rather than where its collection already lives, whichever
group or pool that server belongs to, and it reads an empty disk type as
hdd; under MultiZone it also grows each zone on its own. So the document
keeps zoned instances, and any disk type served by more servers than
copies, at 1000.

BREAKING CHANGE: on the default variant, which runs no platform
migrations, set master.volumeSizeLimitMB to 1000 before upgrading on
every SeaweedFS instance that has no valid value and whose master is an
object, with the step in
docs/operations/seaweedfs-lowering-the-volume-size-limit.md. On the
variants that run platform migrations, migration 57 does this. If the
step is skipped, every such instance drops to 400 on the upgrade: each
of its volumes at or above 400 MB leaves the master's writable set, and
a bucket whose volumes have all crossed that line takes no writes until
it can grow.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-17 00:54:48 +03:00
Andrei Kvapil 4953b512a5 chore(platform): freeze HTTPCache on the last artifact that ships it 
The earlier shape of this migration orphaned every instance into plain
objects, which left its owner without a way to manage or remove it. A
cluster that still runs HTTPCache instead keeps the application, served
from a copy of the packages OCIRepository pinned to the digest of the
last release that contains it. The copy carries no owner references and
no Helm metadata, and the PackageSource is repointed at it, annotated
keep and disowned, so neither Helm nor the operator prunes it. Helm
reads that annotation off the live object during the upgrade prune,
which is what makes disowning in a pre-upgrade hook work at all.

Names are deliberately unchanged: the ApplicationDefinition inside that
artifact hardcodes its chartRef, so renaming the PackageSource would
leave every release pointing at an artifact nobody produces.

A cluster with no instance keeps nothing. The naas bundle installs the
rd component regardless of use, so freezing unconditionally would strand
a pinned source on clusters that never touched the application.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-09-16 11:00:54 +02:00
Andrei Kvapil c0f9f08de7 chore(platform): detach HTTPCache instances before the package is pruned 
Pruning the http-cache PackageSource takes its chart artifacts with it,
so without a migration every http-cache HelmRelease loses its chart and
stays Ready=False for good, with HelmReleaseNotReady firing on every
cluster that runs the naas bundle. Deleting the leftover Package by hand
is worse: it takes the ApplicationDefinition with it while the pods of a
surviving instance are still running.

Migration 57 orphans each tenant release instead, the way migration 55
does, so the workloads stay in the tenant namespace as plain objects,
and then removes the platform side. The Package goes before
hr/http-cache-rd because the operator re-creates a release it owns for
as long as that Package exists.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-09-16 00:34:51 +02:00
Andrei Kvapil 06bd17621f chore(platform): stop shipping the http-cache package 
Drops the cozystack.http-cache-application PackageSource and its entry
in the naas bundle.

No migration accompanies this. Platform Package objects carry
helm.sh/resource-policy: keep, so an existing cluster keeps its
http-cache Package and its http-cache-rd HelmRelease, and with them the
ApplicationDefinition that makes the kind resolvable. Instances that are
already deployed therefore keep running and stay addressable; the
platform simply stops delivering and updating the application. Removing
those objects would be the more invasive choice, not the safer one,
because there is no successor in the catalog to migrate an instance onto.

The httpcaches grant in cozy:tenant:admin:base stays for now. It is the
only rule in the tenant admin chain that can delete an instance, and no
tenant role carries any verb on helm.toolkit.fluxcd.io, so without it
the owner of a surviving HTTPCache could not remove it at all.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-09-14 13:11:00 +02:00
Aleksei Sviridkin b3b280ea73 feat(gateway)!: add TLS-passthrough listeners to TenantGateway (#3342) 
## What this PR does

Phase 1 of external database exposure via Gateway API TLS-passthrough
(#2815): adds a `tlsPassthroughListeners` field to `TenantGateway` and
renders one layer-4 `mode: Passthrough` listener per entry, on a native
database port with a per-engine subdomain hostname, alongside the
Gateway's terminate listeners. No engine is wired: no TLSRoute, no
certificate or CA changes. The existing layer-7 `tlsPassthroughServices`
field gains a 62-entry count bound, an item length and a DNS-1123 item
pattern, and a repeated entry now fails the reconcile with `Ready=False`
instead of rendering the same listener twice. `spec.apex` gains a
253-character bound and nothing else: the CEL rules concatenate it, so
the install-time cost estimator refuses the CRD without a declared
bound. It gains no format pattern, and that is not a claim that a
mixed-case host works. Such a host reaches the field verbatim from a
namespace label, the renderer then composes listener hostnames Gateway
API refuses, and that tenant has no Gateway either way; what a pattern
would change is only that the write is refused, which fails this
HelmRelease where the same fault costs the one Gateway. An apex no
listener hostname can sit within is reported by the controller instead,
and only when `tlsPassthroughListeners` is in use. The constraints and
what they do to objects that already exist are described further down.
The bound on an entry is what the schema can state on its own. An entry
also becomes a listener name and a hostname built from the apex, so the
controller checks both composed values against the 253 characters
Gateway API allows and names whichever one overflows.

Validation rejects duplicate listener names or ports, ports reserved by
the platform Gateway (80 and 443), out-of-range ports, invalid DNS-1123
names, and hostnames outside the tenant apex (mirroring the Gateway
hostname admission policy so an invalid entry cannot make the whole
Gateway rejected).

The SNI-overlap rule refuses two passthrough listeners that answer the
same name on different ports. That is what
[cilium#42898](https://github.com/cilium/cilium/issues/42898) reports:
only one of the two routes ever works. It does not extend to a terminate
listener and a passthrough listener sharing a name across ports. No
failure is reported for that shape, and Cilium counts a
terminate/passthrough pair as conflicting only when they share a port.

The per-engine subdomain hostname scheme follows the phase-1 gate. The
flat-hostname alternative (pending the Cilium listener-isolation fix
targeting 1.20) is a later scheduling decision that does not change this
API field.

One shape differs from #2815 on purpose. The ticket has `hostname`
optional with a `*.<name>.<apex>` default; here it is required. Deriving
the SNI from the listener name hides the one value an operator has to
get right, and the two drift apart the moment the naming scheme changes.

Found while checking that pairing, unrelated to this field: a stock
`dns01` gateway already renders `tls-api` (passthrough, 443) beside
`https` (terminate, 443, `*.<apex>`), one port with overlapping SNI,
which is the shape Cilium 1.19.6 marks `Conflicted`. That belongs to the
Cilium bump and is recorded on #3343.

Also named rather than fixed: the controller-side rules on this field
stop the whole reconcile when they refuse a spec. `renderGateway`
returns an error, so the Gateway is not updated at all, and the tenant
sees `Ready=False` with `ReconcileError` while new app listeners and
every certificate the controller would issue or amend wait behind it.
Certificates already issued go on renewing, since cert-manager drives
that from the objects already in the cluster and an aborted reconcile
writes nothing to them. Dropping only the conflicting listener and
reporting it on status would be the better shape, but that changes the
error model of the whole renderer rather than one rule, so it is out of
scope here. The current behaviour is at least observable: `Ready=False`
is a signal, not silent drift.

On HTTP-01, a hostname a passthrough listener answers earns no
HTTPS-terminate listener once a TLSRoute claiming that hostname can be
served on the listener, matched by SNI overlap so a wildcard entry
reaches a name beneath it as readily as the name written in the spec,
and the field itself is accepted under certMode http01 alone. That
refusal is stated as the set of modes that can carry a passthrough
listener rather than the set that cannot, because a mode added to the
enum would otherwise be admitted by default: dns01 and existingSecret
serve the whole apex from one wildcard terminate listener the pinned
Cilium cannot keep apart from a passthrough listener beneath it, and
edge ends TLS at the class provider and renders no TLS listener for one
to sit beside. The route half of that condition is load-bearing rather
than incidental. On the pinned v1.19.5, tlsPassthroughFilterChains walks
a listener's routes and skips one with no backends, so a declared
passthrough listener nothing attaches to contributes no filter chain and
matches no ClientHello, while the terminate listener's chain is built
from its own Secret and carries the name alone. Keying the withdrawal on
the declaration would therefore take a name that is being served and
hand it to a listener that forwards nowhere, and that is the ordinary
shape rather than a corner: the chart puts api, vm-exportproxy and
cdi-uploadproxy on every tenant Gateway, while there is one platform
TLSRoute per name cluster-wide and each names one tenant's Gateway as
its parent, so every other tenant renders all three entries with a route
on none. The backend half is that same rule reached by a different road:
the chain is built per route out of the backends that survived
resolution, so a route that attached and forwards nowhere puts no chain
on the name either, a backendRef naming a Service that does not exist
yet because manifests applied out of order, one deleted since, a name
with a typo, or one in another namespace with no ReferenceGrant, and
without that half an ordinary backend mistake takes a live endpoint
offline while Cilium builds nothing to replace it. Resolution asks what
v1.19.5 asks and no more: the reference allowed, the Service present,
and neither its ports nor its endpoints read, because the translator
reads neither. Services and ReferenceGrants are watched alongside
routes, so both halves are deferrals rather than exemptions and the
withdrawal lands on the pass that sees the missing piece arrive. The
withdrawal reaches HTTP-01 alone, because `collectHostnameClaims`
returns nothing under the other two modes and they render no
per-hostname listener to withdraw. A TLSRoute that declares no
`spec.hostnames` claims the hostname of each rendered passthrough
listener its parentRef selects, because that is the name Cilium serves
it on: `ComputeHosts` at v1.19.5 substitutes the listener hostname for a
route that names none, so such a route takes the name over exactly as
one that spells it out does. The tenant-namespace hostname policy
requires a route there to name its hostnames, so this shape reaches a
Gateway from the system namespaces where the platform routes live, or
from an administrator, not from a tenant chart. A claim broader than the
listener's counts the same way from the other side: a TLSRoute claiming
`*.<apex>` is served on `api.<apex>` rather than on the wildcard,
because `ComputeHosts` returns the listener's own hostname for a route
hostname covering it, so a claim covering a reserved name is judged as a
claim on that name while the route keeps one entry under the key its
owner wrote. Both restrictions are tied to the Cilium pin, currently
v1.19.5. That version translates the Gateway into a single Envoy
listener whose filter chains match on SNI without the port, so a
terminate chain and a passthrough chain carrying one name collide
regardless of which Gateway port declared them; upstream states the
consequence above NeedsCrossProtocolSplit in operator/pkg/model/model.go
as of v1.19.6, the release that starts splitting Envoy listeners per
port when NeedsPerPortListeners holds. That is a disjunction rather than
a blanket change, and the passthrough side of every one of its three
clauses is counted from attached routes, so a listener with no TLSRoute
behind it contributes to none of them: the split arrives with the route,
not with the bump.

The same single-listener translation has a consequence that is not a
restriction: the declared port is where the listener is published, not a
bound on reach, because the backend answers its hostname's SNI on every
port the Gateway exposes. Firewalling a tenant address down to 80 and
443 does not put a database out of reach; the hostname is the access
surface, and the field's own doc and the chart README now say so.

Both restrictions are validation-only, and they lift together on the pin
plus a configuration the split actually covers rather than on the bump
alone.

A route whose hostname is withdrawn this way is told so under this
controller's own name: Accepted=False with `NoMatchingListenerHostname`,
and a message naming the passthrough hostname that answers it, which for
a wildcard entry the route cannot derive on its own. It matters most
there. A wildcard entry can withdraw the listener from several published
names beneath it at once, each one a TLSRoute claims and can be served
on, and the certificates are garbage-collected with them, and a rollback
is cheaper than that sounds: cert-manager here runs with
`enableCertificateOwnerRef: false`, so deleting a Certificate leaves its
Secret behind and a re-created Certificate adopts a still-valid one
rather than placing an order. Before this the only signal was Cilium's
own RouteParentStatus entry under a different controllerName. A TLSRoute
whose hostname no passthrough listener answers is told so as well, on
the same condition and for the same reason: before this change that
hostname got a terminate listener Cilium refused with
ResolvedRefs=False, so withdrawing the listener would otherwise have
removed the only object saying the route went nowhere.

Two things stay named rather than fixed. A `tlsPassthroughListeners`
hostname of `*.<apex>` is refused only indirectly: the overlap rule
rejects it while `tlsPassthroughServices` carries its defaults, because
the wildcard covers api.<apex>, so emptying that list is enough to let
it through, and the terminate listener then goes from every published
name a TLSRoute claims under the wildcard. The README states this;
refusing the apex and its wildcard on their own terms is a one-line rule
worth adding when the field is wired up to chart values.

A wildcard hostname on an attached HTTPRoute used to render a listener
name the shipped Gateway CRD refuses, `https-*-<hash>`, which failed the
whole Gateway write while the README said the shape worked. Listener and
Certificate names now spell a leading `*` as `wildcard`, with the hash
still covering the whole hostname, and a test reads both the
listener-name and the hostname patterns off the shipped CRD bundle and
renders wildcards through them; every name that was already admissible
is pinned byte for byte, so nothing is renamed on upgrade. Under http01
an HTTPRoute wildcard claim is refused before the race and the overlap
are read, with `NoMatchingListenerHostname`, because the HTTP-01
challenge issues no wildcard certificate and a terminate listener could
never have carried the name.

A route status write that fails no longer leaves the route carrying its
previous condition: `updateRouteStatuses` and the whole-apex pass
collect every failed write and return them after the loop, so the
reconcile fails, the TenantGateway reports `Ready=False` with
`ReconcileError` and controller-runtime requeues, while the loop still
runs to the end so one route's conflict does not hold the others back.
Conditions this controller wrote under http01 are retracted when the
tenant moves to dns01, existingSecret or edge, for HTTPRoutes and
hostname-less TLSRoutes alike, so a refusal does not outlive the mode
that produced it. And a TLSRoute claiming a wildcard passthrough entry
itself leaves the names beneath the wildcard alone, since Envoy ranks an
exact server name above the wildcard and nothing collides.

One thing this does not cover yet: Accepted is computed from the
hostname race and the withdrawals above, so a route is still told it is
fine when an HTTPRoute holds the name and the TLSRoute claiming it
cannot attach to the terminate listener that answers it. That one
predates this change. Two neighbours of it no longer fall in the gap. A
`parentRef` port that names no listener answering the hostname is
compared alongside the `sectionName`, since Gateway API requires every
key a route pins itself with to match the listener it selects. An
HTTPRoute's `parentRef.port` is judged the same way: the terminate
listener sits on 443, so any other pinned port earns no listener and no
certificate through that parentRef and reads the port-mismatch cause;
port 80, or `sectionName: http`, from a namespace the plain-HTTP
listener admits (the publishing tenant and cozy-cert-manager) is served
plain with no terminate listener and no refusal, from any other
namespace it is refused with `NotAllowedByListeners`, and a
`sectionName` and a port that select different listeners select none and
read `NoMatchingParent`. A plain-served route stays in the hostname race
on the same footing as a terminating one, in every recount as well as
the first count. And a `sectionName` naming a `tls-<name>` listener this
Gateway does not render now reports `Accepted=False` with
`NoMatchingParent`, Gateway API's reason for a parentRef matching no
listener: the controller renders every listener in that namespace, so an
absent name there is a fact rather than a guess, while a `sectionName`
naming the plain-HTTP listener is judged like port 80 below, and any
other name outside the prefix is written Accepted like an attached
route, with the class controller's own status entry answering for the
listener it named.

The namespace case is off that list. Native-port listeners admit the
publishing tenant alone, and a TLSRoute elsewhere on one of their
hostnames is refused with `NotAllowedByListeners`. An HTTPRoute that
pins itself by `sectionName` to a rendered `tls-<name>` passthrough
listener is refused with the same reason: that listener forwards the
stream it matched by SNI and terminates nothing, so it takes TLSRoute
alone by protocol and cannot serve the route whatever answers its
hostname. Its claim also leaves the hostname race before ownership is
ranked, so the TLSRoute that can be served is not told it lost to a
route that could never attach. Cilium does not report this refusal at
the pin. The port-443 passthrough listeners name HTTPRoute in
`allowedRoutes.kinds` to keep the port-443 set uniform (cilium#45559),
the Gateway reports that entry back as
`ResolvedRefs=False/InvalidRouteKinds` on the listener, and Cilium's
route-side kind check reads a listener's kinds against every route on
the Gateway rather than against the routes that named it, so it reports
the route Accepted and the condition this controller writes is the only
refusal the route gets. A hostname whose every claimant is refused this
way earns no HTTPS-terminate listener and no certificate, where before
it earned both. The refusal is decided per route rather than per
hostname, which matters once a claim overlaps more than one reserved
name: a wildcard covering both a port-443 service hostname and a
native-port one is attachable through the first and refused through the
second, so the answer comes from the listener the `parentRef` names, its
own hostname and its own attach set. The row that pinned a `sectionName`
naming no rendered listener as a known gap now pins the verdict instead.
On the port-443 listeners the two namespace sets agree by construction,
since `collectHostnameClaims` and the listeners' allowedRoutes selector
run off the same list.

And the justification for the pin is written out at every site that
depends on it rather than once, so the day the pin moves they all have
to move together and the one that does not is a false statement no diff
can catch. That is not hypothetical here: three claims of exactly that
shape were wrong in this branch and were caught by reading the source
they described rather than by any diff, so the volume of restated
reasoning is itself what produces them. One canonical block with
references to it would carry the same knowledge with one place to
update, and rewriting the narrative under review would be a
documentation change rather than a fix, so it is tracked on #3842
instead.

The gateway chainsaw suite now covers the withdrawal, which it did not
before: a tenant owning its own Gateway renders the shipped passthrough
entries with nothing routed to them, an HTTPRoute on api.<apex> keeps
its terminate listener, its certificate and Accepted=True there, and
attaching a TLSRoute to the same listener, pointed at a Service in the
tenant's own namespace, since a route that forwards nowhere is the case
the first phase already measures, withdraws all three and moves the
route to NoMatchingListenerHostname. A third phase patches in a
native-port entry and reads the per-listener readiness flag, which is
the claim no test against a fake client can settle. The first two phases
are one measurement: withdrawing on demand is what dates the earlier
assertion to settled state rather than to a withdrawal that had not
arrived yet.

The `Require API owner review for sizeable API changes` check is red on
purpose and does not clear with a code change. A new field plus
tightened validation on already-shipped fields counts as a sizeable API
change, so `.github/workflows/api-review-gate.yaml` holds the PR until
an API owner approves it. The gate counts each owner's latest review on
the current head commit, so the approval has to land on the head as
pushed and any later push invalidates it; the job re-runs on review
submission, so no push is needed to flip it. Reading that red as "not
ready" is the mistake this paragraph exists to prevent.

Part of #2815.

### Release note

```release-note
feat(gateway): TenantGateway supports layer-4 TLS-passthrough listeners on native database ports for SNI-based external exposure. On HTTP-01 gateways a hostname a TLS-passthrough listener answers, including any name beneath a wildcard entry, no longer earns an HTTPS-terminate listener or a Gateway-issued certificate once a TLSRoute claiming that hostname can be served on the listener and forwards to a backend that resolves; a hostname claimed only by a TLSRoute stops earning one too. A TLSRoute with no spec.hostnames counts as claiming the hostname of each passthrough listener its parentRef selects, and one whose hostname covers a listener's counts as claiming that listener's name, both being the name Cilium serves it on. A declared passthrough listener with no route on it changes nothing, because it emits no filter chain and the terminate listener is what answers the name, and neither does a route whose backendRefs resolve to nothing, a Service that does not exist, or one in another namespace with no ReferenceGrant, since such a route emits no filter chain either. The controller watches Services and ReferenceGrants, so the withdrawal it defers lands once the backend appears. So the tenant whose Gateway the platform TLSRoutes name, tenant-root unless the cluster's expose-ingress says otherwise, sheds the terminate listeners and certificates for api, vm-exportproxy and cdi-uploadproxy on upgrade (those TLSRoutes ship with the cozystack-api, kubevirt and kubevirt-cdi packages, each rendered only where the Gateway integration is enabled and the service is among the platform's exposed services), and a tenant that renders the same three entries with no route on them keeps what it has. The shed certificates leave their TLS Secrets behind: cert-manager here ships enableCertificateOwnerRef: false, so each Secret stays in the namespace with no owner, nothing renewing it and nothing deleting it. That cuts both ways, as stale Secrets to remove by hand and as a cheap way back, since a re-created Certificate adopts a still-valid Secret instead of placing an order. A route the controller renders no listener it can attach to now carries Accepted=False from this controller instead of Accepted=True. Five shapes report NoMatchingListenerHostname and the message tells them apart: the HTTPRoute whose hostname a passthrough listener answers, the TLSRoute with no passthrough listener behind it, the route whose parentRef names a listener that answers a different hostname, the route whose parentRef names a port no listener answering its hostname is published on, and the HTTPRoute claiming a wildcard hostname under http01, which the HTTP-01 challenge cannot certify. A route pinned by sectionName to a tls-<name> listener the Gateway does not render carries NoMatchingParent instead, which is what Gateway API calls a parentRef matching no listener, and so does an HTTPRoute whose sectionName and parentRef port select different listeners. A TLSRoute outside the publishing tenant's namespace claiming a native-port hostname carries NotAllowedByListeners instead, since those listeners admit the tenant alone. So does an HTTPRoute pinned by sectionName to a rendered tls-<name> passthrough listener, which cannot serve its kind, and an HTTPRoute selecting the plain-HTTP listener from a namespace it does not admit; a refused route also stops counting in the hostname race, so a hostname whose every claimant is refused earns no terminate listener and no certificate. Listener and Certificate names derived from a wildcard hostname are admissible now, names that were already admissible do not change, a failed route-status write fails the reconcile instead of being logged away, and conditions written under http01 are retracted when certMode leaves it. The dns01 and existingSecret cert modes shed nothing, because neither renders per-hostname listeners at all, and tlsPassthroughListeners is accepted under http01 alone: those two serve the whole apex from one wildcard terminate listener, and edge terminates TLS upstream and renders no TLS listener a passthrough entry could sit beside. The withdrawal does not take a working path away: on the pinned Cilium both port-443 filter chains already carry these hostnames and nothing on the objects says which one answers a connection, so a race that was previously undetermined and unobservable now resolves the same way every time. Treat this as a change of serving path rather than the removal of something idle: today each of those three hostnames has a terminate listener and a passthrough listener on port 443, both filter chains carry the SNI, and nothing on the objects says which one answers a connection. Gateway API calls for both listeners to report Conflicted; the pinned v1.19.5 has no Conflicted condition at all, so there is nothing to read, and you cannot tell from the cluster whether api.<apex> is terminated by the Gateway or passed through. Afterwards it is passed through, always. Read the Gateway's listener conditions before upgrading to see what those listeners were serving. Before upgrading, confirm the management apiserver serving certificate carries api.<apex> in its SANs, because once the ACME certificate is withdrawn that is the only certificate left on the name; a cluster that came from the ingress-nginx path already meets that, a cluster served by the Gateway's ACME certificate since it moved does not necessarily and loses the name until the SAN is added. If something depends on the terminate path for one of those names, the package README lists three ways forward and what each costs. Moving the consumer to another hostname under the apex is the only one that gives up nothing.
```


### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against this
diff:
`api/gateway/v1alpha1/{tenantgateway_types.go,zz_generated.deepcopy.go}`,
`internal/controller/tenantgateway/{reconciler,renderers,conflict,mapper,status,backends,reconciler_test,crdcel_test,mapper_test,backends_test}.go`,
`internal/controller/tenantgateway/rbac_test.go`,
`cmd/cozystack-controller/main.go`,
`hack/e2e-chainsaw/gateway/chainsaw-test.yaml`,
`packages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml`,
`packages/system/cozystack-controller/templates/rbac.yaml`,
`packages/core/platform/values.yaml`, and
`packages/extra/gateway/README.md`. No trigger fires, and the two
repositories worth ruling out explicitly are below rather than waved
off.

`cozystack/website`: no package under `packages/apps/` or
`packages/extra/` is added, renamed or removed.
`packages/extra/gateway/README.md` is edited in place. The docs
generator regenerates a reference page from a package README only for
the apps hardcoded in the website `Makefile`, and `gateway` is in none
of `APPS`, `K8S`, `VMS`, `NETWORKING` or `SERVICES`, so there is no
generated page for this package to drift out of step with. That gap is
the same shape the trigger map already records for
`packages/apps/opensearch` and `packages/apps/bucket`; it predates this
change and is neither created nor widened here. None of the other
website triggers apply: `packages/core/platform/values.yaml` changes
only in a comment, which now says that api, vm-exportproxy and
cdi-uploadproxy render a TLSRoute rather than an HTTPRoute, with no key
added, moved or given a new default, no platform variant or component
moves, no release asset is renamed, `ApplicationDefinition` semantics
are unchanged, there is no Talos bump and no developer-tooling change.

`cozystack/terraform-provider-cozystack`: this adds an API field, so the
field-level triggers are the ones to check, and they are keyed to app
charts rather than to this CRD. No app is added under `packages/apps/`;
no `values.schema.json` gains, loses or renames a field; no version
enum, `values.yaml` default, `kind`, `plural`, `release.prefix` or
output Secret/Service name changes. `TenantGateway` is not among the
CRDs the provider types by hand. Those are `Package` under
`cozystack.io` and `Plan`/`RestoreJob` under `backups.cozystack.io`, and
the provider carries no reference to `gateway.cozystack.io` at all. It
therefore falls under the map's last bullet: the spec travels as opaque
JSON, so the provider keeps working and simply offers no typed access to
`tlsPassthroughListeners`, which is the position it already holds for
every other field it does not model, and not a break. The provider's
new-top-level-field test guard is scoped to `api/apps/v1alpha1`, which
this diff does not touch.

`cozystack/ansible-cozystack`: nothing in reach.
`packages/core/installer/values.yaml` and the `cozy-system`/PSA contract
are untouched, no platform variant is added or renamed, none of the
`packages/core/platform/values.yaml` keys the role sets are involved
(the file changes in a comment alone), no object the role waits on is
renamed, and `hack/e2e-prepare-cluster.bats` node prerequisites are
unchanged.

The schema constraints this diff adds to already-shipped fields all
reach objects that already exist, and every one of them is a kind the
apiserver ratchets. `spec.apex` gains a 253-character limit (no pattern:
a mixed-case apex is admitted and refused by the controller only when
passthrough listeners are declared), and `spec.tlsPassthroughServices`
gains a DNS-1123 item pattern and a length bound. A stored value that
violates one still passes any write that leaves it unchanged, so the
rule bites only once the value itself changes. A repeated
`tlsPassthroughServices` entry is reported by the controller as
`Ready=False` rather than refused at admission, and the reason is not
the one this section used to give. Ratcheting does reach a list-type
error: `customresource.ValidateUpdate` validates an incoming object's
list-type semantics only when the stored object passes them, so a stored
duplicate is ratcheted through rather than blocking every write. What
makes the marker the wrong tool is the grain of that ratchet, which is
the whole object rather than the field, so one stored duplicate would
switch list-type checking off for every list on the `TenantGateway` at
once and say nothing about it. Documented in
`packages/extra/gateway/README.md`, and pinned by
`TestTightenedConstraintsOnExistingObjects` together with
`TestListTypeSetWouldNotRefuseAStoredDuplicate`, which compiles a copy
of the shipped schema carrying the marker to show what the apiserver
actually does with one. Shipped defaults conform and the chart already
fails on an empty host, so this is a no-op in practice.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added layer-4 TLS passthrough listeners (`tlsPassthroughListeners`)
with native port + SNI hostname, rendered as passthrough listeners and
limited to TLSRoute attachment.
* **Bug Fixes**
* Added stricter admission-time and controller validation (reserved
ports 80/443, distinct listener ports, tenant-apex hostname rules, and
collision prevention with passthrough services).
* Improved listener rendering safety: prevents exceeding the Gateway
listener cap and ensures AllowedRoutes are isolated per listener.
* **Documentation**
* Updated gateway docs with listener-cap considerations and layer-4
passthrough usage details.
* **Tests**
* Added CRD/CEL and reconciliation coverage for valid/invalid
configurations and rendering behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-11 13:02:57 +04:00
Aleksei Sviridkin ae5b626c8f docs(gateway): describe the passthrough listeners and what they shed 
The chart README gains the native-port field, the three limitations it
carries and where each comes from, the one that is a phase-1 narrowing
rather than a consequence of the Cilium pin, and the three ways forward
for anything that depended on a terminate listener the change takes
away. The withdrawal is stated as what it is, a change of serving path
rather than the removal of something idle, because on the pinned
version both filter chains carry the SNI today and nothing on the
objects says which one answers.

The layers section is brought back in step with the listeners: which
label each listener class selects on, that the attach set is the whole
inheriting tenant subtree rather than the tenant plus the admin list,
and the route conditions an operator reads, including the one that says
the sectionName itself names nothing.

The platform values comment on gateway.enabled said every exposed
service renders an HTTPRoute, which the chart value the withdrawal
reads contradicts. It now separates them: the services whose TLS the
Gateway terminates get an HTTPRoute, while api, vm-exportproxy and
cdi-uploadproxy get a TLSRoute pinned to their passthrough listener,
one each from the packages that own those services.

The overlap paragraph no longer describes an HTTPRoute wildcard as
keeping its terminate listener. Such a claim is refused under HTTP-01,
which issues no wildcard certificate, and the paragraph says so and
what the render used to do with it. The route-condition list gains
that shape and the port shapes an HTTPRoute now reaches, and the
per-app naming bullets carry the hash suffix the rendered names have.

Two sentences that were false against the code are corrected: a
TLSRoute on a wildcard entry itself leaves the names beneath it their
terminate listeners, and an unusable apex is refused by the containment
rule at admission before the controller's own check is reached. The
edge section says what happens to the entries the HTTP-01 pass wrote
once the mode moves.

The NoMatchingParent sentence now separates the two kinds: an
HTTPRoute pinned to an absent passthrough section is refused whatever
answers its hostname, while a TLSRoute is judged where a passthrough
listener answers it.

A sectionName outside the passthrough prefix is written Accepted like
any attached route rather than left without a verdict, and the text
says so; the port-80 listener's refusal joins the NotAllowedByListeners
shapes and the name-and-port disagreement the NoMatchingParent ones.

The attachedNamespaces comment described the list as reaching every
listener, which the native-port ones do not: they select on
kubernetes.io/metadata.name and name the publishing tenant alone, so a
namespace listed there cannot attach to a database port. It also
credited the wrong label with keeping the list honest; what a listed
entry gets is the namespace.cozystack.io/gateway label, and what keeps
a tenant from stamping it on itself is that no tenant role carries
patch on Namespace.

The class reference is brought along. Its certificate-mode table
promised one Certificate per published hostname under http01, which a
hostname served by a passthrough listener no longer earns, and its
edge table said every attached TLSRoute is refused, where only a route
declaring hostnames is and the rest have this controller's entry
removed instead.

The refusal-shape list is corrected against the code, without a count
of its own so the next correction does not have to move one. A route
the plain-HTTP listener turns away on its namespace
selector was listed under NoMatchingListenerHostname as well as under
NotAllowedByListeners; the controller reports the second. The port
shape standing beside it swept port 80 back in by naming every port
but 443, where a route selecting the plain-HTTP listener is answered
by the namespace shapes and never reaches the port refusal at all. And a
TLSRoute pinned to an absent section is said to carry
NoMatchingListenerHostname where no passthrough listener answers its
hostname, which holds only while no HTTPRoute claims that hostname:
with one there a terminate listener stands on the name, and the route
is judged like any other attached route.

The wildcard shape gains the same qualifier. A wildcard claim is
refused because no terminate listener can carry a certificate HTTP-01
will not issue, so the refusal reaches a route that wanted a terminate
listener and not one that asked for the plain-HTTP listener, which
carries no hostname and serves the claim there.

The Helm merge paragraph named the two-way path as what happens to a
custom resource Helm has no Go type for. That path is one of two in
createPatch, the version deciding which runs, and the version comes
from helm-controller rather than from this tree, so the claim was not
checkable where it was written. It now names both and says the outcome
it rests on holds either way.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-10 18:48:38 +03:00
Aleksei Sviridkin 1f04e31ed3 feat(backups): add Redis backup/restore strategy driver (#3992) 
## What this PR does

Adds backup/restore support for the `Redis` app (spotahome
`RedisFailover`) via a dedicated `strategy.backups.cozystack.io/Redis`
strategy controller. The spotahome operator ships no backup CRD, so the
driver follows the generic-script-Job pattern (a specialisation of the
Job/Altinity drivers): it runs a one-shot Job that dumps an RDB snapshot
from the current master to the system bucket and replays it on restore.

- **Backup**: discover the current master via the operator's sentinel
Service (`rfs-<name>`, master `mymaster`) → `redis-cli --rdb` → `curl
--aws-sigv4` PUT to `s3://<bucket>/<namespace>/<app>/dump.rdb`.
- **Restore**: GET the object → convert RDB → RESP (`hdt3213/rdb`) →
`FLUSHALL` + `redis-cli --pipe` into the target's master. Both
**in-place** and **to-copy** are implemented; a to-copy restore reads
the source's prefix via `.Backup.ApplicationRef.Name` while writing into
a differently-named target.
- **PITR**: not supported — RDB is a discrete snapshot with no
continuous log to replay to an arbitrary time (the etcd-like case).

The driver is a specialisation of the Job/Altinity script-Job flow with
a `Redis` applicationRef gate and an app-readiness precondition
(surfaces a precise `Ready=False` instead of a silent hang; blocks only
on an explicit `Ready=False` so a missing condition never wedges
backups). Strategy parameters round-trip through the Backup's
`DriverMetadata`; there is no `underlyingResources` snapshot (the Job
pattern re-renders on restore).

### System bucket

Coordinates and credentials reach the Job as env `secretKeyRef` from the
projected `cozy-backups-creds` (the ClickHouse/Altinity delivery model),
so **the app chart is untouched** — no `backup.useSystemBucket`, no new
RBAC (`batch/jobs` and `apps.cozystack.io/*` are already granted). The
bare-host `endpoint` gets `https://` prepended when unscheme'd; against
a self-signed endpoint the strategy verifies with the CA from
`backupStorage.endpointCASecretName` when set, otherwise against the
image's system CA store; skip-verify is a separate explicit
`backupStorage.insecureSkipTLSVerify` opt-in, never a fallback. The
`cozy-default-redis` strategy is gated on `$bucketName` (CRD-ordering)
and bound in the `cozy-default` BackupClass.

### Changes

- `api/backups/strategy/v1alpha1/redis_types.go` + generated CRD and
deepcopy.
- `internal/backupcontroller/redisstrategy_controller.go` (+ unit tests)
wired into both dispatch switches, `supportedBackupStrategyKinds`, and
the no-op restore-cleanup case.
- `cozy-default-redis` strategy template + `cozy-default` BackupClass
route + helm rendering-gate coverage.
- `images/redis-backup` (redis-cli + curl + RDB→RESP converter) +
Makefile target + `backupStrategyController.redisBackupClientImage`.
- `examples/backups/redis/` — an honest marker round-trip for both
restore flows.

### Validation

`go build`/`vet`, `go test ./internal/backupcontroller/...
./api/backups/...`, `make generate` (clean), and `helm unittest` (10/10)
all pass. Not yet cluster-verified.

## Downstream

None.

```release-note
Add backup and restore support for the Redis application (spotahome RedisFailover) via a new Redis backup strategy, wired into the cozy-default BackupClass. Supports in-place and to-copy restore; PITR is not supported.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Redis backup and restore support using RDB snapshots in
S3-compatible storage.
  * Supports restoring data in place or to a separate Redis instance.
* Added configuration for Redis backup images, custom HTTPS certificate
authorities, and TLS verification.
* Added a complete Redis backup and restore demonstration with automated
setup, verification, and cleanup.

* **Documentation**
* Documented Redis storage, credentials, endpoints, TLS, and restore
behavior.
  * Added administrator and tenant scenario guides.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-10 19:17:26 +04:00
Aleksei Sviridkin b083d8e9e6 test(platform): pin the opensearch-operator DNS base wiring 
Two ends of the same path. The platform suite pins that the shared
values Secret carries networking.clusterDomain to the operator under
the subchart key, and the package suite pins that a domain set on that
key reaches the manager container as DNS_BASE, with the upstream
cluster.local default intact when nothing sets it.

Every pattern in the platform suite anchors dnsBase to its
opensearch-operator and manager parents rather than matching the key
anywhere in the payload. An unanchored pattern is satisfied by any
sibling block that happens to carry the same domain, which leaves the
assertion green after the block it is meant to pin has been deleted.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-09 23:57:29 +03:00
Aleksei Sviridkin 857ad4a128 fix(opensearch-operator): resolve in-cluster names on the platform domain 
The operator reads DNS_BASE and builds every in-cluster name it needs
from it: the securityconfig job target, its own OpenSearch API client
URL, the Dashboards host, and the SANs it writes into node
certificates. The vendored chart defaults that to cluster.local, so on
a cluster whose domain is cozy.local the securityconfig job never
resolves its target, the .opendistro_security index is never created,
and the OpenSearch nodes never go Ready.

The domain the platform already holds could not reach the operator
before: _cluster stops at each package's own templates and never
reaches a vendored subchart, and the wrapper's values.yaml is not
templated. Carry it in the shared values Secret under the subchart key
instead, the same way the KEDA cluster domain travels, so the vendored
chart picks it up with no patch and a package rendered without the
Secret keeps the upstream cluster.local fallback.

A global key would have worked for the operator but reaches every
release on the cluster, and the vendored smee chart in bootbox cannot
render once global is non-empty. Scoping the value to the subchart
that consumes it avoids putting every other package on that path.

The diagnosis and the first working fix came from fuad00 in an earlier
pull request.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-09 23:56:12 +03:00
scooby87 afc003c464 feat(postgres): database read-replica autoscaling via KEDA (#3954) 
## What this PR does

Adds automatic horizontal autoscaling of a PostgreSQL database's **read
replicas** in response to load, implementing the merged design proposal
[Database Horizontal
Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling)
(community#44). The mechanism is entirely stock — no bespoke operator,
no new CRD:

- **KEDA as an optional platform component** (`packages/system/keda`,
wired into `core/platform` as a source + system-bundle entry). It claims
the `external.metrics.k8s.io` APIService that nothing served before.
- **`cozy-lib.keda.scaledObject` helper**
(`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA
`ScaledObject`, fail-closed on missing/invalid arguments.
- **`packages/apps/postgres` autoscaling** — an `autoscaling` values
block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`,
`maxReplicationLagSeconds`, `dryRun`, `transition`) validated by
`values.schema.json`; under active autoscaling the chart renders a
`ScaledObject` and seeds `spec.instances` to a constant floor
(`max(replicas, effectiveMin)`), while the postgres release is forced to
client-side (two-way) Helm apply so the seed is a merge no-op on upgrade
and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster`
scale subresource — is not reverted.
- **Single-value metric (§1)** — one PromQL query returns `Σ(read load
over standbys) + target`, so a stock HPA computes `desired =
ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`,
with no per-pod emission.
- **The four static brakes (§5)** — quorum floor `max(minReplicas,
maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down
pacing (`Pods/1/600s`), a replication-lag clamp folded into the query
(freeze both directions while lag exceeds the threshold and the primary
is writing, with `max_over_time` hysteresis), and dry-run (render
paused).
- **Two-phase enable/disable migration** via a `transition` sub-flag:
phase 1 stages `spec.instances` to the live count and brings KEDA up
with its HPA paused in both directions; flipping to phase 2 hands the
runtime count to KEDA, and because the seed re-renders the same count
the flip is a merge no-op with no dip.
- **Observability** — a database-autoscaling Grafana dashboard and
PrometheusRule alerts.

**Live validation (dev cluster, CNPG 1.30.0).** The full path works
end-to-end: the CNPG `Cluster` scale subresource serves
`status.selector`, KEDA renders the HPA, the metric reaches it
(`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2
= 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query
returns the base value on live VictoriaMetrics. Two packaging fixes were
needed and are included: KEDA must run on the cozystack `cozy.local`
cluster domain (the upstream `cluster.local` default makes the metrics
gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`),
and the operator's client-go throttle is raised so a single-replica
operator does not lose its leader lease under scale-subresource polling.

Implements #3953.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two repositories are affected; both are follow-ups to open once
this lands (and after its dependency #3951 merges and the values surface
is final), left unticked here for a maintainer to confirm rather than
claimed with a speculative link:

- **cozystack/terraform-provider-cozystack** — this adds the nested
`autoscaling` block to `packages/apps/postgres/values.schema.json`. The
provider hand-models each app's fields (schema + model +
expand/flatten), so it needs the new block to expose it in Terraform.
- **cozystack/website** — KEDA is a new platform component
(`guides/platform-stack`), and the postgres reference page gains the
`autoscaling` params. The managed-app reference is regenerated from
`README.md` by the release bot; the platform-stack component list is
hand-maintained.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: to open (new platform component + postgres autoscaling
params)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: to open (new `autoscaling` block in postgres
values.schema.json)

### Depends on

- #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG
`Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves
`status.selector`; without it the HPA fails with `InvalidSelector`. This
PR should not merge before #3951.

### Release note

```release-note
feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added configurable PostgreSQL read-replica autoscaling with KEDA.
* Added CPU and read-connection metrics, replica bounds, replication-lag
safeguards, dry-run mode, and transition handling.
* Added optional KEDA platform support, Grafana autoscaling dashboard,
and Prometheus alerts.

* **Documentation**
* Clarified integer thresholds, prerequisites, configuration, and
transition guidance.

* **Bug Fixes**
* Improved validation and actionable errors for unavailable KEDA or
invalid settings.

* **Tests**
* Expanded coverage for autoscaling, observability, package selection,
and validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-09 21:28:34 +03:00
Andrey Kolkov 1289f522fa fix(backups): unbreak install, kill destructive cleanup, reach the resource knob 
Store the Redis strategy Template schemaless, as the Rabbitmq strategy
already does: without the markers controller-gen emitted the full
PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's
Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed
to install at all.

Give the example strategy a cleanup branch, and both scripts an explicit
`elif restore` / `else exit 1`, so a deleted Backup rendered with
Mode=cleanup deletes its object instead of falling into the restore path
and FLUSHALLing the live master.

Forward a backupStrategyController block through the platform bundle
alongside backupStorage, so redisBackupResources — the restore-Pod sizing
knob the loader-not-ready message points at — is reachable via
spec.components.platform.values.backupStrategyController, not merely
settable on a Helm-managed Strategy CR that reverts on the next upgrade.

Correct the endpointCASecretName doc row to match the now-optional mount,
and `int` the activeDeadlineSeconds interpolation so a large values-file
override renders as an integer, not exponent form.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-09 16:53:50 +04:00
Aleksei Sviridkin 2055341ba9 fix(opensearch-operator): raise vm.max_map_count on every node from a DaemonSet (#4152) 
## What this PR does

OpenSearch will not start below `vm.max_map_count=262144` when the store
may use mmap, and nothing in the platform sets it. The Talos profiles
declare no sysctls and no documented node-preparation path sets one, so
nodes sit at the kernel default of 65530. The app chart asks the
operator to cover that with `spec.general.setVMMaxMapCount`, which
appends a privileged init container to every OpenSearch pod. Tenant
namespaces carry no PodSecurity label of their own, so they take the
cluster default, which Talos ships as baseline through the apiserver's
`PodSecurityConfiguration`. That default refuses the container and the
StatefulSet never produces a pod. On vanilla kubeadm, kind or k3s the
default is `privileged` and the same release comes up, which is why this
only bites on a Talos install.

The value cannot come from the pod at all. The kernel does not namespace
`vm.*`, so kubelet rejects it in `securityContext.sysctls`, and refuses
to start if it appears in `--allowed-unsafe-sysctls`. That leaves node
configuration or a privileged pod, and a privileged pod belongs in a
system namespace rather than in every tenant.

This adds a DaemonSet to the `opensearch-operator` package that sets the
value once per node from a privileged init container, and marks the
component privileged so the platform labels `cozy-opensearch-operator`
with `enforce=privileged`. The init container writes to `/proc/sys` and
nothing else, with no hostPath. The pod restarts with the node and
reapplies the value, so there is nothing to persist.

It raises the limit to a floor rather than assigning it, which matters
more than it looks. `vm.max_map_count` is node-global, and a stock
Ubuntu 24.04 node already reads 1048576 from a `sysctl.d` drop-in. An
unconditional write would drop that to 262144 for every workload on the
node, on every boot, and undo whatever the operator set it for. So the
init container reads the current value first and writes only when it is
below the threshold.

The idle main container runs its sleep under a shell that traps SIGTERM,
so a pod exits on its own when the DaemonSet rolls or a node drains,
instead of ignoring the signal and being killed after the full grace
period.

This does not fix OpenSearch on its own. The app chart still requests
the operator's init container through `setVMMaxMapCount: true`, so a
release stays blocked until the flip to `false` lands with #2682, after
this merges. The DaemonSet has to be in place first: flipping it without
one would trade the admission failure for a failed bootstrap check.
Carrying the sysctl in the node contract instead, across the Talos
profiles and the downstream node-prep paths, is tracked in #4144; that
is what would let this DaemonSet be retired.

Two things change on upgrade, and neither is visible from the diff. One
small DaemonSet pod appears per node in `cozy-opensearch-operator`, and
that namespace gains `pod-security.kubernetes.io/enforce=privileged`.
That label has no finer granularity than the namespace, so the operator
Deployment sitting there leaves the cluster default too, not just the
DaemonSet. Keeping the manager at baseline would mean a namespace of its
own and another component entry, which did not look worth it, but it is
a real cost rather than an accident. The operator is in the `paas`
bundle's always-on list, so this lands on every node of every paas
installation and raises a node-global kernel tunable there, whether or
not anyone ever creates an OpenSearchCluster. That is the price of a
node-level sysctl having no cheaper carrier, and it is worth an explicit
decision rather than being noticed later.

The second is `cozypkg add`, which gates privileged components behind a
confirmation or `--allow-privileged`. The gate runs over the resolved
dependency closure, and `opensearch-application` dependsOn the operator,
so installing the app that way now asks one more question. This is not a
new break for scripts: `add` already prompts for a variant on every
package it resolves and fails on EOF, so it was never usable
non-interactively in the first place. Platform-bundle installs go
through the reconciler and are unaffected.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. It adds a template, a test and a values key under
`packages/system/opensearch-operator/`, plus one field in
`packages/core/platform/sources/opensearch-operator.yaml`. No new app
package, no app schema or version enum, no `ApplicationDefinition`, no
installer value, no namespace rename, no variant or bundle, no `hack/`
path or make target, and no node prerequisite in
`hack/e2e-prepare-cluster.bats`. Nothing in the map matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(opensearch-operator): set `vm.max_map_count` on every node from a DaemonSet in the operator's namespace, so OpenSearch no longer depends on a privileged init container that PodSecurity baseline refuses in a tenant namespace. Upgrading adds one small DaemonSet pod per node and labels `cozy-opensearch-operator` with `pod-security.kubernetes.io/enforce=privileged`; installing the package with `cozypkg add` now requires confirmation or `--allow-privileged`. Existing OpenSearch releases are unchanged until a follow-up stops the app chart requesting the operator's own init container.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- OpenSearch Operator installation now runs with the required privileged
configuration.
- OpenSearch deployments automatically ensure nodes meet the required
`vm.max_map_count` setting, improving startup reliability.
  - Existing higher kernel values are preserved.
- Added support for configuring the sysctl image, including use with
air-gapped registries.

- **Tests**
- Added coverage to verify privileged installation, node configuration,
security settings, scheduling behavior, and image customization.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-08 19:45:14 +04:00
Aleksei Sviridkin a6caad7c57 fix(platform): correct what disabledPackages does to ouroboros (#4098) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

`bundles.disabledPackages` does not uninstall anything. Two comments in
the platform chart said it does, offered next to `kubectl delete
package.cozystack.io cozystack.ouroboros` as if the two were
interchangeable.

A name in that list only stops `_helpers.tpl` emitting the Package
document, which is the same suppression you already get from leaving
`publishing.proxyProtocol` false. Every emitted Package carries
`helm.sh/resource-policy: keep`, and that annotation is there precisely
so disabling a package stops deleting it, so Helm leaves the Package CR
alone. The CR stays, the HelmRelease it owns stays, ouroboros keeps
running, and the pre-delete hook never fires because nothing uninstalls
the release. Deleting the Package CR is what gets there: GC takes the
HelmRelease, Flux uninstalls, the hook runs on the way out.

There was a second cost. The guard in the system bundle looks up the
live Package CR and never reads `disabledPackages`, so an operator who
followed the old advice hit the guard anyway with no idea why. New test
case pins that.

Comments only in the templates, so rendered output is unchanged. Chart
suite green at 156 tests.

### Screenshots

Not a UI change.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff. One repo is reached, and I left
its box empty rather than tick it without a link.

`cozystack/website` has the same wrong instruction verbatim in three
files: `content/en/docs/next/networking/hairpin-proxy-protocol.md:122`,
and the identical line under `v1.4/` and `v1.5/`. It tells the reader
that removing the Package CR "(or add it to `bundles.disabledPackages`)"
triggers helm uninstall and fires the pre-delete hook. It also
contradicts `operations/configuration/components.md:72` on the same
site, which says `disabledPackages` will not remove components that are
already installed. The site fix is cozystack/website#683. It turned out
to be four files rather than three: `origin/main` there also carries a
`v1.6` tree, and all four copies were byte-identical.

Nothing else is reached: no package added or renamed, no
`values.schema.json`, no version enum, no default, no key the Ansible
role sets, nothing under `hack/`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/683
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(platform): the platform chart no longer says `bundles.disabledPackages` uninstalls a package. A name in that list only stops the Package document being rendered. `helm.sh/resource-policy: keep` leaves the existing Package CR on the cluster and the component keeps running, so deleting the Package CR is what uninstalls it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Clarified proxy protocol safety guidance, including the effects of
disabling the Ouroboros package and the limitations of using
`bundles.disabledPackages`.
- Documented that disabling package rendering does not remove an
existing live package or stop its associated resources.

- **Tests**
- Added regression coverage confirming that disabling the package does
not bypass the proxy protocol safety guard when the live package remains
active.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-08 02:52:00 +04:00
Aleksei Sviridkin 675163399d fix(platform): accept the container gpu-operator variant in iaas (#4101) 
## What this PR does

`bundles.iaas.gpuOperatorVariant: container` failed the whole platform
render. The `cozystack.gpu-operator` PackageSource ships that variant,
`values-container.yaml` configures it deliberately, and
`hack/check-gpu-operator-variants.bats` covers it, but the `iaas` bundle
guard only accepted `default` and `vgpu`.

The exclusion made sense when it was written and stopped making sense a
few minutes later. The variant and the guard came from parallel
branches, neither an ancestor of the other, variant merged first. Before
the guard the bundle hardcoded `default`, so `container` was selected by
hand-writing a Package CR, and leaving it out of a bundle-side list cost
nothing. The guard made the bundle the variant owner and closed that
route without noticing it was one. Its comment calls itself typo
protection and names `passthrough` as the example typo.

Allowing the value is not the whole fix. The same file puts
`extraFeatureGates: [HostDevices]` on the KubeVirt CR whenever
gpu-operator is enabled, and builds a `permittedHostDevices` table under
it. Both are VM-facing. The container variant keeps the host driver
bound and pins `vfioManager` off, so no GPU reaches a VM: the gate would
advertise something the node cannot serve, and the table would name
resources no sandbox device plugin registers there. Both now render for
`default` and `vgpu` only, and `container` gets the gpu-operator Package
with the kubevirt component untouched.

The helm-unittest case that pinned the rejection is gone. Two replace
it: the Package renders at the container variant with no component
overrides, and the KubeVirt wiring stays absent even with
`gpu.permittedHostDevices` and `gpu.mediatedDevicesConfiguration` set.
Comments and docs that called the mirroring unconditional on variant, or
the container variant unmerged, are fixed in the same pass.

No issue for this one, so the history is in the body.

### Screenshots

No UI change.

### Downstream repositories

Walked the trigger map against the diff. The only file it touches that
appears there is `packages/core/platform/values.yaml`, and the website
hit is not the hand-written key table (that page has no GPU section): it
is the documented workaround this change makes obsolete.
`content/en/docs/next/operations/gpu-container-workloads.md` currently
tells readers the bundle rejects `container` and routes them to a
hand-crafted Package CR, and `virtualization/gpu.md` / `vgpu.md`
describe the KubeVirt mirroring as unconditional on variant.

Filed as an issue rather than a PR because the docs must not land ahead
of this change, and the `v1.6` copies describe released behaviour and
stay correct as they are.

Nothing else matches: no package added, renamed or removed, no
`values.schema.json` or version enum touched, no `ApplicationDefinition`
semantics, no installer values key, no `hack/` layout change, no
platform variant or bundle recomposed. `gpuOperatorVariant` appears in
none of `ccp`, `ansible-cozystack` or `terraform-provider-cozystack`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/685
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): accept `container` as a value for `bundles.iaas.gpuOperatorVariant`, so the gpu-operator variant the platform already ships can be selected from platform values instead of a hand-crafted Package CR. The KubeVirt `HostDevices` feature gate and `permittedHostDevices` table are now emitted for the `default` and `vgpu` variants only; `container` serves GPUs to pods and gets neither.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `container` GPU operator variant for workloads that do not
require VM passthrough.
* Container workloads no longer receive KubeVirt host-device or
mediated-device GPU wiring.
* Clarified GPU configuration behavior for passthrough, vGPU, SR-IOV,
and container workloads.

* **Documentation**
* Updated passthrough guidance, configuration references, examples, and
KubeVirt wiring instructions for all supported variants.
* Added details for manually configured package resources and
platform-specific container setup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-08 02:50:33 +04:00
Aleksei Sviridkin a3ddba441f fix(opensearch-operator): set vm.max_map_count from a node DaemonSet 
OpenSearch refuses to start when vm.max_map_count is below 262144 and
the store may use mmap. The kernel default is 65530 and nothing in the
platform's node configuration raises it: the Talos profiles declare no
sysctls, and neither does any documented node-preparation path.

The value cannot come from the OpenSearch pod. The kernel does not
namespace vm.*, so kubelet refuses it in a pod's securityContext.sysctls
and in --allowed-unsafe-sysctls alike, both of which admit namespaced
sysctls only. The operator's own answer, spec.general.setVMMaxMapCount,
appends a privileged init container to every OpenSearch pod. Tenant
namespaces carry no PodSecurity label of their own and take the cluster
default, which Talos ships as baseline through the apiserver's
PodSecurityConfiguration; that default rejects the container, so the
StatefulSet is created and never produces a pod.

Set the value once per node instead, from a DaemonSet in the operator's
own namespace. The init container raises the limit to a floor rather
than assigning it: the tunable is node-global, so a node already tuned
higher for another mmap-heavy workload keeps its own value. The pod runs
at system-node-critical: it is a precondition for every OpenSearch pod on
the node, so it must not lose a scheduling race to the workload it
enables, though priority is not start ordering: a pod that starts first
just fails its own bootstrap check and retries. The idle container runs
the sleep under a shell with a TERM handler, because PID 1 of a
namespace receives only the signals it installed a handler for and a
bare sleep would be SIGKILLed after the full grace period on every
rollout and drain. The write
goes to /proc/sys only, with no hostPath and no edit to the node's
on-disk configuration; the pod restarts with the node and reapplies the
value. The component is declared privileged so the platform labels that
namespace enforce=privileged, which is what this DaemonSet needs and
what tenant namespaces deliberately do not get.

The file returns to a path bb660b57c emptied. That earlier revision
bind-mounted the host's /etc to append to sysctl.conf and ran an
unpinned image; this one writes no host file and pins the image by
digest.

This removes the reason for an OpenSearch pod to ask the operator for
the sysctl, but the app chart still asks. Dropping that request is a
separate change and is what stops the pods being rejected.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-08 00:35:06 +03:00
Andrei Kvapil 54621bc046 feat(migration): tenant self-service VM import from VMware vSphere 
Hidora's customers are leaving VMware now, and a tenant has to be able to
import their own machines without a platform administrator. The chart
approach could not express that: a tenant can neither create the Secret
holding vCenter credentials nor name the VDDK image, so both had to come
from somewhere they cannot reach.

Two CRDs in a new `forklift.cozystack.io` group, split by lifecycle. A
VMImportSource is a long-lived connection -- type, endpoint, credentials,
readiness -- and a VMImportTask is the one-shot operation that names
machines and produces VMDisks and VMInstances. The outputs carry no owner
reference back to the task, so deleting a finished import leaves the
machines alone. Konveyor Forklift does the transfer underneath; the
controller renders its objects, mirrors its verdicts, and hands each
finished volume into a VMDisk without copying it.

What a live vCenter migration forced out, and no amount of source reading
would have:

- A guest copied verbatim keeps the drivers it had under VMware and no
  virtio, so on a virtio disk it does not fail the import -- it succeeds
  and then does not boot. Imported disks go on SATA.
- VDDK connects straight to the ESXi host at the address vCenter
  advertises, which is routinely unreachable, and on the first cluster
  collided with the Service CIDR. `spec.hosts` redirects per host.
- Firmware must be carried across or a UEFI guest imports as BIOS and
  never boots (stacked on #3002).
- Deleting a task mid-transfer orphaned its DataVolume; a finalizer now
  removes the engine's volumes and leaves the outputs.

The console gains a Migration section: connections are editable, and the
VM field is a picker built from a machine list the controller publishes,
since the aggregated API cannot reach the inventory and should not hold a
source's credentials to try. Two general mechanisms fall out of that --
an option source may take an argument, and one named in a CRD annotation
may reference a sibling field.

Design: cozystack/community#62. Docs: cozystack/website#673.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-09-07 17:03:44 +02:00
myasnikovdaniil 94a202bc5a test(e2e): run the merge-gating lanes on Talos containers (#4020) 
Replaces QEMU substrate with Talos containers on both lanes that gate a
merge. srv1-srv3 run as containers, so tenant worker sits at L2 instead
of L3.

Measured in CI on commits where both substrates ran the same tree:

| | qemu | container |
|---|---|---|
| `kubernetes-latest` / `-previous` | 1143-2040s, red on every
vgif-absent runner | 466-809s, green in both strata |
| whole job | | 24-49 min faster |

Almost all of it is in Chainsaw. Install takes about half an hour either
way.

Node-join soft-red gate goes with it. It was a prosthetic for nested
virt: a worker that registers in minutes elsewhere could miss any
deadline this test can afford, and #3513 established the discriminator
is host `kvm_amd` `vgif`, not anything about the product. Remove the
nesting level and there is nothing left to tolerate, so
`hack/e2e-node-join-soft-red.sh`, the marker and the `soft_red` outputs
are gone from all four workflows.

### What moves off the per-PR path

DRBD, and with it `replicated` StorageClass, its `Immediate` binding
mode, tenant StorageClass propagation (applies only to
remotely-accessible classes), and cozystack talos node image with its
extensions. None of it is lost - `nightly.yaml` and `e2e-tag.yaml` still
run the suite on QEMU, so all four keep nightly and release-candidate
coverage. Live migration is not on that list because nothing tests it
today anyway.

One more belongs on the list and was missing from it: the nocloud disk
build. `build-talos` no longer runs `make -C packages/core/talos
talos-nocloud`, so `images/talos/profiles/nocloud.yaml` is compiled by
`nightly.yaml` and by the tag build's `make assets` rather than by
anything a PR runs, and `promote-rc.yaml` still hard-fails a promotion
whose rc release has no `nocloud-amd64.raw.xz`. Nothing in that chain
breaks, because the tag build supplies the asset as before. What a PR no
longer catches is narrow: it still compiles `installer.yaml` through the
same imager, and the two profiles are generated by the same script and
differ only in `platform`, `kind`, `imageOptions` and `outFormat`.

### Product fixes

Four, all found by measurement while building the lane, each on its own
commit:

The linstor `drbd.enabled` gate. Without it drbd-logger sidecar exits,
satellite is never Ready, piraeus registers zero nodes and every PVC
hangs Pending behind a cluster that looks healthy.

linstor `--strict-topology`. Without it external-provisioner passes
every topology segment as `requisite`, so a node-pinned volume can be
provisioned away from its pod and the pod is then unschedulable forever
with nothing erroring anywhere.

kubevirt-cdi importer resources are configurable now. Default restates
CDI's own values exactly so production behaviour does not move, and the
lane raises the ceiling through its own Package.

vm-disk binds standalone disks immediately. A VMDisk on `local` silently
never populated, while the same disk on `replicated` did.

### Where the run stands

The lane has gone green twice, most recently on `cc70cef7b` where `E2E
(in-tree)` took 2h3m against its 215m cap with 45 Chainsaw tests
passing. The measurements quoted elsewhere in this description come from
the first of those runs, on a commit that has since been rebased away:
the two tenant-Kubernetes suites came in at 569.72s
(`kubernetes-latest`) and 470.19s (`kubernetes-previous`) against a 67m
operation. Both runs answer the four earlier reds, all of which were the
lua-protobuf segfault from #4012 once `vminstance`/`vmdisk` were fixed
on the branch.

ghcr.io mirror removal is carried here too and is the same change as
#4007, whichever lands first makes the other a no-op.

### What review changed

`drbd.enabled=false` now refuses to render together with
`talos.enabled=false`. It removed the logger sidecar only, while the two
DRBD initContainers piraeus contributes unconditionally are deleted by
the Talos configuration alone, so that combination produced a satellite
that can never reach Ready. `isp-full-generic` sets
`talos.enabled=false`, so it was reachable and not theoretical.

vm-disk keeps unconditional immediate binding, a standalone disk has no
consumer to wait for. What it costs on a node-pinned class is in the
chart README now, and the annotation is pinned per source type and on
the existing-DataVolume path where `helm upgrade` retrofits it.

The rest is the harness holding its own promises: HelmRelease gate reads
are bounded and its minimum-count half is pinned by a test that fails
without it, two Chainsaw ops sit above the deadlines they contain, the
fallback-StorageClass render restores `replicated` when it fails,
container lane compose project and pool files are per-checkout, and
three comments that said the opposite of the code now say what is true
(QEMU is still wired into nightly and e2e-tag, the lane replaces two
Packages and not one, the argument-count assertion cannot see upstream
drift).

### Release note

```release-note
fix(linstor): CSI provisioning is constrained to the node the scheduler chose (`--strict-topology`), and the DRBD sidecar can be switched off with `drbd.enabled` on substrates whose kernel cannot run DRBD. On an existing cluster the new default re-templates `spec.csiController.podTemplate`, so piraeus rolls `linstor-csi-controller` once during the upgrade. `drbd.enabled=false` requires `talos.enabled=true` and the chart now refuses the other combination at render time, because only the Talos satellite configuration removes the DRBD initContainers.
fix(kubevirt-cdi): CDI worker pod resources are configurable through `importerResources`, which reaches the importer, the uploader and the host-assisted cloner. The default restates CDI's own built-in values exactly, so an install that sets nothing behaves as before.
fix(vm-disk): every disk requests immediate binding, not only `upload` sources, so a standalone disk on a WaitForFirstConsumer StorageClass is populated instead of waiting for a consumer that may never arrive. On a node-pinned class such as `local` the volume then binds where CDI's worker pod was scheduled rather than where the VM will run; see the chart README before choosing such a class for VM disks. The default `replicated` is unaffected.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added container-based end-to-end testing with local storage support.
* Added optional backup-access preflight checks for database backup
examples.
* Added configurable CDI worker resources and LINSTOR storage settings.
  * Added configurable immediate storage binding for VM disks.
* Added stronger readiness checks, diagnostics, and parallel test
execution.

* **Bug Fixes**
* Improved cleanup failure reporting, capacity validation, and
missing-image detection.
* Improved FoundationDB health verification and storage placement
behavior.

* **Removed**
  * Removed GHCR mirror support and soft-red node-join tolerance.
  * Removed standalone OIDC render test suites.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-07 19:33:20 +05:00
Aleksei Sviridkin 6e8566a94c fix(platform): accept the container gpu-operator variant in iaas 
The cozystack.gpu-operator PackageSource ships three variants: default,
vgpu and container. The iaas bundle's variant guard accepted only the
first two, so selecting container failed the render on a value the
package ships, values-container.yaml configures deliberately, and the
variant bats suite already covers.

The exclusion was coherent when it was written and stopped being so
immediately. The container variant and the guard landed within minutes
of each other from parallel branches, neither an ancestor of the other,
variant first. Until the guard, the bundle hardcoded the default variant
and container was selected by hand-crafting a Package CR, so leaving it
out of a bundle-side list cost nothing. The guard then made the bundle
the variant owner and closed that route without noticing it was one; its
comment describes itself as typo protection and names passthrough as the
example typo.

The KubeVirt side is not a bare allowlist edit. The container variant
serves GPUs to pods through the device plugin and runs no VM-facing
GPU stack: sandboxWorkloads and vfioManager are both off and the host
driver stays bound. Both KubeVirt-side pieces the bundle wires are
VM-facing — the HostDevices feature gate advertises a capability such
a node cannot serve, and permittedHostDevices lists resources no
sandbox device plugin registers there. Both are now emitted only for
default and vgpu, and container yields the gpu-operator Package with
the kubevirt component untouched.

Comments and docs that described the KubeVirt mirroring as
unconditional on variant, or the container variant as unmerged and
unavailable, are corrected to match.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-07 04:58:02 +03:00
Aleksei Sviridkin cd6a9b3a8f fix(platform): correct what disabledPackages does to ouroboros 
Both the publishing.proxyProtocol comment in values.yaml and the
disable-hazard guard comment in the system bundle told operators that
adding a package name to bundles.disabledPackages triggers the vendored
ouroboros chart's pre-delete Corefile cleanup hook, alongside an
explicit `kubectl delete package.cozystack.io cozystack.ouroboros`.

It does not. A name in disabledPackages only makes the rendering
helper in _helpers.tpl skip emitting the Package document, which is
the same suppression that leaving proxyProtocol false already
performs. Every emitted Package carries helm.sh/resource-policy: keep,
so Helm does not prune the Package it has stopped rendering — that
annotation was added precisely to stop disabling a package from
deleting it. The Package CR survives, the HelmRelease it owns via
ownerReference survives with it, ouroboros keeps running and keeps
re-applying its rewrite block to kube-system/coredns, and the
pre-delete hook never runs because nothing uninstalls the release.

No platform values setting reaches that hook; the operator has to
delete the Package CR. Anyone who followed the old advice was left
with ouroboros still installed and, on the proxyProtocol false
transition, still facing the render guard with no idea why.

Pins the behaviour with a case asserting the guard still fails while
cozystack.ouroboros sits in disabledPackages and the Package CR is
live.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-07 01:15:06 +03:00
Myasnikov Daniil 00c6eb18bf docs(e2e): name the container lane's one-per-host limit 
SANDBOX_NAME is per-checkout and these are not, which reads like an
oversight until you notice that hack/e2e-compose.yaml's container names
and the data-srvN zpools are global too. Two checkouts on one host
therefore share nodes and pools, and the second one's teardown destroys
the first one's storage.

Recorded rather than fixed. CI cannot reach it -- the lane already asks
for 24 vCPU and 72 GiB against a 32 vCPU / 128 GiB runner, so a second
one does not fit beside it -- and isolating it properly means renaming
the pools, which the LINSTOR storage-pool registration reads.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
 2026-09-05 14:51:06 +05:00
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers 
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-09-05 13:19:17 +05:00
Aleksei Sviridkin 6935df8c7f chore(kubevirt): update KubeVirt to v1.9.0 (#4015) 
## What this PR does

KubeVirt operator manifest goes from v1.8.4 to v1.9.0. The 1.9 release
turns every Beta feature gate on by default and adds
`disabledFeatureGates` to the CR for opting out, so the CR side needed a
few decisions.

`Template` is disabled. With it on, virt-operator deploys virt-template:
two Deployments, two APIServices, a webhook and a
ValidatingAdmissionPolicy that nothing here uses. One more aggregated
API server is one more way to break discovery. This one is not a knob.
`- Template` is written into the CR unconditionally and there is no
value that takes it back out, because nothing on the platform uses
VirtualMachineTemplate; if something starts to, the static entry becomes
a value then. Asking for it through `extraFeatureGates` is refused at
render time with an error that says so, rather than producing a CR
KubeVirt would reject.

`ImageVolume` is the host-dependent one: off below Kubernetes 1.35, left
on from 1.35, with a platform value for runtimes that cannot serve image
volumes even there (#3190 has the k3s case). Nothing the platform ships
uses containerDisk. The value is `kubevirt.disabledFeatureGates` on the
platform values, which the iaas bundle forwards into the emitted
`cozystack.kubevirt` Package as
`components.kubevirt.values.disabledFeatureGates`:

Closes #3190.

```yaml
kubevirt:
  disabledFeatureGates:
  - ImageVolume
```

The chart refuses to render a CR that names one gate in both lists.
KubeVirt's `kubevirt-update-validator.kubevirt.io` webhook rejects that
combination and runs with `failurePolicy: Fail`, so on a cluster that
already has the CR the write is refused and the `kubevirt` HelmRelease
stops converging with everything else the CR carries. The create
admitter runs no validation at all (its own comment says the update
validator is where that happens, and it only gets installed after the CR
exists), and `IsEnabled` in virt-config checks `FeatureGates` before
`DisabledFeatureGates`, so on a fresh install the same CR is admitted
and the gate silently stays on. Render-time is the only place left where
an operator gets told.

A name repeated inside one list is not an error: upstream compares the
two lists against each other, and both fields are
`x-kubernetes-list-type: atomic` in the CRD, so that CR is accepted and
the guard stays quiet. The gates the chart pins off can arrive a second
time from a supported values file, since the chart writes them itself
and an operator listing the gates their cluster declines has no reason
to know that, so the range skips them and the CR states each opt-out
once. The guard walks its own copy of the seven gates the chart enables.
That copy exists because the rendered list has to stay a literal list
for `make update` to find its `- VMExport` anchor, and a gate quietly
dropped from the copy would keep rendering under `featureGates` while
leaving the guard. The suite carries one case per gate, so removing any
single name turns exactly one of the seven red.

`HotplugVolumes` stays in the list even though 1.9 deprecates it. A
deprecated gate listed in the CR is still enabled, and while it is
enabled virt-controller skips the declarative hotplug path. Dropping it
would switch on declarative hotplug, but then virt-api rejects the
VMI-level `removevolume` subresource on VM-owned VMIs, and kubevirt-csi
uses that as a fallback during teardown. The CSI driver has to move off
that subresource first. Upstream names no removal release for it: the
`Deprecated` state's own doc comment says "discontinued in the following
release", but `MultiArchitecture` and `DisableMDEVConfiguration` have
been deprecated since 1.8.0 and are still registered at 1.9.0, so that
is a direction rather than a date.

What makes that opt-out list matter is a semantics change: on 1.8
`isFeatureGateEnabled` ended in `return false`, so a Beta gate nobody
listed was off; on 1.9 `IsEnabled` ends in `return fg.State == Beta`.
Sixteen gates are Beta at v1.9.0 and none of them is in this chart's
`featureGates`, so all sixteen arrive on with the bump. Two are switched
back off here, one is switched off below Kubernetes 1.35, and I went
through the rest:

- `Template`: off. Reason above.
- `ExternalNetResourceInjection`: off. With it on, virt-controller stops
calling `multus.NetworkToResource`, so a NAD's
`k8s.v1.cni.cncf.io/resourceName` no longer becomes a resource request
on the launcher pod. That is correct for clusters running a
network-resources-injector; this platform ships none, so a VM on an
SR-IOV NAD would schedule without its device.
- `ImageVolume`: off below Kubernetes 1.35, left alone from 1.35 on.
KubeVirt renders containerDisk images as `VolumeSource.Image` volumes
when the gate is on, and it does not check the host: `dropImageVolumes`
in the apiserver strips every image volume and the mounts referencing it
while Kubernetes' own `ImageVolume` gate is off, which is the default
until 1.35 (`kube_features.go` has it `Default: false` at 1.31 and
1.33). The pod is admitted without the disk and virt-launcher fails
looking for it. The management cluster is supported from 1.33, so on
1.33 and 1.34 the chart keeps the KubeVirt gate off and the
containerDisk sidecar path in use; from 1.35 the gate stands and
`extraFeatureGates` can name it. The apiserver version is a proxy for a
runtime capability, not the capability itself, so a cluster at 1.35 or
newer whose container runtime cannot serve image volumes (the k3s case
in #3190) switches the gate off through `kubevirt.disabledFeatureGates`;
that is what the value is for.
- `Snapshot`: on. `VirtualMachineSnapshot` needed the gate listed on
1.8; snapshot and export were exercised on the test cluster.
- `DeclarativeHotplugVolumes`: on and inert.
`handleDeclarativeVolumeHotplug` returns early while `HotplugVolumes` is
enabled, which this chart still does.
- `GPUsWithDRA`, `HostDevicesWithDRA`: on; each only lets a VMI request
a DRA-provisioned device, and nothing here creates one.
- `WorkloadEncryptionSEV`: on; needs SEV hardware and a VM that asks for
it.
- `PasstBinding`: on; needs a VM using the passt binding.
- `RebootPolicy`: on; adds a field on the VMI domain spec, unset means
the old behaviour.
- `VmiMemoryOverheadReport`: on; fills `status.memory.memoryOverhead`,
reporting only.
- `OptOutRoleAggregation`: on; it enables the `RoleAggregationStrategy`
field, and aggregation stops only for a CR that sets it to `Manual`.
This one does not.
- `KubevirtSeccompProfile`: on; installs the profile, and a workload
uses it only where `seccompConfiguration` says so.
- `NodeRestriction`: on; virt-handler node authorization, needs the
Kubernetes `ServiceAccountTokenPodNodeInfo` gate, GA since 1.32.
- `LibvirtHooksServerAndClient`: on; pre-migration hooks on the target
launcher, and no hooks ship here.
- `PodSecondaryInterfaceNamingUpgrade`: flips on; it adds
`--upgrade-ordinal-ifaces` to virt-launcher and a pre-migration hook
that renames the ordinal secondary interfaces of VMs still carrying
them. The test cluster's migrations exercised that path.

Five of the seven gates the chart enables (`ExpandDisks`,
`LiveMigration`, `AutoResourceLimitsGate`, `GPU`, `VMExport`) reached GA
by 1.9 and are inert: `IsEnabled` returns true for a GA gate whether or
not the CR names it. I left the list as it is; dropping the five is a
follow-up, because it changes what the CR says without changing what the
cluster does and it belongs in its own commit.

QEMU is 10.1.0 on both sides of this jump; the el9 build revision moves
from -10 to -20 and libvirt goes 11.9 to 11.10. A moving build revision
is the thing to check rather than a major:
[kubevirt/kubevirt#16386](https://github.com/kubevirt/kubevirt/issues/16386),
where every existing VM lost live migration, was 1.6.2 to 1.7.0 on
qemu-kvm 9.1.0-19 to 9.1.0-20, one build apart within the same QEMU
version. So I tested it: VMs running on the 1.8.4 launcher live-migrated
onto the 1.9.0 one on a three-node cluster.

`make update` in `kubevirt-operator` used to `rm -rf templates` before
the fetch and took the local PrometheusRule template with it. Now it
only overwrites the file it downloads.

`make update` in `kubevirt` now greps for every line of the new block
before it passes: the range directive, the `disabledFeatureGates:` key,
the three static entries, and the guard header that decides which static
entries render. Every insert in that target is guarded by one directive
it also inserts, so a hand-merge that keeps the directive and drops a
sibling line leaves the guard satisfied and the insert skipped. That
shape covers the directive and nothing else. What makes it worth a grep
per line here is the static entries: they are the only lines those
inserts add that are CR content in their own right rather than a wrapper
around a value, so they render on every install and their absence cannot
surface as a value that stopped taking effect. The CR just goes back to
deploying virt-template and to letting KubeVirt skip NAD resource
injection. The target cannot rebuild a partial block, so it names the
missing line and stops. The payload lines of the other four blocks are
exposed the same way; I checked, and both dropping `- {{ . }}` from the
`extraFeatureGates` range and dropping the `permittedHostDevices:` key
from its `with` render green and lose the operator's values. That is a
wider change than this PR and I left it out.

Two sharp edges I know about and left. `kubevirt.disabledFeatureGates:
[HostDevices]` on a cluster that also has `cozystack.gpu-operator` in
`enabledPackages` makes the bundle forward the same gate into both
lists, and the platform renders fine while the child kubevirt
HelmRelease then stops on the guard: fail-closed, with the gate named,
but diagnosed one level below where it was configured. And the
version-conditional opt-out is evaluated at render time, so a cluster
upgraded from 1.34 to 1.35 keeps `ImageVolume` disabled until something
re-renders the chart; the direction is the safe one, since the gate
stays off and containerDisk keeps the path it had on 1.8.4.

Two things I decided against doing here. The guard's copy of the gate
list is pinned against a name being dropped, not against a name being
added to the rendered list only; the copy exists because `make update`
needs the literal `- VMExport` anchor, so closing the other direction
means changing the self-heal contract. And
`hack/run-kubernetes-block-io_test.bats` still names a
`virt-launcher:v1.8.4` image in a fixture: inert, the test matches on
the container name, and the file is outside this change.

Tested on a three-node Talos v1.13.6 / Kubernetes v1.36.2 cluster with
Flux suspended. The upgrade took about ten minutes. virt-operator had
already deployed virt-template before the CR change landed and removed
it afterwards, CRDs included. Two VMs started on the 1.8.4 launcher, one
DataVolume-backed and one containerDisk-backed, live-migrated to the
1.9.0 launcher. VMDisk plus VMInstance through the apps API boots.
`addvolume` on the VM subresource reaches Ready, `removevolume` on the
VMI subresource is accepted. VirtualMachineSnapshot and
VirtualMachineExport from it reach Ready. Restore from snapshot and PVC
clone failed on that cluster inside the LINSTOR API layer, unrelated to
this change. CDI v1.66.0 from #3920 on top of it: HTTP imports to Block
and Filesystem, blank population and hotplug all work. `helm unittest`
is 26/26 for the kubevirt chart and 142/142 for the platform chart, and
the update idempotency script passes its eleven cases.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. No app added or renamed, no
variant, release asset, `ApplicationDefinition` or tooling contract
touched, and none of the `packages/core/platform/values.yaml` keys the
ansible role sets are in the diff.

One repository is reached. `packages/core/platform/values.yaml` gains a
top-level key, `kubevirt.disabledFeatureGates`, defaulting to the empty
list, and `operations/configuration/platform-package.md` on the website
is a hand-written table of `spec.components.platform.values.*` that
nothing generates. The row is in cozystack/website#678, which lands
after this PR merges.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/678
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(kubevirt): update KubeVirt to v1.9.0. Beta feature gates are on by default upstream now. The platform keeps `Template` off, which would deploy virt-template, and `ExternalNetResourceInjection`, which would hand `resourceName` injection for NetworkAttachmentDefinitions to an injector the platform does not run. `ImageVolume` is disabled on management clusters below Kubernetes 1.35: containerDisk-backed VMs are rendered with native image volumes when it is on, and an apiserver whose own ImageVolume gate is off, the default before 1.35, drops those volumes silently, leaving the VM without its disk. On 1.35 and newer the gate stands, and that is the first release where containerDisk-backed VMs are rendered with image volumes instead of sidecars; the bundled Talos ships Kubernetes 1.36, so this is the default there. A `kubevirt.disabledFeatureGates` list in the platform values opts out of any other gate a cluster cannot carry.
```
 2026-09-05 00:30:05 +04:00
Alexey Artamonov daf733ad9c fix(keda): narrow autoscaler RBAC and scope the operator-down alert 
Restrict the KEDA operator ClusterRole: set permissions.operator.restrict.secret
so it drops cluster-wide secret list/watch, and pin rbac.scaledRefKinds to
postgresql.cnpg.io/clusters so it no longer holds a wildcard */scale across every
group.

Gate DatabaseAutoscalerKedaOperatorDown on the presence of keda-hpa-* HPA series
so a tenant running its own Monitoring app (which never scrapes keda_build_info,
root-only) does not fire it forever about a healthy platform component. Make the
two negative containsDocument matchers in the bundles-default test non-vacuous
(any: true) so "KEDA is not installed by default" is actually asserted. Drop the
unused pollingInterval/cooldownPeriod params from the cozy-lib keda helper.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-04 23:19:25 +03:00
Alexey Artamonov 6e3da63586 fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges 
Address the remaining IvanHunters review points on #3954:

- [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum.
  An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from
  effectiveMin while replicas stays at its default, so flipping autoscaling off
  rendered instances: <replicas> next to the quorum and CNPG's webhook rejected
  it ("maxSyncReplicas must be lower than the number of instances"). Add a
  symmetric render-time guard on the static-count path (covers off/dryRun/
  transition): fail with an actionable message when replicas <= maxSyncReplicas.

- [MINOR] Role-scope the freeze brake's currentInstances count
  (label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods
  (cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge
  desired up while the brake is meant to hold it.

- [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the
  non-braking path is validated live, so ship the validated default and let the
  freeze be enabled explicitly until its branch is exercised under load.

- [MINOR] Document the floor-change and non-quorum-disable dips (same three-way-
  merge hazard) and the pin-replicas-first mitigation.

- [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the
  string "false" does not pause the ScaledObject.

- [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8
  verbatim lines into the cozystack-values Secret on every cluster.

Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge
guard, the role-scoped freeze count, and the string-"false" paused coercion.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-04 23:19:25 +03:00
Alexey Artamonov 42a533fc50 fix(platform): inject networking.clusterDomain into the KEDA package 
KEDA's clusterDomain is a vendored-subchart value baked into the metrics-apiserver
gRPC target (keda-operator.<ns>.svc.<clusterDomain>:9666) and the operator's
serving-cert SANs at render time. A chart template cannot derive a subchart value
from _cluster, so the keda package hardcoded cozy.local — which silently breaks
(NXDOMAIN, every DB HPA FailedGetExternalMetric) on any cluster whose domain is
not the default. Inject the authoritative networking.clusterDomain into the shared
cozystack-values Secret under the keda subchart key so the value tracks the
platform config; the keda package keeps cozy.local as a standalone fallback. The
key is ignored by packages without a keda subchart (no app/extra schema sets root
additionalProperties:false). Add platform unittests asserting the Secret carries
keda.clusterDomain and that it tracks a non-default networking.clusterDomain.

Addresses the hardcoded-clusterDomain portability gap found in branch review.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-04 23:18:47 +03:00
Alexey Artamonov 7f8f2c1d86 feat(keda): add KEDA as an optional platform component 
Vendor the upstream KEDA chart (kedacore/keda 2.20.2) as a new optional
system package and wire it into the platform, so database read-replica
autoscaling (which renders KEDA ScaledObjects) has a backend.

- packages/system/keda: umbrella cozy-keda + vendored chart (Makefile
  pins KEDA_CHART_VERSION with a lockstep assert), shim values enabling
  the operator / metrics-server / webhook ServiceMonitors under the
  platform Prometheus.
- core/platform: PackageSource cozystack.keda (namespace cozy-keda,
  dependsOn networking + prometheus-operator-crds; cert-manager is not
  needed, KEDA self-generates its webhook TLS) plus an optional bundle
  entry, opt-in via bundles.enabledPackages. It is not a dependsOn of any
  default package, so the platform HelmRelease never blocks on it.
- KEDA claims the cluster-wide external.metrics.k8s.io APIService
  (distinct from metrics-server's metrics.k8s.io — no conflict).

Covered by helm unittest: platform bundle enable/disable/opt-out, the
PackageSource dependsOn, and the vendored chart's external-metrics
APIService + shim-enabled ServiceMonitor.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-04 23:18:47 +03:00
Aleksei Sviridkin 3a369fed99 feat(redis): add opt-in TLS for Redis and Sentinel (#2729) 
## What this PR does

`tls.enabled` puts Redis and Sentinel on a TLS-only listener with a
per-release cert-manager chain, and hands the tenant the CA certificate
as `redis-<name>.tenant-ca`. Off unless set.

redis-operator v3.3.5 has no TLS support. The work is ported upstream in
freshworks-oss/redis-operator#85, which is open and not in a release, so
this carries it as three patches over the v3.3.5 tarball.
`images/redis-operator/patches/tls.diff` is the source half, cut from
that PR's commit `2ad2ed8d1`, which the patch header pins; it is applied
by the glob the image Dockerfile already runs, after the Dockerfile
strips the tarball's `manifests`, `charts`, `example` and `.github`, so
the image builds from source alone and the upstream tests that read the
CRD YAML skip instead of failing against the stale v3.3.5 copy.
`patches/crd-tls.diff` adds `spec.tls` to the vendored CRD, and with it
the CEL rule that fixes `tls.enabled` at creation; without the schema
block the apiserver prunes the field on admission and the patched binary
waits for something it never sees, and the rule cannot exist without the
block it references. `patches/rbac-tls.diff` grants Create, Get and
Update on Secrets, so the operator can publish the CA-only Secret it
addresses by name. That is the whole delta, and it is narrower than the
ported operator's own chart asks for: those three are the only Secret
calls in the patched source, and cert-manager Certificates get no grant
at all, because the operator only touches one under
`spec.tls.certManager` and this chart pins `certificateSecret` mode
instead. One change in `tls.diff` is not TLS and takes effect with TLS
off, named in the patch header: `MakeSlaveOfWithPort` connects to the
replica on its own port rather than on the master's. It changes a dial,
not a pod template, so shipping the rebuilt image rolls nothing; the
plaintext pod templates stay byte-identical to v3.3.5.

The image build registration this branch used to carry (`make -C
packages/system/redis-operator image` in the root `build:` target)
landed on main with the Valkey service (#3380), so the rebuilt operator
image is produced by CI from main as well; the digest in `values.yaml`
is still refreshed by release preparation, not here.

The chart patches re-apply at the end of `update:`, the same way
reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry
theirs. Otherwise `rm -rf charts` takes TLS back out on the next `make
update` and nothing fails. `make update` reproduces the committed tree
byte for byte.

TLS is chosen when the instance is created. The config gets `port 0` and
the listener moves to `tls-port`, so TLS replaces plaintext rather than
running beside it, and a running instance cannot change mode: a replica
restarted onto the TLS listener cannot replicate from a master still on
plaintext, and Sentinels already on TLS would fail over to a stale
replica. The RedisFailover schema in the carried CRD patch rejects a
change of `tls.enabled`, which on an existing instance surfaces as a
failed HelmRelease upgrade while the instance keeps serving what it did;
the README points at creating a new instance and migrating. That is also
why TLS is not inferred from `external`: an externally reachable
instance is the one most likely to have clients an upgrade must not
drop. Existing releases keep their plaintext listener and are not
restarted by the rebuilt operator image.

The tenant gets `ca.crt` and nothing else, through `tenantsecrets`. The
chart's own Secrets take a dot between the release and the suffix
(`<release>.ca-tls`, `<release>.tls`, `<release>.ca-cert`): the suffixes
are dot-free, so the split is unique and release `foo`'s CA Secret can
no longer be release `foo-ca`'s leaf, the collision the hyphenated form
had. The chart renders a `TenantProjection` pointing at
`<release>.ca-tls`, and the CA-extraction controller re-encodes the one
named key from the parsed certificate, so nothing that is not a
certificate gets through. Same path as postgres, mongodb, nats and
qdrant.

No Secret of this release is granted to the tenant by name beyond the
generated password, and that includes `<release>.ca-cert`, the CA-only
copy the operator publishes for itself. It holds no key today, but the
operator has Create and Update on Secrets cluster-wide, so what sits at
that name is the operator's call rather than the platform's, and RBAC
cannot filter by key. `hack/check-redis-rd-secrets.bats` pins the whole
rendered Secret surface instead of a list of known-bad names, because an
include with no `resourceNames` and no `matchLabels` restricts nothing.

The chainsaw suite for redis gains two cases. A TLS fixture asserts the
cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel
come up, the operator publishes its CA-only Secret under the widened
ClusterRole, the tenant receives exactly `ca.crt` through
`tenantsecrets` while the key-bearing Secrets stay off that path, and a
`redis-cli --tls` PING through the service name completes with the same
CA. A second case applies `tls.enabled` to a running plaintext release
and asserts the refusal: the HelmRelease upgrade fails with the schema's
message, the StatefulSet keeps its revision with both replicas Ready,
plaintext still answers and no TLS listener appears.

Certificate renewal rolls the pods from inside the operator. Redis and
Sentinel read the certificate once at startup, so the patched operator
stamps a hash of the TLS Secret's `tls.crt` and `ca.crt` into the pod
template of both the Redis StatefulSet and the Sentinel Deployment on
every reconcile, as `redis-failover.freshworks.com/tls-secret-hash`. A
renewal changes the hash; the operator's own roller replaces the Redis
pods one at a time and the Deployment controller rolls Sentinel. README
names the fallback: if the roll cannot run, the deadline is the old
certificate's expiry, past which the operator's own TLS connections fail
along with the clients.

`tls.authClients` maps to `tls-auth-clients` and defaults to `no`. The
platform mints no client certificate for the tenant and the CA key is
out of reach, so `yes` is only usable when the client certificate comes
from whoever also controls the CA. Changing the value on a running
release restarts the Redis and Sentinel pods: Redis reads the directive
from its config file only at startup, so the operator folds the
effective value into the certificate hash that drives the rollout, and
the change lands when it is made rather than at the next unrelated
restart.

Follow-ups in the carried operator patch go upstream first, to
freshworks-oss/redis-operator#85 or a successor there, and are picked up
here afterwards; the carried patch does not grow Go that upstream has
not seen. `tlsConfigFor` fetches the Secret on every call, tens of GETs
per reconcile. The renewal roll is static analysis so far: the live
check, `cmctl renew` on a running release while watching the `rfr-*` and
`rfs-*` pods roll, is still owed and belongs with the e2e follow-up.

Two facts about the shipped image rather than the patch.
`packages/system/redis-operator/values.yaml` keeps the digest of the
image built before the patch until release preparation rebuilds it, so
on `main` between this merge and the next release-prep, `tls.enabled`
renders the chain and `spec.tls` while the running operator ignores them
and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026
in `golang.org/x/net` v0.53.0, reached through
`PodService.UpdatePodLabels`; the port neither introduces nor fixes it,
upstream `master` is already on v0.55.0, so it lands with the next tag
bump rather than here.

Outside the patch: `cozyvalues-gen` renders `{}` in the README Value
column for `tls.authClients`: a child property without a default
inherits the parent object's `{}`, and `make generate` reproduces it
byte for byte, so it is a generator bug rather than something to
hand-edit here.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#37
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

I walked the trigger map against the diff file by file.
terraform-provider-cozystack is reached, tracked as
cozystack/terraform-provider-cozystack#37 rather than a speculative PR:
`values.schema.json` gains a `tls` object, and the provider hand-writes
a schema, model and expand/flatten pair per Kind. It already models
`tls` for postgres, kafka, nats and qdrant, not for redis. Redis needs
different wording from those four, whose descriptions say to omit the
block to follow `external`, which is the one thing redis does not do.

Nothing else is reached. The website regenerates this package's
reference page from `README.md` on a stable tag, so the README changes
need no PR there. No package added, renamed or removed.
`packages/core/platform/values.yaml`, variant and bundle composition and
release asset names are untouched, and so are `hack/package.mk`,
`hack/common-envs.mk` and `hack/update-crd.sh`, which is what ccp and
external-apps-example read. The two new files under `hack/` are bats
unit tests. No node prerequisite, installer value, telemetry metric or
cozy-proxy label changes.

### Release note

```release-note
feat(redis): add opt-in TLS for Redis and Sentinel. Set `tls.enabled: true` to serve both on a TLS-only listener backed by a per-release cert-manager chain; the tenant reads the CA certificate as the `<release>.tenant-ca` tenant secret. TLS is off unless set and is not inferred from `external`, so existing releases keep serving plaintext and are not restarted by the rebuilt operator image. TLS is chosen at creation: a running instance cannot switch modes, the change is rejected and surfaces as a failed HelmRelease upgrade, so add TLS by creating a new instance and migrating to it. Requires cert-manager, which the platform's `system` bundle installs on every variant. A renewed certificate rolls the Redis and Sentinel pods through the operator. TLS support is carried as a downstream patch over redis-operator v3.3.5 until freshworks-oss/redis-operator#85 lands in a release.
```
 2026-09-04 19:32:32 +04:00
Aleksei Sviridkin 8adf3a0863 feat(platform): forward kubevirt.disabledFeatureGates to the kubevirt CR 
The kubevirt chart reads disabledFeatureGates from its own values, and
the iaas bundle is the only writer of the cozystack.kubevirt Package, so
nothing could set it: a Package patched by hand is reverted the next
time the platform re-renders it.

Platform values gain a documented kubevirt.disabledFeatureGates list,
forwarded into the emitted Package as
components.kubevirt.values.disabledFeatureGates alongside the entries
derived from resources.cpuAllocationRatio and the gpu block. The
parenthesized value walk is nil-safe, matching the gpu block, so a
cluster that nulls out the whole kubevirt key still renders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-04 15:29:09 +03:00
Aleksei Sviridkin 5976ddf02f fix(gateway): normalise the GatewayClass name and refuse a blank one 
The class lists arrive over the values channel comma-joined, so every
reader trims each entry as it splits them. The scalar those entries are
compared against was not trimmed anywhere. A class name carrying a
stray space therefore failed `has` against its own padded self, and the
two readers of that scalar disagreed about the same class: the gateway
chart put the tenant on an edge-terminated class, while the
cluster-wide ACME issuers still pinned their solver to a listener an
edge Gateway does not render. That stops every HTTP-01 certificate in
the cluster from issuing, with nothing in any status naming the cause.

Normalise once at the single writer so both readers see a clean value,
and keep a defensive trim at each reader. The tenant's own choice does
not travel that channel — it comes from tenant.spec.gatewayClass — so
it is trimmed where the gateway chart reads it.

Unset and whitespace-only are different events and no longer share an
outcome. Unset means the choice was not made, and falling back to the
bundled class stays its documented behaviour. Whitespace-only means a
choice was made and is garbage, in the field that decides where TLS
terminates: resolving it quietly would take a tenant the operator meant
to put behind an edge terminator and publish it directly, with a real
certificate on a real address. Both the platform default and the
tenant's own choice now refuse it by name.

The kind check beside that refusal is part of the same fix, not a tidy
up. Through Helm's values path an unquoted numeric scalar arrives as a
float64, and the whitespace check compares it against a string, so
`className: 123` dies at that comparison — one line before the trim —
with an error naming neither the key nor the file. Refusing whitespace
without checking the kind first would therefore have made the numeric
case worse than it was. `kindIs "string"` is the right test because it
never has to name the type that actually arrived.

The order of the guard's steps is part of the guard. The value is read
before anything normalises it, so a default that fires on any falsy
value cannot launder 0 or false into the empty string and resolve them
to the bundled class ahead of the kind check. Unset is the one case
that still falls back, and each of those four inputs is pinned, so a
normalisation drifting back ahead of a check fails rather than quietly
covering only the loudest bad input.

Also correct a cleanup log line: the wildcard Certificate is deleted
for every mode that is not DNS-01, so naming HTTP-01 in that message is
wrong for existingSecret and for edge alike. And state the listener
name invariant plainly: the cap is 253, not the 63 of a DNS label, and
the longest name the builders produce is 84.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-09-04 14:33:35 +03:00
Aleksei Sviridkin 487cafac3e feat(platform): let a tenant pick its Gateway class from an allowlist 
Tenant Gateways were pinned to the bundled Cilium class. Give the
platform a default in gateway.className, the set a tenant may choose
from in gateway.tenantSelectableClasses, and the set whose provider
terminates TLS upstream of the Gateway in
gateway.edgeTerminatedClasses. A tenant picks its own with
tenant.spec.gatewayClass, read only when it owns a Gateway; an
inheriting tenant runs on its ancestor's.

The selectable set is an allowlist rather than free choice because
Tenant is a tenant-writable object while a GatewayClass is
cluster-scoped: without it a tenant could attach its Gateway to a class
an admin keeps for internal-only traffic. Naming anything outside the
set fails that tenant's own gateway release and reaches no other
tenant.

Where TLS ends is a property of the class, so the gateway chart derives
certMode from whichever class the tenant landed on instead of taking it
as a separate input. The two therefore cannot be configured out of
step.

The publishing tenant cannot cross that line by itself. The
cluster-wide ACME issuers point their HTTP-01 solver at its Gateway and
are rendered from the platform default alone, so a mismatch would pin
the solver to a listener that Gateway does not have. Those issuers now
drop the sectionName pin when the default class is edge-terminated,
letting the challenge attach by hostname instead.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-09-04 14:26:54 +03:00
Andrey Kolkov e94fcf8219 feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990) 
## What this PR does

Adds backup/restore support for the managed `RabbitMQ` application.
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so this backs up broker
**definitions** (vhosts, users, permissions, queues, exchanges,
bindings, policies, parameters) over the management HTTP API
(`GET`/`POST /api/definitions`) and stores them in the system bucket.
Message payloads and point-in-time recovery are intentionally out of
scope; a full-data backup would be a Velero volume snapshot, which is a
separate strategy.

It is delivered as a **first-class
`strategy.backups.cozystack.io/Rabbitmq` driver**, following the
Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

- `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's
`PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`,
pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a
`batch/v1.Job`, covering both restore flows: in-place (into the source
broker) and to-copy (into a separate, differently-named RabbitMQ in the
same namespace, authenticating with the target's own default-user
credentials and reading the source's object key).
- A **Go-read precondition** gates each run on the `RabbitmqCluster`
reporting `AllReplicasReady` (read through a small `rabbitmqtypes`
partial type), so a not-yet-ready broker surfaces a precise
`Ready=False` wait on the BackupJob/RestoreJob rather than a Job that
starts and fails against an unreachable management API.
- The produced `Backup` records the stored object's location on
`status.artifact.uri`, rendered from the strategy's
`spec.artifactURITemplate`. The `Backup` CRD has no status subresource,
so this persists on Create, mirroring the Etcd driver.

Dispatch is wired in both controllers (plus
`supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the
scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job
itself is already covered by the existing `batch/jobs` grant). It ships
`cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and
routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default`
BackupClass, which also enrols it in the `DefaultObjectsGate`. The
RabbitMQ application chart is left untouched — it had no prior backup
wiring, and the strategy carries the S3 coordinates itself.

Unit tests cover the driver's pure helpers (applicationRef validation,
cluster-name derivation, the readiness-message builder, artifact-URI
rendering, parameter round-trip) and the reconcile precondition gate
(defers with `Ready=False` and no Job until the cluster is ready, then
proceeds). A runnable example under `examples/backups/rabbitmq/` and an
honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a
sentinel vhost, back it up, drop it, and assert it round-trips through
object storage on both restore flows.

The Job pod reuses the existing `platform-migrations` image (curl +
ca-certificates) under a new
`backupStrategyController.rabbitmqBackupClientImage` key, stamped in
lockstep with `chBackupClientImage` in `packages/core/platform/Makefile`
so the repository stays pinned at a single digest
(`hack/image-pin-consistency.bats`). A dedicated slim image built by
this package is a follow-up.

**Validation:** `go build`/`go vet`/`gofmt`/`go test
./internal/backupcontroller/... ./internal/template/...`, deepcopy +
Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate
both on and off, `helm lint`, and `helm unittest` (18/18). The backup +
in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a
real management-API export and a real S3 `--aws-sigv4` upload/download
against SeaweedFS), so the driver's core contract is exercised on a real
cluster; the in-CI Chainsaw run is the remaining automated evidence. The
object-cleanup, terminating-namespace teardown and readiness-deadline
paths added during review are unit-tested (including `Reconcile`-level
finalizer retention and the terminating-namespace release) but not yet
cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled
on the `nano` preset, so the example/e2e use `c1.small`.)

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. This is an additive backup
strategy plus a new strategy CRD; no downstream repository is
mechanically forced by the change. A documentation page for RabbitMQ
backup on [cozystack/website](https://github.com/cozystack/website)
would be a reasonable follow-up, but it is not required by this diff —
leaving the boxes unticked for a human to decide rather than ticking "no
downstream" to clear the checklist.

### Release note

```release-note
feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added RabbitMQ backup and restore support for definitions, including
vhosts, users, queues, exchanges, bindings, policies, and related
settings.
  * Added in-place and restore-to-copy workflows with readiness checks.
  * Added scheduled and on-demand backup examples.
  * Integrated RabbitMQ backups with the default backup configuration.
* Backups store definitions only; message payloads and point-in-time
recovery are not included.
* **Documentation**
  * Added a complete RabbitMQ backup and restore demonstration guide.
* **Tests**
* Added end-to-end coverage for readiness and backup/restore round
trips.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-03 16:03:46 +04:00
Aleksei Sviridkin f5206a6e28 docs(redis): declare the TLS-on-upgrade break and fix the parameter sections 
TLS replaces plaintext in this operator rather than running beside it —
with TLS on, Redis and Sentinel are configured with port 0 and serve only
on the TLS port. Since tls.enabled falls back to external, an already
deployed instance with external: true switches to TLS-only on its first
reconcile after the upgrade and every plaintext client breaks. That was
documented as a feature and nowhere as a change of behaviour for existing
instances. State it as a breaking change and name tls.enabled: false as
the opt-out.

Also spell out which hostname the certificate actually covers. The SAN
list carries the four service names and the external tenant hostname, and
its only IP entries are the loopback addresses used by the in-pod probes
and the metrics sidecar, so connecting to the external LoadBalancer by IP
fails for clients that verify the hostname.

The TLS section marker sat above the version parameter, which put version
in the TLS table in the generated README instead of Common parameters.
Move the marker below it. Regenerating also reorders version ahead of tls
in keysOrder, matching how postgres orders the same two fields.

Pin the PackageSource CRD replacement policy the way the mongodb-operator
one is pinned. The RedisFailover CRD gained spec.tls with the fork, and
Helm never upgrades CRDs shipped in crds/, so losing CreateReplace would
leave the live CRD behind and the apiserver would strip spec.tls from
every RedisFailover the chart renders — TLS silently off, no error.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-02 17:56:07 +03:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver 
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-01 18:21:10 +04:00
Andrei Kvapil 53412c3033 fix(valkey): add cozystack-engine dependsOn edge to the PackageSource 
The valkey-application PackageSource ships system/valkey-rd (an
ApplicationDefinition); it must dependOn cozystack.cozystack-engine so the
ApplicationDefinition CRD is registered before the -rd HelmRelease reconciles.
Mirrors redis-application.yaml and satisfies check-applicationdefinition-crd-ordering.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-09-01 11:44:29 +04:00
Andrei Kvapil 4e6313b608 feat(valkey): register Valkey application in the platform 
Wire the managed Valkey app into the platform: add the valkey-rd package
carrying the Valkey ApplicationDefinition, the valkey-application PackageSource
(reusing the existing redis-operator), enable it in the paas bundle, and grant
tenant roles access to the valkeys resource.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-09-01 11:43:10 +04:00
Timofei Larkin f5b957ee27 feat(kubernetes)!: manage worker pools as KubernetesNodes, adopt on upgrade (Phase 2, breaking) (#3315) 
## What this PR does

Phase 2 of the Kubernetes-app split
([cozystack/community#8](https://github.com/cozystack/community/pull/8))
— the **breaking** half. Stacked on #3314 (review that first).

Removes worker node pools from the `kubernetes` chart (they moved to
`kubernetes-nodes` in #3314) and adopts existing pools on upgrade:

- `kubernetes` chart is now **control-plane only**: drops the pool
`range` block, the `talos-reconcile` Job + RBAC (deleted; moved to
`kubernetes-nodes`), the preserve-old-KMT block, the
`kubernetes.nodeGroups` helper, and the per-group dashboard RBAC.
- **`spec.nodeGroups` is removed from the `Kubernetes` CR** (BREAKING).
Worker pools are now separate `KubernetesNodes` resources.
- **Migration 54** (bumps `targetVersion` to 55): pre-upgrade hook that,
for every `kubernetes-<cluster>` release, creates one
`kubernetes-nodes-<cluster>-<pool>` HelmRelease and re-annotates the
existing worker objects onto it with `helm.sh/resource-policy=keep`, so
the now control-plane-only parent upgrade does not prune running
workers. The default `md0` (implicit when `nodeGroups` was empty) is
materialised explicitly. Idempotent. Genuine infrastructure/read
failures fail closed (`exit 1`, retried on the next upgrade); hostile
stored shapes (a pool name over Helm's 53-char release-name cap, an
RFC-1123-invalid or whitespace/slash map key, a non-object `nodeGroups`)
are warned, pinned, and skipped rather than deadlocking the entire
fleet's upgrade.
- The `ingressNginx` addon no longer auto-provisions a node (the
implicit `md0` default is gone). If enabled without a pool carrying
`roles: [ingress-nginx]`, the controller DaemonSet simply schedules
nothing (degraded, not failed) — the requirement is documented in the
chart NOTES.txt rather than enforced at render time.

Because the child chart renders the pool objects **byte-identically** to
the old parent render (golden parity in #3314; the
KubevirtMachineTemplate content-hash name is preserved), adoption does
not churn live worker VMs.

### Migration number

`54` is contiguous with main (latest is `53`) and `targetVersion` is
`55`. The `52`/#3201 collision noted in earlier revisions has been
resolved: this branch was rebased and merged past it, so the numbering
is contiguous and `run-migrations.sh` has no gap to hard-fail on.

### Not yet done (blocking draft → ready)

- [ ] **dev e2e upgrade validation** (`cozystack-pr-test`): upgrade a
custom-`nodeGroups` cluster and a default-`md0` cluster; assert zero
worker-VM churn (KMT hash, Machine uid/creationTimestamp stable) and
that the adopted child HRs reconcile.
- [ ] downstream trigger-map walk (website docs for the `nodeGroups` →
`KubernetesNodes` change; terraform-provider for the removed API field).

### Downstream repositories

Deliberately unticked (draft) — the breaking API change affects docs and
the terraform provider; follow-ups to be filed before ready.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:

```release-note
Worker node pools of a managed Kubernetes cluster are now managed as separate `KubernetesNodes` resources. `spec.nodeGroups` on the `Kubernetes` CR is removed; existing pools are adopted automatically on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Worker pools are now separate `KubernetesNodes` resources with their
own lifecycle.
* Added a render-time compatibility guard to prevent worker pools from
running ahead of the parent control-plane minor version.
* Improved upgrade/uninstall behavior to better preserve already-adopted
worker objects.

* **Breaking Changes**
* Removed per-pool `nodeGroups` and worker node health settings from the
`Kubernetes` chart/CR; pool configuration now lives in
`KubernetesNodes`.
* Ingress requires at least one `KubernetesNodes` pool with `roles:
[ingress-nginx]`.
* `cluster` and `storageClass` are immutable after creation; pool
`version` may lag but must not be ahead of the parent minor.

* **Documentation**
* Updated READMEs, notes, and schemas to reflect the `KubernetesNodes`
split and the immutability/version rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-26 12:06:18 +04:00