admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-06 20:13:43 +09:00
Code Issues Packages Projects Releases Wiki Activity
6,188 Commits 570 Branches 294 Tags
feat/externalip-api
Commit Graph
799 Commits
Author SHA1 Message Date
myasnikovdaniil 94a202bc5a test(e2e): run the merge-gating lanes on Talos containers (#4020) 
Replaces QEMU substrate with Talos containers on both lanes that gate a
merge. srv1-srv3 run as containers, so tenant worker sits at L2 instead
of L3.

Measured in CI on commits where both substrates ran the same tree:

| | qemu | container |
|---|---|---|
| `kubernetes-latest` / `-previous` | 1143-2040s, red on every
vgif-absent runner | 466-809s, green in both strata |
| whole job | | 24-49 min faster |

Almost all of it is in Chainsaw. Install takes about half an hour either
way.

Node-join soft-red gate goes with it. It was a prosthetic for nested
virt: a worker that registers in minutes elsewhere could miss any
deadline this test can afford, and #3513 established the discriminator
is host `kvm_amd` `vgif`, not anything about the product. Remove the
nesting level and there is nothing left to tolerate, so
`hack/e2e-node-join-soft-red.sh`, the marker and the `soft_red` outputs
are gone from all four workflows.

### What moves off the per-PR path

DRBD, and with it `replicated` StorageClass, its `Immediate` binding
mode, tenant StorageClass propagation (applies only to
remotely-accessible classes), and cozystack talos node image with its
extensions. None of it is lost - `nightly.yaml` and `e2e-tag.yaml` still
run the suite on QEMU, so all four keep nightly and release-candidate
coverage. Live migration is not on that list because nothing tests it
today anyway.

One more belongs on the list and was missing from it: the nocloud disk
build. `build-talos` no longer runs `make -C packages/core/talos
talos-nocloud`, so `images/talos/profiles/nocloud.yaml` is compiled by
`nightly.yaml` and by the tag build's `make assets` rather than by
anything a PR runs, and `promote-rc.yaml` still hard-fails a promotion
whose rc release has no `nocloud-amd64.raw.xz`. Nothing in that chain
breaks, because the tag build supplies the asset as before. What a PR no
longer catches is narrow: it still compiles `installer.yaml` through the
same imager, and the two profiles are generated by the same script and
differ only in `platform`, `kind`, `imageOptions` and `outFormat`.

### Product fixes

Four, all found by measurement while building the lane, each on its own
commit:

The linstor `drbd.enabled` gate. Without it drbd-logger sidecar exits,
satellite is never Ready, piraeus registers zero nodes and every PVC
hangs Pending behind a cluster that looks healthy.

linstor `--strict-topology`. Without it external-provisioner passes
every topology segment as `requisite`, so a node-pinned volume can be
provisioned away from its pod and the pod is then unschedulable forever
with nothing erroring anywhere.

kubevirt-cdi importer resources are configurable now. Default restates
CDI's own values exactly so production behaviour does not move, and the
lane raises the ceiling through its own Package.

vm-disk binds standalone disks immediately. A VMDisk on `local` silently
never populated, while the same disk on `replicated` did.

### Where the run stands

The lane has gone green twice, most recently on `cc70cef7b` where `E2E
(in-tree)` took 2h3m against its 215m cap with 45 Chainsaw tests
passing. The measurements quoted elsewhere in this description come from
the first of those runs, on a commit that has since been rebased away:
the two tenant-Kubernetes suites came in at 569.72s
(`kubernetes-latest`) and 470.19s (`kubernetes-previous`) against a 67m
operation. Both runs answer the four earlier reds, all of which were the
lua-protobuf segfault from #4012 once `vminstance`/`vmdisk` were fixed
on the branch.

ghcr.io mirror removal is carried here too and is the same change as
#4007, whichever lands first makes the other a no-op.

### What review changed

`drbd.enabled=false` now refuses to render together with
`talos.enabled=false`. It removed the logger sidecar only, while the two
DRBD initContainers piraeus contributes unconditionally are deleted by
the Talos configuration alone, so that combination produced a satellite
that can never reach Ready. `isp-full-generic` sets
`talos.enabled=false`, so it was reachable and not theoretical.

vm-disk keeps unconditional immediate binding, a standalone disk has no
consumer to wait for. What it costs on a node-pinned class is in the
chart README now, and the annotation is pinned per source type and on
the existing-DataVolume path where `helm upgrade` retrofits it.

The rest is the harness holding its own promises: HelmRelease gate reads
are bounded and its minimum-count half is pinned by a test that fails
without it, two Chainsaw ops sit above the deadlines they contain, the
fallback-StorageClass render restores `replicated` when it fails,
container lane compose project and pool files are per-checkout, and
three comments that said the opposite of the code now say what is true
(QEMU is still wired into nightly and e2e-tag, the lane replaces two
Packages and not one, the argument-count assertion cannot see upstream
drift).

### Release note

```release-note
fix(linstor): CSI provisioning is constrained to the node the scheduler chose (`--strict-topology`), and the DRBD sidecar can be switched off with `drbd.enabled` on substrates whose kernel cannot run DRBD. On an existing cluster the new default re-templates `spec.csiController.podTemplate`, so piraeus rolls `linstor-csi-controller` once during the upgrade. `drbd.enabled=false` requires `talos.enabled=true` and the chart now refuses the other combination at render time, because only the Talos satellite configuration removes the DRBD initContainers.
fix(kubevirt-cdi): CDI worker pod resources are configurable through `importerResources`, which reaches the importer, the uploader and the host-assisted cloner. The default restates CDI's own built-in values exactly, so an install that sets nothing behaves as before.
fix(vm-disk): every disk requests immediate binding, not only `upload` sources, so a standalone disk on a WaitForFirstConsumer StorageClass is populated instead of waiting for a consumer that may never arrive. On a node-pinned class such as `local` the volume then binds where CDI's worker pod was scheduled rather than where the VM will run; see the chart README before choosing such a class for VM disks. The default `replicated` is unaffected.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added container-based end-to-end testing with local storage support.
* Added optional backup-access preflight checks for database backup
examples.
* Added configurable CDI worker resources and LINSTOR storage settings.
  * Added configurable immediate storage binding for VM disks.
* Added stronger readiness checks, diagnostics, and parallel test
execution.

* **Bug Fixes**
* Improved cleanup failure reporting, capacity validation, and
missing-image detection.
* Improved FoundationDB health verification and storage placement
behavior.

* **Removed**
  * Removed GHCR mirror support and soft-red node-join tolerance.
  * Removed standalone OIDC render test suites.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-07 19:33:20 +05:00
Myasnikov Daniil 00c6eb18bf docs(e2e): name the container lane's one-per-host limit 
SANDBOX_NAME is per-checkout and these are not, which reads like an
oversight until you notice that hack/e2e-compose.yaml's container names
and the data-srvN zpools are global too. Two checkouts on one host
therefore share nodes and pools, and the second one's teardown destroys
the first one's storage.

Recorded rather than fixed. CI cannot reach it -- the lane already asks
for 24 vCPU and 72 GiB against a 32 vCPU / 128 GiB runner, so a second
one does not fit beside it -- and isolating it properly means renaming
the pools, which the LINSTOR storage-pool registration reads.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
 2026-09-05 14:51:06 +05:00
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers 
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-09-05 13:19:17 +05:00
Aleksei Sviridkin 6935df8c7f chore(kubevirt): update KubeVirt to v1.9.0 (#4015) 
## What this PR does

KubeVirt operator manifest goes from v1.8.4 to v1.9.0. The 1.9 release
turns every Beta feature gate on by default and adds
`disabledFeatureGates` to the CR for opting out, so the CR side needed a
few decisions.

`Template` is disabled. With it on, virt-operator deploys virt-template:
two Deployments, two APIServices, a webhook and a
ValidatingAdmissionPolicy that nothing here uses. One more aggregated
API server is one more way to break discovery. This one is not a knob.
`- Template` is written into the CR unconditionally and there is no
value that takes it back out, because nothing on the platform uses
VirtualMachineTemplate; if something starts to, the static entry becomes
a value then. Asking for it through `extraFeatureGates` is refused at
render time with an error that says so, rather than producing a CR
KubeVirt would reject.

`ImageVolume` is the host-dependent one: off below Kubernetes 1.35, left
on from 1.35, with a platform value for runtimes that cannot serve image
volumes even there (#3190 has the k3s case). Nothing the platform ships
uses containerDisk. The value is `kubevirt.disabledFeatureGates` on the
platform values, which the iaas bundle forwards into the emitted
`cozystack.kubevirt` Package as
`components.kubevirt.values.disabledFeatureGates`:

Closes #3190.

```yaml
kubevirt:
  disabledFeatureGates:
  - ImageVolume
```

The chart refuses to render a CR that names one gate in both lists.
KubeVirt's `kubevirt-update-validator.kubevirt.io` webhook rejects that
combination and runs with `failurePolicy: Fail`, so on a cluster that
already has the CR the write is refused and the `kubevirt` HelmRelease
stops converging with everything else the CR carries. The create
admitter runs no validation at all (its own comment says the update
validator is where that happens, and it only gets installed after the CR
exists), and `IsEnabled` in virt-config checks `FeatureGates` before
`DisabledFeatureGates`, so on a fresh install the same CR is admitted
and the gate silently stays on. Render-time is the only place left where
an operator gets told.

A name repeated inside one list is not an error: upstream compares the
two lists against each other, and both fields are
`x-kubernetes-list-type: atomic` in the CRD, so that CR is accepted and
the guard stays quiet. The gates the chart pins off can arrive a second
time from a supported values file, since the chart writes them itself
and an operator listing the gates their cluster declines has no reason
to know that, so the range skips them and the CR states each opt-out
once. The guard walks its own copy of the seven gates the chart enables.
That copy exists because the rendered list has to stay a literal list
for `make update` to find its `- VMExport` anchor, and a gate quietly
dropped from the copy would keep rendering under `featureGates` while
leaving the guard. The suite carries one case per gate, so removing any
single name turns exactly one of the seven red.

`HotplugVolumes` stays in the list even though 1.9 deprecates it. A
deprecated gate listed in the CR is still enabled, and while it is
enabled virt-controller skips the declarative hotplug path. Dropping it
would switch on declarative hotplug, but then virt-api rejects the
VMI-level `removevolume` subresource on VM-owned VMIs, and kubevirt-csi
uses that as a fallback during teardown. The CSI driver has to move off
that subresource first. Upstream names no removal release for it: the
`Deprecated` state's own doc comment says "discontinued in the following
release", but `MultiArchitecture` and `DisableMDEVConfiguration` have
been deprecated since 1.8.0 and are still registered at 1.9.0, so that
is a direction rather than a date.

What makes that opt-out list matter is a semantics change: on 1.8
`isFeatureGateEnabled` ended in `return false`, so a Beta gate nobody
listed was off; on 1.9 `IsEnabled` ends in `return fg.State == Beta`.
Sixteen gates are Beta at v1.9.0 and none of them is in this chart's
`featureGates`, so all sixteen arrive on with the bump. Two are switched
back off here, one is switched off below Kubernetes 1.35, and I went
through the rest:

- `Template`: off. Reason above.
- `ExternalNetResourceInjection`: off. With it on, virt-controller stops
calling `multus.NetworkToResource`, so a NAD's
`k8s.v1.cni.cncf.io/resourceName` no longer becomes a resource request
on the launcher pod. That is correct for clusters running a
network-resources-injector; this platform ships none, so a VM on an
SR-IOV NAD would schedule without its device.
- `ImageVolume`: off below Kubernetes 1.35, left alone from 1.35 on.
KubeVirt renders containerDisk images as `VolumeSource.Image` volumes
when the gate is on, and it does not check the host: `dropImageVolumes`
in the apiserver strips every image volume and the mounts referencing it
while Kubernetes' own `ImageVolume` gate is off, which is the default
until 1.35 (`kube_features.go` has it `Default: false` at 1.31 and
1.33). The pod is admitted without the disk and virt-launcher fails
looking for it. The management cluster is supported from 1.33, so on
1.33 and 1.34 the chart keeps the KubeVirt gate off and the
containerDisk sidecar path in use; from 1.35 the gate stands and
`extraFeatureGates` can name it. The apiserver version is a proxy for a
runtime capability, not the capability itself, so a cluster at 1.35 or
newer whose container runtime cannot serve image volumes (the k3s case
in #3190) switches the gate off through `kubevirt.disabledFeatureGates`;
that is what the value is for.
- `Snapshot`: on. `VirtualMachineSnapshot` needed the gate listed on
1.8; snapshot and export were exercised on the test cluster.
- `DeclarativeHotplugVolumes`: on and inert.
`handleDeclarativeVolumeHotplug` returns early while `HotplugVolumes` is
enabled, which this chart still does.
- `GPUsWithDRA`, `HostDevicesWithDRA`: on; each only lets a VMI request
a DRA-provisioned device, and nothing here creates one.
- `WorkloadEncryptionSEV`: on; needs SEV hardware and a VM that asks for
it.
- `PasstBinding`: on; needs a VM using the passt binding.
- `RebootPolicy`: on; adds a field on the VMI domain spec, unset means
the old behaviour.
- `VmiMemoryOverheadReport`: on; fills `status.memory.memoryOverhead`,
reporting only.
- `OptOutRoleAggregation`: on; it enables the `RoleAggregationStrategy`
field, and aggregation stops only for a CR that sets it to `Manual`.
This one does not.
- `KubevirtSeccompProfile`: on; installs the profile, and a workload
uses it only where `seccompConfiguration` says so.
- `NodeRestriction`: on; virt-handler node authorization, needs the
Kubernetes `ServiceAccountTokenPodNodeInfo` gate, GA since 1.32.
- `LibvirtHooksServerAndClient`: on; pre-migration hooks on the target
launcher, and no hooks ship here.
- `PodSecondaryInterfaceNamingUpgrade`: flips on; it adds
`--upgrade-ordinal-ifaces` to virt-launcher and a pre-migration hook
that renames the ordinal secondary interfaces of VMs still carrying
them. The test cluster's migrations exercised that path.

Five of the seven gates the chart enables (`ExpandDisks`,
`LiveMigration`, `AutoResourceLimitsGate`, `GPU`, `VMExport`) reached GA
by 1.9 and are inert: `IsEnabled` returns true for a GA gate whether or
not the CR names it. I left the list as it is; dropping the five is a
follow-up, because it changes what the CR says without changing what the
cluster does and it belongs in its own commit.

QEMU is 10.1.0 on both sides of this jump; the el9 build revision moves
from -10 to -20 and libvirt goes 11.9 to 11.10. A moving build revision
is the thing to check rather than a major:
[kubevirt/kubevirt#16386](https://github.com/kubevirt/kubevirt/issues/16386),
where every existing VM lost live migration, was 1.6.2 to 1.7.0 on
qemu-kvm 9.1.0-19 to 9.1.0-20, one build apart within the same QEMU
version. So I tested it: VMs running on the 1.8.4 launcher live-migrated
onto the 1.9.0 one on a three-node cluster.

`make update` in `kubevirt-operator` used to `rm -rf templates` before
the fetch and took the local PrometheusRule template with it. Now it
only overwrites the file it downloads.

`make update` in `kubevirt` now greps for every line of the new block
before it passes: the range directive, the `disabledFeatureGates:` key,
the three static entries, and the guard header that decides which static
entries render. Every insert in that target is guarded by one directive
it also inserts, so a hand-merge that keeps the directive and drops a
sibling line leaves the guard satisfied and the insert skipped. That
shape covers the directive and nothing else. What makes it worth a grep
per line here is the static entries: they are the only lines those
inserts add that are CR content in their own right rather than a wrapper
around a value, so they render on every install and their absence cannot
surface as a value that stopped taking effect. The CR just goes back to
deploying virt-template and to letting KubeVirt skip NAD resource
injection. The target cannot rebuild a partial block, so it names the
missing line and stops. The payload lines of the other four blocks are
exposed the same way; I checked, and both dropping `- {{ . }}` from the
`extraFeatureGates` range and dropping the `permittedHostDevices:` key
from its `with` render green and lose the operator's values. That is a
wider change than this PR and I left it out.

Two sharp edges I know about and left. `kubevirt.disabledFeatureGates:
[HostDevices]` on a cluster that also has `cozystack.gpu-operator` in
`enabledPackages` makes the bundle forward the same gate into both
lists, and the platform renders fine while the child kubevirt
HelmRelease then stops on the guard: fail-closed, with the gate named,
but diagnosed one level below where it was configured. And the
version-conditional opt-out is evaluated at render time, so a cluster
upgraded from 1.34 to 1.35 keeps `ImageVolume` disabled until something
re-renders the chart; the direction is the safe one, since the gate
stays off and containerDisk keeps the path it had on 1.8.4.

Two things I decided against doing here. The guard's copy of the gate
list is pinned against a name being dropped, not against a name being
added to the rendered list only; the copy exists because `make update`
needs the literal `- VMExport` anchor, so closing the other direction
means changing the self-heal contract. And
`hack/run-kubernetes-block-io_test.bats` still names a
`virt-launcher:v1.8.4` image in a fixture: inert, the test matches on
the container name, and the file is outside this change.

Tested on a three-node Talos v1.13.6 / Kubernetes v1.36.2 cluster with
Flux suspended. The upgrade took about ten minutes. virt-operator had
already deployed virt-template before the CR change landed and removed
it afterwards, CRDs included. Two VMs started on the 1.8.4 launcher, one
DataVolume-backed and one containerDisk-backed, live-migrated to the
1.9.0 launcher. VMDisk plus VMInstance through the apps API boots.
`addvolume` on the VM subresource reaches Ready, `removevolume` on the
VMI subresource is accepted. VirtualMachineSnapshot and
VirtualMachineExport from it reach Ready. Restore from snapshot and PVC
clone failed on that cluster inside the LINSTOR API layer, unrelated to
this change. CDI v1.66.0 from #3920 on top of it: HTTP imports to Block
and Filesystem, blank population and hotplug all work. `helm unittest`
is 26/26 for the kubevirt chart and 142/142 for the platform chart, and
the update idempotency script passes its eleven cases.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. No app added or renamed, no
variant, release asset, `ApplicationDefinition` or tooling contract
touched, and none of the `packages/core/platform/values.yaml` keys the
ansible role sets are in the diff.

One repository is reached. `packages/core/platform/values.yaml` gains a
top-level key, `kubevirt.disabledFeatureGates`, defaulting to the empty
list, and `operations/configuration/platform-package.md` on the website
is a hand-written table of `spec.components.platform.values.*` that
nothing generates. The row is in cozystack/website#678, which lands
after this PR merges.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/678
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(kubevirt): update KubeVirt to v1.9.0. Beta feature gates are on by default upstream now. The platform keeps `Template` off, which would deploy virt-template, and `ExternalNetResourceInjection`, which would hand `resourceName` injection for NetworkAttachmentDefinitions to an injector the platform does not run. `ImageVolume` is disabled on management clusters below Kubernetes 1.35: containerDisk-backed VMs are rendered with native image volumes when it is on, and an apiserver whose own ImageVolume gate is off, the default before 1.35, drops those volumes silently, leaving the VM without its disk. On 1.35 and newer the gate stands, and that is the first release where containerDisk-backed VMs are rendered with image volumes instead of sidecars; the bundled Talos ships Kubernetes 1.36, so this is the default there. A `kubevirt.disabledFeatureGates` list in the platform values opts out of any other gate a cluster cannot carry.
```
 2026-09-05 00:30:05 +04:00
Aleksei Sviridkin 3a369fed99 feat(redis): add opt-in TLS for Redis and Sentinel (#2729) 
## What this PR does

`tls.enabled` puts Redis and Sentinel on a TLS-only listener with a
per-release cert-manager chain, and hands the tenant the CA certificate
as `redis-<name>.tenant-ca`. Off unless set.

redis-operator v3.3.5 has no TLS support. The work is ported upstream in
freshworks-oss/redis-operator#85, which is open and not in a release, so
this carries it as three patches over the v3.3.5 tarball.
`images/redis-operator/patches/tls.diff` is the source half, cut from
that PR's commit `2ad2ed8d1`, which the patch header pins; it is applied
by the glob the image Dockerfile already runs, after the Dockerfile
strips the tarball's `manifests`, `charts`, `example` and `.github`, so
the image builds from source alone and the upstream tests that read the
CRD YAML skip instead of failing against the stale v3.3.5 copy.
`patches/crd-tls.diff` adds `spec.tls` to the vendored CRD, and with it
the CEL rule that fixes `tls.enabled` at creation; without the schema
block the apiserver prunes the field on admission and the patched binary
waits for something it never sees, and the rule cannot exist without the
block it references. `patches/rbac-tls.diff` grants Create, Get and
Update on Secrets, so the operator can publish the CA-only Secret it
addresses by name. That is the whole delta, and it is narrower than the
ported operator's own chart asks for: those three are the only Secret
calls in the patched source, and cert-manager Certificates get no grant
at all, because the operator only touches one under
`spec.tls.certManager` and this chart pins `certificateSecret` mode
instead. One change in `tls.diff` is not TLS and takes effect with TLS
off, named in the patch header: `MakeSlaveOfWithPort` connects to the
replica on its own port rather than on the master's. It changes a dial,
not a pod template, so shipping the rebuilt image rolls nothing; the
plaintext pod templates stay byte-identical to v3.3.5.

The image build registration this branch used to carry (`make -C
packages/system/redis-operator image` in the root `build:` target)
landed on main with the Valkey service (#3380), so the rebuilt operator
image is produced by CI from main as well; the digest in `values.yaml`
is still refreshed by release preparation, not here.

The chart patches re-apply at the end of `update:`, the same way
reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry
theirs. Otherwise `rm -rf charts` takes TLS back out on the next `make
update` and nothing fails. `make update` reproduces the committed tree
byte for byte.

TLS is chosen when the instance is created. The config gets `port 0` and
the listener moves to `tls-port`, so TLS replaces plaintext rather than
running beside it, and a running instance cannot change mode: a replica
restarted onto the TLS listener cannot replicate from a master still on
plaintext, and Sentinels already on TLS would fail over to a stale
replica. The RedisFailover schema in the carried CRD patch rejects a
change of `tls.enabled`, which on an existing instance surfaces as a
failed HelmRelease upgrade while the instance keeps serving what it did;
the README points at creating a new instance and migrating. That is also
why TLS is not inferred from `external`: an externally reachable
instance is the one most likely to have clients an upgrade must not
drop. Existing releases keep their plaintext listener and are not
restarted by the rebuilt operator image.

The tenant gets `ca.crt` and nothing else, through `tenantsecrets`. The
chart's own Secrets take a dot between the release and the suffix
(`<release>.ca-tls`, `<release>.tls`, `<release>.ca-cert`): the suffixes
are dot-free, so the split is unique and release `foo`'s CA Secret can
no longer be release `foo-ca`'s leaf, the collision the hyphenated form
had. The chart renders a `TenantProjection` pointing at
`<release>.ca-tls`, and the CA-extraction controller re-encodes the one
named key from the parsed certificate, so nothing that is not a
certificate gets through. Same path as postgres, mongodb, nats and
qdrant.

No Secret of this release is granted to the tenant by name beyond the
generated password, and that includes `<release>.ca-cert`, the CA-only
copy the operator publishes for itself. It holds no key today, but the
operator has Create and Update on Secrets cluster-wide, so what sits at
that name is the operator's call rather than the platform's, and RBAC
cannot filter by key. `hack/check-redis-rd-secrets.bats` pins the whole
rendered Secret surface instead of a list of known-bad names, because an
include with no `resourceNames` and no `matchLabels` restricts nothing.

The chainsaw suite for redis gains two cases. A TLS fixture asserts the
cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel
come up, the operator publishes its CA-only Secret under the widened
ClusterRole, the tenant receives exactly `ca.crt` through
`tenantsecrets` while the key-bearing Secrets stay off that path, and a
`redis-cli --tls` PING through the service name completes with the same
CA. A second case applies `tls.enabled` to a running plaintext release
and asserts the refusal: the HelmRelease upgrade fails with the schema's
message, the StatefulSet keeps its revision with both replicas Ready,
plaintext still answers and no TLS listener appears.

Certificate renewal rolls the pods from inside the operator. Redis and
Sentinel read the certificate once at startup, so the patched operator
stamps a hash of the TLS Secret's `tls.crt` and `ca.crt` into the pod
template of both the Redis StatefulSet and the Sentinel Deployment on
every reconcile, as `redis-failover.freshworks.com/tls-secret-hash`. A
renewal changes the hash; the operator's own roller replaces the Redis
pods one at a time and the Deployment controller rolls Sentinel. README
names the fallback: if the roll cannot run, the deadline is the old
certificate's expiry, past which the operator's own TLS connections fail
along with the clients.

`tls.authClients` maps to `tls-auth-clients` and defaults to `no`. The
platform mints no client certificate for the tenant and the CA key is
out of reach, so `yes` is only usable when the client certificate comes
from whoever also controls the CA. Changing the value on a running
release restarts the Redis and Sentinel pods: Redis reads the directive
from its config file only at startup, so the operator folds the
effective value into the certificate hash that drives the rollout, and
the change lands when it is made rather than at the next unrelated
restart.

Follow-ups in the carried operator patch go upstream first, to
freshworks-oss/redis-operator#85 or a successor there, and are picked up
here afterwards; the carried patch does not grow Go that upstream has
not seen. `tlsConfigFor` fetches the Secret on every call, tens of GETs
per reconcile. The renewal roll is static analysis so far: the live
check, `cmctl renew` on a running release while watching the `rfr-*` and
`rfs-*` pods roll, is still owed and belongs with the e2e follow-up.

Two facts about the shipped image rather than the patch.
`packages/system/redis-operator/values.yaml` keeps the digest of the
image built before the patch until release preparation rebuilds it, so
on `main` between this merge and the next release-prep, `tls.enabled`
renders the chain and `spec.tls` while the running operator ignores them
and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026
in `golang.org/x/net` v0.53.0, reached through
`PodService.UpdatePodLabels`; the port neither introduces nor fixes it,
upstream `master` is already on v0.55.0, so it lands with the next tag
bump rather than here.

Outside the patch: `cozyvalues-gen` renders `{}` in the README Value
column for `tls.authClients`: a child property without a default
inherits the parent object's `{}`, and `make generate` reproduces it
byte for byte, so it is a generator bug rather than something to
hand-edit here.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#37
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

I walked the trigger map against the diff file by file.
terraform-provider-cozystack is reached, tracked as
cozystack/terraform-provider-cozystack#37 rather than a speculative PR:
`values.schema.json` gains a `tls` object, and the provider hand-writes
a schema, model and expand/flatten pair per Kind. It already models
`tls` for postgres, kafka, nats and qdrant, not for redis. Redis needs
different wording from those four, whose descriptions say to omit the
block to follow `external`, which is the one thing redis does not do.

Nothing else is reached. The website regenerates this package's
reference page from `README.md` on a stable tag, so the README changes
need no PR there. No package added, renamed or removed.
`packages/core/platform/values.yaml`, variant and bundle composition and
release asset names are untouched, and so are `hack/package.mk`,
`hack/common-envs.mk` and `hack/update-crd.sh`, which is what ccp and
external-apps-example read. The two new files under `hack/` are bats
unit tests. No node prerequisite, installer value, telemetry metric or
cozy-proxy label changes.

### Release note

```release-note
feat(redis): add opt-in TLS for Redis and Sentinel. Set `tls.enabled: true` to serve both on a TLS-only listener backed by a per-release cert-manager chain; the tenant reads the CA certificate as the `<release>.tenant-ca` tenant secret. TLS is off unless set and is not inferred from `external`, so existing releases keep serving plaintext and are not restarted by the rebuilt operator image. TLS is chosen at creation: a running instance cannot switch modes, the change is rejected and surfaces as a failed HelmRelease upgrade, so add TLS by creating a new instance and migrating to it. Requires cert-manager, which the platform's `system` bundle installs on every variant. A renewed certificate rolls the Redis and Sentinel pods through the operator. TLS support is carried as a downstream patch over redis-operator v3.3.5 until freshworks-oss/redis-operator#85 lands in a release.
```
 2026-09-04 19:32:32 +04:00
Aleksei Sviridkin 8adf3a0863 feat(platform): forward kubevirt.disabledFeatureGates to the kubevirt CR 
The kubevirt chart reads disabledFeatureGates from its own values, and
the iaas bundle is the only writer of the cozystack.kubevirt Package, so
nothing could set it: a Package patched by hand is reverted the next
time the platform re-renders it.

Platform values gain a documented kubevirt.disabledFeatureGates list,
forwarded into the emitted Package as
components.kubevirt.values.disabledFeatureGates alongside the entries
derived from resources.cpuAllocationRatio and the gpu block. The
parenthesized value walk is nil-safe, matching the gpu block, so a
cluster that nulls out the whole kubevirt key still renders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-04 15:29:09 +03:00
Aleksei Sviridkin 5976ddf02f fix(gateway): normalise the GatewayClass name and refuse a blank one 
The class lists arrive over the values channel comma-joined, so every
reader trims each entry as it splits them. The scalar those entries are
compared against was not trimmed anywhere. A class name carrying a
stray space therefore failed `has` against its own padded self, and the
two readers of that scalar disagreed about the same class: the gateway
chart put the tenant on an edge-terminated class, while the
cluster-wide ACME issuers still pinned their solver to a listener an
edge Gateway does not render. That stops every HTTP-01 certificate in
the cluster from issuing, with nothing in any status naming the cause.

Normalise once at the single writer so both readers see a clean value,
and keep a defensive trim at each reader. The tenant's own choice does
not travel that channel — it comes from tenant.spec.gatewayClass — so
it is trimmed where the gateway chart reads it.

Unset and whitespace-only are different events and no longer share an
outcome. Unset means the choice was not made, and falling back to the
bundled class stays its documented behaviour. Whitespace-only means a
choice was made and is garbage, in the field that decides where TLS
terminates: resolving it quietly would take a tenant the operator meant
to put behind an edge terminator and publish it directly, with a real
certificate on a real address. Both the platform default and the
tenant's own choice now refuse it by name.

The kind check beside that refusal is part of the same fix, not a tidy
up. Through Helm's values path an unquoted numeric scalar arrives as a
float64, and the whitespace check compares it against a string, so
`className: 123` dies at that comparison — one line before the trim —
with an error naming neither the key nor the file. Refusing whitespace
without checking the kind first would therefore have made the numeric
case worse than it was. `kindIs "string"` is the right test because it
never has to name the type that actually arrived.

The order of the guard's steps is part of the guard. The value is read
before anything normalises it, so a default that fires on any falsy
value cannot launder 0 or false into the empty string and resolve them
to the bundled class ahead of the kind check. Unset is the one case
that still falls back, and each of those four inputs is pinned, so a
normalisation drifting back ahead of a check fails rather than quietly
covering only the loudest bad input.

Also correct a cleanup log line: the wildcard Certificate is deleted
for every mode that is not DNS-01, so naming HTTP-01 in that message is
wrong for existingSecret and for edge alike. And state the listener
name invariant plainly: the cap is 253, not the 63 of a DNS label, and
the longest name the builders produce is 84.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-09-04 14:33:35 +03:00
Aleksei Sviridkin 487cafac3e feat(platform): let a tenant pick its Gateway class from an allowlist 
Tenant Gateways were pinned to the bundled Cilium class. Give the
platform a default in gateway.className, the set a tenant may choose
from in gateway.tenantSelectableClasses, and the set whose provider
terminates TLS upstream of the Gateway in
gateway.edgeTerminatedClasses. A tenant picks its own with
tenant.spec.gatewayClass, read only when it owns a Gateway; an
inheriting tenant runs on its ancestor's.

The selectable set is an allowlist rather than free choice because
Tenant is a tenant-writable object while a GatewayClass is
cluster-scoped: without it a tenant could attach its Gateway to a class
an admin keeps for internal-only traffic. Naming anything outside the
set fails that tenant's own gateway release and reaches no other
tenant.

Where TLS ends is a property of the class, so the gateway chart derives
certMode from whichever class the tenant landed on instead of taking it
as a separate input. The two therefore cannot be configured out of
step.

The publishing tenant cannot cross that line by itself. The
cluster-wide ACME issuers point their HTTP-01 solver at its Gateway and
are rendered from the platform default alone, so a mismatch would pin
the solver to a listener that Gateway does not have. Those issuers now
drop the sectionName pin when the default class is edge-terminated,
letting the challenge attach by hostname instead.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-09-04 14:26:54 +03:00
Andrey Kolkov e94fcf8219 feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990) 
## What this PR does

Adds backup/restore support for the managed `RabbitMQ` application.
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so this backs up broker
**definitions** (vhosts, users, permissions, queues, exchanges,
bindings, policies, parameters) over the management HTTP API
(`GET`/`POST /api/definitions`) and stores them in the system bucket.
Message payloads and point-in-time recovery are intentionally out of
scope; a full-data backup would be a Velero volume snapshot, which is a
separate strategy.

It is delivered as a **first-class
`strategy.backups.cozystack.io/Rabbitmq` driver**, following the
Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

- `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's
`PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`,
pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a
`batch/v1.Job`, covering both restore flows: in-place (into the source
broker) and to-copy (into a separate, differently-named RabbitMQ in the
same namespace, authenticating with the target's own default-user
credentials and reading the source's object key).
- A **Go-read precondition** gates each run on the `RabbitmqCluster`
reporting `AllReplicasReady` (read through a small `rabbitmqtypes`
partial type), so a not-yet-ready broker surfaces a precise
`Ready=False` wait on the BackupJob/RestoreJob rather than a Job that
starts and fails against an unreachable management API.
- The produced `Backup` records the stored object's location on
`status.artifact.uri`, rendered from the strategy's
`spec.artifactURITemplate`. The `Backup` CRD has no status subresource,
so this persists on Create, mirroring the Etcd driver.

Dispatch is wired in both controllers (plus
`supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the
scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job
itself is already covered by the existing `batch/jobs` grant). It ships
`cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and
routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default`
BackupClass, which also enrols it in the `DefaultObjectsGate`. The
RabbitMQ application chart is left untouched — it had no prior backup
wiring, and the strategy carries the S3 coordinates itself.

Unit tests cover the driver's pure helpers (applicationRef validation,
cluster-name derivation, the readiness-message builder, artifact-URI
rendering, parameter round-trip) and the reconcile precondition gate
(defers with `Ready=False` and no Job until the cluster is ready, then
proceeds). A runnable example under `examples/backups/rabbitmq/` and an
honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a
sentinel vhost, back it up, drop it, and assert it round-trips through
object storage on both restore flows.

The Job pod reuses the existing `platform-migrations` image (curl +
ca-certificates) under a new
`backupStrategyController.rabbitmqBackupClientImage` key, stamped in
lockstep with `chBackupClientImage` in `packages/core/platform/Makefile`
so the repository stays pinned at a single digest
(`hack/image-pin-consistency.bats`). A dedicated slim image built by
this package is a follow-up.

**Validation:** `go build`/`go vet`/`gofmt`/`go test
./internal/backupcontroller/... ./internal/template/...`, deepcopy +
Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate
both on and off, `helm lint`, and `helm unittest` (18/18). The backup +
in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a
real management-API export and a real S3 `--aws-sigv4` upload/download
against SeaweedFS), so the driver's core contract is exercised on a real
cluster; the in-CI Chainsaw run is the remaining automated evidence. The
object-cleanup, terminating-namespace teardown and readiness-deadline
paths added during review are unit-tested (including `Reconcile`-level
finalizer retention and the terminating-namespace release) but not yet
cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled
on the `nano` preset, so the example/e2e use `c1.small`.)

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. This is an additive backup
strategy plus a new strategy CRD; no downstream repository is
mechanically forced by the change. A documentation page for RabbitMQ
backup on [cozystack/website](https://github.com/cozystack/website)
would be a reasonable follow-up, but it is not required by this diff —
leaving the boxes unticked for a human to decide rather than ticking "no
downstream" to clear the checklist.

### Release note

```release-note
feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added RabbitMQ backup and restore support for definitions, including
vhosts, users, queues, exchanges, bindings, policies, and related
settings.
  * Added in-place and restore-to-copy workflows with readiness checks.
  * Added scheduled and on-demand backup examples.
  * Integrated RabbitMQ backups with the default backup configuration.
* Backups store definitions only; message payloads and point-in-time
recovery are not included.
* **Documentation**
  * Added a complete RabbitMQ backup and restore demonstration guide.
* **Tests**
* Added end-to-end coverage for readiness and backup/restore round
trips.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-03 16:03:46 +04:00
Aleksei Sviridkin f5206a6e28 docs(redis): declare the TLS-on-upgrade break and fix the parameter sections 
TLS replaces plaintext in this operator rather than running beside it —
with TLS on, Redis and Sentinel are configured with port 0 and serve only
on the TLS port. Since tls.enabled falls back to external, an already
deployed instance with external: true switches to TLS-only on its first
reconcile after the upgrade and every plaintext client breaks. That was
documented as a feature and nowhere as a change of behaviour for existing
instances. State it as a breaking change and name tls.enabled: false as
the opt-out.

Also spell out which hostname the certificate actually covers. The SAN
list carries the four service names and the external tenant hostname, and
its only IP entries are the loopback addresses used by the in-pod probes
and the metrics sidecar, so connecting to the external LoadBalancer by IP
fails for clients that verify the hostname.

The TLS section marker sat above the version parameter, which put version
in the TLS table in the generated README instead of Common parameters.
Move the marker below it. Regenerating also reorders version ahead of tls
in keysOrder, matching how postgres orders the same two fields.

Pin the PackageSource CRD replacement policy the way the mongodb-operator
one is pinned. The RedisFailover CRD gained spec.tls with the fork, and
Helm never upgrades CRDs shipped in crds/, so losing CreateReplace would
leave the live CRD behind and the apiserver would strip spec.tls from
every RedisFailover the chart renders — TLS silently off, no error.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-02 17:56:07 +03:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver 
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-01 18:21:10 +04:00
Andrei Kvapil 53412c3033 fix(valkey): add cozystack-engine dependsOn edge to the PackageSource 
The valkey-application PackageSource ships system/valkey-rd (an
ApplicationDefinition); it must dependOn cozystack.cozystack-engine so the
ApplicationDefinition CRD is registered before the -rd HelmRelease reconciles.
Mirrors redis-application.yaml and satisfies check-applicationdefinition-crd-ordering.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-09-01 11:44:29 +04:00
Andrei Kvapil 4e6313b608 feat(valkey): register Valkey application in the platform 
Wire the managed Valkey app into the platform: add the valkey-rd package
carrying the Valkey ApplicationDefinition, the valkey-application PackageSource
(reusing the existing redis-operator), enable it in the paas bundle, and grant
tenant roles access to the valkeys resource.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-09-01 11:43:10 +04:00
Timofei Larkin f5b957ee27 feat(kubernetes)!: manage worker pools as KubernetesNodes, adopt on upgrade (Phase 2, breaking) (#3315) 
## What this PR does

Phase 2 of the Kubernetes-app split
([cozystack/community#8](https://github.com/cozystack/community/pull/8))
— the **breaking** half. Stacked on #3314 (review that first).

Removes worker node pools from the `kubernetes` chart (they moved to
`kubernetes-nodes` in #3314) and adopts existing pools on upgrade:

- `kubernetes` chart is now **control-plane only**: drops the pool
`range` block, the `talos-reconcile` Job + RBAC (deleted; moved to
`kubernetes-nodes`), the preserve-old-KMT block, the
`kubernetes.nodeGroups` helper, and the per-group dashboard RBAC.
- **`spec.nodeGroups` is removed from the `Kubernetes` CR** (BREAKING).
Worker pools are now separate `KubernetesNodes` resources.
- **Migration 54** (bumps `targetVersion` to 55): pre-upgrade hook that,
for every `kubernetes-<cluster>` release, creates one
`kubernetes-nodes-<cluster>-<pool>` HelmRelease and re-annotates the
existing worker objects onto it with `helm.sh/resource-policy=keep`, so
the now control-plane-only parent upgrade does not prune running
workers. The default `md0` (implicit when `nodeGroups` was empty) is
materialised explicitly. Idempotent. Genuine infrastructure/read
failures fail closed (`exit 1`, retried on the next upgrade); hostile
stored shapes (a pool name over Helm's 53-char release-name cap, an
RFC-1123-invalid or whitespace/slash map key, a non-object `nodeGroups`)
are warned, pinned, and skipped rather than deadlocking the entire
fleet's upgrade.
- The `ingressNginx` addon no longer auto-provisions a node (the
implicit `md0` default is gone). If enabled without a pool carrying
`roles: [ingress-nginx]`, the controller DaemonSet simply schedules
nothing (degraded, not failed) — the requirement is documented in the
chart NOTES.txt rather than enforced at render time.

Because the child chart renders the pool objects **byte-identically** to
the old parent render (golden parity in #3314; the
KubevirtMachineTemplate content-hash name is preserved), adoption does
not churn live worker VMs.

### Migration number

`54` is contiguous with main (latest is `53`) and `targetVersion` is
`55`. The `52`/#3201 collision noted in earlier revisions has been
resolved: this branch was rebased and merged past it, so the numbering
is contiguous and `run-migrations.sh` has no gap to hard-fail on.

### Not yet done (blocking draft → ready)

- [ ] **dev e2e upgrade validation** (`cozystack-pr-test`): upgrade a
custom-`nodeGroups` cluster and a default-`md0` cluster; assert zero
worker-VM churn (KMT hash, Machine uid/creationTimestamp stable) and
that the adopted child HRs reconcile.
- [ ] downstream trigger-map walk (website docs for the `nodeGroups` →
`KubernetesNodes` change; terraform-provider for the removed API field).

### Downstream repositories

Deliberately unticked (draft) — the breaking API change affects docs and
the terraform provider; follow-ups to be filed before ready.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:

```release-note
Worker node pools of a managed Kubernetes cluster are now managed as separate `KubernetesNodes` resources. `spec.nodeGroups` on the `Kubernetes` CR is removed; existing pools are adopted automatically on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Worker pools are now separate `KubernetesNodes` resources with their
own lifecycle.
* Added a render-time compatibility guard to prevent worker pools from
running ahead of the parent control-plane minor version.
* Improved upgrade/uninstall behavior to better preserve already-adopted
worker objects.

* **Breaking Changes**
* Removed per-pool `nodeGroups` and worker node health settings from the
`Kubernetes` chart/CR; pool configuration now lives in
`KubernetesNodes`.
* Ingress requires at least one `KubernetesNodes` pool with `roles:
[ingress-nginx]`.
* `cluster` and `storageClass` are immutable after creation; pool
`version` may lag but must not be ahead of the parent minor.

* **Documentation**
* Updated READMEs, notes, and schemas to reflect the `KubernetesNodes`
split and the immutability/version rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-26 12:06:18 +04:00
IvanHunters 32ec1b5a00 fix(migrations): correct unadopted-record count semantics and stale comments 
- The unadopted-record WARNING said "N pool(s)" but counted SKIPPED_FILE lines,
  and the foreign-owner branch writes one line per object (so a single pool with
  MD+MHC+WM reported "3 pools"). Reword to "N record(s) ... one line per object
  or pool ... not a distinct-pool count" so the number does not mislead.
- The inline-chart restamp comment implied the cozystack-apps source is enforced;
  it is not (matching by chart name only is a deliberate trade-off: gating on the
  historical sourceRef risks under-matching a real legacy parent, which loses its
  workers, whereas the loose match's worst case is a harmless mislabel). Spell out
  the trade-off and that the render guard is the data-loss backstop.
- The overflow comment cited "up to 42 chars per rest.go"; the enforced cap is 32
  (maxKubernetesClusterName = 53 - 17 - 4). Correct it and note legacy/hand-created
  names can still be longer, which is why the overflow branch exists.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-25 14:40:20 +03:00
IvanHunters c0fcdd767c fix(migrations): discover pre-chartRef parents via inline spec.chart in the restamp 
The restamp discovered unlabelled parent kubernetes releases by spec.chartRef
only, but that misses the population it exists for. The apps.cozystack.io/
application.kind label was introduced 2025-12-19 (fdca49838); spec.chartRef only
2026-01-14 (43da779ee). So any parent new enough to carry chartRef already
carries the label and is handled by the label-selected sweep, while every
genuinely unlabelled parent predates chartRef and instead carries an inline
spec.chart (chart name "kubernetes" from the cozystack-apps HelmRepository).
The chartRef-only selector therefore matched none of the real targets. Match both
shapes. This is a seamless-adoption improvement, not a data-loss fix: the chart's
own render-time guard already fail-closes (blocks the upgrade rather than pruning)
for any parent the restamp misses. Exact chart-name match plus the kubernetes-
name-prefix guard keep it from touching other apps or kubernetes-nodes children;
covered by an inline-shape restamp test and a non-kubernetes decoy test.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-25 13:50:14 +03:00
IvanHunters c6b6755eec fix(migrations): record foreign-owner skips, harden record_skip, scope restamp to kubernetes- names 
Three hardening fixes on migration 56's new code:
- The foreign-owner branch in adopt_one pins the object prune-proof but does not
  adopt it (the child release can then stay Failed on the ownership conflict),
  yet it was the one warn+pin+skip path that did not record itself in the durable
  unadopted ConfigMap -- so an operator relying on that record to find stuck
  pools would miss it. Record it like every sibling branch.
- record_skip ran a bare append under set -e inside skip branches that are
  designed never to abort; a failed write (e.g. /tmp full) could deadlock the
  fleet upgrade for one tenant. Make it fail-open, matching the branches' intent.
- The restamp selector matched any release by chartRef; scope it to names
  starting with kubernetes- so it cannot mislabel (and then have the sweep skip)
  a non-standard release, keeping restamp and the sweep's contract in sync.
Also fix the previously-vacuous negative restamp test (its fixture lacked the
chartRef, so it never reached the label guard it claimed to check).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-25 13:28:58 +03:00
IvanHunters 3d1a83f419 fix(migrations): make the unadopted-pools record publish best-effort 
The durable-record ConfigMap write sat under set -euo pipefail right before the
version stamp, so a permanent failure (e.g. the record exceeding etcd's object
size limit at a very large skip count) would abort the migration and deadlock
the fleet upgrade with no path out -- even though every skipped pool was already
pinned prune-proof earlier in the sweep, so the workers are safe regardless of
whether the record lands. The record is a convenience artifact, not a safety
mechanism, so swallow its failure with a warning instead of failing closed.
Covered by a bats case that fails the publish and asserts the run still stamps.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-25 12:38:28 +03:00
IvanHunters 952901776a fix(migrations): adopt unlabelled parent kubernetes releases and record unadopted pools in migration 56 
The worker-adoption sweep selected parents by the apps.cozystack.io/
application.kind label, which postdates v1.2. A parent kubernetes HelmRelease
older than that, or one restored by hand, lacks the label: the sweep skipped
it, stamped, and the parent's control-plane-only re-render then pruned its
un-pinned workers with no pin and no warning. Discover such releases by their
chartRef instead and restamp the three application labels before the sweep
(additive, cannot corrupt a release, and repairs it for cozystack-api too).

Every warn+pin+skip branch leaves a pool pinned prune-proof but unadopted,
recorded only in the Job log, which the next upgrade reaps; the migration does
not re-run, so an operator cannot find those pools afterward. Accumulate the
skips and publish them as configmap/cozystack-migration-56-unadopted before
the stamp.

Extend the fake kubectl and add bats coverage for the restamp (positive and
negative) and the unadopted-ConfigMap paths.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-25 12:28:46 +03:00
IvanHunters 5b08d48632 docs(migrations): pin DEFAULT_MD0 provenance to the pre-removal helper commit 
The kubernetes.nodeGroups helper that DEFAULT_MD0 reproduces was deleted in this
line of work, so nothing in-tree diffs against it and a future edit to the
constant (instanceType/diskSize/...) would silently rename the content-hashed
KubevirtMachineTemplate and roll every implicit-md0 pool. Reference the
pre-removal definition (git show 92b9785da^:.../kubernetes/templates/_helpers.tpl)
so the canonical default stays diffable.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-24 11:26:44 +03:00
IvanHunters d93e136e23 fix(computeplane): provision worker pools via kubernetes-nodes after Phase 2 split 
Phase 2 moved worker nodeGroups out of apps/kubernetes into apps/kubernetes-nodes,
but the ComputePlane module still passed nodeGroups to the wrapped kubernetes
chart, which now ignores it: a fresh ComputePlane cluster came up with zero
workers (and the pinned ingress-nginx addon stuck Pending), while existing ones
were unreachable by migration 56 (it only adopts kubernetes-* releases, not
computeplane-cluster).

The module now emits one kubernetes-nodes child HelmRelease per nodeGroup,
shaped like a natively created pool, and materialises the pre-split default md0
(roles: [ingress-nginx]) when nodeGroups is empty. The pool chart's clusterName
gains an internal clusterReleaseName override so pools attach to the module's
fixed computeplane-cluster release, which is off the aggregated API's
kubernetes-<cluster> convention; the default keeps every API-created pool
byte-identical.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-24 11:19:54 +03:00
Aleksei Sviridkin 9ccb133b66 fix(ci): de-escalate select-e2e and name every escalation (#3817) 
## Why

I replayed the 150 most recently merged pull requests through
`hack/select-e2e.sh` with every escalation branch instrumented. 118 of
them (78.7%) ran the full 21-suite Chainsaw run, and the causes, counted
once per escalating PR with the count where that cause was the only one
in brackets, were `FULL_PATTERN` 88 [68], `NO_SUITE_FOR_GROUP` 24 [24],
`CHAINSAW_SHARED` 18 [1] and `UNCLASSIFIED` 7 [4]. `YQ_BROKEN`,
`NO_GRAPH_OWNER` and `BACKSTOP_EMPTY` never fired.

Two problems come out of that. Most of the escalation is real but
unnecessary. Whole classes of path that provably cannot affect a
Chainsaw suite were escalating because no rule claimed them. And the
commonest cause of a full run was absent from the log, so "why did this
pull request run everything" could only be answered by re-deriving the
selection by hand.

## What changes

**Every escalation now names its cause on stderr.** Seven branches could
reach the full suite and four did it in silence, `full_suite_pattern`
among them, the commonest cause by a wide margin, so the usual answer
was the one the log never carried. The lines go to stderr and must stay
there: stdout is the suite list and both e2e lanes parse it, so a reason
line on stdout would be read as a suite name. A unit test asserts every
message, verified by muting each of the eleven in turn and confirming
the file goes red for all of them. A reason that regresses to silence
changes no selection and is otherwise invisible.

**`cozystack.etcd-application` gains its missing `dependsOn:
cozystack.etcd-operator`.** `packages/extra/etcd` renders `kind:
EtcdCluster` from `etcd-operator.cozystack.io/v1alpha2`, and this was
the only operator-backed application source in the tree with no edge to
its operator. It is a latent install bug as much as a selection gap:
without the edge `etcd-rd` registers the ApplicationDefinition as soon
as the engine is up, so a tenant can create an `Etcd` before the
operator exists and its HelmRelease fails on `no matches for kind
"EtcdCluster"`, and `hack/select-install.sh`'s forward closure for the
`etcd` suite omitted the operator for the same reason. On the selection
side `cozystack.etcd-operator` reached no runnable suite, so every
change to the operator ran all 21.

**`vm-disk-application` maps to the `vminstance` suite.** It resolved to
a `vm-disk` suite that does not exist, `intersect_suites()` dropped the
name, and the source reached nothing runnable. Neither did
`cozystack.kubevirt-cdi` above it, whose only other dependents are
`vm-default-images` and `vm-disk-application` itself, so every CDI
change ran all 21. The coverage was there and the table did not know it:
the `vminstance` suite creates a `VMDisk` and asserts the `DataVolume`
behind it (`hack/e2e-chainsaw/vminstance/vmdisk.yaml`,
`vmdisk-vmi.yaml`). Same defect class as the etcd edge above, on the
mapping side rather than the graph side, and with no production blast
radius of its own, since `src_to_suites` is read by the selector and by
nothing else.

**`cozystack.cozystack-basics` joins `cozystack.cozystack-engine` as a
propagation hub excluded from the reverse-dependency walk.** Every edge
into it exists so a namespace or a platform-wide policy is in place
before the dependent installs, which `kubevirt-cdi` states in its own
source ("Depend on cozystack-basics so the target namespace exists
first"), and that is install ordering rather than behaviour. The damage
runs opposite to the engine's: the engine fans one change out to every
app, basics narrows instead. It reaches no suite today so it escalates,
but it sits upstream of `kubevirt-cdi`, so the first suite to land under
CDI silently converts the platform's namespace-and-policy package from
the full run to that one suite. Not hypothetical — on #3426 enabling the
site-router suite takes `cozystack-basics` from 22 suites to
`site-router` alone, and nothing in the output says coverage was lost.
Dropping the reverse edges keeps the package reachable, stops it
propagating, and leaves a change to it running everything through the
per-path escalation.

**`hack/e2e-apps/<name>.bats` maps to the `<name>` suite**, off the
basename, exactly as the per-suite rule takes the name off a
`hack/e2e-chainsaw/<app>/` directory. Deliberately mapped rather than
marked inert: what remains there is wired to nothing after the Chainsaw
migration, and inert would bake that orphan status into the rule and go
quietly wrong the day a lane runs those files again. The basename is
membership-tested against the suite list on the spot, so an unmatched
one escalates there and then rather than having the verdict deferred to
the final intersection, where the rest of the diff would decide it.

**`packages/tests/` and three `.gitattributes` join
`inert_config_pattern`,** which is what the script's own header says to
do with a genuinely inert path instead of widening the fall-through.
`packages/tests/` is a helm-unittest fixture chart, and changing a test
*of* `cozy-lib` does not change `cozy-lib`, no PackageSource lists those
paths as a component, and nothing installs them, while a change to the
library itself still escalates through `packages/library/`. The
`.gitattributes` entries are enumerated rather than matched by filename,
because the justification is what those files contain (only
`linguist-generated` markers) and the name does not carry it:
`.gitattributes` can also set `filter`, `eol`, `working-tree-encoding`
and `export-subst`, each of which changes what lands in the working tree
and therefore what gets built.

**`full_suite_pattern` escalates only `hack/e2e-*.bats`, not every
`hack/*.bats`.** The root `Makefile` is the authority on the split and
draws it at exactly that prefix, `BATS_UNIT_FILES := $(filter-out
hack/e2e-%.bats,$(wildcard hack/*.bats))`, so the 60 files it keeps are
the unit lane and the e2e sandbox runs none of them, while the three it
filters out are what `packages/core/testing`'s recipes execute and those
still escalate. Being inert here does not leave them untested: `make
unit-tests` is gated on the `plan` job's `code` output, which
`pull-requests.yaml` computes as "any changed path outside `docs/`" and
never from `select-e2e.sh`, so a bats-only pull request still runs the
whole unit lane, plus install and the OpenAPI tests, since the `e2e` job
reads the same output rather than the selection.

## Measured effect

The before-and-after comparison below is a second measurement over a
slightly different population, stated separately because the two are not
interchangeable: it replays the last 150 first-parent merge commits on
`main` (`git diff --name-only $sha^1 $sha`) through the base selector
with base sources versus this branch with its own, where the histogram
above walks the 150 merged pull requests as GitHub lists them. The
populations overlap heavily and the direction is the same, and the
totals differ by two commits.

| | full suite | scoped | nothing |
|---|---|---|---|
| before | 120 | 20 | 10 |
| after | 106 | 25 | 19 |

Full-suite rate 80.0% → 70.7%, verdict changed on 15 commits. Fourteen
de-escalated and I read every one of their file lists: they are
unit-lane bats files, non-e2e workflows, `docs/`, `CODEOWNERS`,
helm-unittest fixtures and linguist markers. The fifteenth gets one
suite *wider*: a `cert-manager` change now also selects `etcd`, which is
correct, since `cert-manager` is a dependency of `etcd-operator` and the
etcd app now genuinely reaches it through the new edge.

That replay predates the two reachability rules, and replaying the same
150 first-parent merges across those two on their own — the selector at
`73bc4c0ae` against the selector at `adf231513`, same sources on both
sides — changes no verdict at all. Two of the 150 touch
`cozystack-basics` and both ran the full suite before and after, which
is the hub exclusion keeping something true rather than failing to do
anything, and none of the 150 touch `kubevirt-cdi` or `vm-disk`, so the
mapping shows up only in a direct selection.

A few individual selections, before → after:

| changed files | before | after |
|---|---|---|
| `hack/select-e2e_test.bats` | 21 suites | *nothing* |
| `hack/select-e2e_test.bats` +
`packages/apps/redis/templates/redis.yaml` | 21 suites | `redis` |
| `hack/e2e-install-cozystack.bats` | 21 suites | 21 suites |
| `packages/tests/cozy-lib-tests/**` | 21 suites | *nothing* |
| `packages/system/.gitattributes` | 21 suites | *nothing* |
| `hack/e2e-apps/postgres.bats` | 21 suites | `postgres` |
| `packages/system/etcd-operator/values.yaml` | 21 suites | `etcd` |
| `packages/system/kubevirt-cdi/values.yaml` | 21 suites | `vminstance`
|
| `packages/apps/vm-disk/values.yaml` | 21 suites | `vminstance` |
| `packages/system/cozystack-basics/values.yaml` | 21 suites | 21 suites
|

## Blast radius of the `dependsOn` edge

A `dependsOn` on a PackageSource is an install-ordering edge in
production, not only a test-selection hint, so this is the part worth
reviewing hardest. Both Packages are emitted by
`packages/core/platform/templates/bundles/system.yaml` under the single
`bundles.system.enabled` guard with no intervening conditional between
them, in every variant, so no supported configuration has the app
without the operator. `hack/select-install.sh --validate` reports the
graph still has no cycle and nothing dangling. The one new exposure is
an admin who lists `cozystack.etcd-operator` in
`bundles.disabledPackages` while keeping the app, which now leaves it
`DependenciesNotReady`. That is the same exposure postgres, mariadb,
kafka and redis already carry, with `Package.spec.ignoreDependencies` as
the escape hatch. On upgrade `etcd-rd` waits for the operator's two
HelmReleases, and an unhealthy `etcd-operator` already means broken etcd
apps.

## Testing

Every rule added or changed has a test in `hack/select-e2e_test.bats`,
the install-side change has one in `hack/select-install_test.bats`, and
the new graph edge has a helm-unittest guard at
`packages/core/platform/tests/sources_etcd_application_dependson_test.yaml`.
The round-trip test that walks every suite through both mapping tables
stays green, `vminstance` included now that two sources map onto it.

The two reachability rules were each checked by removing the rule and
confirming its own test reddens: without the mapping the CDI test falls
back to the full-suite escalation, and without the hub exclusion
`cozystack-basics` selects `redis vminstance` instead of everything. The
hub test has to seed its own hazard, since no suite sits downstream of
basics in this tree yet, so it gives `redis-application` an edge onto
basics in a copied sources directory and pins that a redis change still
selects redis, or the assertion would pass against a graph that resolved
nothing.

`make bats-unit-tests` aborts on the first failing file (`for f in …; do
… || exit 1; done`), and `hack/ghcr-mirror_test.bats` fails on `main`
today and sorts before `select-e2e_test.bats`, so the aggregate target
never reaches these tests. I ran the 60 unit files individually instead:
59 pass, and the one failure is that pre-existing one, byte-identical to
the base commit and referencing nothing this branch touches. Also green:
`make helm-unit-tests`, `make rd-presets-check migrations-target-check
test-check-readiness`, `hack/select-install.sh --validate`, and
`pre-commit run --all-files` leaving a clean tree.

## Two things a reviewer should know, neither addressed here

`assert_selection` in `hack/select-e2e_test.bats` aborts the whole file
with `exit 1` rather than failing a single test. That is the only thing
that works under `hack/cozytest.sh`, which provides no real `run`,
`$status` or `skip`, and it is the idiom the existing tests in that file
already use. It becomes wrong when #3497 and #3498 flatten these onto
real `bats(1)`, where a failing assert should end one test and let the
rest report. Worth converting with that work rather than ahead of it.

`cozystack.mongodb-application` carries the identical missing operator
edge: `packages/apps/mongodb` renders `psmdb.percona.com/v1` while the
source depends only on `cozystack.networking` and
`cozystack.cozystack-engine`, so `cozystack.mongodb-operator` still
reaches no runnable suite and every change to it escalates to all 21.
Same defect class and same fix shape as the etcd edge, but it is a
second production install-ordering change and belongs in its own pull
request.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- The default etcd application now includes its operator and required
platform dependencies, ensuring the EtcdCluster provider is available.

- **Bug Fixes**
- Improved end-to-end test selection for application-specific changes,
shared components, inert paths, and unresolved or unclassified files.
- Full-suite escalations now provide clear reasons through standard
error.

- **Tests**
- Expanded coverage for test selection, dependency reachability, and
etcd application dependency declarations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-20 14:12:56 +03:00
IvanHunters f035da8a0f Merge branch 'main' into feat/kubernetes-nodes-phase2b 
Resolve conflicts introduced by main:
- Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in
  favour of the GPU type main added; main removed the fluxcd addon wholesale.
- Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's
  redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57.
- Accept the branch's removal of the in-chart talos-reconcile Job; it moved to
  the kubernetes-nodes chart, main only touched its comment.
- Regenerate kubernetes values.schema.json / README and the kubernetes-rd
  ApplicationDefinition schema.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-20 14:04:32 +03:00
Andrei Kvapil 851a86dcf1 feat(platform): orphan tenant FluxCD HelmReleases on upgrade 
Add migration 54 for the removed addons.fluxcd feature. For every tenant
that had it enabled, the FluxCD HelmReleases in the management cluster
are suspended, stripped of their Flux finalizer and deleted, so the
delete does not trigger a Helm uninstall inside the tenant cluster. The
Flux release already running in the tenant keeps running untouched.

Runs as a pre-upgrade hook, before the addon-less Kubernetes chart
reaches the tenant HelmReleases. Mirrors the orphan idiom of migration
29. Bumps migrations.targetVersion to 55.

Fail closed. Dropping the finalizer is the only thing that keeps the
delete inert, and migrations never re-run, so the script runs under
`set -euo pipefail` with no `|| true` on the critical path and gates the
delete on the finalizer actually being gone. The verify read uses
`--ignore-not-found` rather than `2>/dev/null || true`, so a failed read
aborts instead of reading as an empty finalizer list that passes the
gate on an object whose state was never checked. A migration that stops
is recoverable; one that half-orphans a fleet is not.

Cover it with a docker + fake-kubectl unit test running the real script
by path inside the migrations image's own alpine base (busybox ash, jq
baked on), so it exercises the shipped interpreter and load-bearing
pipefail. The suite pins ordering and selection (only the addon
HelmReleases are touched, each suspend -> finalizer-drop -> delete in
order) and the fail-closed paths (a surviving finalizer, a failing
verify read, and a failing fleet scan each abort before deleting and
before stamping).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-20 13:37:09 +03:00
Andrei Kvapil 7cca3d75e7 feat(kubernetes)!: remove the FluxCD tenant addon 
The optional `addons.fluxcd` toggle of the Kubernetes app deployed a
Flux Operator + FluxInstance into a tenant cluster. It was served by two
vendored charts, packages/system/fluxcd-operator and packages/system/
fluxcd, which were the last AGPL-3.0 content in this Apache-2.0
repository and a governance blocker for CNCF Incubation.

The management cluster does not use these charts (it runs classic Flux
via internal/fluxinstall; migration 21 already removed the operator from
it), and the addon defaulted to disabled. Rather than maintain an AGPL
payload for an unused feature, drop it entirely:

- delete both system packages and their platform source entries
- remove the addons.fluxcd HelmReleases, values and generated schema
  from the Kubernetes app
- drop the addons.fluxcd block still submitted by the e2e harness
- drop the removed fluxcd package entries from CODEOWNERS

BREAKING CHANGE: the addons.fluxcd option of the Kubernetes app is
removed. Tenant clusters that had it enabled keep their running Flux
release; see the companion migration for the upgrade path.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-20 13:34:32 +03:00
Aleksei Sviridkin 4385990869 feat(redis-operator): migrate from archived spotahome to freshworks-oss fork (#3406) 
## What this PR does

`spotahome/redis-operator` was archived on 2026-06-11 (read-only; last
release v1.2.4, 2022). This migrates the operator to the
actively-maintained **freshworks-oss/redis-operator** fork (v3.3.5,
Apache-2.0).

### Why freshworks-oss

It keeps the same `RedisFailover` CRD and API group
(`databases.spotahome.com`) — so existing managed Redis is unaffected —
and adds a native `spec.engine: Redis|Valkey` switch, letting managed
Valkey drop its Redis-compatibility shim image and run the official
`valkey/valkey` image directly.

### Changes

- Re-vendor the chart from
`oci://ghcr.io/freshworks-oss/charts/redis-operator` (v3.3.5), which
carries the `engine`-aware CRD.
- Build the operator image from freshworks-oss v3.3.5 source.
- Re-target the label/annotation-preservation patch onto the freshworks
Go module (`github.com/freshworks/redis-operator`).

### Verification

- freshworks-oss source + our `labels.diff` patch compiles natively with
Go 1.26.3 (`service` and `cmd` packages).
- `git apply --check` of the re-targeted patch against v3.3.5 succeeds.
- The operator chart renders with the cozystack-registry image override.
- The image is built (freshworks-oss v3.3.5 source + our patch,
multi-arch amd64/arm64), pushed to
`ghcr.io/cozystack/cozystack/redis-operator`, and pinned by digest in
`values.yaml`.
- `hack/promote-retag_test.bats` and `hack/nightly-mirror_test.bats`
pass with the pinned digest.

### Note for maintainers

`packages/system/redis-operator` is not part of the automated `make
build`; its image is built out-of-band and the digest is pinned in
`values.yaml`. That image is already built and pinned in this PR; re-run
`make -C packages/system/redis-operator image` only if you bump the
operator version.

### Follow-up

The cozystack/redis-operator fork carries cert-manager TLS work
(cozystack/cozystack#2729) that freshworks-oss does not have yet; the
plan is to upstream it to freshworks-oss so it lands in the shared
maintained fork rather than a cozystack-only one.

### Release note

```release-note
feat(redis-operator): migrate from the archived spotahome/redis-operator to the actively-maintained freshworks-oss/redis-operator fork (same RedisFailover CRD, adds native spec.engine: Redis|Valkey)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for injecting additional environment variables into the
Redis Operator (including a default operator group id).
  * Added default configurable labels for the Redis Operator.
* **Bug Fixes**
* Preserves and merges existing Service labels and annotations during
Redis Operator updates.
* **Chores**
* Updated the Redis Operator and Helm chart to the Freshworks-maintained
3.3.5 release, aligned chart/image sources, and simplified Helm chart
retrieval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-20 13:11:41 +03:00
IvanHunters cd72d81bb4 Merge branch 'main' into feat/kubernetes-nodes-phase2b 
Reconcile the phase-2b split with main's configurable worker-health
feature (MHC condition timeouts, maxNodeProvisionTime, podCpuLimit/
podCpuRequest), which landed on both charts after this branch forked:

- kubernetes (parent): pool rendering stays removed; main's parent
  worker-pool additions do not apply post-split.
- kubernetes-nodes (pool): absorb the feature; keep the directional
  parent-version skew guard and the immutable cluster/storageClass rules.
  KubevirtMachineTemplate content hashes are unchanged (render snapshot).
- migration 54: carry podCpuLimit/podCpuRequest (KMT-hash inputs) and the
  cluster-wide readyUnknownTimeout/readyFalseTimeout/maxNodeProvisionTime
  into adopted pools, so adoption neither rolls nor resets a tuned pool.
  Covered by a new adopt-values bats case.
- drop the parent chart's obsolete MHC-timeout suites; the
  kubernetes-nodes twins cover the feature.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-19 13:14:11 +03:00
Aleksei Sviridkin 0b8837e552 fix(telemetry): make the documented opt-out cover both collectors (#3791) 
## What this PR does

Telemetry is reported by two components — `cozystack-operator` (cluster
facts) and `cozystack-controller` (application counts) — but only the
operator had a reachable switch.

The documented opt-out sets `cozystackOperator.disableTelemetry` on the
`cozy-installer` chart. That chart deploys the operator; it cannot reach
a package the platform deploys. `packages/system/cozystack-controller`
has its own independent `disableTelemetry` key that nothing propagated
to, so after an operator had followed the published procedure the
controller kept posting `cozy_application_count` every 15 minutes.
Nothing failed, and nothing indicated the opt-out was partial.

Two changes:

- `packages/core/platform` gains a `telemetry.disabled` key and threads
it into the `cozystack-controller` component of the
`cozystack.cozystack-engine` Package — the same mechanism the bundle
already uses for cilium, multus and linstor values.
- `packages/core/installer/values.yaml` declares
`cozystackOperator.disableTelemetry`. It was only ever referenced from
the template, so the `--set` target the documentation names did not
appear in the chart's own values.

Both keys default to `false`. Nobody who has not opted out sees a
behaviour change.

The opt-out remains two steps, because the two collectors are installed
by two different charts. Collapsing it into one knob would mean the
operator writing the value into the platform Package it does not own,
which is a design change rather than a fix.

`helm unittest` on the platform chart: 137 passed (133 before, plus the
four added here). `helm lint` clean on both charts.

### Screenshots

Not a UI change.

### Downstream repositories

Trigger map walked against the diff.
`packages/core/platform/values.yaml` changed, which the map routes to
the website's hand-written platform-package table; the same change also
makes the published opt-out procedure wrong, so the docs follow-up
covers both.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/651
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(telemetry): the documented telemetry opt-out now covers both collectors. Setting `cozystackOperator.disableTelemetry=true` alone left `cozystack-controller` reporting application counts; the platform gains a `telemetry.disabled` key that silences it. Both default to false.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Added a platform telemetry opt-out setting, disabled by default.
* Added a separate operator telemetry opt-out setting, also disabled by
default.
* Applying the platform setting now disables telemetry reporting for the
relevant controller.

* **Bug Fixes**
* Ensured telemetry settings are correctly applied across supported
platform variants while preserving existing configuration overrides.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-19 12:51:55 +03:00
Andrei Kvapil c9ee4f6de8 test(platform): pin redis-operator upgradeCRDs=CreateReplace 
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-19 10:15:59 +02:00
Andrei Kvapil c7a3320bb4 fix(migrations): pin the engine and fail-close the CRD probe in migration 54 
Two defects in migration 54's adoption pass:

(1) The label landed before the image pins, so the operator adopted a
labelled CR with empty image fields and rolled Redis to the fork's 7.2.4
default — rewriting dump.rdb into a format the later 6.2.6 chart pin can no
longer load. Pin spec.redis.image/spec.sentinel.image to redis:6.2.6-alpine
in the same pass, only where empty.

(2) The CRD probe used a bare `get crd >/dev/null 2>&1` that could not tell
NotFound from an API blip and stamped anyway; switch to migration 50's
`--ignore-not-found -o name` so a real error aborts under set -e.

Also self-contained the renumber note and qualified the resource.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-19 10:15:59 +02:00
Andrei Kvapil caf9511365 fix(platform): CreateReplace the redis-operator CRDs on upgrade 
helm-controller skips CRD upgrades by default, so an upgraded cluster would
keep the old spotahome schema and prune the fork's spec.engine field on
admission. The schema diff is additive, so CreateReplace is safe. Also refresh
the harbor e2e wait comment for the freshworks operator.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-19 10:15:58 +02:00
Andrei Kvapil 0a3a3bfa77 fix(migrations): adopt existing RedisFailovers onto the operator group 
The freshworks-oss operator watches RedisFailovers through a server-side
operator-group label selector, so a CR without the label is silently dropped
from its informer. The chart templates now stamp the label, but an existing CR
only gets it when its own HelmRelease next upgrades — after the operator rolls
— leaving a window with no healing, and a suspended/stuck HelmRelease never
gets it at all. Label every RedisFailover in a pre-upgrade migration, ahead of
the new operator.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-19 10:15:58 +02:00
Aleksei Sviridkin b76cb2091c test(e2e): record the runner VM shape and region in the e2e report 
An e2e lane asks for a runner class, not for a machine. A flex shape is
filled from whatever capacity the pool has at the time, so one lane can
be served by two processor generations, and the region is a preference
with a fallback. None of that reached the report, so a red run and a
green one could not be compared by the hardware they ran on.

The read is taken on the runner VM, before the sandbox container is
created, because the metadata service answers on a link-local address
the container has no guaranteed route to: an "unreachable" answered from
in there would look exactly like a capture that had tried. It is wired
into the sandbox apply target rather than into a workflow step, so every
lane picks it up through `make prepare-env` on both the passing and the
failing path, and cozyreport folds the file into the artifact from
inside without going near the endpoint.

The keys are fetched one at a time and the root instance document never
is. That document carries the instance and compartment OCIDs, the image
OCID, the display name, the hostname and the user metadata bag, and the
report is a public artifact, so asking per key makes that property
something the wire settles rather than something a filter has to be
trusted for. It also matches where the shape sizing is published: it has
its own subpath and is not in the root document at all. From the sizing
object only the numbers are kept, so the rule holds for keys nobody has
seen yet, and an answer that is not one scalar is named rather than
printed.

Every read carries a wall-clock ceiling, since a read that never returns
costs the step its whole budget without reaching any exit status, and the
first key doubles as the probe so a machine that is not an OCI instance
pays one ceiling rather than one per key. Nothing here is fatal: this
runs inside the one retried step in the job, so a runner with no
metadata service leaves an "imds unreachable" line and an exit status
of 0.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-18 03:31:42 +03:00
Myasnikov Daniil 42bd45a87c fix(platform): make etcd-application depend on etcd-operator 
extra/etcd renders kind: EtcdCluster from
etcd-operator.cozystack.io/v1alpha2, but cozystack.etcd-application
declared no dependsOn edge to cozystack.etcd-operator -- the only
operator-backed application source in the tree that did not. The
edge has been absent since the sources were first added.

Two consequences, one per side of the graph.

On install this is a latent ordering bug. Without the edge the
etcd-rd HelmRelease registers the ApplicationDefinition as soon as
cozystack-engine is up, so a tenant can create an Etcd before the
operator exists and its HelmRelease then fails with `no matches for
kind "EtcdCluster"`. hack/select-install.sh's forward closure for the
etcd suite omitted the operator for the same reason, which would
install the suite against a cluster with no CRD to apply to.

On test selection cozystack.etcd-operator reached no runnable suite,
so every change to the operator escalated to all 21 suites -- 5 of
the last 150 merged pull requests. The etcd suite exists; only the
edge that makes it reachable from the operator was missing.

Scope of the ordering change: both Packages are emitted
unconditionally by the same bundle (templates/bundles/system.yaml,
under the single bundles.system.enabled guard, in every variant), so
no supported configuration has one without the other. An admin who
lists cozystack.etcd-operator in bundles.disabledPackages while
keeping the app now leaves it DependenciesNotReady -- the same
exposure postgres, mariadb, kafka and redis already carry, with
Package.spec.ignoreDependencies as the escape hatch.
select-install.sh --validate confirms the edge adds no cycle and
nothing dangling.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-15 17:58:07 +05:00
IvanHunters f9065a9455 fix(migrations): correct the non-scalar-field skip remedy 
The non-scalar scalar-field skip branch told the operator to restore the field
and 're-run adoption', but migration 54 stamps the version unconditionally and
does not re-run, so that remedy is a dead end. Align it with the sibling skip
branches: restore the scalar, then recreate the pool, and state plainly that
the migration has already stamped and will not re-run.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-14 17:21:14 +03:00
IvanHunters 8815468c9b fix(migrations): route non-scalar nodeGroup scalar-fields through warn+pin+skip 
Migration 54 copied per-group scalar fields (storageClass, instanceType,
diskSize, minReplicas, maxReplicas, logSerialConsole, maxUnhealthy,
nodeStartupTimeout) verbatim into the child HelmRelease values. The nodeGroups
map value was never fully schema-constrained, so a kubectl/dashboard edit could
store one of these as an object or array; copied through, it fails the
KubernetesNodes schema on the child install and leaves that child release stuck
Failed.

Validate the scalar fields before building the child values and route a
non-scalar value through the same warn + pin + skip contract as the invalid
name, overflow and non-object branches: pin the pool's objects prune-proof and
skip adoption, so the pool keeps running unmanaged until the operator restores
the stored value. Container-typed fields (roles, resources, gpus, kubelet) are
left untouched.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-13 18:44:07 +03:00
Andrei Kvapil c3cead8f64 fix(telemetry): make the documented opt-out cover both collectors 
Telemetry is reported by two components, cozystack-operator (cluster facts)
and cozystack-controller (application counts), but only the operator had a
reachable switch. The documented opt-out sets
cozystackOperator.disableTelemetry on the cozy-installer chart, which cannot
reach a package the platform deploys, so cozystack-controller kept reporting
after an operator had followed the procedure — silently, with nothing to
indicate the opt-out was partial.

The platform now exposes telemetry.disabled and threads it into the
cozystack-controller component of the cozystack-engine Package, the same way
the bundle already threads cilium, multus and linstor values. The installer
key is also declared in values.yaml: it was only ever referenced from the
template, so the documented --set target did not appear in the chart's own
values.

Both keys stay false by default; this changes no behaviour for anyone who has
not opted out.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-08-12 19:48:37 +02:00
IvanHunters 9ec5f8ceca fix(migration): carry logSerialConsole through worker-pool adoption 
Migration 54's adopt pick-list copied every worker NodeGroup field that feeds
the content-hashed KubevirtMachineTemplate except logSerialConsole. That field
renders into the same hashed template (kubernetes-nodes nodegroup.yaml), so an
adopted pool that set logSerialConsole would render a KMT without it, change the
content-hash, rename the KMT, and roll every worker VM of the pool on upgrade
(and silently drop the setting). Add logSerialConsole to the pick-list (guarded
by has(), so pools without it are unaffected) and assert it carries through.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-11 10:56:07 +03:00
IvanHunters b4a2f6f791 Merge remote-tracking branch 'origin/main' into pr-3315-lgtm 
# Conflicts:
#	api/apps/v1alpha1/kubernetes/types.go
#	api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go
#	hack/e2e-chainsaw/_lib/run-kubernetes.sh
#	packages/apps/kubernetes-nodes/tests/render-parity.sh
#	packages/apps/kubernetes/README.md
#	packages/apps/kubernetes/templates/cluster.yaml
#	packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml
#	packages/apps/kubernetes/tests/talos_templates_test.yaml
#	packages/apps/kubernetes/values.schema.json
#	packages/apps/kubernetes/values.yaml
#	packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
 2026-08-10 22:11:21 +03:00
Aleksei Sviridkin f4d812cfb9 fix(velero): apply CRD updates on upgrade via CreateReplace 
velero evolves its CRD schema between versions; v1.18 adds the Queued
and ReadyToStart backup phases. Helm never upgrades CRDs shipped in
crds/, the generated HelmRelease left spec.upgrade.crds at the Skip
default, and the package disables the chart's upgrade-crds Job, so live
CRDs stayed frozen at first-install state: the server upgrades, the
apiserver rejects the new phases, and backups sit in New while the
HelmRelease stays green. Opt the velero PackageSource into
CreateReplace, consistent with mongodb-operator, add a platform test
pinning the policy, and correct the stale Job-disable rationale in the
package.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-10 12:23:16 +03:00
myasnikovdaniil 7dca642894 fix(ci): restore e2e for fork PRs and close the merge-gate bypass (#3262) 
## What this PR does

Fixes #3257 — fork PRs never ran e2e, and the skipped required check let
them merge anyway.

Fork PRs get no secrets on `pull_request`, so `make image` pushed
anonymously and died with `denied`. That failed every build, collapsed
the `build → finalize → e2e` chain, and left the required **E2E Tests**
job *skipped* — which branch protection treats as satisfied, so an
approved fork PR could merge with e2e never having run.

This implements **Option A** from the issue (privileged `workflow_run`
split), pushing fork e2e images to the existing CI registry (OCIR):

**Build primitive (`hack/common-envs.mk`)**
- New `OCI_EXPORT_DIR` mode: build each image to a per-image OCI archive
instead of pushing, and force `PUSH`/`LOAD` off. The digest is captured
via `--metadata-file` regardless of output type, so the refs baked into
`pr.patch` match what the privileged run later pushes. Threaded through
the `image-tags` macro every package already uses — no per-package
Makefile change.

**`pull-requests.yaml` (unprivileged, fork branch)**
- `build` / `build-talos` / `finalize` export images to OCI archives and
upload them as artifacts instead of pushing → the red `denied` build
wall disappears and the chain no longer collapses.
- The in-tree e2e job is renamed off `"E2E Tests"` and guarded to
same-repo PRs. A new `e2e-report` job concludes the required `"E2E
Tests"` context as an **explicit check-run** (success on a docs-only PR
or when e2e passed, failure otherwise). A skipped job can no longer
satisfy the required check — this also closes the same hole for
same-repo PRs whose build fails.

**`e2e-fork.yaml` (new, privileged `workflow_run` from the default
branch)**
- Resolves the fork PR, opens a pending `"E2E Tests"` check-run, and
fails-closed if the fork build failed.
- Pushes the pre-built image archives to OCIR **by digest** and
flux-pushes the digest-pinned packages tree. The OCIR token is live only
for trusted binaries (`git`/`skopeo`/`flux`/`yq`) over data — never for
`make` or any fork-authored script.
- Runs the e2e suite against the pushed closure with **no push token**
and concludes the check-run on the PR head SHA.

Fork build code never receives registry credentials; the fork's images
and test scripts run only in the credential-less, ephemeral e2e job — a
stronger posture than a `pull_request_target` run that would hand fork
code the secrets directly.

### Validation

Local: buildx OCI-archive output + digest capture, macro expansion for
single- and multi-image packages, non-fork push path unchanged,
`actionlint` clean.

Needs live validation (why this is a **draft**): `workflow_run` artifact
download across runs; that the `"E2E Tests"` commit status satisfies the
required context and keeps a fork PR non-mergeable until the privileged
run concludes it; `refs/pull/<N>/merge` fetch + `git apply --3way`;
`skopeo --preserve-digests` end-to-end. `workflow_run` only runs from
the default branch, so the privileged half is exercised by mirroring the
workflow onto a personal fork's default branch and opening a throwaway
fork PR into it — not by merging first.

_Since review_: the fork `e2e` job ran the pre-#2826 BATS app loop that
the merged Chainsaw migration deleted (already broken); it now runs the
same Chainsaw + TIA path as the in-tree job, with suite selection and
the `full-e2e` override sourced from `resolve`'s trusted base-repo file
list / labels (not a merge-ref diff). Publish job hardened per the
non-blocking follow-ups: base-tree image allowlist, `skopeo copy --all`,
pinned flux CLI. The required `E2E Tests` gate is now published as a
commit **status** (default token + `statuses:write`), not a COZYSTACK_CI
app check-run — the app lacked `checks:write` (`checks.create` 403'd)
and a default-token check-run floated under a labeler suite; a commit
status has neither problem, and forks can't post one (their
`pull_request` token is read-only). Two review-bot findings fixed too:
fork `OCI_EXPORT_DIR` is now absolute (a relative path misplaced
archives under `make -C <pkg>` and every fork build would fail at
upload), and the "≥1 archive" guard no longer no-ops under `nullglob`.

### Repo-admin prerequisites

One repository setting lives outside this diff:

1. **Branch protection: keep `E2E Tests` required.** The gate is now a
commit **status** named `E2E Tests` (posted with the default
`GITHUB_TOKEN` — see `docs/agents/e2e-testing.md` §10), not a job or an
app check-run. Branch protection already requires `E2E Tests` (currently
the e2e job's name, renamed here to `E2E (in-tree)`), so the required
context now resolves to the status — **no GitHub App, no `checks:write`
grant, no app-id pin needed**. Just don't pin the context to a specific
app.
2. **Actions → "Require approval for all outside collaborators".** Each
fork run must be a deliberate maintainer click before it executes, since
the fork's unprivileged run produces the artifacts the privileged
`e2e-fork.yaml` consumes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the diff
(7 files: two CI workflows, `docs/agents/e2e-testing.md`,
`hack/common-envs.mk` + its bats suite, and two package `Makefile`s).

The one trigger worth naming is "change developer tooling
(`cozyvalues-gen`, `cozypkg`, the package `Makefile`s) →
`development.md`" on the website. It does not fire here:
`OCI_EXPORT_DIR` is a CI-only build mode that defaults off, and the
local developer workflow is unchanged — `make image` still builds and
pushes exactly as before, which the new default-path case in
`hack/common-envs_test.bats` now pins. No chart values, no API surface,
no app list, no platform variant, no Talos version bump and no release
asset rename are touched.

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(ci): fork pull requests now run e2e via a privileged workflow, and a fork PR can no longer merge on a skipped (never-run) required E2E Tests check
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a dedicated fork E2E workflow that runs after the base PR and
reports results via the required **E2E Tests** commit status.
- Fork PRs now export build images as OCI archive artifacts for later
publishing.
- Added a same-repo E2E reporting job and renamed the in-tree E2E flow
to **E2E (in-tree)**.
- **Bug Fixes**
- Added tighter, fail-closed gating to ensure merge blocking when **E2E
Tests** isn’t correctly finalized.
- **Documentation**
- Documented fork vs same-repo E2E split, OCI export behavior, and
required fork security settings.
- **Tests**
- Added Bats coverage validating `OCI_EXPORT_DIR` and preventing
unintended registry pushes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-08 04:26:25 +03:00
Aleksei Sviridkin 879d0f6db6 fix(installer): make operator readiness honest in the chart and the install gate (#3466) 
## What this PR does

The operator Deployment ran `replicas: 1` with `maxUnavailable: 1`, so
the Deployment controller took the minimum available replica count as `1
- 1 = 0` and the `Available` condition reported `True` with no running
pod at all. Measured against an operator whose image tag does not exist:
the condition was `True` with `reason: MinimumReplicasAvailable` while
the only pod sat in `ImagePullBackOff`, `unavailableReplicas: 1`, and no
`availableReplicas` key in the status.

That broke two separate things, so this fixes both.

The E2E install gate asked `kubectl wait --for=condition=Available` and
passed in 156 ms against that dead operator. The install then carried on
and died about two minutes later on the `packages.cozystack.io` CRD wait
further down the file, which sends you after CRD installation when the
operator never started. It now uses `kubectl rollout status`, which
reads updated and available replicas directly, and dumps the deployment,
pods, operator logs and namespace events when it fails. This half is
deliberately independent of the chart: `rollout status` is correct under
any rollout strategy, so a later change to that strategy cannot quietly
make the gate vacuous again.

The chart is the other half, and fixing the gate does not cover it.
`Available` is read by more than this test, and until the strategy
changes an alert on that condition, a health dashboard, or a person
reading `kubectl get deploy` all show green while the operator is dead.
The strategy moves to `maxSurge: 1` / `maxUnavailable: 0`, which
restores the ordinary meaning of the condition. Kubernetes refuses
`maxSurge: 0` and `maxUnavailable: 0` together, so the surge has to move
for the unavailability to reach zero.

That pair is worth reading twice, because it is not a configuration I
picked. At `replicas: 1` the Kubernetes default of `25%` / `25%`
resolves to exactly `maxSurge: 1` and `maxUnavailable: 0`, since the
surge rounds up and the unavailability rounds down. What this PR does is
stop overriding the default, not introduce a tuning of its own, and the
risk of the change should be judged on that basis.

Moving the surge is safe here, and I checked rather than assumed,
because the pair came from somewhere. It was carried over from the
earlier `cozystack` Deployment, where a sidecar published host port 8123
and `hostNetwork` left the surge pod unschedulable on a single-node
cluster. It got here in two steps, which is worth knowing if you go
looking for the reason: by the commit before this file existed the
sidecar was already gone and no port was declared, yet the pair stayed.
It had outlived its own reason before it was ever copied, which is why
there is no justification to find in this file's own history. On a
single-node cluster the surge pod is scheduled next to the outgoing one
and the rollout completes; adding a declared container port reproduces
the original `didn't have free ports for the requested pod ports`
failure exactly. Nothing else in the pod holds a port either: metrics
and health probes are bound to `0`, and the configured webhook server
never starts because the operator registers no webhook and never asks
the manager for the server.

The surge makes leader election load-bearing, though it covers less of
the overlap than it first appears. While `maxSurge: 0` capped total pods
at `replicas` a second manager process could not exist, so
`--leader-elect=true` was defence in depth. With a surge pod the lease
is what keeps the incoming manager's reconcile loops off the cluster
while the outgoing one is still leader, and because there is no
readiness probe the incoming pod counts as available as soon as its
container starts, so that overlap covers the whole image pull. It does
not cover everything: the operator installs CRDs, installs Flux and
creates the platform PackageSource before `mgr.Start()`, and the lease
is only taken inside `mgr.Start()`, so those three privileged installs
run outside leader election entirely. They are safe because both pods
write the same objects through server-side apply under the same field
manager, which is a different guarantee from the lease. The
`k8s-await-election` wrapper this operator replaced did hold the whole
process until it won leadership, so if that property sounds familiar it
belongs to the predecessor rather than to this binary.

The sibling `opensearch-operator` chart turns leader election off at one
replica for a real reason: a transient lease-renewal blip makes
controller-runtime exit the process, which at one replica reads as a
crashlooping Deployment. This operator installs first, when the
apiserver is at its shakiest, so that scenario applies here too. I am
accepting the risk knowingly, because after `maxSurge: 1` the
alternative is two managers reconciling the same cluster, which is
worse. The flag is pinned in the unit test so it cannot be turned off
without the test going red.

One consequence is permanent, so I am writing it down now instead of
leaving it to be found in a year. Because any bound port breaks the
surge pod under `hostNetwork`, an `httpGet` readiness probe is off the
table for this Deployment for good and only an `exec` probe could ever
work. So `Available` still answers "a container started", not "the
operator is healthy": an operator that starts and then dies partway
through installing CRDs is caught by the CRD waits below the gate, not
by the gate itself.

The surge also opens one new window. The operator installs CRDs and Flux
before the manager starts, which puts that work outside leader election,
so the incoming pod runs those applies while the outgoing pod is still
leader. Both apply the same objects server-side and the outgoing pod
exits moments later.

`--wait` is dropped from the helm step in the same change. Helm decides
a Deployment is ready with the same `replicas - maxUnavailable`
arithmetic, so on the old strategy it was satisfied at zero ready pods
and gated nothing, which is why the broken install got past it as well.
Once the strategy is honest it would gate for real, but on its own 2m
budget, and it would fail the install before the rollout gate ran and
hide the diagnostics behind a bare Helm timeout. Hooks are waited for
with or without the flag, and the release otherwise carries only the
Deployment and its RBAC objects, so the rollout gate is the better
single place to decide the operator is up.

A helm unittest suite pins the strategy and all four preconditions the
surge makes load-bearing: no container or initContainer declares a port,
both listener arguments stay disabled, leader election stays on, and no
probe uses `httpGet`. That last one follows from the two disabled
listeners leaving no port for a probe to reach, and it is the worst of
the four to get wrong, because the others wedge an upgrade while a
`livenessProbe` against a disabled listener crashloops the pod on every
cluster, fresh install included. Only the `httpGet` field is forbidden
rather than the probes themselves, so an `exec` probe stays available.
All four run against all three chart variants, because the template
branches on variant inside the container spec and a port added under one
branch would render for that variant alone.

I checked the suite fails for the right reasons instead of trusting it.
Reverting the strategy, declaring a `containerPort` on the operator
container, declaring one under a single variant branch, declaring one on
an initContainer, pointing a bind-address at a real address, turning
leader election off, and adding a readiness, liveness or startup probe
with `httpGet` each turns it red on its own assert. Inserting a
container ahead of the operator trips the container-name pin before the
argument asserts can start describing the wrong container. An `exec`
probe correctly stays green.

The rationale is split by audience. What an operator applying the
rendered manifest has to respect stays in plain comments and travels
into `_out/assets/cozystack-operator-*.yaml`: why the strategy is what
it is, that no port may be declared or bound, and that leader election
has to stay on. Everything that is only about this repository moves into
a `{{/* */}}` template comment that stops at the chart boundary. That
covers the pointer to the test and which assert pins what, where the
strategy was copied from, why the webhook server on 9443 is not a third
port binder, and why `opensearch-operator` reaches the opposite
conclusion on leader election. The other templates in this chart already
keep maintainer notes that way.

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

I left the boxes empty on purpose rather than to skip the question,
because the honest answer is not one of them and someone else should
make the call.

Walking the trigger map in `docs/agents/contributing.md`, nothing fires.
The `hack/` triggers name `hack/e2e-prepare-cluster.bats`,
`hack/package.mk`, `hack/update-crd.sh` and moves or renames under
`hack/`, and this PR only edits the body of
`hack/e2e-install-cozystack.bats`. The `ansible-cozystack` trigger is
renaming an object the role waits on, and
`deployment/cozystack-operator` keeps its name.

The behaviour that role sees does change, though. It runs `kubectl wait
deployment/cozystack-operator --for=condition=Available`, the same
vacuous check this PR replaces here, so the chart change turns its wait
from one that returns instantly into one that means something. Its
`cozystack_operator_wait_timeout` defaults to 300 seconds, the same
budget I gave the gate here, so a cold image pull fits and I do not
think anything needs to change over there. No follow-up PR seemed
warranted for a change that only makes an existing check work, but that
is a judgement call rather than a trigger, which is why the boxes are
empty.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(installer): the cozystack-operator Deployment no longer reports Available with zero running pods. It ran replicas: 1 with maxUnavailable: 1, which made the minimum available replica count 0, so the condition stayed True while the operator was dead and anything reading it, including the E2E install gate, saw green. The rollout strategy is now maxSurge: 1 / maxUnavailable: 0 and the install gate waits for the rollout instead of the condition.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved operator installation checks by monitoring deployment
rollouts and providing deployment, pod, log, and event details when a
rollout fails.
* Updated operator upgrades to maintain availability during rollout,
including in single-node environments.
* Documented rollout requirements and behavior across supported
installation configurations.

* **Tests**
* Added coverage for rollout settings and upgrade compatibility across
supported installation variants.
  * Clarified Helm unit-test directory coverage documentation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-06 15:25:39 +03:00
Aleksei Sviridkin 115d9bbaaf fix(installer): let the operator Deployment report Available honestly 
At replicas: 1 with maxUnavailable: 1 the Deployment controller takes
the minimum available replica count as 1 - 1 = 0, so the Available
condition reports True with no running pod at all. That reading is what
every consumer of the status sees: an alert on Available, a health
dashboard, or a person reading kubectl get deploy all show green while
the operator is dead.

The replacement is not a tuned pair, it is the platform default written
down. At replicas: 1 the default 25% / 25% resolves to maxSurge 1 and
maxUnavailable 0, because the surge rounds up and the unavailability
rounds down. So this change stops overriding the default rather than
introducing a configuration of its own, and Kubernetes rejects
maxUnavailable: 0 together with maxSurge: 0, so the surge could not
have gone the other way regardless.

The overridden pair was carried over from the earlier cozystack
Deployment, where a darkhttpd sidecar published a host port and
hostNetwork left a surge pod unschedulable on a single-node cluster. It
arrived here in two steps: by the commit before this file was created
the sidecar was already gone and no port was declared, yet the pair
stayed, so it had outlived its own reason before it was ever copied.
That is why looking for a justification in this file's history finds
none. With no port declared anywhere in the pod there is no hostNetwork
port conflict left, so the surge pod schedules beside the outgoing one;
declaring one reintroduces exactly the scheduling failure the pair was
added for. Not exercised on a live cluster in this rework.

Stay on RollingUpdate rather than move to Recreate, which would fix the
condition too, and by a shorter route: MaxUnavailable() reports 0 for
any strategy that is not RollingUpdate, so the minimum available
replica count becomes replicas - 0 = 1 and the condition is honest
without any surge pod at all.
The transition was not attempted, and the failure this project fixed in
ouroboros 0.8.1 does not transfer to it. There the rollingUpdate block
was the one the apiserver defaults onto a Deployment created without
one, so it belonged to no applier and outlived the switch to Recreate.
This Deployment has carried an explicit block since the commit that
created it, and nothing applies it server-side, so whether Recreate
would upgrade cleanly is an open question about the field-ownership
history of the live object rather than a known rejection. What keeps
RollingUpdate here is availability rather than apply semantics:
maxUnavailable: 0 leaves the outgoing operator running when the
replacement never starts, where Recreate deletes it first and leaves
none. Editing the numbers inside the block is
upgrade-safe from every state the field can already be in.

The surge introduces new windows, and leader election covers less of
them than it looks. The operator runs four privileged install steps
before mgr.Start(), and the lease is only acquired inside mgr.Start():
the CRD install, the Flux install, the platform source resource and the
platform PackageSource. All four therefore run in the incoming pod
outside leader election entirely. What the lease does cover is the
reconcile loops, the part that would otherwise conflict continuously
instead of once at startup.

That leaves three seams, and only the first two happen on a healthy
upgrade. The routine one is the incoming pod's install steps against
the outgoing pod's live reconcile loops, since the outgoing pod ran its
own installs at its startup and has been inside mgr.Start() ever since.
Those do not contend for the same fields: the installs write as field
manager cozystack-operator, while the reconcilers write as
cozystack-packagesource-controller and cozystack-package-controller and
go mostly to status subresources. Different owners, different objects,
so this seam is safe by separation rather than by any shared manager.

The second is version skew rather than field contention, and maxSurge:
1 widens it rather than opening it. Under maxSurge: 0 the surge budget
counts desired replicas rather than live pods, so the incoming pod is
created while the outgoing one may still be Terminating - the overlap
existed. What it did not have was a live counterparty: scale-down
sends SIGTERM before the new pod is created, and that cancels the
manager context, so the old loops were already unwinding. Here the
outgoing pod is not signalled until the incoming one is available, so
the overlap runs against a fully live reconciler for a termination
grace period instead of a vestigial one. The incoming pod reports
Ready the instant its
container starts, because there is no readiness probe, so its CRD
install runs while the outgoing binary's reconcile loops are still live
and now writing against a schema the incoming pod has already moved.
Old code against a new schema, which no field manager separates. It is
bounded rather than benign: the outgoing pod exits on SIGTERM within
seconds, a rejected write is transient and the next reconcile repairs
it, and no object is left corrupt. Widening the skew, by adding a probe
that delays Ready or by a migration the old code cannot round-trip, is
what would make this seam matter.

The third seam is install against install, and it needs the outgoing
pod to be restarting, since a crashlooping pod re-runs its four steps
on every restart, which is exactly the broken operator this gate exists
to catch. Three of the four steps use server-side apply under the same
field manager, so they never conflict and resolve last-writer-wins
whenever the two pods carry different manifests. That ordering matters
in one direction, and it is accepted here rather than guarded against:
the CRD install and the Flux install both apply with Force: true, so an
outgoing pod that writes last silently overwrites what the incoming pod
has just applied, with no conflict and no error to exit on. The
incoming pod is inside mgr.Start() by then and will not apply again
until it restarts, so that revert of the upgrade's install phase
persists rather than healing itself. Reaching it needs a restart
landing in the seconds between the incoming pod's installs and the
outgoing pod's SIGTERM. installPlatformSourceResource is the remaining
exception, a read-modify-write of a Get followed by Create or by Update
with the observed resourceVersion, so two pods running it at once can
collide on AlreadyExists or on a conflict; that failure is loud and
self-healing, since any error there exits the process and the pod
restarts, and the object is not left corrupt.

Pin the strategy and all four preconditions the surge makes
load-bearing in a unit test: no container or initContainer declares a
port, both listener arguments stay disabled, leader election stays on,
and no probe uses httpGet. They fail differently and none is visible to
the others, since a declared port stops the scheduler, a bound one only
fails once the incoming process starts, and a second manager without a
lease fails nothing at all until it writes. The httpGet precondition
follows from the two disabled listeners leaving no port to target, and
it is the worst of the four to get wrong: the other three wedge an
upgrade, while a livenessProbe against a disabled listener crashloops
the pod on every cluster, including a fresh install where no surge
happens. Only the httpGet field is forbidden rather than the probes
themselves, because an exec probe stays legitimate. All four are
checked against all three variants, because the template branches on
variant inside the container spec and a port added under one branch
renders for that variant alone. The strategy type is pinned alongside
its two values, since maxSurge and maxUnavailable say nothing on their
own once the type is no longer RollingUpdate.

The listener preconditions are pinned less tightly than the other two,
and the test says so rather than implying coverage it does not have. A
membership assert has no notion of position while Go's flag package
keeps the last occurrence, so a duplicate after the disabled one wins
and the assert still passes, while a duplicate before it loses and the
assert passes just the same - the test cannot tell those two apart.
Moving the disabled flag last fixes the process without making the
assert catch anything. Rejecting the two addresses this
binary defaults to closes the shape a values-gated block to enable
metrics would take, and nothing available here closes an arbitrary
address, since helm-unittest cannot express that a flag appears at most
once.

Note in hack/helm-unit-tests.sh that the loop covers packages/core,
which its comment omitted. The suite added here lives there, so a later
edit reconciling the loop with the stale comment would have silently
stopped running it.

Split the rationale by audience rather than keeping it all in shipped
comments. What an operator applying the rendered manifest has to respect
stays in plain comments and travels into the install artifacts: that the
pair is the default, why maxUnavailable must stay 0, that no port may be
declared or bound, that no probe may use httpGet, and that leader
election has to stay on. Most of what stays in the template comment is
what the chart's own suite pins, so it cannot drift out of step with the
chart without a test going red: the four preconditions and which assert
covers each, and why the variant loop is needed. Three things there are
deliberate exceptions, kept because anyone editing the strategy has to
know them and no assert can carry them - that the webhook server
configured on 9443 is not a port binder, since controller-runtime adds
it to the runnables only inside GetWebhookServer() and this binary never
calls it; that two overlaps between the pods sit outside the lease; and
why RollingUpdate is kept rather than Recreate. Those name what exists.
The seam analysis above, which is how each one works, deliberately stays
here instead. It describes another file's internals, nothing pins it,
and a commit message is fixed in time, so it cannot quietly fall out of
date the way a comment can.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-05 18:15:41 +03:00
Aleksei Sviridkin 7739d1e294 feat(multus)!: stage reference CNI plugins from the multus image 
Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (bridge, macvlan, ipvlan, ...),
and nothing guarantees a node has them in the directory Multus
searches.  On k3s they live under /var/lib/rancher/k3s/data/cni while
Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails
there with `failed to find plugin "bridge" in path [/opt/cni/bin]`.

Bundle 14 of the release's plugins into the multus image and add an
install-cni-plugins init container that installs them into the host
/opt/cni/bin, mirroring the existing install-multus-binary container.
loopback, dhcp, tap and dummy are left out; the Dockerfile records why.

Gate it on a new chart value, stageCniPlugins, resolved per bundle and
on for both.  Talos bundles only bridge, firewall, flannel, host-local,
loopback and portmap, so the other ten are missing there exactly as on
generic Linux; the four that overlap are rewritten with the same
upstream version, v1.9.1 on both sides, rather than an older one.  A
marker in the image Dockerfile records the Talos release that pin was
compared against, and a test holds it to the release the platform
ships.  multus therefore moves out of system.common-packages into the
two variant branches, the way linstor already does for talos.enabled.
networking.stageCniPlugins overrides the variant default in either
direction.

Where staging is on the install is unconditional: a plugin is replaced
on every pod recreation, not only when the destination is missing.  A
destination that is already a directory is reported and left alone,
since replacing it is not something this can do safely.  Turning the
value off afterwards does not undo it; nothing here removes or
restores a plugin.

The container is not privileged and takes no mount propagation.  It
runs in the spc_t SELinux domain, without which an enforcing node
refuses the write.  Each plugin is installed under a temporary name and
swapped in with a rename: /opt/cni/bin is shared with cilium and
kube-ovn.  The plugin tarball is checksummed before extraction.

A plugin that cannot be installed is reported on stderr and to
/dev/termination-log, and the container still exits 0: Multus stays the
node's primary CNI and nothing removes its config, so an init container
that never completes would leave the shim with no daemon behind it.

The manifest change lives in customize-deployment.patch so it survives
make update.

BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI
plugins in /opt/cni/bin on existing nodes, Talos included -- four of the
names it writes are ones the Talos node image also carries.  Opt out
before upgrading: declining afterwards does not restore what was
replaced.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-05 14:19:47 +03:00
IvanHunters c010745e8e Merge branch 'main' into feat/kubernetes-nodes-phase2b 
Bring the worker-pool split branch up to date with main (111 commits),
picking up the node-join and CI fixes it was missing: the 18m tenant
node-join deadline, guest-fsync-off on ephemeral CI disks (#3455),
kubevirt-csi client rate-limit raise (#3428), the Cilium ingress-IP
race guard (#3430), and etcd-operator v0.5.4.

Conflict resolution: main's #3535 (render the talos-reconcile Job for
the default md0 group) modified packages/apps/kubernetes/templates/talos/
talos-reconcile-job.yaml, which this branch deletes because the worker
split moves that Job into the kubernetes-nodes chart. Kept the parent
template deleted and dropped the parent-scoped test
talos_reconcile_nodegroups_test.yaml: the empty-nodeGroups md0-default
gap it guards cannot occur in the per-pool child chart, and migration 54
materialises the implicit md0 into an explicit child HelmRelease.

targetVersion resolves to 55 (migration 54 on top of main's 53).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-04 21:52:36 +03:00
IvanHunters e969d71379 chore(platform): use bash shebang for migration 54 
Migration 54 uses set -o pipefail, a bashism. The migrations image ships
bash (alpine:3.24 + apk add bash) and sibling migrations 45/47/50/51 already
use #!/bin/bash for the same reason. Align 54 so the interpreter matches the
options it sets and cannot silently no-op pipefail if /bin/sh were ever a
non-bash shell.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-03 23:02:36 +03:00
IvanHunters cb56f140ad fix(platform): pin keep on a foreign-owned worker object before skipping it 
The foreign-owner skip branch warned and returned without applying
helm.sh/resource-policy=keep, on the premise that a foreign release-name
means the parent will not prune the object. That premise is unsound: Helm
computes the upgrade prune set from the release's own stored old manifest
minus the new manifest and gates deletion only on resource-policy=keep, not
on the live meta.helm.sh/release-name. The worker object is in the parent's
old pre-split manifest, so the control-plane-only re-render would prune it
and cascade-destroy the tenant's worker VMs.

Pin keep before returning, exactly as every sibling skip branch does, while
still refusing to rewrite the foreign release-name (which could corrupt that
release). Also correct the printed remedy: migration 54 stamps 55
unconditionally and never re-runs, so recovery is manual, not a re-run.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-03 22:19:35 +03:00
IvanHunters 4f323bfce5 fix(platform): pin live worker pools when migration-54 sees non-object nodeGroups 
The top-level non-object nodeGroups branch warned and skipped the cluster
without pinning any objects, unlike every sibling skip branch (RFC-1123
invalid key, 53-char overflow, per-group non-object) which pins first. If a
cluster's pools were rendered under an earlier good revision and its
nodeGroups was later mangled out-of-band into an array/string/number, the
worker objects stay live but un-pinned; the parent's control-plane-only
re-render then prunes them, destroying the tenant's running worker VMs.

The declared pool keys are unreadable in this shape, so discover the live
pools from their KubevirtMachineTemplates (kubernetes-<cluster>-<group>-<hash>)
and pin each via pin_pool_objects before skipping adoption. Fail-closed
through list_kmts so a transient read never masquerades as no-pools.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-03 21:52:40 +03:00
IvanHunters 2d73043518 fix(platform): scope migration-54 foreign-owner refusal to one object 
A worker object whose meta.helm.sh/release-name is neither the parent nor
the child release made adopt_one exit 1. Under set -e that aborts the whole
per-cluster pre-upgrade hook, so run-migrations.sh never stamps the version
and one tenant's out-of-band-edited annotation deadlocks the platform
upgrade for every tenant, with no path out.

This is the same hostile-stored-shape class the non-object, overflow and
RFC-1123-invalid branches were already reshaped to warn-and-skip rather than
deadlock the fleet. A foreign owner also means the parent release no longer
owns the object, so its control-plane-only re-render will not prune it:
skipping is prune-safe. Refuse to adopt (rewriting a foreign release-name
could corrupt that release) but scope the blast radius to the single object:
warn, skip it, document the operator remedy, and let the run complete.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-03 21:51:28 +03:00
myasnikovdaniil 3f967ec7e0 fix(platform): restore the isp-hosted dependency closure (#3391) 
## Problem

Fixes #3376. The `isp-hosted` preset shipped `bundles.system.enabled:
false` while `paas` and `naas` were enabled, so it enabled an app
catalog whose dependency closure it simultaneously disabled. Every
emitted Package sat in `DependenciesNotReady`, the `*-rd` HelmReleases
failed with `no matches for kind "ApplicationDefinition" in version
"cozystack.io/v1alpha1"`, and the `cozystack-platform` HelmRelease never
went Ready.

Rendering the preset at `origin/main` shows the scale: **24 emitted
Packages, 24 of them blocked**, by 6 missing dependencies — two more
than the issue lists (`cert-manager` and `prometheus-operator-crds` are
also absent). `isp-full` renders 75 Packages with a fully satisfied
closure, which is a clean control.

## Root cause

`templates/bundles/system.yaml` is gated end to end on
`bundles.system.enabled`, but it carries two separable things: a
variant-selected **data-plane head** (kubeovn/cilium/linstor on
`isp-full`, a noop networking Package on `isp-hosted`) and a
variant-agnostic **control-plane body** — `cozystack-engine`, the sole
source of the `application-definition-crd` component, plus
`cert-manager`, `prometheus-operator-crds`, `postgres-operator` and
`objectstorage-controller`.

"Hosted" means the host already provides CNI, storage and
virtualisation, so only the data-plane head should be dropped. Disabling
the whole bundle also deleted the control-plane body that every variant
needs.

Three things show the disable was the mistake rather than the intent:
the preset still sets `variant: "isp-hosted"`; `system.yaml` contains an
`isp-hosted` branch (noop networking plus default scheduler) that could
never render; and `iaas.yaml` already refuses to render on this variant,
encoding the same "system bundle minus iaas" contract. The preset has
never worked — it was born this way in `2d022e38e` (January 2026), the
refactor that replaced `bundles/paas-hosted.yaml`, whose contents
confirm hosted was always meant to ship the engine, `cert-manager` and
`prometheus-operator-crds`.

## Changes

1. **`values-isp-hosted.yaml`** — enable the system bundle. Hosted now
emits 53 Packages with every `dependsOn` satisfied, installing 59
components against the 60 of the pre-refactor `paas-hosted` bundle, with
no LINSTOR, KubeVirt, CAPI or Kamaji leakage.
2. **Move `cozystack.bucket-application`** from the iaas bundle into the
system bundle. Its own `dependsOn` closure is `networking` +
`objectstorage-controller` + `cozystack-engine`, none of them iaas,
while `cozystack.backupstrategy-controller` — a *default* package of the
system bundle — hard-depends on it. Any system-on/iaas-off configuration
therefore wedged `backupstrategy-controller`; both `isp-full` presets
masked this by enabling iaas. This is a second, independent bug not
reported in the issue.
3. **Guards** — `paas`, `naas` and `iaas` now also require
`bundles.system.enabled`. The pre-existing checks looked only at
`bundles.system.variant`, which keeps its value when the bundle is
disabled, so they passed on exactly this configuration. `iaas` had the
identical hole (12 Packages, 6 unsatisfied deps). This turns a silent
multi-hour `DependenciesNotReady` hang into a render-time failure that
names the remedy.
4. **Two helm-unittest suites** — one pinning the dependency closure the
catalog needs, one pinning the guards.

## Verification

- All four operator-registered variants render dependency-closed:
`default` 0 Packages, `isp-hosted` 53, `isp-full` 75, `isp-full-generic`
75. The two `isp-full` variants are byte-identical before and after, so
the bucket move is a no-op for them.
- Hosted is a strict subset of full with zero data-plane leakage, and
its exclusions match the pre-refactor hosted bundle.
- 23 suites / 97 tests pass in `packages/core/platform`; repo-wide
`hack/helm-unit-tests.sh` is green.
- Every change was **mutation-tested** — reverting each one in turn
makes the suite fail, so the tests are not vacuous.
- The workaround documented in #3376 (`paas`/`naas` disabled) still
renders cleanly; the new guards do not fire on it.
- Hosted runtime spot-checks: `cozystack-api` is a Deployment with only
a *preferred* control-plane affinity plus blanket tolerations, and
`cozystack-scheduler` uses its `default` variant (plain Deployment, no
LINSTOR extender) — both schedule fine on a cluster with no
control-plane nodes.

## What this does not fix

This restores the dependency closure. It is **not** by itself sufficient
for a fully converged hosted install. Review surfaced further
hosted-unaware defaults that the dead bundle had been masking, filed
separately:

- #3387 — `apps/tenant` renders 14 `cilium.io/v2` policies with no
`.Capabilities` guard, so `tenant-root` fails on a non-Cilium cluster.
This is the blocker for actual hosted convergence, and it needs a
deliberate decision rather than a reflexive capability guard, because
those policies implement tenant network isolation.
- #3388 — `extra/monitoring` defaults `logsStorages` to the LINSTOR-only
`replicated` StorageClass.
- #3389 — `bundles.disabledPackages` can still break the closure with no
render-time error; the guards here cover the bundle flags only.
- #3390 — nothing in CI exercises `isp-hosted`, which is why all of the
above accumulated undetected.

## Upgrade note

Existing `*-application` Package CRs carry `helm.sh/resource-policy:
keep`. An operator who applied the #3376 workaround but did not also
delete the retained Package CRs (as that issue instructs) will find them
install once this change makes their dependencies Ready, since the
reconciler re-enqueues dependents on readiness. Deleting the unwanted
Packages before upgrading avoids that.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added fail-fast validation that requires the **System bundle** when
enabling **IaaS**, **NaaS**, or **PaaS** to prevent incomplete catalogs.
* Updated bundle composition so the **bucket application** is provided
by the **System bundle**.
* Enabled the **System bundle** by default for the `isp-hosted` variant.
* **Bug Fixes**
* Improved error messaging to clearly indicate the required
bundle/variant configuration.
* **Tests**
* Added Helm unittest coverage for rendering-gate failures,
dependency/package expectations, and hosted data-plane exclusions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-03 21:27:56 +05:00