mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
feat/externalip-api
490
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
6935df8c7f |
chore(kubevirt): update KubeVirt to v1.9.0 (#4015)
## What this PR does KubeVirt operator manifest goes from v1.8.4 to v1.9.0. The 1.9 release turns every Beta feature gate on by default and adds `disabledFeatureGates` to the CR for opting out, so the CR side needed a few decisions. `Template` is disabled. With it on, virt-operator deploys virt-template: two Deployments, two APIServices, a webhook and a ValidatingAdmissionPolicy that nothing here uses. One more aggregated API server is one more way to break discovery. This one is not a knob. `- Template` is written into the CR unconditionally and there is no value that takes it back out, because nothing on the platform uses VirtualMachineTemplate; if something starts to, the static entry becomes a value then. Asking for it through `extraFeatureGates` is refused at render time with an error that says so, rather than producing a CR KubeVirt would reject. `ImageVolume` is the host-dependent one: off below Kubernetes 1.35, left on from 1.35, with a platform value for runtimes that cannot serve image volumes even there (#3190 has the k3s case). Nothing the platform ships uses containerDisk. The value is `kubevirt.disabledFeatureGates` on the platform values, which the iaas bundle forwards into the emitted `cozystack.kubevirt` Package as `components.kubevirt.values.disabledFeatureGates`: Closes #3190. ```yaml kubevirt: disabledFeatureGates: - ImageVolume ``` The chart refuses to render a CR that names one gate in both lists. KubeVirt's `kubevirt-update-validator.kubevirt.io` webhook rejects that combination and runs with `failurePolicy: Fail`, so on a cluster that already has the CR the write is refused and the `kubevirt` HelmRelease stops converging with everything else the CR carries. The create admitter runs no validation at all (its own comment says the update validator is where that happens, and it only gets installed after the CR exists), and `IsEnabled` in virt-config checks `FeatureGates` before `DisabledFeatureGates`, so on a fresh install the same CR is admitted and the gate silently stays on. Render-time is the only place left where an operator gets told. A name repeated inside one list is not an error: upstream compares the two lists against each other, and both fields are `x-kubernetes-list-type: atomic` in the CRD, so that CR is accepted and the guard stays quiet. The gates the chart pins off can arrive a second time from a supported values file, since the chart writes them itself and an operator listing the gates their cluster declines has no reason to know that, so the range skips them and the CR states each opt-out once. The guard walks its own copy of the seven gates the chart enables. That copy exists because the rendered list has to stay a literal list for `make update` to find its `- VMExport` anchor, and a gate quietly dropped from the copy would keep rendering under `featureGates` while leaving the guard. The suite carries one case per gate, so removing any single name turns exactly one of the seven red. `HotplugVolumes` stays in the list even though 1.9 deprecates it. A deprecated gate listed in the CR is still enabled, and while it is enabled virt-controller skips the declarative hotplug path. Dropping it would switch on declarative hotplug, but then virt-api rejects the VMI-level `removevolume` subresource on VM-owned VMIs, and kubevirt-csi uses that as a fallback during teardown. The CSI driver has to move off that subresource first. Upstream names no removal release for it: the `Deprecated` state's own doc comment says "discontinued in the following release", but `MultiArchitecture` and `DisableMDEVConfiguration` have been deprecated since 1.8.0 and are still registered at 1.9.0, so that is a direction rather than a date. What makes that opt-out list matter is a semantics change: on 1.8 `isFeatureGateEnabled` ended in `return false`, so a Beta gate nobody listed was off; on 1.9 `IsEnabled` ends in `return fg.State == Beta`. Sixteen gates are Beta at v1.9.0 and none of them is in this chart's `featureGates`, so all sixteen arrive on with the bump. Two are switched back off here, one is switched off below Kubernetes 1.35, and I went through the rest: - `Template`: off. Reason above. - `ExternalNetResourceInjection`: off. With it on, virt-controller stops calling `multus.NetworkToResource`, so a NAD's `k8s.v1.cni.cncf.io/resourceName` no longer becomes a resource request on the launcher pod. That is correct for clusters running a network-resources-injector; this platform ships none, so a VM on an SR-IOV NAD would schedule without its device. - `ImageVolume`: off below Kubernetes 1.35, left alone from 1.35 on. KubeVirt renders containerDisk images as `VolumeSource.Image` volumes when the gate is on, and it does not check the host: `dropImageVolumes` in the apiserver strips every image volume and the mounts referencing it while Kubernetes' own `ImageVolume` gate is off, which is the default until 1.35 (`kube_features.go` has it `Default: false` at 1.31 and 1.33). The pod is admitted without the disk and virt-launcher fails looking for it. The management cluster is supported from 1.33, so on 1.33 and 1.34 the chart keeps the KubeVirt gate off and the containerDisk sidecar path in use; from 1.35 the gate stands and `extraFeatureGates` can name it. The apiserver version is a proxy for a runtime capability, not the capability itself, so a cluster at 1.35 or newer whose container runtime cannot serve image volumes (the k3s case in #3190) switches the gate off through `kubevirt.disabledFeatureGates`; that is what the value is for. - `Snapshot`: on. `VirtualMachineSnapshot` needed the gate listed on 1.8; snapshot and export were exercised on the test cluster. - `DeclarativeHotplugVolumes`: on and inert. `handleDeclarativeVolumeHotplug` returns early while `HotplugVolumes` is enabled, which this chart still does. - `GPUsWithDRA`, `HostDevicesWithDRA`: on; each only lets a VMI request a DRA-provisioned device, and nothing here creates one. - `WorkloadEncryptionSEV`: on; needs SEV hardware and a VM that asks for it. - `PasstBinding`: on; needs a VM using the passt binding. - `RebootPolicy`: on; adds a field on the VMI domain spec, unset means the old behaviour. - `VmiMemoryOverheadReport`: on; fills `status.memory.memoryOverhead`, reporting only. - `OptOutRoleAggregation`: on; it enables the `RoleAggregationStrategy` field, and aggregation stops only for a CR that sets it to `Manual`. This one does not. - `KubevirtSeccompProfile`: on; installs the profile, and a workload uses it only where `seccompConfiguration` says so. - `NodeRestriction`: on; virt-handler node authorization, needs the Kubernetes `ServiceAccountTokenPodNodeInfo` gate, GA since 1.32. - `LibvirtHooksServerAndClient`: on; pre-migration hooks on the target launcher, and no hooks ship here. - `PodSecondaryInterfaceNamingUpgrade`: flips on; it adds `--upgrade-ordinal-ifaces` to virt-launcher and a pre-migration hook that renames the ordinal secondary interfaces of VMs still carrying them. The test cluster's migrations exercised that path. Five of the seven gates the chart enables (`ExpandDisks`, `LiveMigration`, `AutoResourceLimitsGate`, `GPU`, `VMExport`) reached GA by 1.9 and are inert: `IsEnabled` returns true for a GA gate whether or not the CR names it. I left the list as it is; dropping the five is a follow-up, because it changes what the CR says without changing what the cluster does and it belongs in its own commit. QEMU is 10.1.0 on both sides of this jump; the el9 build revision moves from -10 to -20 and libvirt goes 11.9 to 11.10. A moving build revision is the thing to check rather than a major: [kubevirt/kubevirt#16386](https://github.com/kubevirt/kubevirt/issues/16386), where every existing VM lost live migration, was 1.6.2 to 1.7.0 on qemu-kvm 9.1.0-19 to 9.1.0-20, one build apart within the same QEMU version. So I tested it: VMs running on the 1.8.4 launcher live-migrated onto the 1.9.0 one on a three-node cluster. `make update` in `kubevirt-operator` used to `rm -rf templates` before the fetch and took the local PrometheusRule template with it. Now it only overwrites the file it downloads. `make update` in `kubevirt` now greps for every line of the new block before it passes: the range directive, the `disabledFeatureGates:` key, the three static entries, and the guard header that decides which static entries render. Every insert in that target is guarded by one directive it also inserts, so a hand-merge that keeps the directive and drops a sibling line leaves the guard satisfied and the insert skipped. That shape covers the directive and nothing else. What makes it worth a grep per line here is the static entries: they are the only lines those inserts add that are CR content in their own right rather than a wrapper around a value, so they render on every install and their absence cannot surface as a value that stopped taking effect. The CR just goes back to deploying virt-template and to letting KubeVirt skip NAD resource injection. The target cannot rebuild a partial block, so it names the missing line and stops. The payload lines of the other four blocks are exposed the same way; I checked, and both dropping `- {{ . }}` from the `extraFeatureGates` range and dropping the `permittedHostDevices:` key from its `with` render green and lose the operator's values. That is a wider change than this PR and I left it out. Two sharp edges I know about and left. `kubevirt.disabledFeatureGates: [HostDevices]` on a cluster that also has `cozystack.gpu-operator` in `enabledPackages` makes the bundle forward the same gate into both lists, and the platform renders fine while the child kubevirt HelmRelease then stops on the guard: fail-closed, with the gate named, but diagnosed one level below where it was configured. And the version-conditional opt-out is evaluated at render time, so a cluster upgraded from 1.34 to 1.35 keeps `ImageVolume` disabled until something re-renders the chart; the direction is the safe one, since the gate stays off and containerDisk keeps the path it had on 1.8.4. Two things I decided against doing here. The guard's copy of the gate list is pinned against a name being dropped, not against a name being added to the rendered list only; the copy exists because `make update` needs the literal `- VMExport` anchor, so closing the other direction means changing the self-heal contract. And `hack/run-kubernetes-block-io_test.bats` still names a `virt-launcher:v1.8.4` image in a fixture: inert, the test matches on the container name, and the file is outside this change. Tested on a three-node Talos v1.13.6 / Kubernetes v1.36.2 cluster with Flux suspended. The upgrade took about ten minutes. virt-operator had already deployed virt-template before the CR change landed and removed it afterwards, CRDs included. Two VMs started on the 1.8.4 launcher, one DataVolume-backed and one containerDisk-backed, live-migrated to the 1.9.0 launcher. VMDisk plus VMInstance through the apps API boots. `addvolume` on the VM subresource reaches Ready, `removevolume` on the VMI subresource is accepted. VirtualMachineSnapshot and VirtualMachineExport from it reach Ready. Restore from snapshot and PVC clone failed on that cluster inside the LINSTOR API layer, unrelated to this change. CDI v1.66.0 from #3920 on top of it: HTTP imports to Block and Filesystem, blank population and hotplug all work. `helm unittest` is 26/26 for the kubevirt chart and 142/142 for the platform chart, and the update idempotency script passes its eleven cases. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. No app added or renamed, no variant, release asset, `ApplicationDefinition` or tooling contract touched, and none of the `packages/core/platform/values.yaml` keys the ansible role sets are in the diff. One repository is reached. `packages/core/platform/values.yaml` gains a top-level key, `kubevirt.disabledFeatureGates`, defaulting to the empty list, and `operations/configuration/platform-package.md` on the website is a hand-written table of `spec.components.platform.values.*` that nothing generates. The row is in cozystack/website#678, which lands after this PR merges. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/678 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note chore(kubevirt): update KubeVirt to v1.9.0. Beta feature gates are on by default upstream now. The platform keeps `Template` off, which would deploy virt-template, and `ExternalNetResourceInjection`, which would hand `resourceName` injection for NetworkAttachmentDefinitions to an injector the platform does not run. `ImageVolume` is disabled on management clusters below Kubernetes 1.35: containerDisk-backed VMs are rendered with native image volumes when it is on, and an apiserver whose own ImageVolume gate is off, the default before 1.35, drops those volumes silently, leaving the VM without its disk. On 1.35 and newer the gate stands, and that is the first release where containerDisk-backed VMs are rendered with image volumes instead of sidecars; the bundled Talos ships Kubernetes 1.36, so this is the default there. A `kubevirt.disabledFeatureGates` list in the platform values opts out of any other gate a cluster cannot carry. ``` |
||
|
Aleksei Sviridkin
|
3a369fed99 |
feat(redis): add opt-in TLS for Redis and Sentinel (#2729)
## What this PR does `tls.enabled` puts Redis and Sentinel on a TLS-only listener with a per-release cert-manager chain, and hands the tenant the CA certificate as `redis-<name>.tenant-ca`. Off unless set. redis-operator v3.3.5 has no TLS support. The work is ported upstream in freshworks-oss/redis-operator#85, which is open and not in a release, so this carries it as three patches over the v3.3.5 tarball. `images/redis-operator/patches/tls.diff` is the source half, cut from that PR's commit `2ad2ed8d1`, which the patch header pins; it is applied by the glob the image Dockerfile already runs, after the Dockerfile strips the tarball's `manifests`, `charts`, `example` and `.github`, so the image builds from source alone and the upstream tests that read the CRD YAML skip instead of failing against the stale v3.3.5 copy. `patches/crd-tls.diff` adds `spec.tls` to the vendored CRD, and with it the CEL rule that fixes `tls.enabled` at creation; without the schema block the apiserver prunes the field on admission and the patched binary waits for something it never sees, and the rule cannot exist without the block it references. `patches/rbac-tls.diff` grants Create, Get and Update on Secrets, so the operator can publish the CA-only Secret it addresses by name. That is the whole delta, and it is narrower than the ported operator's own chart asks for: those three are the only Secret calls in the patched source, and cert-manager Certificates get no grant at all, because the operator only touches one under `spec.tls.certManager` and this chart pins `certificateSecret` mode instead. One change in `tls.diff` is not TLS and takes effect with TLS off, named in the patch header: `MakeSlaveOfWithPort` connects to the replica on its own port rather than on the master's. It changes a dial, not a pod template, so shipping the rebuilt image rolls nothing; the plaintext pod templates stay byte-identical to v3.3.5. The image build registration this branch used to carry (`make -C packages/system/redis-operator image` in the root `build:` target) landed on main with the Valkey service (#3380), so the rebuilt operator image is produced by CI from main as well; the digest in `values.yaml` is still refreshed by release preparation, not here. The chart patches re-apply at the end of `update:`, the same way reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry theirs. Otherwise `rm -rf charts` takes TLS back out on the next `make update` and nothing fails. `make update` reproduces the committed tree byte for byte. TLS is chosen when the instance is created. The config gets `port 0` and the listener moves to `tls-port`, so TLS replaces plaintext rather than running beside it, and a running instance cannot change mode: a replica restarted onto the TLS listener cannot replicate from a master still on plaintext, and Sentinels already on TLS would fail over to a stale replica. The RedisFailover schema in the carried CRD patch rejects a change of `tls.enabled`, which on an existing instance surfaces as a failed HelmRelease upgrade while the instance keeps serving what it did; the README points at creating a new instance and migrating. That is also why TLS is not inferred from `external`: an externally reachable instance is the one most likely to have clients an upgrade must not drop. Existing releases keep their plaintext listener and are not restarted by the rebuilt operator image. The tenant gets `ca.crt` and nothing else, through `tenantsecrets`. The chart's own Secrets take a dot between the release and the suffix (`<release>.ca-tls`, `<release>.tls`, `<release>.ca-cert`): the suffixes are dot-free, so the split is unique and release `foo`'s CA Secret can no longer be release `foo-ca`'s leaf, the collision the hyphenated form had. The chart renders a `TenantProjection` pointing at `<release>.ca-tls`, and the CA-extraction controller re-encodes the one named key from the parsed certificate, so nothing that is not a certificate gets through. Same path as postgres, mongodb, nats and qdrant. No Secret of this release is granted to the tenant by name beyond the generated password, and that includes `<release>.ca-cert`, the CA-only copy the operator publishes for itself. It holds no key today, but the operator has Create and Update on Secrets cluster-wide, so what sits at that name is the operator's call rather than the platform's, and RBAC cannot filter by key. `hack/check-redis-rd-secrets.bats` pins the whole rendered Secret surface instead of a list of known-bad names, because an include with no `resourceNames` and no `matchLabels` restricts nothing. The chainsaw suite for redis gains two cases. A TLS fixture asserts the cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel come up, the operator publishes its CA-only Secret under the widened ClusterRole, the tenant receives exactly `ca.crt` through `tenantsecrets` while the key-bearing Secrets stay off that path, and a `redis-cli --tls` PING through the service name completes with the same CA. A second case applies `tls.enabled` to a running plaintext release and asserts the refusal: the HelmRelease upgrade fails with the schema's message, the StatefulSet keeps its revision with both replicas Ready, plaintext still answers and no TLS listener appears. Certificate renewal rolls the pods from inside the operator. Redis and Sentinel read the certificate once at startup, so the patched operator stamps a hash of the TLS Secret's `tls.crt` and `ca.crt` into the pod template of both the Redis StatefulSet and the Sentinel Deployment on every reconcile, as `redis-failover.freshworks.com/tls-secret-hash`. A renewal changes the hash; the operator's own roller replaces the Redis pods one at a time and the Deployment controller rolls Sentinel. README names the fallback: if the roll cannot run, the deadline is the old certificate's expiry, past which the operator's own TLS connections fail along with the clients. `tls.authClients` maps to `tls-auth-clients` and defaults to `no`. The platform mints no client certificate for the tenant and the CA key is out of reach, so `yes` is only usable when the client certificate comes from whoever also controls the CA. Changing the value on a running release restarts the Redis and Sentinel pods: Redis reads the directive from its config file only at startup, so the operator folds the effective value into the certificate hash that drives the rollout, and the change lands when it is made rather than at the next unrelated restart. Follow-ups in the carried operator patch go upstream first, to freshworks-oss/redis-operator#85 or a successor there, and are picked up here afterwards; the carried patch does not grow Go that upstream has not seen. `tlsConfigFor` fetches the Secret on every call, tens of GETs per reconcile. The renewal roll is static analysis so far: the live check, `cmctl renew` on a running release while watching the `rfr-*` and `rfs-*` pods roll, is still owed and belongs with the e2e follow-up. Two facts about the shipped image rather than the patch. `packages/system/redis-operator/values.yaml` keeps the digest of the image built before the patch until release preparation rebuilds it, so on `main` between this merge and the next release-prep, `tls.enabled` renders the chain and `spec.tls` while the running operator ignores them and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026 in `golang.org/x/net` v0.53.0, reached through `PodService.UpdatePodLabels`; the port neither introduces nor fixes it, upstream `master` is already on v0.55.0, so it lands with the next tag bump rather than here. Outside the patch: `cozyvalues-gen` renders `{}` in the README Value column for `tls.authClients`: a child property without a default inherits the parent object's `{}`, and `make generate` reproduces it byte for byte, so it is a generator bug rather than something to hand-edit here. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#37 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: I walked the trigger map against the diff file by file. terraform-provider-cozystack is reached, tracked as cozystack/terraform-provider-cozystack#37 rather than a speculative PR: `values.schema.json` gains a `tls` object, and the provider hand-writes a schema, model and expand/flatten pair per Kind. It already models `tls` for postgres, kafka, nats and qdrant, not for redis. Redis needs different wording from those four, whose descriptions say to omit the block to follow `external`, which is the one thing redis does not do. Nothing else is reached. The website regenerates this package's reference page from `README.md` on a stable tag, so the README changes need no PR there. No package added, renamed or removed. `packages/core/platform/values.yaml`, variant and bundle composition and release asset names are untouched, and so are `hack/package.mk`, `hack/common-envs.mk` and `hack/update-crd.sh`, which is what ccp and external-apps-example read. The two new files under `hack/` are bats unit tests. No node prerequisite, installer value, telemetry metric or cozy-proxy label changes. ### Release note ```release-note feat(redis): add opt-in TLS for Redis and Sentinel. Set `tls.enabled: true` to serve both on a TLS-only listener backed by a per-release cert-manager chain; the tenant reads the CA certificate as the `<release>.tenant-ca` tenant secret. TLS is off unless set and is not inferred from `external`, so existing releases keep serving plaintext and are not restarted by the rebuilt operator image. TLS is chosen at creation: a running instance cannot switch modes, the change is rejected and surfaces as a failed HelmRelease upgrade, so add TLS by creating a new instance and migrating to it. Requires cert-manager, which the platform's `system` bundle installs on every variant. A renewed certificate rolls the Redis and Sentinel pods through the operator. TLS support is carried as a downstream patch over redis-operator v3.3.5 until freshworks-oss/redis-operator#85 lands in a release. ``` |
||
|
Aleksei Sviridkin
|
8adf3a0863 |
feat(platform): forward kubevirt.disabledFeatureGates to the kubevirt CR
The kubevirt chart reads disabledFeatureGates from its own values, and the iaas bundle is the only writer of the cozystack.kubevirt Package, so nothing could set it: a Package patched by hand is reverted the next time the platform re-renders it. Platform values gain a documented kubevirt.disabledFeatureGates list, forwarded into the emitted Package as components.kubevirt.values.disabledFeatureGates alongside the entries derived from resources.cpuAllocationRatio and the gpu block. The parenthesized value walk is nil-safe, matching the gpu block, so a cluster that nulls out the whole kubevirt key still renders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5976ddf02f |
fix(gateway): normalise the GatewayClass name and refuse a blank one
The class lists arrive over the values channel comma-joined, so every reader trims each entry as it splits them. The scalar those entries are compared against was not trimmed anywhere. A class name carrying a stray space therefore failed `has` against its own padded self, and the two readers of that scalar disagreed about the same class: the gateway chart put the tenant on an edge-terminated class, while the cluster-wide ACME issuers still pinned their solver to a listener an edge Gateway does not render. That stops every HTTP-01 certificate in the cluster from issuing, with nothing in any status naming the cause. Normalise once at the single writer so both readers see a clean value, and keep a defensive trim at each reader. The tenant's own choice does not travel that channel — it comes from tenant.spec.gatewayClass — so it is trimmed where the gateway chart reads it. Unset and whitespace-only are different events and no longer share an outcome. Unset means the choice was not made, and falling back to the bundled class stays its documented behaviour. Whitespace-only means a choice was made and is garbage, in the field that decides where TLS terminates: resolving it quietly would take a tenant the operator meant to put behind an edge terminator and publish it directly, with a real certificate on a real address. Both the platform default and the tenant's own choice now refuse it by name. The kind check beside that refusal is part of the same fix, not a tidy up. Through Helm's values path an unquoted numeric scalar arrives as a float64, and the whitespace check compares it against a string, so `className: 123` dies at that comparison — one line before the trim — with an error naming neither the key nor the file. Refusing whitespace without checking the kind first would therefore have made the numeric case worse than it was. `kindIs "string"` is the right test because it never has to name the type that actually arrived. The order of the guard's steps is part of the guard. The value is read before anything normalises it, so a default that fires on any falsy value cannot launder 0 or false into the empty string and resolve them to the bundled class ahead of the kind check. Unset is the one case that still falls back, and each of those four inputs is pinned, so a normalisation drifting back ahead of a check fails rather than quietly covering only the loudest bad input. Also correct a cleanup log line: the wildcard Certificate is deleted for every mode that is not DNS-01, so naming HTTP-01 in that message is wrong for existingSecret and for edge alike. And state the listener name invariant plainly: the cap is 253, not the 63 of a DNS label, and the longest name the builders produce is 84. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
487cafac3e |
feat(platform): let a tenant pick its Gateway class from an allowlist
Tenant Gateways were pinned to the bundled Cilium class. Give the platform a default in gateway.className, the set a tenant may choose from in gateway.tenantSelectableClasses, and the set whose provider terminates TLS upstream of the Gateway in gateway.edgeTerminatedClasses. A tenant picks its own with tenant.spec.gatewayClass, read only when it owns a Gateway; an inheriting tenant runs on its ancestor's. The selectable set is an allowlist rather than free choice because Tenant is a tenant-writable object while a GatewayClass is cluster-scoped: without it a tenant could attach its Gateway to a class an admin keeps for internal-only traffic. Naming anything outside the set fails that tenant's own gateway release and reaches no other tenant. Where TLS ends is a property of the class, so the gateway chart derives certMode from whichever class the tenant landed on instead of taking it as a separate input. The two therefore cannot be configured out of step. The publishing tenant cannot cross that line by itself. The cluster-wide ACME issuers point their HTTP-01 solver at its Gateway and are rendered from the platform default alone, so a mismatch would pin the solver to a listener that Gateway does not have. Those issuers now drop the sectionName pin when the default class is edge-terminated, letting the challenge attach by hostname instead. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Andrey Kolkov
|
e94fcf8219 |
feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990)
## What this PR does Adds backup/restore support for the managed `RabbitMQ` application. RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no consistent online export of message payloads, so this backs up broker **definitions** (vhosts, users, permissions, queues, exchanges, bindings, policies, parameters) over the management HTTP API (`GET`/`POST /api/definitions`) and stores them in the system bucket. Message payloads and point-in-time recovery are intentionally out of scope; a full-data backup would be a Velero volume snapshot, which is a separate strategy. It is delivered as a **first-class `strategy.backups.cozystack.io/Rabbitmq` driver**, following the Altinity pattern (a dedicated driver that renders a one-shot Job for an HTTP-API engine): - `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's `PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`, pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a `batch/v1.Job`, covering both restore flows: in-place (into the source broker) and to-copy (into a separate, differently-named RabbitMQ in the same namespace, authenticating with the target's own default-user credentials and reading the source's object key). - A **Go-read precondition** gates each run on the `RabbitmqCluster` reporting `AllReplicasReady` (read through a small `rabbitmqtypes` partial type), so a not-yet-ready broker surfaces a precise `Ready=False` wait on the BackupJob/RestoreJob rather than a Job that starts and fails against an unreachable management API. - The produced `Backup` records the stored object's location on `status.artifact.uri`, rendered from the strategy's `spec.artifactURITemplate`. The `Backup` CRD has no status subresource, so this persists on Create, mirroring the Etcd driver. Dispatch is wired in both controllers (plus `supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job itself is already covered by the existing `batch/jobs` grant). It ships `cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default` BackupClass, which also enrols it in the `DefaultObjectsGate`. The RabbitMQ application chart is left untouched — it had no prior backup wiring, and the strategy carries the S3 coordinates itself. Unit tests cover the driver's pure helpers (applicationRef validation, cluster-name derivation, the readiness-message builder, artifact-URI rendering, parameter round-trip) and the reconcile precondition gate (defers with `Ready=False` and no Job until the cluster is ready, then proceeds). A runnable example under `examples/backups/rabbitmq/` and an honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a sentinel vhost, back it up, drop it, and assert it round-trips through object storage on both restore flows. The Job pod reuses the existing `platform-migrations` image (curl + ca-certificates) under a new `backupStrategyController.rabbitmqBackupClientImage` key, stamped in lockstep with `chBackupClientImage` in `packages/core/platform/Makefile` so the repository stays pinned at a single digest (`hack/image-pin-consistency.bats`). A dedicated slim image built by this package is a follow-up. **Validation:** `go build`/`go vet`/`gofmt`/`go test ./internal/backupcontroller/... ./internal/template/...`, deepcopy + Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate both on and off, `helm lint`, and `helm unittest` (18/18). The backup + in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a real management-API export and a real S3 `--aws-sigv4` upload/download against SeaweedFS), so the driver's core contract is exercised on a real cluster; the in-CI Chainsaw run is the remaining automated evidence. The object-cleanup, terminating-namespace teardown and readiness-deadline paths added during review are unit-tested (including `Reconcile`-level finalizer retention and the terminating-namespace release) but not yet cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled on the `nano` preset, so the example/e2e use `c1.small`.) ### Screenshots Not applicable — no UI changes. ### Downstream repositories Walked the trigger map against the diff. This is an additive backup strategy plus a new strategy CRD; no downstream repository is mechanically forced by the change. A documentation page for RabbitMQ backup on [cozystack/website](https://github.com/cozystack/website) would be a reasonable follow-up, but it is not required by this diff — leaving the boxes unticked for a human to decide rather than ticking "no downstream" to clear the checklist. ### Release note ```release-note feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added RabbitMQ backup and restore support for definitions, including vhosts, users, queues, exchanges, bindings, policies, and related settings. * Added in-place and restore-to-copy workflows with readiness checks. * Added scheduled and on-demand backup examples. * Integrated RabbitMQ backups with the default backup configuration. * Backups store definitions only; message payloads and point-in-time recovery are not included. * **Documentation** * Added a complete RabbitMQ backup and restore demonstration guide. * **Tests** * Added end-to-end coverage for readiness and backup/restore round trips. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
f5206a6e28 |
docs(redis): declare the TLS-on-upgrade break and fix the parameter sections
TLS replaces plaintext in this operator rather than running beside it — with TLS on, Redis and Sentinel are configured with port 0 and serve only on the TLS port. Since tls.enabled falls back to external, an already deployed instance with external: true switches to TLS-only on its first reconcile after the upgrade and every plaintext client breaks. That was documented as a feature and nowhere as a change of behaviour for existing instances. State it as a breaking change and name tls.enabled: false as the opt-out. Also spell out which hostname the certificate actually covers. The SAN list carries the four service names and the external tenant hostname, and its only IP entries are the loopback addresses used by the in-pod probes and the metrics sidecar, so connecting to the external LoadBalancer by IP fails for clients that verify the hostname. The TLS section marker sat above the version parameter, which put version in the TLS table in the generated README instead of Common parameters. Move the marker below it. Regenerating also reorders version ahead of tls in keysOrder, matching how postgres orders the same two fields. Pin the PackageSource CRD replacement policy the way the mongodb-operator one is pinned. The RedisFailover CRD gained spec.tls with the fork, and Helm never upgrades CRDs shipped in crds/, so losing CreateReplace would leave the live CRD behind and the apiserver would strip spec.tls from every RedisFailover the chart renders — TLS silently off, no error. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrey KolkovandClaude Opus 4.8
|
ba32df69c0 |
feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.
Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):
- reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
--aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
- A Go-read precondition gates each run on the RabbitmqCluster reporting
AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
precise Ready=False wait instead of a Job that starts and fails.
- The produced Backup records the object location on status.artifact.uri
(rendered from spec.artifactURITemplate; the Backup CRD has no status
subresource, so it persists on Create, as in the Etcd driver).
Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.
Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.
The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrei Kvapil
|
53412c3033 |
fix(valkey): add cozystack-engine dependsOn edge to the PackageSource
The valkey-application PackageSource ships system/valkey-rd (an ApplicationDefinition); it must dependOn cozystack.cozystack-engine so the ApplicationDefinition CRD is registered before the -rd HelmRelease reconciles. Mirrors redis-application.yaml and satisfies check-applicationdefinition-crd-ordering. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
4e6313b608 |
feat(valkey): register Valkey application in the platform
Wire the managed Valkey app into the platform: add the valkey-rd package carrying the Valkey ApplicationDefinition, the valkey-application PackageSource (reusing the existing redis-operator), enable it in the paas bundle, and grant tenant roles access to the valkeys resource. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Timofei Larkin
|
f5b957ee27 |
feat(kubernetes)!: manage worker pools as KubernetesNodes, adopt on upgrade (Phase 2, breaking) (#3315)
## What this PR does Phase 2 of the Kubernetes-app split ([cozystack/community#8](https://github.com/cozystack/community/pull/8)) — the **breaking** half. Stacked on #3314 (review that first). Removes worker node pools from the `kubernetes` chart (they moved to `kubernetes-nodes` in #3314) and adopts existing pools on upgrade: - `kubernetes` chart is now **control-plane only**: drops the pool `range` block, the `talos-reconcile` Job + RBAC (deleted; moved to `kubernetes-nodes`), the preserve-old-KMT block, the `kubernetes.nodeGroups` helper, and the per-group dashboard RBAC. - **`spec.nodeGroups` is removed from the `Kubernetes` CR** (BREAKING). Worker pools are now separate `KubernetesNodes` resources. - **Migration 54** (bumps `targetVersion` to 55): pre-upgrade hook that, for every `kubernetes-<cluster>` release, creates one `kubernetes-nodes-<cluster>-<pool>` HelmRelease and re-annotates the existing worker objects onto it with `helm.sh/resource-policy=keep`, so the now control-plane-only parent upgrade does not prune running workers. The default `md0` (implicit when `nodeGroups` was empty) is materialised explicitly. Idempotent. Genuine infrastructure/read failures fail closed (`exit 1`, retried on the next upgrade); hostile stored shapes (a pool name over Helm's 53-char release-name cap, an RFC-1123-invalid or whitespace/slash map key, a non-object `nodeGroups`) are warned, pinned, and skipped rather than deadlocking the entire fleet's upgrade. - The `ingressNginx` addon no longer auto-provisions a node (the implicit `md0` default is gone). If enabled without a pool carrying `roles: [ingress-nginx]`, the controller DaemonSet simply schedules nothing (degraded, not failed) — the requirement is documented in the chart NOTES.txt rather than enforced at render time. Because the child chart renders the pool objects **byte-identically** to the old parent render (golden parity in #3314; the KubevirtMachineTemplate content-hash name is preserved), adoption does not churn live worker VMs. ### Migration number `54` is contiguous with main (latest is `53`) and `targetVersion` is `55`. The `52`/#3201 collision noted in earlier revisions has been resolved: this branch was rebased and merged past it, so the numbering is contiguous and `run-migrations.sh` has no gap to hard-fail on. ### Not yet done (blocking draft → ready) - [ ] **dev e2e upgrade validation** (`cozystack-pr-test`): upgrade a custom-`nodeGroups` cluster and a default-`md0` cluster; assert zero worker-VM churn (KMT hash, Machine uid/creationTimestamp stable) and that the adopted child HRs reconcile. - [ ] downstream trigger-map walk (website docs for the `nodeGroups` → `KubernetesNodes` change; terraform-provider for the removed API field). ### Downstream repositories Deliberately unticked (draft) — the breaking API change affects docs and the terraform provider; follow-ups to be filed before ready. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: ```release-note Worker node pools of a managed Kubernetes cluster are now managed as separate `KubernetesNodes` resources. `spec.nodeGroups` on the `Kubernetes` CR is removed; existing pools are adopted automatically on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Worker pools are now separate `KubernetesNodes` resources with their own lifecycle. * Added a render-time compatibility guard to prevent worker pools from running ahead of the parent control-plane minor version. * Improved upgrade/uninstall behavior to better preserve already-adopted worker objects. * **Breaking Changes** * Removed per-pool `nodeGroups` and worker node health settings from the `Kubernetes` chart/CR; pool configuration now lives in `KubernetesNodes`. * Ingress requires at least one `KubernetesNodes` pool with `roles: [ingress-nginx]`. * `cluster` and `storageClass` are immutable after creation; pool `version` may lag but must not be ahead of the parent minor. * **Documentation** * Updated READMEs, notes, and schemas to reflect the `KubernetesNodes` split and the immutability/version rules. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
32ec1b5a00 |
fix(migrations): correct unadopted-record count semantics and stale comments
- The unadopted-record WARNING said "N pool(s)" but counted SKIPPED_FILE lines, and the foreign-owner branch writes one line per object (so a single pool with MD+MHC+WM reported "3 pools"). Reword to "N record(s) ... one line per object or pool ... not a distinct-pool count" so the number does not mislead. - The inline-chart restamp comment implied the cozystack-apps source is enforced; it is not (matching by chart name only is a deliberate trade-off: gating on the historical sourceRef risks under-matching a real legacy parent, which loses its workers, whereas the loose match's worst case is a harmless mislabel). Spell out the trade-off and that the render guard is the data-loss backstop. - The overflow comment cited "up to 42 chars per rest.go"; the enforced cap is 32 (maxKubernetesClusterName = 53 - 17 - 4). Correct it and note legacy/hand-created names can still be longer, which is why the overflow branch exists. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
c0fcdd767c |
fix(migrations): discover pre-chartRef parents via inline spec.chart in the restamp
The restamp discovered unlabelled parent kubernetes releases by spec.chartRef only, but that misses the population it exists for. The apps.cozystack.io/ application.kind label was introduced 2025-12-19 ( |
||
|
IvanHunters
|
c6b6755eec |
fix(migrations): record foreign-owner skips, harden record_skip, scope restamp to kubernetes- names
Three hardening fixes on migration 56's new code: - The foreign-owner branch in adopt_one pins the object prune-proof but does not adopt it (the child release can then stay Failed on the ownership conflict), yet it was the one warn+pin+skip path that did not record itself in the durable unadopted ConfigMap -- so an operator relying on that record to find stuck pools would miss it. Record it like every sibling branch. - record_skip ran a bare append under set -e inside skip branches that are designed never to abort; a failed write (e.g. /tmp full) could deadlock the fleet upgrade for one tenant. Make it fail-open, matching the branches' intent. - The restamp selector matched any release by chartRef; scope it to names starting with kubernetes- so it cannot mislabel (and then have the sweep skip) a non-standard release, keeping restamp and the sweep's contract in sync. Also fix the previously-vacuous negative restamp test (its fixture lacked the chartRef, so it never reached the label guard it claimed to check). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
3d1a83f419 |
fix(migrations): make the unadopted-pools record publish best-effort
The durable-record ConfigMap write sat under set -euo pipefail right before the version stamp, so a permanent failure (e.g. the record exceeding etcd's object size limit at a very large skip count) would abort the migration and deadlock the fleet upgrade with no path out -- even though every skipped pool was already pinned prune-proof earlier in the sweep, so the workers are safe regardless of whether the record lands. The record is a convenience artifact, not a safety mechanism, so swallow its failure with a warning instead of failing closed. Covered by a bats case that fails the publish and asserts the run still stamps. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
952901776a |
fix(migrations): adopt unlabelled parent kubernetes releases and record unadopted pools in migration 56
The worker-adoption sweep selected parents by the apps.cozystack.io/ application.kind label, which postdates v1.2. A parent kubernetes HelmRelease older than that, or one restored by hand, lacks the label: the sweep skipped it, stamped, and the parent's control-plane-only re-render then pruned its un-pinned workers with no pin and no warning. Discover such releases by their chartRef instead and restamp the three application labels before the sweep (additive, cannot corrupt a release, and repairs it for cozystack-api too). Every warn+pin+skip branch leaves a pool pinned prune-proof but unadopted, recorded only in the Job log, which the next upgrade reaps; the migration does not re-run, so an operator cannot find those pools afterward. Accumulate the skips and publish them as configmap/cozystack-migration-56-unadopted before the stamp. Extend the fake kubectl and add bats coverage for the restamp (positive and negative) and the unadopted-ConfigMap paths. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
5b08d48632 |
docs(migrations): pin DEFAULT_MD0 provenance to the pre-removal helper commit
The kubernetes.nodeGroups helper that DEFAULT_MD0 reproduces was deleted in this line of work, so nothing in-tree diffs against it and a future edit to the constant (instanceType/diskSize/...) would silently rename the content-hashed KubevirtMachineTemplate and roll every implicit-md0 pool. Reference the pre-removal definition (git show 92b9785da^:.../kubernetes/templates/_helpers.tpl) so the canonical default stays diffable. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
d93e136e23 |
fix(computeplane): provision worker pools via kubernetes-nodes after Phase 2 split
Phase 2 moved worker nodeGroups out of apps/kubernetes into apps/kubernetes-nodes, but the ComputePlane module still passed nodeGroups to the wrapped kubernetes chart, which now ignores it: a fresh ComputePlane cluster came up with zero workers (and the pinned ingress-nginx addon stuck Pending), while existing ones were unreachable by migration 56 (it only adopts kubernetes-* releases, not computeplane-cluster). The module now emits one kubernetes-nodes child HelmRelease per nodeGroup, shaped like a natively created pool, and materialises the pre-split default md0 (roles: [ingress-nginx]) when nodeGroups is empty. The pool chart's clusterName gains an internal clusterReleaseName override so pools attach to the module's fixed computeplane-cluster release, which is off the aggregated API's kubernetes-<cluster> convention; the default keeps every API-created pool byte-identical. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
9ccb133b66 |
fix(ci): de-escalate select-e2e and name every escalation (#3817)
## Why
I replayed the 150 most recently merged pull requests through
`hack/select-e2e.sh` with every escalation branch instrumented. 118 of
them (78.7%) ran the full 21-suite Chainsaw run, and the causes, counted
once per escalating PR with the count where that cause was the only one
in brackets, were `FULL_PATTERN` 88 [68], `NO_SUITE_FOR_GROUP` 24 [24],
`CHAINSAW_SHARED` 18 [1] and `UNCLASSIFIED` 7 [4]. `YQ_BROKEN`,
`NO_GRAPH_OWNER` and `BACKSTOP_EMPTY` never fired.
Two problems come out of that. Most of the escalation is real but
unnecessary. Whole classes of path that provably cannot affect a
Chainsaw suite were escalating because no rule claimed them. And the
commonest cause of a full run was absent from the log, so "why did this
pull request run everything" could only be answered by re-deriving the
selection by hand.
## What changes
**Every escalation now names its cause on stderr.** Seven branches could
reach the full suite and four did it in silence, `full_suite_pattern`
among them, the commonest cause by a wide margin, so the usual answer
was the one the log never carried. The lines go to stderr and must stay
there: stdout is the suite list and both e2e lanes parse it, so a reason
line on stdout would be read as a suite name. A unit test asserts every
message, verified by muting each of the eleven in turn and confirming
the file goes red for all of them. A reason that regresses to silence
changes no selection and is otherwise invisible.
**`cozystack.etcd-application` gains its missing `dependsOn:
cozystack.etcd-operator`.** `packages/extra/etcd` renders `kind:
EtcdCluster` from `etcd-operator.cozystack.io/v1alpha2`, and this was
the only operator-backed application source in the tree with no edge to
its operator. It is a latent install bug as much as a selection gap:
without the edge `etcd-rd` registers the ApplicationDefinition as soon
as the engine is up, so a tenant can create an `Etcd` before the
operator exists and its HelmRelease fails on `no matches for kind
"EtcdCluster"`, and `hack/select-install.sh`'s forward closure for the
`etcd` suite omitted the operator for the same reason. On the selection
side `cozystack.etcd-operator` reached no runnable suite, so every
change to the operator ran all 21.
**`vm-disk-application` maps to the `vminstance` suite.** It resolved to
a `vm-disk` suite that does not exist, `intersect_suites()` dropped the
name, and the source reached nothing runnable. Neither did
`cozystack.kubevirt-cdi` above it, whose only other dependents are
`vm-default-images` and `vm-disk-application` itself, so every CDI
change ran all 21. The coverage was there and the table did not know it:
the `vminstance` suite creates a `VMDisk` and asserts the `DataVolume`
behind it (`hack/e2e-chainsaw/vminstance/vmdisk.yaml`,
`vmdisk-vmi.yaml`). Same defect class as the etcd edge above, on the
mapping side rather than the graph side, and with no production blast
radius of its own, since `src_to_suites` is read by the selector and by
nothing else.
**`cozystack.cozystack-basics` joins `cozystack.cozystack-engine` as a
propagation hub excluded from the reverse-dependency walk.** Every edge
into it exists so a namespace or a platform-wide policy is in place
before the dependent installs, which `kubevirt-cdi` states in its own
source ("Depend on cozystack-basics so the target namespace exists
first"), and that is install ordering rather than behaviour. The damage
runs opposite to the engine's: the engine fans one change out to every
app, basics narrows instead. It reaches no suite today so it escalates,
but it sits upstream of `kubevirt-cdi`, so the first suite to land under
CDI silently converts the platform's namespace-and-policy package from
the full run to that one suite. Not hypothetical — on #3426 enabling the
site-router suite takes `cozystack-basics` from 22 suites to
`site-router` alone, and nothing in the output says coverage was lost.
Dropping the reverse edges keeps the package reachable, stops it
propagating, and leaves a change to it running everything through the
per-path escalation.
**`hack/e2e-apps/<name>.bats` maps to the `<name>` suite**, off the
basename, exactly as the per-suite rule takes the name off a
`hack/e2e-chainsaw/<app>/` directory. Deliberately mapped rather than
marked inert: what remains there is wired to nothing after the Chainsaw
migration, and inert would bake that orphan status into the rule and go
quietly wrong the day a lane runs those files again. The basename is
membership-tested against the suite list on the spot, so an unmatched
one escalates there and then rather than having the verdict deferred to
the final intersection, where the rest of the diff would decide it.
**`packages/tests/` and three `.gitattributes` join
`inert_config_pattern`,** which is what the script's own header says to
do with a genuinely inert path instead of widening the fall-through.
`packages/tests/` is a helm-unittest fixture chart, and changing a test
*of* `cozy-lib` does not change `cozy-lib`, no PackageSource lists those
paths as a component, and nothing installs them, while a change to the
library itself still escalates through `packages/library/`. The
`.gitattributes` entries are enumerated rather than matched by filename,
because the justification is what those files contain (only
`linguist-generated` markers) and the name does not carry it:
`.gitattributes` can also set `filter`, `eol`, `working-tree-encoding`
and `export-subst`, each of which changes what lands in the working tree
and therefore what gets built.
**`full_suite_pattern` escalates only `hack/e2e-*.bats`, not every
`hack/*.bats`.** The root `Makefile` is the authority on the split and
draws it at exactly that prefix, `BATS_UNIT_FILES := $(filter-out
hack/e2e-%.bats,$(wildcard hack/*.bats))`, so the 60 files it keeps are
the unit lane and the e2e sandbox runs none of them, while the three it
filters out are what `packages/core/testing`'s recipes execute and those
still escalate. Being inert here does not leave them untested: `make
unit-tests` is gated on the `plan` job's `code` output, which
`pull-requests.yaml` computes as "any changed path outside `docs/`" and
never from `select-e2e.sh`, so a bats-only pull request still runs the
whole unit lane, plus install and the OpenAPI tests, since the `e2e` job
reads the same output rather than the selection.
## Measured effect
The before-and-after comparison below is a second measurement over a
slightly different population, stated separately because the two are not
interchangeable: it replays the last 150 first-parent merge commits on
`main` (`git diff --name-only $sha^1 $sha`) through the base selector
with base sources versus this branch with its own, where the histogram
above walks the 150 merged pull requests as GitHub lists them. The
populations overlap heavily and the direction is the same, and the
totals differ by two commits.
| | full suite | scoped | nothing |
|---|---|---|---|
| before | 120 | 20 | 10 |
| after | 106 | 25 | 19 |
Full-suite rate 80.0% → 70.7%, verdict changed on 15 commits. Fourteen
de-escalated and I read every one of their file lists: they are
unit-lane bats files, non-e2e workflows, `docs/`, `CODEOWNERS`,
helm-unittest fixtures and linguist markers. The fifteenth gets one
suite *wider*: a `cert-manager` change now also selects `etcd`, which is
correct, since `cert-manager` is a dependency of `etcd-operator` and the
etcd app now genuinely reaches it through the new edge.
That replay predates the two reachability rules, and replaying the same
150 first-parent merges across those two on their own — the selector at
`73bc4c0ae` against the selector at `adf231513`, same sources on both
sides — changes no verdict at all. Two of the 150 touch
`cozystack-basics` and both ran the full suite before and after, which
is the hub exclusion keeping something true rather than failing to do
anything, and none of the 150 touch `kubevirt-cdi` or `vm-disk`, so the
mapping shows up only in a direct selection.
A few individual selections, before → after:
| changed files | before | after |
|---|---|---|
| `hack/select-e2e_test.bats` | 21 suites | *nothing* |
| `hack/select-e2e_test.bats` +
`packages/apps/redis/templates/redis.yaml` | 21 suites | `redis` |
| `hack/e2e-install-cozystack.bats` | 21 suites | 21 suites |
| `packages/tests/cozy-lib-tests/**` | 21 suites | *nothing* |
| `packages/system/.gitattributes` | 21 suites | *nothing* |
| `hack/e2e-apps/postgres.bats` | 21 suites | `postgres` |
| `packages/system/etcd-operator/values.yaml` | 21 suites | `etcd` |
| `packages/system/kubevirt-cdi/values.yaml` | 21 suites | `vminstance`
|
| `packages/apps/vm-disk/values.yaml` | 21 suites | `vminstance` |
| `packages/system/cozystack-basics/values.yaml` | 21 suites | 21 suites
|
## Blast radius of the `dependsOn` edge
A `dependsOn` on a PackageSource is an install-ordering edge in
production, not only a test-selection hint, so this is the part worth
reviewing hardest. Both Packages are emitted by
`packages/core/platform/templates/bundles/system.yaml` under the single
`bundles.system.enabled` guard with no intervening conditional between
them, in every variant, so no supported configuration has the app
without the operator. `hack/select-install.sh --validate` reports the
graph still has no cycle and nothing dangling. The one new exposure is
an admin who lists `cozystack.etcd-operator` in
`bundles.disabledPackages` while keeping the app, which now leaves it
`DependenciesNotReady`. That is the same exposure postgres, mariadb,
kafka and redis already carry, with `Package.spec.ignoreDependencies` as
the escape hatch. On upgrade `etcd-rd` waits for the operator's two
HelmReleases, and an unhealthy `etcd-operator` already means broken etcd
apps.
## Testing
Every rule added or changed has a test in `hack/select-e2e_test.bats`,
the install-side change has one in `hack/select-install_test.bats`, and
the new graph edge has a helm-unittest guard at
`packages/core/platform/tests/sources_etcd_application_dependson_test.yaml`.
The round-trip test that walks every suite through both mapping tables
stays green, `vminstance` included now that two sources map onto it.
The two reachability rules were each checked by removing the rule and
confirming its own test reddens: without the mapping the CDI test falls
back to the full-suite escalation, and without the hub exclusion
`cozystack-basics` selects `redis vminstance` instead of everything. The
hub test has to seed its own hazard, since no suite sits downstream of
basics in this tree yet, so it gives `redis-application` an edge onto
basics in a copied sources directory and pins that a redis change still
selects redis, or the assertion would pass against a graph that resolved
nothing.
`make bats-unit-tests` aborts on the first failing file (`for f in …; do
… || exit 1; done`), and `hack/ghcr-mirror_test.bats` fails on `main`
today and sorts before `select-e2e_test.bats`, so the aggregate target
never reaches these tests. I ran the 60 unit files individually instead:
59 pass, and the one failure is that pre-existing one, byte-identical to
the base commit and referencing nothing this branch touches. Also green:
`make helm-unit-tests`, `make rd-presets-check migrations-target-check
test-check-readiness`, `hack/select-install.sh --validate`, and
`pre-commit run --all-files` leaving a clean tree.
## Two things a reviewer should know, neither addressed here
`assert_selection` in `hack/select-e2e_test.bats` aborts the whole file
with `exit 1` rather than failing a single test. That is the only thing
that works under `hack/cozytest.sh`, which provides no real `run`,
`$status` or `skip`, and it is the idiom the existing tests in that file
already use. It becomes wrong when #3497 and #3498 flatten these onto
real `bats(1)`, where a failing assert should end one test and let the
rest report. Worth converting with that work rather than ahead of it.
`cozystack.mongodb-application` carries the identical missing operator
edge: `packages/apps/mongodb` renders `psmdb.percona.com/v1` while the
source depends only on `cozystack.networking` and
`cozystack.cozystack-engine`, so `cozystack.mongodb-operator` still
reaches no runnable suite and every change to it escalates to all 21.
Same defect class and same fix shape as the etcd edge, but it is a
second production install-ordering change and belongs in its own pull
request.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- The default etcd application now includes its operator and required
platform dependencies, ensuring the EtcdCluster provider is available.
- **Bug Fixes**
- Improved end-to-end test selection for application-specific changes,
shared components, inert paths, and unresolved or unclassified files.
- Full-suite escalations now provide clear reasons through standard
error.
- **Tests**
- Expanded coverage for test selection, dependency reachability, and
etcd application dependency declarations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
IvanHunters
|
f035da8a0f |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Resolve conflicts introduced by main: - Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in favour of the GPU type main added; main removed the fluxcd addon wholesale. - Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57. - Accept the branch's removal of the in-chart talos-reconcile Job; it moved to the kubernetes-nodes chart, main only touched its comment. - Regenerate kubernetes values.schema.json / README and the kubernetes-rd ApplicationDefinition schema. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Andrei Kvapil
|
851a86dcf1 |
feat(platform): orphan tenant FluxCD HelmReleases on upgrade
Add migration 54 for the removed addons.fluxcd feature. For every tenant that had it enabled, the FluxCD HelmReleases in the management cluster are suspended, stripped of their Flux finalizer and deleted, so the delete does not trigger a Helm uninstall inside the tenant cluster. The Flux release already running in the tenant keeps running untouched. Runs as a pre-upgrade hook, before the addon-less Kubernetes chart reaches the tenant HelmReleases. Mirrors the orphan idiom of migration 29. Bumps migrations.targetVersion to 55. Fail closed. Dropping the finalizer is the only thing that keeps the delete inert, and migrations never re-run, so the script runs under `set -euo pipefail` with no `|| true` on the critical path and gates the delete on the finalizer actually being gone. The verify read uses `--ignore-not-found` rather than `2>/dev/null || true`, so a failed read aborts instead of reading as an empty finalizer list that passes the gate on an object whose state was never checked. A migration that stops is recoverable; one that half-orphans a fleet is not. Cover it with a docker + fake-kubectl unit test running the real script by path inside the migrations image's own alpine base (busybox ash, jq baked on), so it exercises the shipped interpreter and load-bearing pipefail. The suite pins ordering and selection (only the addon HelmReleases are touched, each suspend -> finalizer-drop -> delete in order) and the fail-closed paths (a surviving finalizer, a failing verify read, and a failing fleet scan each abort before deleting and before stamping). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
7cca3d75e7 |
feat(kubernetes)!: remove the FluxCD tenant addon
The optional `addons.fluxcd` toggle of the Kubernetes app deployed a Flux Operator + FluxInstance into a tenant cluster. It was served by two vendored charts, packages/system/fluxcd-operator and packages/system/ fluxcd, which were the last AGPL-3.0 content in this Apache-2.0 repository and a governance blocker for CNCF Incubation. The management cluster does not use these charts (it runs classic Flux via internal/fluxinstall; migration 21 already removed the operator from it), and the addon defaulted to disabled. Rather than maintain an AGPL payload for an unused feature, drop it entirely: - delete both system packages and their platform source entries - remove the addons.fluxcd HelmReleases, values and generated schema from the Kubernetes app - drop the addons.fluxcd block still submitted by the e2e harness - drop the removed fluxcd package entries from CODEOWNERS BREAKING CHANGE: the addons.fluxcd option of the Kubernetes app is removed. Tenant clusters that had it enabled keep their running Flux release; see the companion migration for the upgrade path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
4385990869 |
feat(redis-operator): migrate from archived spotahome to freshworks-oss fork (#3406)
## What this PR does `spotahome/redis-operator` was archived on 2026-06-11 (read-only; last release v1.2.4, 2022). This migrates the operator to the actively-maintained **freshworks-oss/redis-operator** fork (v3.3.5, Apache-2.0). ### Why freshworks-oss It keeps the same `RedisFailover` CRD and API group (`databases.spotahome.com`) — so existing managed Redis is unaffected — and adds a native `spec.engine: Redis|Valkey` switch, letting managed Valkey drop its Redis-compatibility shim image and run the official `valkey/valkey` image directly. ### Changes - Re-vendor the chart from `oci://ghcr.io/freshworks-oss/charts/redis-operator` (v3.3.5), which carries the `engine`-aware CRD. - Build the operator image from freshworks-oss v3.3.5 source. - Re-target the label/annotation-preservation patch onto the freshworks Go module (`github.com/freshworks/redis-operator`). ### Verification - freshworks-oss source + our `labels.diff` patch compiles natively with Go 1.26.3 (`service` and `cmd` packages). - `git apply --check` of the re-targeted patch against v3.3.5 succeeds. - The operator chart renders with the cozystack-registry image override. - The image is built (freshworks-oss v3.3.5 source + our patch, multi-arch amd64/arm64), pushed to `ghcr.io/cozystack/cozystack/redis-operator`, and pinned by digest in `values.yaml`. - `hack/promote-retag_test.bats` and `hack/nightly-mirror_test.bats` pass with the pinned digest. ### Note for maintainers `packages/system/redis-operator` is not part of the automated `make build`; its image is built out-of-band and the digest is pinned in `values.yaml`. That image is already built and pinned in this PR; re-run `make -C packages/system/redis-operator image` only if you bump the operator version. ### Follow-up The cozystack/redis-operator fork carries cert-manager TLS work (cozystack/cozystack#2729) that freshworks-oss does not have yet; the plan is to upstream it to freshworks-oss so it lands in the shared maintained fork rather than a cozystack-only one. ### Release note ```release-note feat(redis-operator): migrate from the archived spotahome/redis-operator to the actively-maintained freshworks-oss/redis-operator fork (same RedisFailover CRD, adds native spec.engine: Redis|Valkey) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for injecting additional environment variables into the Redis Operator (including a default operator group id). * Added default configurable labels for the Redis Operator. * **Bug Fixes** * Preserves and merges existing Service labels and annotations during Redis Operator updates. * **Chores** * Updated the Redis Operator and Helm chart to the Freshworks-maintained 3.3.5 release, aligned chart/image sources, and simplified Helm chart retrieval. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
cd72d81bb4 |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Reconcile the phase-2b split with main's configurable worker-health feature (MHC condition timeouts, maxNodeProvisionTime, podCpuLimit/ podCpuRequest), which landed on both charts after this branch forked: - kubernetes (parent): pool rendering stays removed; main's parent worker-pool additions do not apply post-split. - kubernetes-nodes (pool): absorb the feature; keep the directional parent-version skew guard and the immutable cluster/storageClass rules. KubevirtMachineTemplate content hashes are unchanged (render snapshot). - migration 54: carry podCpuLimit/podCpuRequest (KMT-hash inputs) and the cluster-wide readyUnknownTimeout/readyFalseTimeout/maxNodeProvisionTime into adopted pools, so adoption neither rolls nor resets a tuned pool. Covered by a new adopt-values bats case. - drop the parent chart's obsolete MHC-timeout suites; the kubernetes-nodes twins cover the feature. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Andrei Kvapil
|
c9ee4f6de8 |
test(platform): pin redis-operator upgradeCRDs=CreateReplace
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
c7a3320bb4 |
fix(migrations): pin the engine and fail-close the CRD probe in migration 54
Two defects in migration 54's adoption pass: (1) The label landed before the image pins, so the operator adopted a labelled CR with empty image fields and rolled Redis to the fork's 7.2.4 default — rewriting dump.rdb into a format the later 6.2.6 chart pin can no longer load. Pin spec.redis.image/spec.sentinel.image to redis:6.2.6-alpine in the same pass, only where empty. (2) The CRD probe used a bare `get crd >/dev/null 2>&1` that could not tell NotFound from an API blip and stamped anyway; switch to migration 50's `--ignore-not-found -o name` so a real error aborts under set -e. Also self-contained the renumber note and qualified the resource. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
caf9511365 |
fix(platform): CreateReplace the redis-operator CRDs on upgrade
helm-controller skips CRD upgrades by default, so an upgraded cluster would keep the old spotahome schema and prune the fork's spec.engine field on admission. The schema diff is additive, so CreateReplace is safe. Also refresh the harbor e2e wait comment for the freshworks operator. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
0a3a3bfa77 |
fix(migrations): adopt existing RedisFailovers onto the operator group
The freshworks-oss operator watches RedisFailovers through a server-side operator-group label selector, so a CR without the label is silently dropped from its informer. The chart templates now stamp the label, but an existing CR only gets it when its own HelmRelease next upgrades — after the operator rolls — leaving a window with no healing, and a suspended/stuck HelmRelease never gets it at all. Label every RedisFailover in a pre-upgrade migration, ahead of the new operator. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Myasnikov Daniil
|
42bd45a87c |
fix(platform): make etcd-application depend on etcd-operator
extra/etcd renders kind: EtcdCluster from etcd-operator.cozystack.io/v1alpha2, but cozystack.etcd-application declared no dependsOn edge to cozystack.etcd-operator -- the only operator-backed application source in the tree that did not. The edge has been absent since the sources were first added. Two consequences, one per side of the graph. On install this is a latent ordering bug. Without the edge the etcd-rd HelmRelease registers the ApplicationDefinition as soon as cozystack-engine is up, so a tenant can create an Etcd before the operator exists and its HelmRelease then fails with `no matches for kind "EtcdCluster"`. hack/select-install.sh's forward closure for the etcd suite omitted the operator for the same reason, which would install the suite against a cluster with no CRD to apply to. On test selection cozystack.etcd-operator reached no runnable suite, so every change to the operator escalated to all 21 suites -- 5 of the last 150 merged pull requests. The etcd suite exists; only the edge that makes it reachable from the operator was missing. Scope of the ordering change: both Packages are emitted unconditionally by the same bundle (templates/bundles/system.yaml, under the single bundles.system.enabled guard, in every variant), so no supported configuration has one without the other. An admin who lists cozystack.etcd-operator in bundles.disabledPackages while keeping the app now leaves it DependenciesNotReady -- the same exposure postgres, mariadb, kafka and redis already carry, with Package.spec.ignoreDependencies as the escape hatch. select-install.sh --validate confirms the edge adds no cycle and nothing dangling. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
f9065a9455 |
fix(migrations): correct the non-scalar-field skip remedy
The non-scalar scalar-field skip branch told the operator to restore the field and 're-run adoption', but migration 54 stamps the version unconditionally and does not re-run, so that remedy is a dead end. Align it with the sibling skip branches: restore the scalar, then recreate the pool, and state plainly that the migration has already stamped and will not re-run. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
8815468c9b |
fix(migrations): route non-scalar nodeGroup scalar-fields through warn+pin+skip
Migration 54 copied per-group scalar fields (storageClass, instanceType, diskSize, minReplicas, maxReplicas, logSerialConsole, maxUnhealthy, nodeStartupTimeout) verbatim into the child HelmRelease values. The nodeGroups map value was never fully schema-constrained, so a kubectl/dashboard edit could store one of these as an object or array; copied through, it fails the KubernetesNodes schema on the child install and leaves that child release stuck Failed. Validate the scalar fields before building the child values and route a non-scalar value through the same warn + pin + skip contract as the invalid name, overflow and non-object branches: pin the pool's objects prune-proof and skip adoption, so the pool keeps running unmanaged until the operator restores the stored value. Container-typed fields (roles, resources, gpus, kubelet) are left untouched. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Andrei Kvapil
|
c3cead8f64 |
fix(telemetry): make the documented opt-out cover both collectors
Telemetry is reported by two components, cozystack-operator (cluster facts) and cozystack-controller (application counts), but only the operator had a reachable switch. The documented opt-out sets cozystackOperator.disableTelemetry on the cozy-installer chart, which cannot reach a package the platform deploys, so cozystack-controller kept reporting after an operator had followed the procedure — silently, with nothing to indicate the opt-out was partial. The platform now exposes telemetry.disabled and threads it into the cozystack-controller component of the cozystack-engine Package, the same way the bundle already threads cilium, multus and linstor values. The installer key is also declared in values.yaml: it was only ever referenced from the template, so the documented --set target did not appear in the chart's own values. Both keys stay false by default; this changes no behaviour for anyone who has not opted out. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
IvanHunters
|
9ec5f8ceca |
fix(migration): carry logSerialConsole through worker-pool adoption
Migration 54's adopt pick-list copied every worker NodeGroup field that feeds the content-hashed KubevirtMachineTemplate except logSerialConsole. That field renders into the same hashed template (kubernetes-nodes nodegroup.yaml), so an adopted pool that set logSerialConsole would render a KMT without it, change the content-hash, rename the KMT, and roll every worker VM of the pool on upgrade (and silently drop the setting). Add logSerialConsole to the pick-list (guarded by has(), so pools without it are unaffected) and assert it carries through. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
b4a2f6f791 |
Merge remote-tracking branch 'origin/main' into pr-3315-lgtm
# Conflicts: # api/apps/v1alpha1/kubernetes/types.go # api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go # hack/e2e-chainsaw/_lib/run-kubernetes.sh # packages/apps/kubernetes-nodes/tests/render-parity.sh # packages/apps/kubernetes/README.md # packages/apps/kubernetes/templates/cluster.yaml # packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml # packages/apps/kubernetes/tests/talos_templates_test.yaml # packages/apps/kubernetes/values.schema.json # packages/apps/kubernetes/values.yaml # packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml # packages/system/kubernetes-rd/cozyrds/kubernetes.yaml |
||
|
Aleksei Sviridkin
|
f4d812cfb9 |
fix(velero): apply CRD updates on upgrade via CreateReplace
velero evolves its CRD schema between versions; v1.18 adds the Queued and ReadyToStart backup phases. Helm never upgrades CRDs shipped in crds/, the generated HelmRelease left spec.upgrade.crds at the Skip default, and the package disables the chart's upgrade-crds Job, so live CRDs stayed frozen at first-install state: the server upgrades, the apiserver rejects the new phases, and backups sit in New while the HelmRelease stays green. Opt the velero PackageSource into CreateReplace, consistent with mongodb-operator, add a platform test pinning the policy, and correct the stale Job-disable rationale in the package. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
7739d1e294 |
feat(multus)!: stage reference CNI plugins from the multus image
Multus-backed NetworkAttachmentDefinitions delegate to the upstream containernetworking reference plugins (bridge, macvlan, ipvlan, ...), and nothing guarantees a node has them in the directory Multus searches. On k3s they live under /var/lib/rancher/k3s/data/cni while Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails there with `failed to find plugin "bridge" in path [/opt/cni/bin]`. Bundle 14 of the release's plugins into the multus image and add an install-cni-plugins init container that installs them into the host /opt/cni/bin, mirroring the existing install-multus-binary container. loopback, dhcp, tap and dummy are left out; the Dockerfile records why. Gate it on a new chart value, stageCniPlugins, resolved per bundle and on for both. Talos bundles only bridge, firewall, flannel, host-local, loopback and portmap, so the other ten are missing there exactly as on generic Linux; the four that overlap are rewritten with the same upstream version, v1.9.1 on both sides, rather than an older one. A marker in the image Dockerfile records the Talos release that pin was compared against, and a test holds it to the release the platform ships. multus therefore moves out of system.common-packages into the two variant branches, the way linstor already does for talos.enabled. networking.stageCniPlugins overrides the variant default in either direction. Where staging is on the install is unconditional: a plugin is replaced on every pod recreation, not only when the destination is missing. A destination that is already a directory is reported and left alone, since replacing it is not something this can do safely. Turning the value off afterwards does not undo it; nothing here removes or restores a plugin. The container is not privileged and takes no mount propagation. It runs in the spc_t SELinux domain, without which an enforcing node refuses the write. Each plugin is installed under a temporary name and swapped in with a rename: /opt/cni/bin is shared with cilium and kube-ovn. The plugin tarball is checksummed before extraction. A plugin that cannot be installed is reported on stderr and to /dev/termination-log, and the container still exits 0: Multus stays the node's primary CNI and nothing removes its config, so an init container that never completes would leave the shim with no daemon behind it. The manifest change lives in customize-deployment.patch so it survives make update. BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI plugins in /opt/cni/bin on existing nodes, Talos included -- four of the names it writes are ones the Talos node image also carries. Opt out before upgrading: declining afterwards does not restore what was replaced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
IvanHunters
|
c010745e8e |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Bring the worker-pool split branch up to date with main (111 commits), picking up the node-join and CI fixes it was missing: the 18m tenant node-join deadline, guest-fsync-off on ephemeral CI disks (#3455), kubevirt-csi client rate-limit raise (#3428), the Cilium ingress-IP race guard (#3430), and etcd-operator v0.5.4. Conflict resolution: main's #3535 (render the talos-reconcile Job for the default md0 group) modified packages/apps/kubernetes/templates/talos/ talos-reconcile-job.yaml, which this branch deletes because the worker split moves that Job into the kubernetes-nodes chart. Kept the parent template deleted and dropped the parent-scoped test talos_reconcile_nodegroups_test.yaml: the empty-nodeGroups md0-default gap it guards cannot occur in the per-pool child chart, and migration 54 materialises the implicit md0 into an explicit child HelmRelease. targetVersion resolves to 55 (migration 54 on top of main's 53). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
e969d71379 |
chore(platform): use bash shebang for migration 54
Migration 54 uses set -o pipefail, a bashism. The migrations image ships bash (alpine:3.24 + apk add bash) and sibling migrations 45/47/50/51 already use #!/bin/bash for the same reason. Align 54 so the interpreter matches the options it sets and cannot silently no-op pipefail if /bin/sh were ever a non-bash shell. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
cb56f140ad |
fix(platform): pin keep on a foreign-owned worker object before skipping it
The foreign-owner skip branch warned and returned without applying helm.sh/resource-policy=keep, on the premise that a foreign release-name means the parent will not prune the object. That premise is unsound: Helm computes the upgrade prune set from the release's own stored old manifest minus the new manifest and gates deletion only on resource-policy=keep, not on the live meta.helm.sh/release-name. The worker object is in the parent's old pre-split manifest, so the control-plane-only re-render would prune it and cascade-destroy the tenant's worker VMs. Pin keep before returning, exactly as every sibling skip branch does, while still refusing to rewrite the foreign release-name (which could corrupt that release). Also correct the printed remedy: migration 54 stamps 55 unconditionally and never re-runs, so recovery is manual, not a re-run. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
4f323bfce5 |
fix(platform): pin live worker pools when migration-54 sees non-object nodeGroups
The top-level non-object nodeGroups branch warned and skipped the cluster without pinning any objects, unlike every sibling skip branch (RFC-1123 invalid key, 53-char overflow, per-group non-object) which pins first. If a cluster's pools were rendered under an earlier good revision and its nodeGroups was later mangled out-of-band into an array/string/number, the worker objects stay live but un-pinned; the parent's control-plane-only re-render then prunes them, destroying the tenant's running worker VMs. The declared pool keys are unreadable in this shape, so discover the live pools from their KubevirtMachineTemplates (kubernetes-<cluster>-<group>-<hash>) and pin each via pin_pool_objects before skipping adoption. Fail-closed through list_kmts so a transient read never masquerades as no-pools. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
2d73043518 |
fix(platform): scope migration-54 foreign-owner refusal to one object
A worker object whose meta.helm.sh/release-name is neither the parent nor the child release made adopt_one exit 1. Under set -e that aborts the whole per-cluster pre-upgrade hook, so run-migrations.sh never stamps the version and one tenant's out-of-band-edited annotation deadlocks the platform upgrade for every tenant, with no path out. This is the same hostile-stored-shape class the non-object, overflow and RFC-1123-invalid branches were already reshaped to warn-and-skip rather than deadlock the fleet. A foreign owner also means the parent release no longer owns the object, so its control-plane-only re-render will not prune it: skipping is prune-safe. Refuse to adopt (rewriting a foreign release-name could corrupt that release) but scope the blast radius to the single object: warn, skip it, document the operator remedy, and let the run complete. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
myasnikovdaniil
|
3f967ec7e0 |
fix(platform): restore the isp-hosted dependency closure (#3391)
## Problem Fixes #3376. The `isp-hosted` preset shipped `bundles.system.enabled: false` while `paas` and `naas` were enabled, so it enabled an app catalog whose dependency closure it simultaneously disabled. Every emitted Package sat in `DependenciesNotReady`, the `*-rd` HelmReleases failed with `no matches for kind "ApplicationDefinition" in version "cozystack.io/v1alpha1"`, and the `cozystack-platform` HelmRelease never went Ready. Rendering the preset at `origin/main` shows the scale: **24 emitted Packages, 24 of them blocked**, by 6 missing dependencies — two more than the issue lists (`cert-manager` and `prometheus-operator-crds` are also absent). `isp-full` renders 75 Packages with a fully satisfied closure, which is a clean control. ## Root cause `templates/bundles/system.yaml` is gated end to end on `bundles.system.enabled`, but it carries two separable things: a variant-selected **data-plane head** (kubeovn/cilium/linstor on `isp-full`, a noop networking Package on `isp-hosted`) and a variant-agnostic **control-plane body** — `cozystack-engine`, the sole source of the `application-definition-crd` component, plus `cert-manager`, `prometheus-operator-crds`, `postgres-operator` and `objectstorage-controller`. "Hosted" means the host already provides CNI, storage and virtualisation, so only the data-plane head should be dropped. Disabling the whole bundle also deleted the control-plane body that every variant needs. Three things show the disable was the mistake rather than the intent: the preset still sets `variant: "isp-hosted"`; `system.yaml` contains an `isp-hosted` branch (noop networking plus default scheduler) that could never render; and `iaas.yaml` already refuses to render on this variant, encoding the same "system bundle minus iaas" contract. The preset has never worked — it was born this way in `2d022e38e` (January 2026), the refactor that replaced `bundles/paas-hosted.yaml`, whose contents confirm hosted was always meant to ship the engine, `cert-manager` and `prometheus-operator-crds`. ## Changes 1. **`values-isp-hosted.yaml`** — enable the system bundle. Hosted now emits 53 Packages with every `dependsOn` satisfied, installing 59 components against the 60 of the pre-refactor `paas-hosted` bundle, with no LINSTOR, KubeVirt, CAPI or Kamaji leakage. 2. **Move `cozystack.bucket-application`** from the iaas bundle into the system bundle. Its own `dependsOn` closure is `networking` + `objectstorage-controller` + `cozystack-engine`, none of them iaas, while `cozystack.backupstrategy-controller` — a *default* package of the system bundle — hard-depends on it. Any system-on/iaas-off configuration therefore wedged `backupstrategy-controller`; both `isp-full` presets masked this by enabling iaas. This is a second, independent bug not reported in the issue. 3. **Guards** — `paas`, `naas` and `iaas` now also require `bundles.system.enabled`. The pre-existing checks looked only at `bundles.system.variant`, which keeps its value when the bundle is disabled, so they passed on exactly this configuration. `iaas` had the identical hole (12 Packages, 6 unsatisfied deps). This turns a silent multi-hour `DependenciesNotReady` hang into a render-time failure that names the remedy. 4. **Two helm-unittest suites** — one pinning the dependency closure the catalog needs, one pinning the guards. ## Verification - All four operator-registered variants render dependency-closed: `default` 0 Packages, `isp-hosted` 53, `isp-full` 75, `isp-full-generic` 75. The two `isp-full` variants are byte-identical before and after, so the bucket move is a no-op for them. - Hosted is a strict subset of full with zero data-plane leakage, and its exclusions match the pre-refactor hosted bundle. - 23 suites / 97 tests pass in `packages/core/platform`; repo-wide `hack/helm-unit-tests.sh` is green. - Every change was **mutation-tested** — reverting each one in turn makes the suite fail, so the tests are not vacuous. - The workaround documented in #3376 (`paas`/`naas` disabled) still renders cleanly; the new guards do not fire on it. - Hosted runtime spot-checks: `cozystack-api` is a Deployment with only a *preferred* control-plane affinity plus blanket tolerations, and `cozystack-scheduler` uses its `default` variant (plain Deployment, no LINSTOR extender) — both schedule fine on a cluster with no control-plane nodes. ## What this does not fix This restores the dependency closure. It is **not** by itself sufficient for a fully converged hosted install. Review surfaced further hosted-unaware defaults that the dead bundle had been masking, filed separately: - #3387 — `apps/tenant` renders 14 `cilium.io/v2` policies with no `.Capabilities` guard, so `tenant-root` fails on a non-Cilium cluster. This is the blocker for actual hosted convergence, and it needs a deliberate decision rather than a reflexive capability guard, because those policies implement tenant network isolation. - #3388 — `extra/monitoring` defaults `logsStorages` to the LINSTOR-only `replicated` StorageClass. - #3389 — `bundles.disabledPackages` can still break the closure with no render-time error; the guards here cover the bundle flags only. - #3390 — nothing in CI exercises `isp-hosted`, which is why all of the above accumulated undetected. ## Upgrade note Existing `*-application` Package CRs carry `helm.sh/resource-policy: keep`. An operator who applied the #3376 workaround but did not also delete the retained Package CRs (as that issue instructs) will find them install once this change makes their dependencies Ready, since the reconciler re-enqueues dependents on readiness. Deleting the unwanted Packages before upgrading avoids that. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added fail-fast validation that requires the **System bundle** when enabling **IaaS**, **NaaS**, or **PaaS** to prevent incomplete catalogs. * Updated bundle composition so the **bucket application** is provided by the **System bundle**. * Enabled the **System bundle** by default for the `isp-hosted` variant. * **Bug Fixes** * Improved error messaging to clearly indicate the required bundle/variant configuration. * **Tests** * Added Helm unittest coverage for rendering-gate failures, dependency/package expectations, and hosted data-plane exclusions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
ebeeab77aa |
docs(platform): correct the computeplane dependsOn and variant comments
The dependsOn comment claimed the kubernetes-application edge orders the addon artifacts the wrapped chart references; variant dependsOn only gates install: components — the operative effect is ordering the computeplane-rd install so the ComputePlane kind is not registered before the kubernetes machinery exists. Also document why the variant is named kubevirt. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
aa5198b758 |
fix(platform): add cozystack-engine dependency to the computeplane package
The computeplane-rd component installs an ApplicationDefinition, whose CRD is served by cozystack-engine — hack/check-applicationdefinition-crd-ordering.bats enforces the edge for every package with a system/*-rd component. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
4c3262a4ee |
feat(computeplane): deliver ComputePlane as a tenant module wrapping the kubernetes app
Add the operator-owned computeplane tenant module per the revised design (cozystack/community design-proposals/compute-plane): an extra/computeplane chart that provisions a single-tenant, Cozystack-managed cluster for untrusted-code apps by deploying the ordinary apps/kubernetes chart with operator-fixed values (fluxcd, cert-manager, ingress-nginx Proxied). The package re-declares apps/kubernetes as its own source-only component, so the module is self-contained; the cluster release is named computeplane so Kamaji writes the admin kubeconfig to computeplane-admin-kubeconfig — the contract consumed by placement: ComputePlane applications. The tenant holds no admin kubeconfig and can set cluster shape (nodeGroups) only. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
8bffa34b54 |
feat(platform): add opt-in in-cluster east-west encryption toggle (#2984)
## What this PR does Adds `networking.encryption.enabled` to the platform chart — a single operator-facing opt-in, off by default, that turns on in-cluster (east-west) transport encryption for the active datapath. When enabled, pod-to-pod traffic across nodes is encrypted at the network layer regardless of whether the workload implements TLS: defense-in-depth for the whole mesh without per-application changes. The default `isp-full` bundle selects the `kubeovn-cilium` datapath (and `isp-full-generic` selects `kubeovn-cilium-generic`), where KubeOVN owns the datapath and the overlay is a Geneve tunnel. The toggle therefore maps to KubeOVN's overlay IPsec (`kube-ovn` `ENABLE_OVN_IPSEC`). KubeOVN's own controller provisions the IPsec CA and per-node certificates through the Kubernetes CSR API, so no cert-manager wiring is needed. This complements, and does not replace, application-level / end-to-end TLS: in-cluster encryption never reaches an external client and gives a tenant no verifiable endpoint certificate. The values comments document the per-datapath scope — including that Kilo's WireGuard mesh (`meshGranularity: location`) covers only cross-location traffic and leaves the intra-location gap this toggle targets, and that the Cilium-native WireGuard/IPsec path on the `cilium` / `cilium-generic` variants is not reached because the system bundle does not select those variants — plus the MTU/throughput overhead. Scope: only the default KubeOVN datapath is wired through the platform toggle, which is what the system bundle actually selects. Wiring the Cilium-native path would require the bundle to expose cilium-only variant selection; that is out of scope here and called out in the values comments. Validation: `helm unittest` on the platform chart passes (66 tests, 7 new) covering on / off / absent / null and verifying the IPsec flag deep-merges without dropping the existing `kube-ovn` overrides, for both `isp-full` and `isp-full-generic`. Closes #2977 Part of #2811 ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(platform): add opt-in `networking.encryption.enabled` (off by default) that enables in-cluster east-west transport encryption for the default KubeOVN datapath via overlay IPsec; complements, and does not replace, application-level TLS. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced opt-in in-cluster network encryption using KubeOVN IPsec for east-west traffic protection. * Added `networking.encryption` configuration toggle (disabled by default) with full backward compatibility. * **Tests** * Added comprehensive test coverage for network encryption configurations across system variants and edge cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
eeee87f0b1 |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Resolve the migrations pin conflict in packages/core/platform/values.yaml: keep targetVersion 55 (this branch owns migration 54) and take the newer platform-migrations image v1.6.0 from main. The in-tree image pin intentionally lags targetVersion until the release retag, and run-migrations.sh fails loud if a targeted migration is missing from the image, so the lag is safe by design. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Timofei Larkin
|
730d006ce8 |
fix(platform): skip pinning names no object can have
The pin path composed "kubernetes-<cluster>-<group>" and read it back, but a group key that cannot form a valid object name (a slash, a space, an underscore...) provably never had objects applied under it -- the old chart's apply would have been rejected by the apiserver. Worse, kubectl rejects a name containing '/' client-side, deterministically, so pin_keep's fail-closed read classified it as a real failure and the migration exited 1 on every retry: a permanent fleet-wide pre-upgrade deadlock over a pool that cannot exist. Lean on the invariant instead of the read: pin_pool_objects statically checks the composed name against the DNS-1123 subdomain shape and skips with a note when it cannot name an object -- nothing to pin, no kubectl call to misread. Keys that are label-invalid but still valid object names (e.g. dotted) may have live objects and keep today's pin behavior; the pin-path transient-error test now uses a dotted key so it still exercises a genuine read. The fake kubectl reproduces real kubectl's client-side slash rejection, and a new bats case pins the skip: a "pool/a" key must warn, pin nothing, apply nothing, and still reach the version stamp. Red against the previous script, which deadlocked at the first pin read. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
bdadb3ff04 |
fix(platform): detect absence via --ignore-not-found
The stderr-based classification this branch introduced had two holes of its own. Capturing stderr into the data variable (2>&1) contaminates the SUCCESS path: kubectl prints warnings on successful reads (API deprecation headers -- deterministic for a deprecated GVK, and CAPI v1beta1 is one -- plus client-side throttling notices), so the warning glued onto the owner annotation would trip adopt_one's foreign-owner refusal on every tenant and permanently deadlock the pre-upgrade hook. And the *"not found"* pattern over-matches: a transient conversion-webhook outage fails a read with '... service "capi-webhook-service" not found', which the classifier would treat as absence -- skipping the keep pin and re-opening the exact fail-open prune hazard this branch exists to close. Detect absence with --ignore-not-found instead: a missing object is SUCCESS with empty output, so a non-zero exit can only be a real failure and aborts before the version stamp, with no stderr parsing at all. adopt_one reads -o json and extracts the owner annotation with jq (already a dependency), keeping absence distinct from present-but- unannotated; pin_keep reads -o name. list_kmts still needs stderr for the one legitimate classification -- the two deterministic missing-CRD messages -- so it captures stderr to a temp file, never into data, and no longer accepts NotFound-shaped text for a list. The fake kubectl honors --ignore-not-found the way real kubectl does, gains FAKE_WARN_STDERR (success-path stderr warnings) and FAKE_GET_FAIL_MSG (transient error with arbitrary text). Two new bats tests, each red against the previous commit: a deprecation warning must not break adoption or fake a foreign owner, and a failure whose message contains "not found" must still abort without stamping. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
f8258c8859 |
fix(platform): fail migration 54 reads closed
kubectl exits 1 for NotFound and for transient server failures (timeout, 429, etcd hiccup) alike, and migration 54 treated every non-zero read as "object absent". A transient error while reading a worker object would skip the keep pin and the ownership rewrite, let the migration complete and stamp version 55 (so it never re-runs), and the parent's control-plane-only upgrade would then prune the un-pinned live MachineDeployment -- deleting every running worker VM in the pool. The doubly-masked KubevirtMachineTemplate listings (2>/dev/null plus `|| true`) had the same shape: a failed list adopted or pinned zero KMTs, permanently stranding a preserved older-revision KMT still referenced by an in-flight MachineSet, since the child chart only re-emits KMTs already annotated with its own release name. The `cur=$(... || true)` owner read additionally bypassed the foreign-owner refusal guard. Classify reads by stderr, the migration-47 precedent: NotFound means genuinely absent and is skipped; anything else exits 1 so run-migrations.sh leaves CURRENT_VERSION at 54 and retries on the next upgrade. adopt_one and pin_keep fold the existence check into the annotation read (one request instead of two); KMT listings go through a capture-first list_kmts helper because `for x in $(...)` discards the substitution's exit status even under set -e, leaving `|| true` to cover only the empty-grep-match case, mirroring the unpin hook. The child-HR existence check stays deliberately tolerant: its fallback is an idempotent apply that itself fails closed. adopt_one now also stamps meta.helm.sh/release-namespace so an object whose Helm annotations were stripped out-of-band cannot fail the child's ownership validation. The fake kubectl now reports absent objects the way the real apiserver does (NotFound on stderr, exit 1) and gains FAKE_GET_FAIL to inject transient errors; four new bats tests pin the classification: NotFound still completes and stamps, while a transient error on the adopt path, the KMT listing, or the pin path aborts before the version stamp. Each red against the previous script. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |