mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 02:57:27 +09:00
ci/shellcheck-gate
1198
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
b4e203164a |
fix(kubernetes): retry the tenant CSI install instead of uninstalling it (#3581)
## What this PR does The tenant CSI HelmRelease carried `remediation.retries: -1` with no strategy, so Flux applied its default, `RemediateOnFailure`. Install remediation is an uninstall in every case the API offers, so an install that missed its budget was torn down between attempts: the tenant lost its CSI node plugin and the StorageClasses this release propagates until a later attempt applied them again, and that attempt restarted the DaemonSet rollout instead of resuming one already under way. That teardown races the rollout it exists to recover, and the run recorded in #3576 shows how narrowly. Both DaemonSet pods were scheduled at the same second on both workers, so nothing was waiting on capacity; the ten minutes went to image pulls. The slower worker started its last container six seconds after the install timeout fired, and the uninstall removed it one second after that. `RetryOnFailure` leaves the manifests applied and retries the failed release. Both actions carry it because the controller picks the active retry from `.status.lastAttemptedReleaseAction` and performs that retry as an upgrade: set on install alone, the retry still happens, but its own failure is then handled by upgrade remediation, whose default is a rollback with no previous release to roll back to. The `remediation` blocks stay, because an absent release consults `install.remediation` and an out-of-sync one consults `upgrade`, both by retry count rather than by strategy. This is the change #3552 makes for the tenant CNI, and the reason for putting the strategy on both actions is the one given there. It is also the shape upstream prescribes: the "Recommended settings" block in the [HelmRelease spec shipped with the helm-controller this installs](https://github.com/fluxcd/helm-controller/blob/v1.5.1/docs/spec/v2/helmreleases.md#recommended-settings) puts `strategy.name: RetryOnFailure` under both `install` and `upgrade` for production deployments. Nothing changes on the test side, and that is deliberate rather than deferred. Because no manifest is removed at the attempt boundary, the rollout continues across it and the kubelet finishes what it started, while the retry re-applies the same manifests every 30s. The release therefore reports Ready once the rollout completes rather than once a whole attempt has fit inside the timeout, so what the existing e2e wait has to outlast is the rollout and no longer an attempt boundary that discards it. That widens what the wait tolerates without making it unbounded: a rollout slower than the wait itself still fails, and this does not claim otherwise. Scope. #3576 records more than this fixes, so this does not close it. The other tenant addon HelmReleases this chart renders keep the remediation-only shape; CSI is the one this changes. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: The trigger map was walked against the diff. The change is a HelmRelease template and a chart test, not a `values.schema.json` field, a `values.yaml` default, a version enum, a `release.prefix`, an `ApplicationDefinition`, a chart source kind or a namespace. ### Release note ```release-note fix(kubernetes): the tenant CSI HelmRelease now retries a failed install instead of being uninstalled between attempts, so a failed install no longer takes the tenant's CSI node plugin and its propagated StorageClasses away while it retries. The same strategy applies to the upgrade action, which it has to: a retried install runs as an upgrade. One consequence is operator-visible — a failed CSI upgrade in a tenant cluster now retries the new manifests every 30s instead of rolling back to the previous release. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when installing or upgrading the CSI component by automatically retrying failed actions every 30 seconds. * Retains unlimited remediation retries to help recover from transient failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
0c4c11c845 |
fix(kubernetes): retry the tenant CSI install instead of uninstalling it
Flux defaults a HelmRelease to the RemediateOnFailure strategy, whose install remediation is always an uninstall. A tenant CSI install that misses its timeout was therefore torn down between attempts, taking the CSI node plugin and the StorageClasses this release propagates with it, and the next attempt restarted the DaemonSet rollout from nothing instead of resuming one already under way. That teardown races the rollout it is meant to recover. In one observed run both DaemonSet pods were scheduled immediately and spent the whole budget pulling images; the slower node started its last container six seconds after the install timeout fired, and the uninstall removed it one second later. Retrying in place keeps that work. RetryOnFailure leaves the manifests applied and retries the failed release. Both actions carry it because the controller picks the active retry by .status.lastAttemptedReleaseAction and runs that retry as an upgrade: set on install alone, the retry happens but its own failure falls through to upgrade remediation, which rolls back by default with no previous release to roll back to. The cost is that a real upgrade failure now retries rather than rolls back, and that gives up no stable safety net: with retries: -1 and the default rollback strategy, upgrade remediation already rolled back and retried without end. The remediation blocks stay for a separate reason. An absent release consults install.remediation and an out-of-sync one consults upgrade, both by retry count rather than by strategy, so retries: -1 is what keeps those two branches acting. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
80a2090444 |
fix(kubernetes): retry the tenant CNI install instead of uninstalling it
The tenant cilium HelmRelease carried no install strategy, so Flux applied its default, RemediateOnFailure, which uninstalls the release between attempts. Cilium is the tenant cluster's only CNI, so every cycle takes it away and gives it back when the next install applies the manifests again, and retries: -1 lets that repeat without end. What sets a cycle off is an install that outruns its budget, which can happen while the tenant still has no registered node, since the operator Deployment cannot become Available until one exists. Remediation neither causes that delay nor is fixed by this change; what it adds is a repeated teardown of the CNI on top of a delay that resolves itself once the node arrives. This removes the amplifier, not the cause. RetryOnFailure keeps the applied manifests in place and retries a failed install as an upgrade, so the rollout completes on its own once a worker registers. Both actions carry the strategy because the retry of a failed install runs as an upgrade, which would otherwise fall back to upgrade remediation. It is the strategy and retry interval cozystack-api and cozystack-operator already set on every HelmRelease they generate, except that they leave remediation nil where this release keeps it. The other tenant addons the chart renders still carry the remediation-only shape; this one is where the teardown removes the cluster's only CNI rather than churning a single component. The cost is that a genuine upgrade failure no longer rolls back either, it retries on the failed manifests instead. That is forced, since the controller cannot tell a retried install from a real upgrade, and what it replaces was an unbounded rollback-and-retry flap rather than a stable safety net. The remediation blocks stay, since they cover the branches no strategy reaches: an absent release, which consults the install block, and an out-of-sync one, which consults upgrade. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
4e9c5ce8ca |
fix(kubernetes): render the talos-reconcile Job for the default md0 group
The Job in templates/talos/talos-reconcile-job.yaml is the only producer of the TalosConfigTemplate that each worker MachineDeployment names in spec.template.spec.bootstrap.configRef; cluster.yaml renders the reference but deliberately not the object. Its loop read .Values.nodeGroups directly, while the MachineDeployment loop reads the effective set through the kubernetes.nodeGroups helper. For a cluster that supplies no nodeGroups the two disagree: the helper's else-branch emits the built-in md0 group, so the md0 MachineDeployment renders, but the raw map is empty and no Job renders at all. Every Machine the cluster-autoscaler adds to md0 then blocks indefinitely on a TalosConfigTemplate that nothing will ever create, and the KamajiControlPlane spec.network.certSANs patch the same Job performs is skipped too. Range over the helper so the Job set tracks the MachineDeployment set exactly. No change for a cluster that declares its own groups, and md0 stays removable: the helper's if-branch keeps a user-supplied map authoritative. The gap survived because every helm-unittest fixture and e2e suite declares md0 explicitly. tests/nodegroups_default_test.yaml does render the empty-nodeGroups case but lists only templates/cluster.yaml, so it never looked at the Job. The new suite pins both halves of the helper contract and fails without this fix. Fixes #3504 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
2ff9c30d04 |
docs(kubernetes): note the etcd gate in the Proxied Service comment
The Proxied ingress Service is gated on _namespace.etcd as well as the addon toggle and exposeMethod, but the comment above it still described only the addon+exposeMethod gate — comment drift from the etcd-gate change. State both gates so the comment matches the condition it sits on. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8fc8d23dfd |
fix(computeplane): address review round three
Gate the Proxied ingress Service on _namespace.etcd like the addon HelmRelease it fronts, so an etcd-less tenant with the addon pinned on does not get a selector-less Service (+ test). Correct the secrets framing: the tenant-visible surface is an allowlist (include: [] withholds everything), so drop the never-load-bearing exclude entry, the bats cross-check and the rename-coupling comments, and state the real mechanism. Fix the etcd prerequisite docs (an ancestor's etcd satisfies the wrapped chart; the awaiting-etcd beacon is the diagnostic), reword the tenant toggle to stop advertising the not-yet-shipped placement routing, require maxReplicas >= 1 on the ingress-role node group, and fix comment drift (Service name after the release rename, placement-hook label, package-guard lookup semantics). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
366268d9f5 |
test(kubernetes): pin that a disabled ingress addon renders nothing with hosts set
hosts with the addon disabled used to render the Ingress; the addon gate makes 'the addon is off' mean nothing renders. Pin the combination so the behaviour change reads as a decision. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
6dcf5373e7 |
fix(computeplane): let Flux derive both release names
The aggregated API cannot represent spec.releaseName: the conversion rebuilds HelmRelease specs without it, so the first tenant edit through the API dropped the module's explicit computeplane-module release name, let it default to the object name computeplane, and collided with the Helm storage record the cluster release owned — with the tenant chart writing the old name back every interval. Drop releaseName from both HelmReleases and derive names from object names, like every other tenant module. Kamaji now writes the admin kubeconfig to computeplane-cluster-admin-kubeconfig; the computeplane-rd secrets.exclude entry follows in the same change, and a bats test pins the exclusion to the name the cluster release actually produces so a future rename cannot silently re-expose the credential. Also add the check-release-name guard the other extra modules carry, and pin the fluxcd addon's distribution artifact to the flux-operator version vendored in system/fluxcd-operator instead of :latest. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
76374f6472 |
fix(tenant): fail loudly when computeplane is enabled without its package
The computeplane-application package ships in the iaas bundle while the toggle arrives with tenant-rd in the system bundle, and bundles are independent booleans. Refuse the render with a clear diagnostic instead of producing a HelmRelease that retries a nonexistent artifact forever. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
bca5b3f9fb |
fix(kubernetes): render the Proxied ingress Service only when the addon is enabled
exposeMethod defaults to Proxied while the addon defaults to off, so gating the Service on exposeMethod alone emitted a selector-less Service — surfaced in the tenant dashboard — on every cluster that runs no ingress-nginx at all. Gate on the addon instead; ComputePlane sets enabled: true explicitly, and the Service still renders without a hosts list. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
9cb4f2922d |
fix(kubernetes): create the Proxied ingress Service independently of hosts
The proxy Service was only rendered when addons.ingressNginx.hosts was non-empty, but external consumers — e.g. a ComputePlane app's per-app management Ingress — need to route to the tenant cluster's ingress-nginx without the module knowing every app host in advance. Render the Service whenever exposeMethod is Proxied; the Ingress still requires hosts. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
b2232904b6 |
feat(tenant): add the computeplane module toggle
Wire the computeplane module into the tenant chart as a bool toggle, matching the other tenant modules. The module's own Helm release takes the -module suffix so the canonical release name computeplane (and with it the computeplane-admin-kubeconfig Secret) stays reserved for the cluster HelmRelease the module renders. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8b461f0c0b |
feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins (#3378)
## What this PR does Cozystack builds and publishes `ghcr.io/cozystack/cozystack/grafana` — a rebuilt Grafana image — and uses it by default in the monitoring stack. Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a redistributor of an AGPL Grafana build. This is a governance concern for CNCF Incubation (there is no precedent for a CNCF project redistributing an AGPL binary). The only reason the image was ever rebuilt was to bundle the VictoriaLogs datasource plugin, which was not in the official Grafana plugin catalog at the time. It is now published there as a signed plugin (`victoriametrics-logs-datasource`), so the rebuild is no longer necessary. This PR moves the deployment to the upstream `grafana/grafana` image (digest-pinned, routed through `cozy-lib.image` so mirrored-registry installs keep working) and installs the plugins at container startup via `GF_INSTALL_PLUGINS`, with versions pinned: - `victoriametrics-logs-datasource 0.29.0` — the VictoriaLogs datasource (signed; the reason the image was rebuilt) - `natel-discrete-panel 0.1.1`, `grafana-worldmap-panel 1.0.6`, `marcusolsson-dynamictext-panel 6.2.0` — reproduce the panel set the removed Dockerfile vendored **Air-gapped installs keep working.** The plugin zips are not downloaded from the internet at pod startup: they are mirrored from the official catalog into the existing `grafana-dashboards` image at build time and served by that in-cluster static file server, next to the dashboards it already serves. `GF_INSTALL_PLUGINS` points at `http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…`, so the runtime dependency stays inside the cluster and the artifact is mirrored into private registries together with every other first-party image. A new `allow-to-grafana-dashboards` CiliumNetworkPolicy (mirroring `allow-to-keycloak`) opens tenant egress to that server, since tenant egress is default-deny. As a result, `ghcr.io/cozystack/cozystack/grafana` is no longer built or published: the Dockerfile, the pinned image tag, the `image` targets in both monitoring Makefiles, and the build step in the root Makefile are removed. Notes: - The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored plugin zips are Apache-2.0 licensed and are served unmodified from the official catalog builds (signatures intact). - `allow_loading_unsigned_plugins` and the custom `/var/lib/grafana-plugins` path are dropped: the catalog datasource is signed, and plugins install into the writable default `GF_PATHS_PLUGINS` (`/var/lib/grafana/plugins`). - The VictoriaLogs datasource provisioning (the `GrafanaDatasource` CRs pointing at `vlselect`) is unchanged. - Plugins are re-installed on every pod start from the in-cluster server (no PVC involved, ~77 MB per start per replica over the cluster network); a failed download fails the pod loudly instead of silently dropping the datasource. - `natel-discrete-panel` and `grafana-worldmap-panel` are Angular plugins; Angular is disabled by default in Grafana 11, so they do not load — this is unchanged from the current rebuilt image (same Grafana 11.6.15, same config). They are kept for parity and can be dropped as a follow-up. ### How this was verified - `helm unittest` across the repo passes, including new regression tests: the monitoring chart asserts the exact upstream image reference (with and without `images-registry` set) and the exact `GF_INSTALL_PLUGINS` value; the tenant chart asserts the new egress policy. - Local end-to-end of the full chain in Docker: built the modified `grafana-dashboards` image, served it on a shared network, and started upstream `grafana/grafana:11.6.15` with the rendered `GF_INSTALL_PLUGINS` value. All four plugins download from the in-cluster-style URL (no internet egress from Grafana), and `GET /api/plugins` reports `victoriametrics-logs-datasource` v0.29.0 as `enabled: true` with `signature: valid`. ### Screenshots Not applicable — no UI change. Grafana renders the same datasources and panels; only the image source and plugin delivery mechanism change. ### Downstream repositories Walked the trigger map against the diff. This is an internal change to how the monitoring stack sources the Grafana image and its plugins (`packages/system/monitoring`, `packages/system/grafana-operator`, `packages/apps/tenant` network policy). No package is added, renamed or removed; no `values.schema.json`, `ApplicationDefinition`, platform values, release asset, shared `hack/` tooling or `package.mk` contract changes; the added `grafana.image` key is an internal system-chart value, not a user-facing app field. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Grafana in monitoring is now based on a digest-pinned upstream image for more consistent deployments. * Required Grafana plugins are installed automatically at startup via `GF_INSTALL_PLUGINS`; unsigned-plugin configuration was removed. * Tenant networking now includes a new egress rule specifically for Grafana dashboards. * Grafana dashboards image now includes required plugin assets prepared during image build. * **Tests** * Added/updated checks covering digest pinning, mirrored-registry behavior, plugin-installation behavior, and the new Grafana dashboards network policy. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
2dfe380ac3 |
fix(kubernetes): raise kubevirt-csi API client rate limits
The kubevirt-csi controller built its infra- and tenant-cluster Kubernetes clients on a rest.Config that left QPS/Burst unset, so client-go applied its 5 QPS / 10 burst default. ControllerPublishVolume polls the infra PVC once per second for up to two minutes; when a tenant scales out and several volumes attach at once, the default limiter starves and the poll fails with "client rate limiter Wait returned an error: context deadline exceeded", which the tenant kubelet surfaces as FailedAttachVolume. Set QPS/Burst to 100/200, overridable via --kube-api-qps and --kube-api-burst, on both the infra and tenant configs so a concurrent-attach burst is admitted promptly. Reject non-positive flag values, which client-go would otherwise turn into the starving default (0) or unlimited (<0). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
1c7eb22918 |
feat(tenant): allow egress to the grafana-dashboards plugin server
Tenant egress is default-deny plus explicit allow-to-* policies, so the Grafana pods deployed per tenant could not reach the in-cluster plugin server in cozy-grafana-operator. Mirror the allow-to-keycloak pattern and cover the policy with a test. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Timofei Larkin
|
2823b76802 |
feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners (#3398)
## What this PR does Browsers negotiate HTTP/2 exclusively through TLS ALPN: if the terminating listener does not advertise `h2`, every client silently falls back to HTTP/1.1, with no error surfaced anywhere. Cozystack's Gateway API path is served by Cilium's embedded Envoy, and Cilium ships with `enable-gateway-api-alpn=false` — so every Gateway listener advertised no ALPN protocols and all browser traffic through a Gateway ran over HTTP/1.1 (observable in browser dev tools as `http/1.1` on every request). The ingress-nginx path this replaces advertised `h2` out of the box, so the migration to Gateway API silently downgraded clients: head-of-line blocking and the ~6-connections-per-origin limit are back for anything served through a Gateway. This PR enables ALPN (`h2`, then `http/1.1`) on Gateway API listeners, default-on, with an opt-out at each management layer: - `packages/system/cilium/values.yaml` now defaults `gatewayAPI.enableAlpn: true`, which renders `enable-gateway-api-alpn: "true"` into the `cilium-config` ConfigMap. - The platform chart gains `gateway.http2` (default `true`) next to the existing `gateway.enabled`/`gateway.attachedNamespaces`. The `isp-full` and `isp-full-generic` bundles thread it as inline component values into the `cozystack.networking` Package CR (`components.cilium.values.cilium.gatewayAPI.enableAlpn`); inline HelmRelease values override the package `valuesFiles`, so an explicit `gateway.http2: false` actually reaches the dataplane rather than being shadowed by the new chart default. On `isp-hosted` the hosting-side CNI owns this setting and the flag is a no-op. - Tenant Kubernetes clusters (`packages/apps/kubernetes`) get the same fix: when the `gatewayAPI` addon is enabled, the tenant cilium HelmRelease now sets `enableAlpn: true`; individual clusters opt out via `addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false`. ### Why default-on rather than opt-in HTTP/2 for browsers was the platform's behavior before the Gateway API migration, so keeping ALPN off would preserve a regression, not a status quo. Defaulting on is safe because ALPN is a negotiation: clients that cannot speak HTTP/2 simply keep negotiating `http/1.1`, exactly as before. Operators who need the old wire behavior can set `gateway.http2: false` platform-wide. ### Frontend vs backend hop This change affects the client↔gateway hop only. Gateway↔backend connections stay HTTP/1.1 unless a Service explicitly opts in per GEP-1911 by declaring `appProtocol: kubernetes.io/h2c` on its port. Cilium couples that backend-protocol support to the ALPN flag (`enable-gateway-api-app-protocol` flips to true together with it), but it only acts on Services that declare a recognized `appProtocol` value — and no cozystack-shipped Service does today (only plain `http`/`https` values exist in-tree, which Cilium ignores for protocol selection). So upstreams see no change from this PR alone; h2c/gRPC backends become possible to configure, not configured. The setting is cluster-wide Cilium agent config, so it covers the root Gateway and all tenant Gateways at once — Cilium offers no per-Gateway granularity here. Flipping it re-rolls the cilium DaemonSet on upgrade (`rollOutCiliumPods: true`), the same disruption profile as any other cilium config change. ### How it was verified - New helm-unittest suites: `packages/core/platform/tests/bundles_gateway_http2_wiring_test.yaml` asserts the Package CR carries `enableAlpn` true by default and false with `gateway.http2: false` for both cilium bundles; `packages/apps/kubernetes/tests/cilium_gateway_alpn_test.yaml` asserts the tenant HelmRelease default, the addon-off case, and the `valuesOverride` opt-out. Full suites for both charts pass (93 and 184 tests). - `helm template` of `packages/system/cilium` confirms the value is consumed, not just rendered: default produces `enable-gateway-api-alpn: "true"` (and `enable-gateway-api-app-protocol: "true"`) in `cilium-config`; the threaded `false` produces `"false"` for both. ### Screenshots Not a UI change. ### Downstream repositories The website's hand-written platform values table (`content/en/docs/next/operations/configuration/platform-package.md`) needs a row for `gateway.http2` — filed as cozystack/website#625. No other downstream repository restates anything in this diff (no app `values.schema.json`, ansible-consumed platform key, or hack/ tooling changed). - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/625 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(gateway): Gateway API listeners now advertise HTTP/2 via TLS ALPN (h2, then http/1.1) by default, restoring the browser HTTP/2 support lost in the migration from ingress-nginx. Opt out platform-wide with gateway.http2=false, or per tenant Kubernetes cluster via addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false. Backend (gateway-to-upstream) connections are unchanged and stay HTTP/1.1 unless a Service declares appProtocol kubernetes.io/h2c. ``` |
||
|
Aleksei Sviridkin
|
6ecb36ffc7 |
feat(cozystack-controller): extract application CA into a key-free tenant Secret (#3407)
## What this PR does Adds an engine-agnostic controller that publishes a key-free `<release>.tenant-ca` Secret (only `ca.crt`) for a managed application, so a tenant can verify the application's TLS without ever seeing a private key. The source is declared, not guessed: a chart renders a namespaced `TenantProjection` sentinel (group `internal.cozystack.io`) naming the CA Secret to lift; the controller watches the sentinel, extracts `ca.crt`, and writes the projection owner-referenced to the sentinel, so garbage collection is native. The projection carries the `internal.cozystack.io/tenant-ca` label the lineage webhook turns into tenant visibility. Postgres is the first consumer. The trust boundary is RBAC: no tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot forge a sentinel. A companion `ValidatingAdmissionPolicy` that pins the writer to helm-controller ships separately as defence in depth. This is the controller-and-CRD slice, split out from #3299 for focused review. It is the only piece required for the feature to work; the render-time helper, the writer policy, the capability-gate, and the release-prefix validation land as their own PRs. ### Downstream repositories Walked the trigger map against the diff. This adds an internal CRD and controller and a sentinel to the Postgres chart; no app is added or renamed and no `values.schema.json` changes, so the typed provider and Ansible are unaffected. The one to confirm is the Postgres README edit — if the website mirrors it, it needs a refresh. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: ### Release note ```release-note feat(cozystack-controller): publish a key-free per-application CA trust anchor (`<release>.tenant-ca`) that tenants read to verify managed-application TLS, declared through a new `TenantProjection` CRD. Postgres is the first consumer. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added tenant-facing CA certificate projections for PostgreSQL deployments. * Tenant secrets now provide only the CA certificate, excluding private keys. * Added support for configuring and monitoring tenant CA projections. * **Documentation** * Updated PostgreSQL TLS instructions to retrieve certificates through tenant secrets. * **Bug Fixes** * Improved protection against invalid certificates, secret collisions, and accidental exposure of private key material. * Added validation to ensure projected certificates remain synchronized with their sources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
f077420fbe |
fix(http-cache,vpa): fix haproxy 3.3 backend name conflict and vpa-for-vpa OOM (#2654)
## What this PR does Fixes two independent issues found during cluster operations: **http-cache (HAProxy 3.3 breaking change)** HAProxy 3.3 no longer allows a frontend and backend to share the same name. The existing config had both `frontend http` and `backend http`, which causes haproxy to refuse to start on upgrade. Rename `backend http` → `backend cache` and update the `default_backend` directive accordingly. **vpa-for-vpa (OOMKilled recommender)** The VPA recommender deployed by `vpa-for-vpa` was hitting its 512Mi memory limit and being OOMKilled. Increase the limit to 1Gi to keep the recommender running stably. ### Release note ```release-note fix(http-cache): rename haproxy backend to avoid HAProxy 3.3 name conflict between frontend and backend fix(vpa): increase vpa-for-vpa recommender memory limit from 512Mi to 1Gi to prevent OOMKilled restarts ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Corrected HAProxy routing so HTTP traffic is directed to the cache backend. * Increased the VPA recommender’s memory limit to improve reliability under higher workloads. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
c503ae29d9 |
fix(backups): request S3 checksum only when required for barman-cloud (non-AWS S3 / Ceph RGW) (#3417)
## What this PR does Fixes CNPG **barman-cloud plugin** backups to non-AWS, S3-compatible object stores (Ceph RADOS Gateway, and some MinIO / Cloudflare R2 builds). The plugin's sidecar uploads via **boto3**. Since botocore ~1.36 (early 2025) the default `RequestChecksumCalculation` is `when_supported`, so a flexible checksum (CRC32) plus the corresponding `x-amz-content-sha256` handling is attached to every `PutObject`. AWS S3 accepts it, but several S3-compatible backends reject the request: ``` InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value. ``` so every backup / WAL-archive upload fails against them and the `ScheduledBackup` never stores anything. This sets `spec.instanceSidecarConfiguration.env` on the rendered `ObjectStore`s to carry `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` (botocore reads this env var from the sidecar's environment). `when_required` computes a checksum only for operations that mandate one; a plain `PutObject` then carries none, which both AWS S3 and the affected backends accept — a safe default everywhere. No API/CRD change, no new value, no new dependency. Applied to every barman-cloud `ObjectStore` Cozystack creates: - **chart-rendered** — `packages/system/keycloak/templates/db.yaml` (Keycloak system DB) and `packages/apps/postgres/templates/db.yaml` (postgres app — backup **and** bootstrap-recovery `ObjectStore`s), via a shared `cozy-lib` helper (`cozy-lib.barman.checksumSidecarConfiguration`); - **Go-constructed** — the platform-managed `useSystemBucket=true` `ObjectStore` SSA-applied by `internal/backupcontroller/cnpgstrategy_controller.go` (`barmanSidecarConfiguration`), whose default system bucket is SeaweedFS. Regression coverage: a Go unit test on the SSA-applied `ObjectStore`, plus helm-unittest assertions on `spec.instanceSidecarConfiguration.env` for all three rendered `ObjectStore`s (mutation-checked — reverting the change fails them). Mirrors the same fix already applied to the etcd path in `cozystack/etcd-operator#342` (released in `v0.5.3`) for the same Ceph RGW backend. **Verification:** reproduced end-to-end against a real Ceph RGW endpoint. With default botocore, `PutObject` fails with the `InvalidArgument` above; with `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` in the environment it succeeds. Deployed the plugin + a throwaway CNPG cluster whose `ObjectStore` carried this env: continuous WAL archiving reported `ContinuousArchiving=True` and an on-demand `Backup` completed, with the base backup and WAL segments present in the bucket. `helm template` renders the env on all three `ObjectStore`s; the `packages/apps/postgres` unit-test suite still passes. ### Screenshots N/A — no UI changes. ### Downstream repositories <!-- Walked the trigger map in docs/agents/contributing.md against the diff, file by file. The diff touches only two Helm templates (ObjectStore renders); it adds no package, no values.yaml / values.schema.json field, no API/CRD change, no variant/component/asset change, no ApplicationDefinition change, and no developer-tooling change. None of the trigger-map entries match. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): CNPG barman-cloud backups now work against non-AWS S3-compatible object stores (Ceph RADOS Gateway, some MinIO/Cloudflare R2). The backup sidecar requests an S3 request checksum only when required, avoiding the "x-amz-content-sha256 must be UNSIGNED-PAYLOAD, ..." error that previously broke every backup and WAL-archive upload to those backends. No change for AWS S3. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved compatibility with S3-compatible, non-AWS storage backends for PostgreSQL and Keycloak backups and recovery. * Pin the barman-cloud sidecar to compute AWS request checksums only when required (`AWS_REQUEST_CHECKSUM_CALCULATION=when_required`) to prevent failures with strict S3 gateways. * **Tests** * Updated/added assertions in controller and Helm/YAML suites to verify the checksum pin is applied to generated and recovery ObjectStores (Postgres and Keycloak). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d3815a4ce6 |
Merge branch 'main' into release-1.6.0
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
235d38dae1 |
Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
80a03b1e09 |
test(backups): assert the barman-cloud sidecar checksum env on rendered ObjectStores
helm-unittest coverage for the S3 request-checksum fix (was mutation-negative before): assert spec.instanceSidecarConfiguration.env pins AWS_REQUEST_CHECKSUM_CALCULATION=when_required on all rendered barman-cloud ObjectStores — postgres backup + bootstrap-recovery (packages/apps/postgres/tests/backup_storage_test.yaml) and the Keycloak DB (new packages/system/keycloak/tests/db_backup_test.yaml). Verified by mutation: reverting the template change now fails these assertions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Andrey KolkovandClaude Opus 4.8
|
b574e2176f |
fix(backups): cover Go-driven system-bucket ObjectStore + share the checksum helper
Address review on the S3 request-checksum fix: - Extend the fix to the platform-managed useSystemBucket=true flow, whose ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via barmanSidecarConfiguration(). The platform's own default system bucket is SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it. - Factor the duplicated instanceSidecarConfiguration block (3x across two charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Andrey KolkovandClaude Opus 4.8
|
0c586b9d72 |
fix(backups): request S3 checksum only when required for barman-cloud
The barman-cloud plugin's sidecar uploads via boto3. Since botocore ~1.36 (early 2025) the default RequestChecksumCalculation is when_supported, which attaches a flexible checksum to every PutObject. Non-AWS S3-compatible backends (Ceph RADOS Gateway, some MinIO / Cloudflare R2 builds) reject the accompanying header with "InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value", so every backup/WAL-archive upload fails against them. Pin the sidecar to when_required via the ObjectStore's spec.instanceSidecarConfiguration.env (botocore honors the AWS_REQUEST_CHECKSUM_CALCULATION env var) on both the keycloak system DB and the postgres app (backup + recovery ObjectStores). when_required is also accepted by AWS S3 on a plain PutObject, so it is a safe default everywhere. This mirrors the etcd-operator fix (cozystack/etcd-operator#342, released in v0.5.3), which applied the same WhenRequired policy to its aws-sdk-go-v2 S3 client for the same Ceph RGW backend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Aleksei Sviridkin
|
858c1c04fe |
feat(postgres): declare the tenant CA trust anchor
CloudNativePG stores its self-signed CA in <release>-ca alongside the private key and offers no way to label it, so a tenant cannot be handed the trust anchor without also being handed the key. Render a TenantProjection that lifts ca.crt alone into the key-free <release>.tenant-ca Secret, and select that label from the postgres tenant-resource definition so the projection is served through the tenantsecrets API. Document the new retrieval path in the chart README. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Ivan Okhotnikov
|
b2e1707632 |
fix(kubernetes-nodes): drop dead ephemeralStorage guard, clarify parity scope
The ephemeralStorage guard could never fire (the pool group is built from a fixed set of chart values that never includes ephemeralStorage), so it read as protection that is not there — remove it. Note in the parity script header that only the four pool objects are compared, not the talos-reconcile Job. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
326dbe0a5b |
test(kubernetes-nodes): cover GPU and kubelet pool shapes in render parity
Extend the golden-parity matrix beyond the resources-sized pool to also diff a GPU pool and a kubelet-override pool against the parent render, so future drift in those branches (which would roll the content-hash-named KubevirtMachineTemplate) is caught in CI. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
f21365c4f0 |
fix(kubernetes-nodes): address review nits in guard comment and NOTES
Reference the adoption migration by role (Phase 2b) instead of a number that does not exist on this branch, and correct the NOTES operational-note count. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
9e2f7cc953 |
test(kubernetes-nodes): run render parity in make test
Wire tests/render-parity.sh into the chart's test target so CI verifies the pool objects stay byte-identical to the parent render. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
3d10a1c431 |
fix(kubernetes-nodes): keep applied TalosConfigTemplate byte-identical to the parent
Restore the inline machineconfig comments (nameservers, GPU node labels) inside the reconcile Job's data block so the applied TalosConfigTemplate matches the parent's, avoiding drift on adoption. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
73be7032a3 |
fix(kubernetes-nodes): drop unbacked cluster option source
The cluster field referenced x-cozystack-options source 'kubernetes', which has no provider (cozystack-api returns NotFound), so the dashboard dropdown 404s. Drop the annotation so the field is a plain text input; a real provider listing Kubernetes CRs is a follow-up. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
fe32fd8e58 |
fix(kubernetes-nodes): scope KMT preservation by release and guard foreign pool names
Replace the collision-prone name-prefix match (pool 'gpu' matched sibling 'gpu-large-<hash>') with a release-name annotation scope, and add a render-time guard that fails cleanly when a pool's MachineDeployment already exists under a different release (e.g. naming a pool after a parent nodeGroup such as the default md0). Document the naming caveat in NOTES.txt. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
1e8d44995c |
fix(kubernetes): scope worker KMT preservation to the release
The preserve-old-KubevirtMachineTemplate block re-emitted any KMT owned by the CAPI Cluster and referenced by a MachineSet. A kubernetes-nodes pool's KMT lives in the same namespace under the same Cluster, so the parent would re-emit it with a foreign meta.helm.sh/release-name and its next upgrade would fail Helm ownership validation. Restrict the block to KMTs whose release-name annotation matches the parent release. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
277b32306c |
test(kubernetes-nodes): add unit tests and golden-parity check
Unit tests cover naming/selectors, clusterName wiring, autoscaler annotations, omitted replicas, bootstrap TCT reference, KMT hash name, worker WorkloadMonitor, and the empty-cluster / mis-prefixed-release render guards. render-parity.sh diffs the pool objects against the parent kubernetes chart and asserts byte-identity. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
fd14126bea |
feat(kubernetes-nodes): add worker node pool chart
Introduce the kubernetes-nodes application chart: one HelmRelease per worker node pool, attached to a parent Kubernetes cluster by name convention (kubernetes-<cluster>). Renders the KubevirtMachineTemplate, MachineDeployment, MachineHealthCheck, worker WorkloadMonitor and the talos-reconcile Job with its RBAC, extracted from the monolithic kubernetes chart. Pool objects are byte-identical to the parent's render for the same group, so a later adoption never churns live worker VMs. This is the additive half of the Phase 2 split (community#8); the parent kubernetes chart is left unchanged. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Myasnikov Daniil
|
14af87296f |
fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux
quay.io serves only currently-tagged manifests: when a rolling tag moves, the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned `centos:stream9` digest has died twice in six days (#3303, #3386), each time breaking `make build` on main and on every PR that triggers a full rebuild. Repinning is not a fix -- the newly pinned digest is always the live tag, so it is next in line to be retired. Move the runtime stage to a registry that provably retains superseded manifests instead, which makes the digest pin durable rather than decorative. Pinning policy is unchanged. Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`, both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4, fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le and s390x -- the same architecture set as the previous base. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Timofei Larkin
|
ebe111c439 |
feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners
Browsers negotiate HTTP/2 exclusively through TLS ALPN "h2". Cilium's Gateway API implementation ships with enable-gateway-api-alpn=false, so every Gateway listener advertised no ALPN protocols and every browser silently fell back to HTTP/1.1 — a regression against the ingress-nginx path, which advertised h2 out of the box. Enable ALPN (h2, then http/1.1) by default on the client-facing hop: - packages/system/cilium: default gatewayAPI.enableAlpn=true. - platform chart: new gateway.http2 value (default true), threaded as inline component values into the cozystack.networking Package for the isp-full and isp-full-generic bundles, so an explicit false overrides the system/cilium valuesFiles default either way. - apps/kubernetes: tenant-cluster cilium gets enableAlpn=true whenever the gatewayAPI addon is enabled; opt out per cluster via addons.cilium.valuesOverride. The backend hop is unchanged: gateway-to-upstream stays HTTP/1.1 unless a Service opts in per GEP-1911 via appProtocol kubernetes.io/h2c (that support is switched on together with ALPN; no cozystack-shipped Service declares such a value today). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Andrey Kolkov
|
e73ad68a85 |
fix(backups): support cnpg-operator with barman plugin (#3313)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Fixes #3300. Migrates CNPG Postgres backups off the deprecated native
`spec.backup.barmanObjectStore` (removed in CNPG 1.29; the
`barman-cloud-*` binaries it needs are absent from the `standard` image
variant keycloak-db pins, and the bare system-flavor tags apps/postgres
pins are deprecated upstream) onto the [CloudNativePG Barman Cloud
plugin](https://github.com/cloudnative-pg/plugin-barman-cloud), across
every Postgres backup surface: `apps/postgres`, `system/keycloak`, and
the `cozy-default-cnpg` BackupClass strategy driver. The chart renders
`spec.plugins` + a `barmancloud.cnpg.io/ObjectStore` CR; the
backup-controller driver SSA-applies an `ObjectStore` and patches
`spec.plugins`; `Backup`/`ScheduledBackup` use `method: plugin`. The
`postgres-operator` chart vendors and co-locates the
`plugin-barman-cloud` operator (with its cert-manager-issued mTLS) so
CNPG can discover it.
One platform-level change is required for the plugin to run and ships
with this PR (see the upgrade notes in
`docs/operations/backup-classes.md` and the release note below): the
CloudNativePG operator is bumped to 1.28.1 (also fixing an operator/CRD
version skew that broke all backups). The other prerequisite — the
LINSTOR scheduler admission webhook stripping
`initContainers[].restartPolicy` cluster-wide, which broke the plugin's
restartable sidecar — is already fixed on `main` (
|
||
|
cozystack-ci[bot]
|
d95685c427 |
Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
9c3bae012a |
fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver
quay.io/centos/centos:stream9 is a rolling tag; upstream deletes the pinned
digest whenever it publishes a new stream9 build. The digest re-pinned in
|
||
|
myasnikovdaniil
|
1d2418e4c4 |
fix(mariadb): derive my.cnf sizing from the instance's own resources (#3345)
## What this PR does The chart shipped a my.cnf with sizing hardcoded for a large dedicated host — `innodb_buffer_pool_size=60G`, `innodb_log_file_size=4096M`, `key_buffer_size=1024M`, `max_connections=4096` — while the default preset `t1.nano` gives the pod 250m CPU and 128Mi of memory, and the default volume is 10Gi. Nothing connected the two, so the defaults asked a 128Mi container for a 60G buffer pool and made every first boot preallocate a redo log worth 40% of the volume. That is not only a slow start. On the current default, MariaDB 10.6 is OOM-killed before it finishes checking its configuration, and on 11.8 the server refuses to start at higher limits with `Cannot map innodb_buffer_pool_size_max`. It survives in practice only because a large node's overcommit heuristics allow the mapping and a small database never touches most of it. Sizing now derives from the instance's own limits through the same helper that fills `resources` on the CR, so both `resourcesPreset` and an explicit `resources` block work. The buffer pool takes half the memory limit, rounded down to a 128 MiB multiple because InnoDB on the 10.6 line allocates in chunks of that size and rounds requests up — a naive value would hand the pod a pool larger than its own limit. Half rather than the conventional 70-80% because that advice assumes a dedicated host where the remainder is page cache and overshoot means swapping; under a cgroup limit the same budget also has to cover per-connection buffers, temporary tables, the key cache and mysqld's own heap, and overshoot means the kernel kills the process. The redo log follows the smaller of the pool and 5% of the volume, and the remaining allocations scale from memory or CPU with floors at the server's own defaults. `innodb_buffer_pool_instances` and `innodb_thread_concurrency` are removed rather than scaled: every shipped server version logs that they were removed and now do nothing, so this only drops a startup warning. Settings that describe durability or disk behaviour rather than instance size — `innodb_flush_log_at_trx_commit`, `innodb_flush_method`, `innodb_io_capacity`, `innodb_fast_shutdown` — are deliberately left alone, since the chart cannot derive them from a pod's resources. ### Upgrade safety An existing instance has a 4096M redo log and will get a smaller one. Shrinking it was verified on every shipped server version — 10.6.24, 10.11.15, 11.4.9 and 11.8.5 — after both a clean shutdown and a SIGKILL with writes still unflushed. In all cases the server replayed crash recovery first and resized the log afterwards, with data intact by checksum and `CHECK TABLE ... EXTENDED`. The ConfigMap carries no operator watch label, so a running healthy instance keeps its current sizing until its next restart; an instance already crash-looping on the old values picks the new ones up when it restarts. ### Release note ```release-note fix(mariadb): size my.cnf from the instance's own CPU, memory and volume instead of hardcoded values that assumed a large dedicated host. Instances on small presets previously advertised `max_connections=4096`, which they could not serve; the limit now scales with memory and starts at the server default of 151. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - MariaDB server settings now scale automatically based on CPU, memory, storage, and configured resource limits. - Improved buffer pool, redo log, connection, cache, temporary table, and thread sizing across instance presets. - Large instances can use buffer pools beyond the previous fixed limit. - **Bug Fixes** - Improved sizing for small-volume deployments. - Removed obsolete MariaDB configuration options for improved version compatibility. - Explicit resource settings now take precedence over preset values. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
2830ad977c |
chore(packages): drop duplicate test targets (#3352)
## What this PR does Three package Makefiles — `apps/mariadb`, `apps/kafka` and `extra/monitoring` — declared an identical `test` target twice, so every invocation printed: ```text Makefile:19: warning: overriding commands for target `test' Makefile:7: warning: ignoring old commands for target `test' ``` Make names the lines in that warning itself: it was already running the second recipe and discarding the first, so the earlier declaration was dead in each case and removing it changes nothing but the noise. All six recipes were `helm unittest .`. Which copy is dead is not the same across the three, so this is not a mechanical edit. In `monitoring` the `.PHONY: test` line sits above the *dead* recipe rather than the live one; since `.PHONY` applies to the target name wherever it is declared, the declaration stays and only the recipe beneath it goes. Default goals are unaffected: `hack/package.mk` sets `.DEFAULT_GOAL=help` and is included ahead of any target in these files, so `test` was never selected by first-target-wins. Confirmed with `make -p --dry-run` before and after, and each package's tests still run — 11, 35 and 9 respectively, with no warnings left. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Streamlined build workflows for MariaDB, Kafka, and Monitoring applications. * Removed redundant test target definitions while preserving the existing Helm unit test capability. * Other generation, update, and image-building workflows remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
055c8c755c |
chore(packages): drop duplicate test targets
Three package Makefiles declared an identical `test` target twice, so every invocation printed "overriding commands for target 'test'" and "ignoring old commands". Make names the lines in that warning: it was already running the second recipe and discarding the first, so the first declaration was dead in each case and removing it changes nothing but the noise. Which copy is dead is not the same in all three, so the edit is not mechanical. In monitoring the `.PHONY: test` line sits above the dead recipe rather than the live one; `.PHONY` applies to the target name wherever it is declared, so the declaration stays and only the recipe under it goes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
ba2623f31d |
chore(mariadb): drop the duplicate test target
The package Makefile declared an identical `test` target twice, so every invocation printed "overriding commands for target 'test'" and "ignoring old commands". Make already ran the second recipe and discarded the first, so removing the first is behaviour-preserving. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4f9ff665da |
fix(mariadb): derive my.cnf sizing from the instance's own resources
The shipped my.cnf hardcoded the tuning of a large bare-metal host: a 60G buffer pool, a 4096M redo log, a 1024M MyISAM key cache, 4096 connections and 24 pool threads. The chart defaults to t1.nano, which is 250m of CPU and 128Mi of memory with a 10Gi volume, so by default a 128Mi container was asked to map 60G of buffer pool and preallocate a redo log worth 40% of its data volume. That is not merely wasteful. With the Linux default heuristic overcommit the 60G mapping is refused outright on a node without the memory to back it, and mariadbd exits with "Cannot map innodb_buffer_pool_size_max"; where the mapping is permitted, the eagerly allocated key cache alone exceeds a small instance's limit and the pod is OOM killed during startup. A nano instance only survives on a node large enough for the mapping to be granted and with a working set small enough never to touch it. Every value that is a memory allocation or a thread count now derives from the memory and CPU limits resolved by the same cozy-lib helper that fills the MariaDB CR's resource stanza, so an explicit resources block and a preset both flow through. Settings that encode durability or storage behaviour rather than size -- the flush method, flush_log_at_trx_commit, io_capacity, fast_shutdown -- are left alone, because the chart cannot infer them from the pod. The buffer pool takes half of the memory limit. The usual advice to give InnoDB 70-80% of RAM assumes a dedicated host where the rest is page cache and overshooting merely swaps; under a cgroup limit the same budget must also cover per-connection buffers, temp tables, the key cache and mysqld's own heap, and overshooting is a kill rather than a slowdown. Sizes above 128 MiB are rounded down to a whole number of 128 MiB chunks. On the 10.6 line InnoDB allocates the pool in 128 MiB chunks and rounds a request up, which would hand the pod more pool than its limit allows; the newer lines either autosize the chunk or ignore it outright, so there the rounding only costs a little pool on a limit that is at least 256 MiB and not a multiple of it -- the rounding applies to half the limit, and below one whole chunk it does not fire at all -- and buys one effective size across every version the chart ships. Most caps are the previously shipped value, so an instance large enough to reach one is sized as it is today, and most floors are the server's own default, so a small instance is not configured more restrictively than an untuned MariaDB. Three clamps deliberately break that symmetry. The buffer pool has no cap: pinning it to the old 60G would keep the largest instances sized for a host they do not run on, so an instance with 128Gi or more of memory is the one case where a value goes up rather than down, to half of a limit it already has. The MyISAM key cache goes the other way, capped at 128M against the 1024M shipped and floored at 8M against a 128M default, because it is allocated eagerly at startup and MariaDB's own system tables are Aria rather than MyISAM, so on a small instance it is close to dead weight charged against the limit. The IO threads are the third, floored at 2 against a default of 4, because they should follow the CPU limit rather than the host. The one user-visible narrowing is max_connections on small instances, which falls from 4096 to the stock default of 151 -- a limit those instances could not have served anyway, and one that now rises automatically as the instance is sized up. Changing the redo log size is safe on every version this chart ships. InnoDB replays the existing log during crash recovery and only then rewrites it at the new size. This was verified on all four shipped versions -- 10.6.24, 10.11.15, 11.4.9 and 11.8.5 -- after both a clean shutdown and a SIGKILL mid-write, with the data intact afterwards. The size is also bounded by a twentieth of the data volume, so first boot no longer preallocates a large part of it -- below roughly 2Gi of volume the 96M floor wins instead and the server's own default is what lands. The ConfigMap is not labelled k8s.mariadb.com/watch, so the operator does not roll an instance when it changes. That is left as it was. An instance crashlooping on the old config picks the new one up as soon as the kubelet restarts it, which is the case this fixes, while a healthy instance keeps its current sizing until it next restarts for some other reason. Rolling every tenant database on a chart upgrade would be the larger risk. innodb_buffer_pool_instances and innodb_thread_concurrency are dropped rather than scaled. Both were removed from the server in 10.6, below the oldest version this chart ships, and each start logged "was removed. It does nothing now and exists only for compatibility with old my.cnf files". Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bbed4892c9 |
fix(mariadb): widen startup probe budget so bootstrap cannot be killed
The operator builds the MariaDB startup probe without setting failureThreshold, so it falls back to the Kubernetes default of 3. The kubelet only starts probing once initialDelaySeconds has elapsed and kills the container on the Nth consecutive failure, so with the operator's initialDelaySeconds of 20 and periodSeconds of 10 the budget is 20 + (3 - 1) * 10 = 40 seconds -- exactly what the liveness probe allows, which leaves the startup probe deferring nothing. Bootstrapping a fresh datadir does not reliably fit in 40 seconds under a constrained CPU limit or slow replicated storage. Overrunning the budget is not a slow start that recovers on the next attempt: the kubelet kills the container after mariadb-install-db has created the system tables but before the entrypoint's temporary server applies the root grant. The restarted entrypoint keys DATABASE_ALREADY_EXISTS off that directory and skips initialization for good, so the server comes up with the install-time root account -- local only and passwordless -- rather than the credentials the operator holds. Every probe from then on fails with "Access denied for user 'root'", so the pod crash-loops permanently with no path back and the install times out waiting for a MariaDB that can never become ready. Raise failureThreshold to 30, giving first boot 310 seconds. Only the threshold is set, so the operator keeps its own probe handler -- the agent HTTP endpoint when replication is enabled, the exec check otherwise -- and the liveness and readiness probes keep their defaults, so a server that hangs after a successful start is still caught as quickly as before. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrey KolkovandClaude Fable 5
|
572492c276 |
docs(backups): scope the barman-binaries claim, drop the stale linstor webhook note
The db.yaml plugins-block comment still carried the over-generalized claim that the pinned postgresql image variants no longer ship the barman-cloud-* binaries; mirror the scoped wording from docs/operations/backup-classes.md (only the standard variant keycloak-db pins lacks them; this chart's bare system-flavor tags still ship them but the flavor is deprecated upstream). After the rebase onto main this PR no longer touches linstor-scheduler: main re-vendored it to upstream chart 0.3.1 / extender v0.3.6, which keeps the admission webhook and stops stripping unknown pod fields (including initContainers[].restartPolicy the barman sidecar needs). The upgrade note claiming the webhook is removed and locality scheduling becomes advisory is rewritten as a hold-back warning. Also: afterwards -> afterward (American English, review nit). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
0b584a1e6e |
chore(postgres): regenerate README after the bootstrap.serverName description
make -C packages/apps/postgres generate re-pads the Bootstrap parameters table for the new bootstrap.serverName row; the committed README still carried the old padding and kept the pre-commit generate check red. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Fable 5
|
67db8f7da7 |
fix(backups): make the barman-cloud serverName stable across every flow
Two defects of the same class — the WAL-archive S3 prefix (serverName) changing or defaulting differently between components: Recovery default regression (review blocker): native CNPG defaulted an absent externalClusters serverName to the entry name (= bootstrap.oldName), but the barman-cloud plugin defaults it to the NEW cluster's name, so a legacy-flow restore with bootstrap.serverName unset would read the wrong S3 prefix (hard failure, or silently restoring another cluster's data). The chart now always emits serverName, defaulting to bootstrap.oldName. Live-cluster serverName flip (found by the gated postgres backup e2e on a real cluster): the chart renders spec.plugins without serverName (effective prefix = cluster name) while a strategy template may name a different one; the driver's ForceOwnership patch then flipped the prefix mid-stream, splitting the WAL archive across two prefixes — the WAL holding the backup's start was lost between them and the restore failed with "WAL not found". The driver now preserves the live Cluster's effective serverName (explicit parameter, or the cluster name when the plugin is attached without one) and only applies the strategy's value to a Cluster with no plugin attached; both charts pin serverName explicitly, and the example strategy no longer sets one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
a1c135c2a1 |
fix(backups): omit empty retentionPolicy from the ObjectStore
Review follow-up (cozystack#3300, non-blocking): the barman-cloud ObjectStore CRD
validates spec.retentionPolicy against ^[1-9][0-9]*[dwm]$, so an operator who
explicitly sets backup.retentionPolicy="" would render retentionPolicy: "" and
the API server would reject the ObjectStore, breaking backup provisioning. Guard
the field in the apps/postgres and keycloak chart ObjectStores with
{{- if .Values.backup.retentionPolicy }} (the Go driver path is already safe via
omitempty). Add a helm-unittest asserting an empty retentionPolicy is omitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|