mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
cc7f98142ae5d9a331d32331d11f02c297aae921
2543
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
889bb79236 |
feat(harbor): restrict the portal and API to listed source ranges (#4300)
## What this PR does Adds `managementWhitelist` to the Harbor app, a list of IP addresses or CIDR ranges allowed to reach the web portal and the Harbor API, and I made it a typed list on purpose. The obvious alternative, a free-form `ingress.annotations` map, gives tenants a way into the shared ingress controller. The platform ingress-nginx runs with `allowSnippetAnnotations: true` (the kubernetes app needs a `configuration-snippet`), and v1.11.5 defaults `annotations-risk-level` to `Critical`, so it accepts any annotation, `server-snippet` included. Tenants can't create Ingress objects themselves, but they can write any field of an app spec. An annotation map on the Harbor spec would let any tenant put arbitrary nginx config into a controller that serves every tenant. A typed list is also what the repo already does for this: `filer.whitelist` in `extra/seaweedfs`, `whitelist` in `extra/ingress` and `tcp-balancer`. The use case is a registry that serves customers: pulls and pushes come from anywhere, the admin UI and API should answer only to an office or VPN range. Right now the portal, `/api/v2.0` and `/v2/` sit on one Ingress and one public host, and a tenant can't restrict any of them. With a non-empty list the existing Ingress gets `nginx.ingress.kubernetes.io/whitelist-source-range`, and `/v2` and `/service/token` move to a second Ingress, `<release>-registry-ingress`, on the same host. ingress-nginx applies that annotation only to the locations of the Ingress that carries it, so those two stay open. They are what a registry client needs: Harbor answers an anonymous `/v2/` request with a Bearer challenge whose realm is `/service/token` on the same host. Everything else under `/service/` is `/service/notifications/*`, internal job callbacks that the bundled nginx already answers with 404, so it stays on the restricted side. The second Ingress uses the same TLS secret and has no cert-manager annotations, otherwise ingress-shim gets two owners for one Certificate. The HTTP-01 solver is its own Ingress and is not restricted. With the default empty list the chart renders exactly what it renders today. An HTTPRoute can't express a source restriction, so in Gateway API mode a non-empty list fails the render. Dropping it silently would leave the portal public while the spec says it is restricted. Each entry has to be an IP address or CIDR range, and a bad one fails the render. ingress-nginx reads an empty `whitelist-source-range` as no restriction and turns an unparseable one into a 503 for the whole location, and the values schema generator can't validate list items. Three things users should know. Harbor-to-Harbor replication that pulls from this registry goes through its API, so the other registry's address has to be on the list. The check uses the client address that ingress-nginx sees: with `externalTrafficPolicy: Local` or PROXY protocol that is the real client, behind another proxy it is the proxy. And for the portal and API the list replaces the tenant ingress `whitelist` rather than narrowing it. Disabling the portal is not here. The vendored Harbor chart (1.19.1) always renders the portal Deployment and its internal nginx always proxies `/` to it, so there is no small switch to wire. The new Ingress is added to the dashboard Role and to the ingress selector in the ApplicationDefinition. ### Screenshots No UI code change. The field reaches the dashboard form through the regenerated schema and `keysOrder`. ### Downstream repositories The terraform provider models Harbor by hand, so the new field needs a follow-up there: cozystack/terraform-provider-cozystack#46. The website reference page is generated from `README.md` on release. Nothing else in the trigger map is touched: no package added or renamed, no platform, installer or `hack/` change. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/46 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(harbor): add `managementWhitelist`, a list of IP addresses or CIDR ranges allowed to reach the Harbor web portal and API. Only `/v2/` and `/service/token` stay reachable from any address. The default is an empty list, which keeps the current behaviour. Not supported with Gateway API publishing: a non-empty list fails the render there. Each entry must be an IP address or CIDR range; for the portal and API the list replaces the tenant ingress `whitelist`. ``` |
||
|
|
54159736bc |
feat(talos)!: bump Talos to v1.14.2 (#4598)
## What this PR does Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4. ZFS moves from 2.4.3 to 2.4.4. On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning on a failed send (EPIPE). What followed was RCU stalls, processes stuck in D-state, a hanging `drbdsetup disconnect` and two-phase-commit timeouts. The [9.3.4 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has fixes for a sender thread pinning a CPU while its connection is down, for two-phase-commit issues and a hanging `drbdadm disconnect`, and for two nodes ending UpToDate with different data after a reconnect. None of them are in 9.3.3. I matched the trace to those entries by their descriptions only, not against the commits. The base installer image had to change. Talos v1.14 no longer publishes `ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses `ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager profiles. I built the amd64 installer from the new profile with `imager:v1.14.2` locally and it builds without errors. PR CI will not boot the new image. It builds the installer and matchbox images, but the e2e runs on the upstream `talos:v1.13.5` container from `hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD and ZFS is the nightly after merge, which builds the nocloud disk from these profiles and runs it in QEMU. On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on demand. After upgrading a node from v1.13.6, `drbd` was loaded and the transport module was not, `drbdsetup new-peer` failed with "Failed to create transport (drbd_transport_xxx module missing?)", and every DRBD resource on the node stayed in Connecting. Listing the module in `machine.kernel.modules` fixed it without a reboot. The cause is [siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565), in Talos since v1.14.0: the kernel is built with an empty modprobe path, so it no longer loads modules on `request_module()`, which is how DRBD asks for its transport. Any module that used to load that way now has to be listed explicitly (reported upstream as siderolabs/talos#14501). Outside DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2 kernel builds as modules, so LVM-thin or multipath users on Talos need them listed too. The e2e node config now loads `drbd_transport_tcp` explicitly, and the talm preset (cozystack/talm#252) and the install docs (cozystack/website#720) get the same line. The module ships in the drbd extension on v1.13 too, so the extra line is safe before the upgrade. The order matters for host nodes. Either update the cozystack preset in the talm project to a version with cozystack/talm#252 and re-render and apply the node config, or add `drbd_transport_tcp` to `machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2. A node upgraded first loses DRBD replication until the module is added. The host network policy now also denies port 2383 to `world`. Talos v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from 2379 to a separate listener on 2383, and the policy only blocked 2379 and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes would be reachable from outside, protected only by client mTLS. Two smaller changes come with the bump. The multus `talos-cni-plugins-checked-against` marker moves to v1.14.2, because the pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as multus. The system memory limits doc now describes the v1.14 OOM trigger, since v1.14.0 dropped its global memory PSI clause ([siderolabs/talos#13895](https://github.com/siderolabs/talos/pull/13895)). Before upgrading hosts, operators should know a few things from the [v1.14.0 release notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the Talos compatibility code: - Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a cluster on Kubernetes 1.31 has to upgrade Kubernetes first. - etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to 2383. The Cozystack etcd scrape proxy reads a separate metrics listener on `127.0.0.1:2381`, which this change does not move. Anything else that scraped or health-checked etcd on 2379 has to switch to 2383, and a firewall outside the cluster that blocked 2379 should block 2383 too. - etcd and kube-apiserver now require TLS 1.3, so a client that can only speak TLS 1.2 stops connecting. - Workload isolation (`SecurityProfileConfig`) stays off on upgraded clusters, but `talosctl gen config` turns it on for new ones. I haven't tested Cozystack with it enabled. - `talosctl apply-config --mode=reboot` is gone. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#720 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: cozystack/talm#252 - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The website PR refreshes the `next` version pins for v1.14.2 and adds the module to the Talos install pages. The talm PR adds it to the cozystack preset. ### Release note ```release-note feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Upgraded host node support to Talos v1.14.2 across machine images, with refreshed firmware and storage extensions. * Updated memory-pressure guidance for Talos v1.14, including version-specific behavior and configuration recommendations. * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383. * Updated cluster preparation to load the TCP transport module required for DRBD. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
031ded71ee |
feat(seaweedfs): alert before an instance stops taking writes (#4573)
## What this PR does A SeaweedFS instance that runs out of room stops taking writes without anything paging. A volume server below its `minFreeSpacePercent` reserve turns every volume it holds read-only; before that, it can run out of slots, and a bucket whose volumes are all full then cannot get a new one, even with half the disk free. Backups archiving into the instance fail in turn, and a Postgres cluster doing continuous WAL archiving keeps the segments on its own volume until that fills too. The generic `KubePersistentVolumeFillingUp` alert does not catch this: it pages below 3% free, while a volume server stalls at its 5% reserve or, when slots run out, with the disk still half empty; and its early warning needs the disk to keep filling, which stops once writes fail. This PR adds per-instance rules to the SeaweedFS system chart, on signals the servers already publish and the platform already scrapes, rendered under the same `monitoring.enabled` gate as the chart's ServiceMonitors: - `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less than 10% of its disk left above the reserve. - `SeaweedFSVolumeServerFull` (warning): a volume server cannot host another volume, because it is at the reserve or has no slot left. - `SeaweedFSWritesFailing` (critical): the master has failed to place a write in every 5-minute window for 15 minutes. Writes may still land in between; the usual cause is a bucket with no writable volume left that the master cannot grow. A new bucket's first writes fail a few picks while its volumes grow, which the 15-minute hold does not reach unless new buckets keep arriving. Every expression is pinned to the release namespace, because the root vmalert evaluates each instance's rules against every namespace's metrics. It also adds `docs/operations/seaweedfs-capacity-for-backups.md`: how to size an instance for continuous WAL archiving, what the rules mean, and how to grow an instance. The expressions were validated live rather than by unit tests, since the repository has no rule-evaluation tooling: on a test cluster, a standalone SeaweedFS 4.45 on a 1 GiB volume was scraped by the platform's vmagent and the rules rendered from this chart were evaluated by the root vmalert. All three loaded healthy and each reached `firing`: `SeaweedFSVolumeServerDiskLow` with 5.7% of the disk left above the reserve, `SeaweedFSVolumeServerFull` both at the reserve and on slot exhaustion, which stopped one bucket at 50% disk use, and `SeaweedFSWritesFailing` under continuous failing writes, resolving once they stopped. A new bucket's first successful writes on a free disk did fail a few picks, which is why that rule holds for 15 minutes. On the healthy platform instance all three stay quiet. helm-unittest pins what a render can break: the namespace selector of each rule, the shared label set of the two arms of the full-server rule, the gate, the critical rule's hold, and that alert templating reaches vmalert unrendered; each assertion was checked to fail when its line is mutated. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches a system chart template, its test and a document under `docs/operations/`; no app values, platform values, ApplicationDefinition, tooling or `hack/` path from the trigger map. ### Release note ```release-note feat(seaweedfs): alert when a SeaweedFS volume server nears its free-space reserve, cannot host another volume, or keeps failing to place writes, and document how to size an instance for continuous WAL archiving. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added namespace-scoped SeaweedFS monitoring alerts for low or exhausted disk capacity, unavailable volume slots, and write-placement failures. Capacity alerts use a 10% disk threshold, and alerts trigger after 15 minutes; write failures are marked critical. - **Documentation** - Added guidance on estimating SeaweedFS storage needs for Postgres backups, interpreting capacity alerts, and expanding storage when supported by the StorageClass. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c983b5c404 |
fix(linstor): use severity warning instead of warn in piraeus-datastore alerts (#3799)
## What this PR does
Seven of the piraeus-datastore alert rules label themselves `severity:
warn`. Alerta, which the monitoring package ships as the alert sink,
validates that value and `warn` is not on its list. Every webhook
delivery for these alerts dies with a 500.
```
Severity (warn) is not one of security, critical, major, minor, warning,
indeterminate, informational, normal, ok, cleared, debug, trace, unknown
```
Alertmanager retries seven times and drops the notification, and the
alert never shows up anywhere. The affected rules are the LINSTOR/DRBD
ones, such as storage pool at capacity, lost quorum, unintentional
diskless and stuck resync. These are exactly the alerts you least want
silently dropped.
Found this on a v1.4.6 cluster where `linstorStoragePoolAtCapacity` was
genuinely firing. The cluster had 153 dropped notifications in 40
minutes and nothing in Alerta. Renaming the label to `warning`, which
the other three rules in the same file already use, stopped the failures
immediately and the alerts came through.
The same labels sit on `main` and in `v1.6.1`, so a backport to
`release-1.6` would be appreciated.
The test commit adds a bats guard so the next one gets caught in CI. It
walks every `PrometheusRule` and `VMRule` under `packages/` outside
vendored `charts/`, helm templates included, and fails on a severity
outside Alerta's default alarm model, which is exactly where `warn`,
`info`, `error` and `fatal` fall. Helm actions are stripped before the
file is read, so a rule behind a `{{- if }}` or an unquoted `{{
.Release.Namespace }}` is checked rather than skipped, and a file that
still does not parse fails the contract instead of thinning it out. On
the current tree that is 194 alerting rules in 41 scanned files.
The guard found one more on `main`. `DatabaseAutoscalerAtMax` in
`packages/system/keda/templates/alerts.yaml` carries `severity: info`,
which Alerta refuses the same way. It is renamed to `informational`, the
value the model has for that level and the one seven other rules in the
tree already use.
Moving that alert off `info` also stops `InfoInhibitor` in
`packages/system/monitoring-agents/alerts/general.rules.yaml` from
firing. That rule matches any `severity="info"` alert and carries
`severity: major`, so on `main` it usually reached Alerta whenever
`DatabaseAutoscalerAtMax` fired, even though the keda alert itself was
refused. After this change no rule carries `info`, so operators no
longer see that `InfoInhibitor` notification.
Fixes #2411.
### Screenshots
Not a UI change.
### Downstream repositories
Walked the trigger map against the diff. It touches severity labels
inside `packages/system/piraeus-operator/alerts/` and
`packages/system/keda/templates/`, plus a bats file under `hack/`. No
package is added or renamed, and there is no values schema, CRD, tooling
the satellites gate on or node contract change. Nothing downstream
restates these labels.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(linstor): LINSTOR/DRBD alerts carried severity `warn`, which Alerta rejects, so they were never delivered. They now use the standard `warning` severity, and the keda at-max autoscaler alert moves from `info` to `informational` for the same reason.
```
|
||
|
|
2c2f2c194e |
fix(dashboard, foundationdb): show the connection ConfigMap to tenants (#4612)
## What this PR does Fixes #4611. Since #4148 a tenant may read the `<release>-config` ConfigMap that holds the FoundationDB cluster file, but the console has no view for ConfigMaps, so the connection string was still reachable only with kubectl. Chart: - `<release>-resourcemap` is rendered regardless of `monitoring.enabled`, because the console now reads it; - the tenant Role grants `get`, `list` and `watch` on `<release>-resourcemap` next to `<release>-config`, still by `resourceNames`, with no wider ConfigMap access. Console: - the application page gets a ConfigMaps tab when the application's resource map lists at least one ConfigMap in the same namespace; - each listed ConfigMap is read and watched with a `metadata.name` field selector, which RBAC checks against `resourceNames`, so a tenant never needs `list` on all ConfigMaps; - values are shown as plain text with a copy button, the cluster file needs no decoding; - a resource map that is missing or forbidden hides the tab, a listed ConfigMap that cannot be read shows the API error instead of looking empty, and a malformed map or an invalid name is reported. The tab is generic, but today only FoundationDB ships a resource map, so it is the only application that gets it. #3490 keeps `monitoring.enabled` as the switch for the resource map and asks whether the map should render unconditionally. This PR answers that for FoundationDB by rendering it always, so the wording in #3490 would need to follow if both land. Tests: `helm unittest` for the chart (6 tests), `tsc --noEmit`, `vitest run` (438 tests in 60 files) and `eslint` on the changed files pass. The new tests cover reading only the listed names in the active namespace, the watch, copying, a clipboard failure, a read denial, and malformed maps. The chart renders with `monitoring.enabled` true and false. ### Screenshots The ConfigMaps tab of a FoundationDB application, rendered by the console from this branch against a local stub API with synthetic data:  ### Downstream repositories Walked the trigger map against the diff: it changes a FoundationDB chart template and its test, and the console. No `values.yaml`, `values.schema.json`, README, CRD or `hack/` file is touched. The website FoundationDB page is generated from the chart README and says nothing about connecting, and the Terraform provider does not read these ConfigMaps. The console is vendored in this repository. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(dashboard, foundationdb): the application page shows the FoundationDB connection ConfigMap with the cluster file to tenants who may read it ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Application details now include a ConfigMaps tab when configuration maps are available. View their values and copy them to the clipboard. * The tab displays loading, empty, unavailable, and error states. * **Bug Fixes** * The resource map remains available when monitoring is disabled, and its access permissions are included. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
51155f1003 |
feat(postgres-operator): alert when the WAL archive backlog does not drain (#4572)
## What this PR does `LastFailedArchiveTime` compares a cluster's last failed WAL archival with its last successful one, so it resolves as soon as any segment gets through. Archiving that fails intermittently, or runs slower than the cluster writes WAL, keeps clearing that alert while the queue of segments waiting on the Postgres volume grows, until the volume is full and the database stops. The alert is also only a warning. This PR adds `CNPGWALArchiveBacklog`, a critical alert that fires when a primary's `archive_status` queue has not been empty once in 30 minutes. A healthy archiver empties the queue within seconds of each WAL switch, so a queue that never drains means point-in-time recovery is falling behind, whatever the archiver reports. Idle clusters and replicas have an empty queue and stay quiet. Validated live, since the repository has no rule-evaluation tooling: on a test cluster, two CNPG clusters archived into an object store that refused connections and into one that accepted them and never answered, while writing WAL continuously. The rule, evaluated by the root vmalert, fired for both. Over the previous seven days of the same cluster's metrics, none of its healthy clusters would have matched; those clusters carry light load, so a cluster whose archiver runs near its limit all the time has not been observed and would page, which is the intended reading of a queue that never drains. The unit test pins only that the rule ships in the rendered PrometheusRule as critical, since the rules reach the cluster through a file glob that drops a moved file silently. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches an operator chart's alert rules and its test; no path from the trigger map. ### Release note ```release-note feat(postgres-operator): page when a Postgres cluster's WAL archive queue has not drained for 30 minutes, which the existing archiving alert misses while some archives still succeed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a critical alert for sustained PostgreSQL WAL archive backlogs. It triggers when the minimum observed ready archive status remains above zero over a 30-minute window, then stays pending for 10 minutes before firing. The alert describes potential impacts on point-in-time recovery and storage usage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
97fa2a619f |
feat(dashboard): show a connection hostname for LoadBalancer services
Users who provision a service through the dashboard had no hostname to connect to: the Services tab showed only the raw LoadBalancer IP, while the pre-1.4.0 UI composed <service>.<root host> for every Service. Render <service>.<suffix> under the External IP of every LoadBalancer Service (also while the IP is still Pending), and nothing for ClusterIP Services. The suffix is config.json serviceDomain, set by the platform value publishing.serviceDomain. There is deliberately no fallback: the app charts fix their external name as <release>.<tenant host> while the LoadBalancer Services are named <release>-external-write and similar, so a name derived from the tenant apex would match neither the DNS nor the certificate. Without a configured suffix nothing is rendered. Runtime config previously reached only the Shell as a prop; it is now published through AppConfigContext and read from there by the Shell and the Services tab alike, so a single test on the rendered version pins the wiring. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
67adfbce3c |
feat(dashboard): surface service-domain in the console config.json
The console loads config.json once at startup and treats a missing serviceDomain key as "show no hostname", so the chart copies _cluster.service-domain into config.json only when it is non-empty rather than shipping an empty string the SPA would have to special-case. The helm-unittest suite pins the unset, empty and set cases. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0bc3f9f25c |
fix(kubernetes): name TalosConfigTemplate by content hash (#3523)
## What this PR does Fixes #3515. `TalosConfigTemplate.spec` is immutable — the `vtalosconfigtemplate.cluster.x-k8s.io` webhook denies any mutation with `TalosConfigTemplate.Spec is immutable` — but the `talos-reconcile` Job applied it under a fixed `<release>-<group>` name on every reconcile. Once the template existed, **no change to the rendered worker machine config could ever reach the cluster**: the apply was rejected server-side while the HelmRelease stayed `Ready`. Reported with `machine.network.searchDomains` after a `clusterDomain` correction; the same held for `certSANs`, `machine.registries` and nameservers. - Name the `TalosConfigTemplate` `<release>-<group>-<6-char hash of its spec>`, mirroring the `KubevirtMachineTemplate` content-hash idiom the chart already uses, and point `MachineDeployment.spec.template.spec.bootstrap.configRef` at that name. A content change now creates a **new** template instead of attempting a rejected mutation, and CAPI rolls the workers onto it. - Extract the machine config into a single named template (`_talosconfigtemplate.tpl`) shared by the two consumers that have to agree on the name byte-for-byte: the Job, which embeds it in the heredoc it `kubectl apply`s, and the `MachineDeployment`, which hashes it into `configRef`. - Assemble the node group once, in `kubernetes-nodes.group`, and read it back at both call sites. Sharing the rendering was not enough: each file also built its own `$group` dict, so the two names agreed only by coincidence of the keys the spec helper happens to read. - The runtime values the Job substitutes (`${TALOS_CA_B64}`, `${SVC_IP}`, `${COREDNS_IP}`, tokens) stay shell placeholders and are deliberately **not** part of the hash: they are facts of an already-provisioned cluster, not chart inputs, so a certificate rotation or a Service re-address does not roll workers. ### What the shared spec carries Worker node pools live in the `kubernetes-nodes` chart, which owns `_talosconfigtemplate.tpl` outright: `make update` copies nothing from the parent chart. The helper renders everything `main` puts into the worker machine config, so the hashed spec is the one the cluster gets: - The heredoc escaping from #3513: backslash, dollar and backtick escaping on `talosVersion`, `installerRepository` and `schematicID`, the escaped `machine.registries.mirrors` block, and the `INVARIANT` comment above `data:`, which belongs in that file since it is where the next field will be added. The spec is still emitted from an unquoted heredoc in a host-cluster Pod, so an unescaped copy would be command injection. - The per-pool install image. `_talosconfigtemplate.tpl` calls `kubernetes-nodes.resolveOsImage` itself instead of reading `talos.schematicID` and `talos.version`, so a pool on `osImage.builtin` or `osImage.factory` reinstalls the schematic and version it actually booted. The install image is part of the hashed spec, so the hash has to be taken over the resolved image, or an `osImage` change would render a different config under an unchanged name. - The Proxmox branches: `nameservers` from `proxmox.dnsServers` where a KubeVirt pool points at the cluster's CoreDNS, and the kubelet's `cloud-provider: external`, without which the Proxmox cloud-controller-manager never initialises the Node. - `machine.kernel.modules`, emitted only when the pool resolves to a non-empty list. `kubernetes-nodes.group` carries every key those read, `podCpuLimit`, `podCpuRequest`, `osImage`, `proxmox` and `kernelModules` included. A key missing there is the failure this PR exists to prevent: with `kernelModules` read at one call site only, an explicit module list hashes to two different names and the `MachineDeployment` points at a template nothing creates. The `kubectl apply` stays `--server-side --force-conflicts` (#4163), so the Job is not OOMKilled on the merge path; with a content-hash name the apply is normally a create or a no-op, and server-side apply keeps the no-op cheap too. ### The replace-on-immutable path and the Role The runtime values stay outside the hash, so one failure is left that the hash cannot absorb: a new apiserver address renders a different spec under an unchanged name, and the webhook rejects it. The replace path from #4089 handles exactly that case and is kept, with its `kubectl delete` pointed at `${TCT_NAME}`. The talos-reconcile Role pins its TalosConfigTemplate `get`, `patch`, `update` and `delete` rights to the pool's own template, so it names the hashed template too. A server-side apply of a template that does not exist yet is still a PATCH on its name: a rule left on `<release>-<group>` would leave the Job Forbidden and the pool with no template to scale onto, with nothing turning red. The name-agreement guard below therefore covers the Role as well. ## Upgrade impact — please read **Existing clusters roll their Talos workers once.** The `configRef` moves from `<release>-<group>` to the hashed name; CAPI creates a new `MachineSet` for it and rolls machines onto it under the existing `RollingUpdate` strategy (`maxSurge: maxReplicas`, `maxUnavailable: 1`) plus the group's `MachineHealthCheck`. The rendered machine config itself is byte-identical to `main` for the same values, so this particular roll delivers no config change — it is the one-time cost of adopting the content-hash name, the same cost `KubevirtMachineTemplate` already paid. Proxmox pools roll the same way: the `ProxmoxMachineTemplate` name does not move, but the `configRef` does. **Until the new template exists, the pool cannot scale.** The Helm upgrade points the `MachineDeployment`'s `configRef` at the hashed name before the `talos-reconcile` Job has created that template, and there is no render-time ordering fix, since the template needs values Helm does not have. Existing workers are untouched in that window, but the pool is frozen: in CAPI v1.10.1, which this repository ships, `getTemplatesAndSetOwner` records the missing bootstrap template, and `rolloutRolling` then fails in `getAllMachineSetsAndSyncRevision` with `cannot create a new MachineSet when templates do not exist` and returns before it scales either the new or the old `MachineSet`. A replica change from the cluster-autoscaler therefore reaches no `MachineSet`: scale-up and scale-down both wait. On an upgraded cluster the Job's inputs already exist, so the window is normally a few seconds. **A failed `talos-reconcile` Job now blocks autoscaling, and nothing turns red.** After `backoffLimit: 30` the Job is not retried until the next Helm upgrade of the pool release, and the HelmRelease stays `Ready` meanwhile, as it did in #3515. Before this PR the same failure left the pool on its old template and still scaling; with it, the pool cannot scale at all until the template exists. Where to look: - the `MachineDeployment` `kubernetes-<cluster>-<group>`: its `ScalingUp` condition under `status.v1beta2.conditions` reads `Scaling up from N to M replicas is blocked because TalosConfigTemplate does not exist` (or `Scaling up would be blocked because …` while no scale-up is pending), and the capi-controller-manager logs carry `cannot create a new MachineSet when templates do not exist` for it; - whether a `TalosConfigTemplate` named after its `spec.template.spec.bootstrap.configRef.name` exists in the tenant namespace; - the `talos-reconcile` Job's status and Pod logs — but the Job has `ttlSecondsAfterFinished: 600`, so a failed Job and its logs are deleted ten minutes after it gives up. Past that, the `MachineDeployment` condition and the missing template are the only evidence left. The pre-existing fixed-name template is **not** deleted — the outgoing `MachineSet` still references it — and is garbage-collected with the `KamajiControlPlane` when the cluster is deleted. Hashed templates therefore accumulate, one per distinct machine config. Pruning them safely is filed as #4355 rather than widened into this PR. **The one roll is not the only one.** Hashing the whole rendered spec makes anything inside it a roll trigger, including two things that are not edits to this cluster's resource. For a pool sized by `instanceType`, the kubelet reservations come from a live `lookup` of the `VirtualMachineClusterInstancetype`, so a later platform-side edit to that instancetype rotates the name and rolls the pool again — and the `KubevirtMachineTemplate` hash this mirrors does not have that property, since it embeds the instancetype by name and never its resolved size. And `talos.registryMirrors` is inside the hashed spec, so setting or changing a mirror now rolls the pool rather than applying only to nodes created later; the field's documentation said nothing about this and now says it. The same holds on Proxmox for `proxmox.dnsServers`, and on every substrate for `kernelModules`: both are inside the hashed spec, so changing either rolls the pool, and the `kernelModules` documentation now says so instead of describing the fixed name as the current state. **The roll is not free on a node group holding a scarce resource.** From exercising this same rollout path on a production fleet of 13 tenant clusters, by rotating the `KubevirtMachineTemplate` hash on a GPU node group. `maxSurge: maxReplicas` is dangerous when the resource is constrained: with 6 GPUs and 1 free, CAPI created all 5 new machines at once and 3 sat `Pending` on `Insufficient nvidia.com/...`, each having already imported a 200 GiB disk. It converges, but it wastes a lot of import. The natural reflex, `maxSurge: 0` with `maxUnavailable: 1`, deadlocks instead: CAPI computes `maxScaledDown = available - (desired - maxUnavailable)`, which was `4 - 4 = 0`, so it removed no old machine while the new ones waited for exactly those machines to release their GPUs; nothing moved until `maxUnavailable` was loosened to 2. And the `MachineHealthCheck` amplifies both — at `maxUnhealthy: 100%` and `nodeStartupTimeout: 10m`, machines queuing for a GPU were declared unhealthy and remediated in a loop, three deleted and recreated while perfectly healthy, each iteration re-importing the disk. The per-node-group `nodeStartupTimeout` override fixes that, but the default is shorter than a plausible queue wait. Anyone upgrading a fleet with a GPU pool hits all three, and this PR is what starts the roll. ## Validation - `helm unittest packages/apps/kubernetes-nodes` passes, snapshots included. Pointing `configRef` back at the fixed name fails `nodegroup_test.yaml`, the snapshot and the name-agreement bats; dropping `osImage` from `kubernetes-nodes.group` fails the `worker_image_*` suites; removing the Proxmox nameserver branch fails `substrate_branches_test.yaml` and the Proxmox heredoc case. - `hack/check-talosconfigtemplate-name-agreement.bats` is new. The unittest suites pin each name against its own literal in its own file, so a divergence shows up as one literal having moved, which reads as a stale fixture and invites re-pasting the value. The guard renders the chart once and compares the Job's `TCT_NAME`, the Role's TalosConfigTemplate `resourceNames` and the `MachineDeployment`'s `configRef.name` against each other, so the failure states the invariant rather than a number. Asserted on a GPU-shaped pool as well as the default one, because the default renders `gpus []`, `kubelet {}` and `instanceType ""` as falsy and a consumer reading a subset of the group is indistinguishable there from one reading all of it; on an explicit `kernelModules` list, which neither of those shapes sets; and a last case pins that the shapes hash differently, so the others cannot hold vacuously. Verified by injecting each divergence: dropping `gpus` at one call site, dropping `kernelModules` at one call site, and pointing the Role back at `<release>-<group>` each fail it, the last one also failing the Role cases in `talos_tct_immutable_test.yaml`, as does pointing the `delete` back at the fixed name. - `hack/cozytest.sh hack/talos-reconcile-heredoc_test.bats` passes, the Proxmox cases included. This is the execution-level evidence that the escaping survived the move into `_talosconfigtemplate.tpl`: it renders with a hostile `registryMirrors` endpoint and a hostile `installerRepository`, runs the extracted heredoc through a real shell, and asserts the values come out literal rather than command-substituted; hostile `schematicID` and `version` values, and `proxmox.dnsServers` entries that are not addresses, are asserted refused at render; and the Proxmox heredoc is asserted to emit the nameservers it was given. A rendered-string regex cannot catch a heredoc the shell refuses to emit; this does. - `make generate` in `packages/apps/kubernetes-nodes` — no diff. - The "byte-identical to `main`" claim was measured against `main` on six pool shapes: default, GPU, an explicit `kernelModules` list, `kernelModules: []` on a GPU pool, `osImage.factory` and Proxmox. For each, the `TalosConfigTemplate` manifest the Job applies differs only in its name (`${RELEASE}-${GROUP_NAME}` → `${TCT_NAME}`); the spec is unchanged, the Job's `TCT_NAME` equals the `MachineDeployment`'s `configRef.name`, and the `KubevirtMachineTemplate` and `ProxmoxMachineTemplate` names do not move. - **Not verified:** no live-cluster or e2e run (needs bare-metal nodes). The upgrade sequence above, the scaling freeze included, comes from reading the CAPI v1.10.1 `MachineDeployment` controller source and the chart's existing `KubevirtMachineTemplate` precedent, not from watching it end to end. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Every box is left empty on purpose, including the first, because I am not confident about one line of the map and would rather a maintainer decided than have me tick it. The change adds no field and changes no default: the only schema movement is the description text of `talos.registryMirrors` and `kernelModules`, regenerated from their `values.yaml` doc comments. Nothing else downstream restates this chart's templates, the hash naming is internal to the chart, and no image reference, CI behaviour or CRD shape moves. The open question is whether `terraform-provider-cozystack` restates field descriptions as well as field shapes; if it does, those description changes reach it and want a follow-up. ## Release note ```release-note fix(kubernetes): name the worker TalosConfigTemplate after a hash of its content so machine-config changes (searchDomains, certSANs, registries, nameservers) actually propagate instead of being silently rejected by the immutability webhook. Upgrading rolls existing Talos workers once, on KubeVirt and Proxmox pools alike. Until the talos-reconcile Job has created the new template, the pool cannot scale, so a failed talos-reconcile Job now blocks cluster-autoscaler scale-up and scale-down while the HelmRelease stays Ready; check the MachineDeployment's ScalingUp condition ("blocked because TalosConfigTemplate does not exist") and the Job status, which is deleted ten minutes after it fails. On a node group holding a scarce resource such as GPUs, review the MachineDeployment surge settings and the MachineHealthCheck nodeStartupTimeout first: `maxSurge: 0` can deadlock the roll and the default startup timeout can trigger spurious remediation while machines queue for the resource. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Worker Talos configuration templates now use content-hashed names for consistent reconciliation and bootstrap references. * Changes to worker machine configuration, including GPU and registry mirror settings, automatically roll existing workers. * **Documentation** * Updated registry mirror documentation to explain that configuration changes roll all workers in the pool. * **Tests** * Added coverage for hashed template names, configuration-triggered worker rotations, and matching references across worker resources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
2fa7fea29b |
fix(dashboard): show ConfigMaps listed in the application resource map
FoundationDB keeps its connection string in a ConfigMap that its operator writes, and tenants may read it, but the console had no view for ConfigMaps. Tenant roles grant ConfigMaps by name only, so the tab takes the names from the application's resource map and reads each one with a metadata.name field selector, which RBAC checks against resourceNames. An application without a readable resource map shows no tab, while a listed ConfigMap that cannot be read is reported instead of looking empty. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
17aaf3e038 |
feat(monitoring): add optional email alert delivery
Platform alerts need an email delivery path alongside Alerta without exposing arbitrary namespace Secrets through tenant-controlled values. SMTP credentials therefore use a fixed Secret name. Alertmanager parses the addresses and connects to the SMTP host only when it sends a notification, so a malformed value there drops alerts instead of failing the release. The chart validates these fields when it renders. Addresses must be literal dot-atom mailboxes, and a domain literal must hold a dotted-quad IPv4 address, because IPv6 literal syntax differs across the supported Go mail parsers. A bracketed smarthost must be a valid IPv4 or IPv6 address, and its port must be between 1 and 65535 without leading zeros. The repeat interval follows the Alertmanager duration grammar and range, so a value that renders is one Alertmanager loads. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM |
||
|
|
0cf6e8c98b |
fix(keda): use severity informational instead of info in the at-max alert
Alerta validates the severity label against its alarm model, which has no info in it, so the DatabaseAutoscalerAtMax notification is refused the same way the piraeus warn alerts were. informational is the value the model carries for this level, and the one the other rules in the tree already use. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM |
||
|
|
3ade16a380 |
fix(linstor): use severity warning instead of warn in piraeus-datastore alerts
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
2c5c230435 |
fix(postgres-operator): count every scraped empty queue in the WAL backlog rule
The per-pod subquery stepped once a minute and took the latest sample at each step, so with a 30-second scrape it looked at half the samples. A queue that emptied only in a skipped sample did not count, and the rule could page on a cluster that was keeping up. While a recreated pod overlapped its old series, taking the maximum also hid the new series' zero. The minimum of each raw series, then the minimum per pod name, sees every sample and still keeps the window across a new instance label. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9afe7f1293 |
fix(kubernetes): assemble the node group once in kubernetes-nodes
The TalosConfigTemplate name is a hash of the machine-config spec, and two
templates have to agree on it: talos-reconcile-job.yaml creates the object,
nodegroup.yaml points MachineDeployment.bootstrap.configRef at it. Sharing the
rendering was not enough — each file also assembled its own $group dict, one
carrying every pool value and one only the few the spec reads, so the two
names agreed only by coincidence of the keys the spec helper happens to read.
Dropping "gpus" from the Job's dict diverged them on a GPU pool with the whole
suite still green, which is the "templates do not exist" deadlock this fix
exists to make impossible.
Move the assembly into a single kubernetes-nodes.group helper, read back with
fromYaml at both call sites, so there is one input as well as one rendering.
Cover it too, since the seam can be reintroduced. The chart's helm-unittest
suites pin each name against its own literal in its own file, which catches a
divergence but reports it as one literal having moved — indistinguishable from
a stale fixture, so re-pasting the new value is the obvious and wrong repair.
hack/check-talosconfigtemplate-name-agreement.bats renders the chart once and
compares the two values against each other, so the failure states the invariant
instead of a number. It asserts on a GPU-shaped pool as well as the default
one: the default renders gpus [], kubelet {} and instanceType "" as falsy, so a
consumer reading a subset of the group is indistinguishable there from one
reading all of it. kernelModules gets a case of its own, because it has no
default and stays unset on both of those shapes. A last case pins that the
shapes hash differently, so the others cannot hold vacuously.
The same guard compares the talos-reconcile Role's resourceNames on
TalosConfigTemplates against that name. The Job applies server-side, and
that is a PATCH on the name even when it creates the object, so a Role
naming anything else leaves the Job Forbidden and the pool without a
template, with the same silent outcome.
Document that talos.registryMirrors and kernelModules sit inside the hashed
spec, so changing either rolls the pool's existing workers instead of only
affecting nodes created later. The kernelModules text described the
fixed-name template as the present state and the content-hash name as a
future one.
No rendered output changes: the golden render snapshots are unchanged.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
|
||
|
|
4570672186 |
fix(cilium): deny world ingress to etcd's HTTP port on Talos v1.14
Talos v1.14 serves etcd's /metrics, /health and gRPC-gateway JSON API on a dedicated listener on 2383 instead of the client port, and its release notes ask for 2383 to be blocked wherever 2379 was. The host policy denied 2379 and 2380 to world, so on v1.14 nodes the JSON KV API would be reachable from outside behind client mTLS alone. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e002f8d5e5 |
feat(talos)!: bump Talos to v1.14.2
Talos v1.14.2 ships DRBD 9.3.4 and ZFS 2.4.4. The v1.13 line tops out at DRBD 9.3.3, which lacks the 9.3.4 fixes for a sender thread pinning a CPU while its connection is down, for two-phase-commit retries and hangs that leave drbdadm disconnect stuck, and for two nodes ending UpToDate with different data after a reconnect. Talos v1.14 no longer publishes ghcr.io/siderolabs/installer, so the imager profiles take installer-base as the base installer, which is also what upstream's own v1.14 imager profiles use. BREAKING CHANGE: Talos v1.14 no longer loads kernel modules on demand, so every node must list drbd_transport_tcp in machine.kernel.modules before it is upgraded, or DRBD cannot connect to its peers. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d902421f47 |
feat(kubernetes): deploy the Proxmox cloud-controller-manager with the cluster
On this substrate the worker kubelet runs with --cloud-provider=external, which taints every node node.cloudprovider.kubernetes.io/uninitialized until a cloud provider clears it. Without a cloud-controller-manager the taint is permanent: nothing but tolerating DaemonSets schedules, and the CSI node plugin — which reads topology.kubernetes.io/region and zone off its own Node and exits fatally without them — never starts. It runs in the management cluster for the same reason as the CSI controller beside it: it authenticates to the hypervisor, and the tenant is cluster-admin of its own cluster. Its whole job is to write on the tenant's Nodes, and it does that over the tenant's admin kubeconfig from out of cluster, which also means there is no tenant-side RBAC left to install. capmox mode, not the default one. The default resolves a node to its VM by name and writes providerID as proxmox://<node>/<vmid>, while capmox reads the CAPI Machine and writes proxmox://<SMBIOS UUID> — the form capmox itself puts on the Machine. With the default the Node and its Machine carry two different identifiers for the same VM. The tolerations are spelled out rather than a blanket `operator: Exists`. That one also matches node.kubernetes.io/unreachable:NoExecute with no tolerationSeconds, which stops admission adding the bounded 300s one and lets the Pod outlive the Node it runs on. `proxmox.insecure` defaults to false: the controllers verify the hypervisor's certificate unless an operator turns that off, and both run here, so the value describes what the management cluster trusts, not the tenant. The README gains a section on what the two tokens can reach — the controller acts on whatever VMID and volume handle the tenant's objects name — and on the per-tenant Proxmox scope that contains it: a storage of the tenant's own and one ACL on the resource pool proxmox.pool puts its workers in, which follows every VM capmox creates — VM.Audit there for the CCM too, which deletes a NotReady Node whose VM it cannot see. ccm, csi and insecure are optional in the schema. Each has a default, and what the substrate needs is enforced at render time — a proxmox cluster without ccm.credentialsSecretName does not render — so a Kubernetes object written before these fields existed stays valid, and adding them is not a breaking change to the API. The kubelet flag is asserted on both substrates: present on proxmox, absent on kubevirt, where nothing would ever clear the taint it causes. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
93c89c8bf6 |
feat(kubernetes): add the Proxmox CSI driver, with its controller in this cluster
Tenant clusters on the proxmox substrate get persistent volumes backed by the
hypervisor's storage. The driver is sergelogvinov/proxmox-csi-plugin, vendored
as a chart that reproduces byte-for-byte from `helm pull`.
It is deployed in halves, and which half goes where is the whole design. The
controller authenticates to Proxmox, so it runs HERE, in the tenant's namespace
of the management cluster, and drives the tenant over its admin kubeconfig —
the same shape templates/csi/deploy.yaml already uses for kubevirt-csi. The
tenant gets the node plugin, the CSIDriver object and the storage classes; its
DaemonSet mounts no credentials.
Putting the controller in the tenant would hand the token to the tenant. The
resource definition lists the admin kubeconfig in secrets.include, so the
tenant is cluster-admin of its own cluster and can read any Secret there, and
the chart writes the driver's config.yaml — token included — wherever it is
installed. The privileges the driver documents are granted on `/`, so that
token reaches every VM and every volume on the Proxmox cluster, not just this
tenant's.
Three consequences of the split are handled here rather than discovered later:
- The vendored chart has no switch for its controller, so the tenant release
sets replicaCount: 0. The Deployment object still renders; no Pod does.
- Its Secret template is gated on a non-empty config.clusters, so an empty
list is what keeps the credential out.
- CSIStorageCapacity is published by the provisioner sidecar, which now runs
here and owns those objects through a Pod that does not exist in the
tenant. Left on, CSIDriver.storageCapacity=true and the scheduler refuses
every Pod with "node(s) did not have enough free storage". The chart
hardcodes that field, so a kustomize postRenderer patches it rather than
forking the chart.
The controller follows proxmox.csi.enabled, like the tenant release: with the
driver switched off there is nothing for it to serve and no token to hold.
Volumes are owned per tenant. The driver names them vm-<controllerVmID>-pvc-…
and defaults the id to 9999; on shared storage that made every tenant's volumes
indistinguishable to an operator and to a VM-level ACL. Each cluster derives its
own from its release name.
What this does not do is make one tenant unable to reach another's volumes: a
tenant can still create a static PV naming a foreign volumeHandle. That needs a
per-tenant Proxmox scope — a dedicated storage and a VM ACL per tenant — which
is a deployment requirement rather than a chart change.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
|
fc729299f4 |
feat(kubernetes-nodes): put Proxmox workers into a resource pool
The privileges a Proxmox token needs — VM.Audit and VM.Config.Disk for the CSI controller, VM.Audit and VM.GuestAgent.Audit for the cloud-controller-manager — are granted on an ACL path, and the only path that follows every VM capmox creates, replacements included, is a resource pool. Without one the choices are `/`, which reaches every VM on the hypervisor, or each VMID, which has to be granted again after every scale-up and every Machine replacement, and until it is the new worker gets no disks and no providerID. proxmox.pool names the pool; capmox adds each worker to it as it clones the VM (ProxmoxMachineTemplate.spec.template.spec.pool). The pool has to exist on the hypervisor — capmox does not create it — and every node pool of a tenant cluster sets the same one, or a worker outside it is a worker the tenant's tokens cannot reach. Empty adds the workers to no pool, which is what the chart did before. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
08c344ab5f |
feat(vpc): expose allowSubnets per subnet (#4478)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
Adds an optional `allowSubnets` list to each entry of `subnets` in the
VirtualPrivateCloud chart, rendered into `spec.allowSubnets` of the
kube-ovn `Subnet`.
Every VPC subnet is rendered `private: true`, so kube-ovn drops any
packet whose source is outside the subnet's own CIDR. A tenant who runs
a VPN or router VM on a subnet already gets the forward path through
`routes` (a static route with the VM as `nextHopIP`), but traffic
arriving from the remote network is dropped by the private ACL. Today
the only fix is an operator patching the cluster-scoped `Subnet` by
hand: the tenant cannot do it, no application value records it, and it
is lost whenever the subnet is recreated.
```yaml
spec:
subnets:
- name: gateway
cidr: 172.16.1.0/24
allowSubnets:
- 192.168.1.0/24
routes:
- cidr: 192.168.1.0/24
nextHopIP: 172.16.1.10
```
The subnet stays private; the list only widens the ingress ACL of the
subnet that declares it, inside the tenant's own VPC. An absent or empty
list renders nothing, so existing releases produce the same manifests.
The chart had no helm-unittest suite, so this adds a `test` target and
`tests/subnet_test.yaml`, covering a subnet without the field, one with
it, the field staying scoped to its own subnet, and an empty list. Two
of the four cases fail against the template on `main`.
`allowSubnets` is a kube-ovn field. #4034 teaches this chart to render
Cozyplane objects when kube-ovn is absent, so whichever of the two lands
second has to decide how the other backend treats the list.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#61 (the provider
models `subnets` by hand in `vpc_schema.go` and `vpc_model.go`, so the
new nested field needs a schema and expand/flatten change there; an
issue rather than a PR, since it should follow this one merging)
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
feat(vpc): subnets accept an optional `allowSubnets` list of CIDRs admitted into the private subnet, so a VM on the subnet can act as a VPN or router gateway without a manual patch of the kube-ovn Subnet.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Private subnets can now allow additional source CIDR ranges, such as
remote networks routed through a gateway. This setting is optional and
can be configured for individual subnets.
* **Tests**
* Added checks for rendering subnet CIDRs, including cases where the
setting is omitted or applied to only one subnet.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
3e4030d4d9 |
feat(kubernetes): per-pool kernel modules for Talos worker nodes (#3571)
## What this PR does Implements gap 1 of #3563: a pool-scoped `kernelModules` field on `KubernetesNodes`, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a kernel module but does not load it — loading is `machine.kernel.modules`' job. The worker machine config had no such block and no values key could add one, so a GPU pool on Talos workers was unusable as shipped: the extension supplied the NVIDIA module, nothing loaded it, and the failure was silent end to end. The VM held the PCI device, the node advertised no GPU, and no component logged an error. Found in a production 1.6.0 environment, where the worker rollover took the 4 GPUs of a tenant cluster out of service with no error anywhere. ### Design: three states, and no default The shape was arbitrated in #3563 (option 3 of the three proposed there). - **absent** — the chart chooses. A pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset`; any other pool gets no `kernel` block at all. - **non-empty list** — taken verbatim, replacing the chart's choice outright. - **`[]`** — explicit opt-out: no modules even on a GPU pool. The order is not cosmetic: Talos loads the list in sequence, `nvidia` has to come first because the other three depend on it, and `nvidia_uvm` is what CUDA unified memory needs. This is the order validated against a production GB202 passthrough node. `NVreg_NvLinkDisable=1` is what the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be off, so nothing else would carry it, and without it the driver waits forever for an NVLink fabric that the passed-through GPU does not have. The three states are distinguishable only because the field carries no default — no entry in `values.yaml`, no `default` in `values.schema.json`. A default of `[]` would collapse "absent" into "opted out" for every pool and make the automatic set unreachable. The generated parameter table prints `[]` in the Default column for any array with no default; the field description says so explicitly. For the same reason `keysOrder` does not list the field, so the dashboard renders it at the bottom of the form. Automatic emission is keyed on the `nvidia.com/` resource prefix, not on the mere presence of `gpus`, so an AMD pool is not given NVIDIA modules on a guess. The field loads modules; it does not put them in the image. Which extensions the image carries is the pool's schematic, so a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic carrying the NVIDIA extensions. Module names and parameters reach the reconcile Job's unquoted heredoc, so both are validated at render, and the emitted list is rebuilt from those two fields alone: the schema does not prune unknown keys, and any other key passed through `toYaml` would reach the shell unchecked. A parameter must be printable ASCII with no whitespace, and must not contain `$`, a backtick, a backslash or quotes. The ASCII half is there because RE2's `\s` and `[:cntrl:]` are ASCII-only: without it U+2028 and U+2029 pass, `toYaml` writes them raw, and Helm's YAML parser reads them as line breaks that end the Job's `command` block scalar early. Inside ASCII the guard denies what the shell expands rather than allowing a fixed set, because NVIDIA's documented multi-value parameter is semicolon-separated. ### Rollout, and #3523 A pool that resolves to no modules renders the machine config it rendered before, byte for byte: `tests/render_snapshot_test.yaml` and its committed golden snapshot pass unmodified, including the `talos-reconcile` Job's content-hash name. For a pool that does get modules, what happens depends on how its `TalosConfigTemplate` is named, and the field description states both cases: - **On main today** the template has a fixed name and an immutable spec, so the reconcile Job's apply of a changed machine config is refused (#3515). Running workers keep what they booted with, and new workers boot from the old template too. This covers the upgrade itself: every existing pool holding an `nvidia.com/*` GPU with the field unset gets the automatic set, and its apply is refused the same way. Only pools created afterwards pick the modules up. - **Once #3523 lands** the template is named by its content, so the same change renames it and CAPI rolls the pool. On the upgrade that brings both, every existing pool holding an `nvidia.com/*` GPU with the field unset rolls onto the automatic set. That roll is the intended outcome — those pools have no working driver without it — but it is a roll, and the release note says so. ### Tests - `packages/apps/kubernetes-nodes/tests/kernel_modules_test.yaml`, 12 cases: the NVIDIA set and its order for an `nvidia.com/*` pool, no `kernel` block for a pool with neither GPUs nor modules, an explicit list taken verbatim including `parameters`, an explicit list replacing the NVIDIA default, `[]` opting out while leaving the `gpu=on` label intact, no NVIDIA assumption for another vendor, a semicolon-separated parameter accepted, a hostile module name and a hostile parameter each rejected, a parameter carrying U+2028 and one carrying U+2029 each rejected with the kernelModules message rather than a YAML parse error, and an unvalidated extra key on an item dropped. The patterns pin absolute indentation, because `modules:` one level out is still valid YAML and silently loads nothing. - `hack/talos-reconcile-heredoc_test.bats` lists `kernelModules` in its validated arm and gains two execution-level cases: a hostile name and a hostile parameter each refused at render, and a semicolon-separated parameter surviving the real shell literally. Loosening the guards to `^.+$` makes the refusal case fail. - `helm unittest packages/apps/kubernetes-nodes`: 31 suites, 174 tests, 12 snapshots, all passing. `hack/cozytest.sh hack/talos-reconcile-heredoc_test.bats`: 8 passed. - `make generate` in the package, and `controller-gen object` for the `api/apps/v1alpha1` submodule. ### Not in scope The rest of #3563: the `nvidia-operator-validator` → device-plugin chain that cannot validate an OS-provided driver, CDI spec generation and the `nvidia` containerd runtime default, and the `maxSurge` / `nodeStartupTimeout` defaults. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/643 - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/20 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Both follow-ups are issues rather than PRs. `terraform-provider-cozystack` is hand-written, so the field needs a schema entry, a model entry and an expand/flatten pair on `KubernetesNodes`; the undefaulted three-state behaviour is exactly where that provider's habit of sending its own defaults would break users silently, since materialising `kernel_modules = []` would disable the automatic NVIDIA set for every GPU pool managed through Terraform. `website` needs the three-state behaviour written down, the GPU pool example pointed at `osImage.factory.schematicID`, and the Blackwell constraint documented: on GB202 the schematic must carry `siderolabs/nvidia-open-gpu-kernel-modules-production`, because with the proprietary extension the module loads, `/dev/nvidia0` appears, and `nvidia-smi -L` then reports `No devices found`. The `hack/talos-reconcile-heredoc_test.bats` change adds test cases and does not change node prerequisites, so `ansible-cozystack` and `talm` are not reached. No other repository in the map is touched: no package, platform component, bundle, variant or release asset is added, and `ApplicationDefinition` semantics are unchanged. ### Release note ```release-note feat(kubernetes): Worker node pools (`KubernetesNodes`) accept a `kernelModules` field, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a kernel module without loading it, so a GPU pool previously came up with no working driver and no way to fix it through values. Left unset, a pool holding at least one `nvidia.com/*` GPU now loads `nvidia` (with `NVreg_NvLinkDisable=1`), `nvidia_uvm`, `nvidia_drm` and `nvidia_modeset` automatically; an explicit list replaces that set, and an explicit `[]` opts out. The modules still have to be in the image: point the GPU pool's `osImage.factory.schematicID` at a schematic carrying the NVIDIA extensions, and on Blackwell (GB202) use the open kernel modules extension, not the proprietary one. Pools without GPUs and without the field render exactly what they rendered before. NOTE: while the worker `TalosConfigTemplate` keeps a fixed name (#3515), a changed module list does not reach a pool that already exists — including the automatic set on existing GPU pools — and only pools created afterwards pick it up. Once that template is named by its content (#3523), the same change rolls the pool, so existing GPU pools that leave the field unset are rolled onto the automatic set. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Worker node pools can now configure Talos kernel modules, including optional module parameters. * When no list is specified, NVIDIA GPU pools automatically receive NVIDIA modules; other pools receive none. A custom list replaces this selection, while an empty list disables modules. * Invalid module names or parameters are rejected during configuration rendering. * **Documentation** * Added guidance on module selection, parameters, image-extension requirements, and pool rollout behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
877d2ff43c |
fix(linstor): build piraeus-server's protobuf code on an arm64 builder (#4576)
## What this PR does The arm64 half of a release fails to build piraeus-server. linstor-server's build always downloads the `linux-x86_64` protoc, and the builder stage runs on the build platform. Since the release builds arm64 on a native arm64 runner, that protoc cannot execute there, and `./gradlew generateJava` fails with `Syntax error: ")" unexpected`. The first multi-arch pre-release run, v1.7.0-alpha.2, stopped on it. A new patch to linstor-server picks the protoc build for the host's architecture. An amd64 builder still gets the same x86_64 protoc as before. The generated code is Java, so one native builder serves both target platforms. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Builds on ARM-based hosts now select a compatible Protocol Buffers compiler download, avoiding failures caused by using an x86_64-only build. Builds on other supported hosts continue to use the x86_64 compiler. This improves compatibility across build architectures without changing the resulting application behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
e56478cf2f |
feat(keycloak): expose organizationsEnabled realm toggle (#3031)
## What this PR does - Adds an `organizationsEnabled` value to the `cozy-keycloak-configure` chart (default `false`, no behaviour change). - Maps it to `ClusterKeycloakRealm.spec.organizationsEnabled` so the Keycloak [Organizations](https://www.keycloak.org/docs/latest/server_admin/#_managing_organizations) feature can be turned on declaratively for the platform realm. ### Why The chart hardcoded the realm without exposing this field. Enabling Organizations through the Keycloak admin console is not durable: the EDP keycloak-operator reconciles `ClusterKeycloakRealm` by `PUT`ting the chart-rendered spec back to the realm, so the console change is reverted on the next reconcile (operator restart, periodic resync, or any CR change). Verified on Keycloak 26.5.2. ### Release note ```release-note feat(keycloak): add `organizationsEnabled` value to the keycloak-configure chart to enable the Keycloak Organizations feature on the platform realm (default: false) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a configuration option to enable or disable organizations in the platform-managed Keycloak realm. * Organizations remain disabled by default. Set the option to `true` to enable them. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
018b49929f |
feat(monitoring): auto-route tenant logs using namespace labels (#2195)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A tenant that runs its own monitoring stack gets a VictoriaLogs instance, but the node fluent-bit in `cozy-monitoring` sends every container log to `global.target` (`tenant-root`), so that instance stays empty. This PR routes each namespace's container logs to the VictoriaLogs named by its `namespace.cozystack.io/monitoring` label, the label the tenant chart already sets and the per-tenant vmagent already selects on. **Chart (`packages/system/monitoring-agents`).** `templates/_tenant-log-routes.tpl` looks up the namespaces and keeps those whose label is set and names a tenant other than `global.target`. For those, fluent-bit gets one `rewrite_tag` filter with a `Rule` per target (a single emitter however many tenants are routed), one `http` output per target to `vlinsert-generic.<target>.svc:9481`, and a per-target `modify Set tenant <target>`, because the re-tagged record re-enters the filter chain where the `Match *` `Add tenant` cannot overwrite the key. Unlabelled namespaces, empty labels and labels equal to `global.target` stay on the default outputs. **Controller (`internal/controller/tenantlogrouting`).** `lookup` results are not part of the digest helm-controller compares, so the chart alone would not pick up a tenant that turns monitoring on, or drop one that is removed, until an unrelated upgrade. The new reconciler in cozystack-controller digests the monitoring label across all namespaces and, when the digest moves, sets `reconcile.fluxcd.io/requestedAt` and `reconcile.fluxcd.io/forceAt` on `cozy-monitoring/monitoring-agents`, which forces an upgrade and re-runs the lookup. It digests every labelled namespace rather than repeating the chart's selection, so it cannot drift from the chart; the cost is an occasional upgrade that renders the same config. The digest it acted on is kept in its own annotation, so a manual `flux reconcile` does not make it force another upgrade. The existing ClusterRole already grants the verbs it needs. **Behaviour change for the platform operator.** Routed records are re-tagged with `keep false`: once a tenant runs its own monitoring, its namespaces' container logs go to that tenant's VictoriaLogs and no longer to the root one. Events and audit logs are not routed and stay on the root store. **Why not a per-tenant collector.** An earlier comment on this PR proposed a fluent-bit per tenant, following the per-tenant vmagent and the OTLP collector from #3763. That collector receives OTLP over the network; a log collector has to tail `/var/log` through `hostPath`, which the PodSecurity level of tenant namespaces rejects, and running it in a privileged system namespace instead would mean one DaemonSet per tenant tailing every node's logs. The routing therefore stays in the one node DaemonSet, with the reconciler closing the reconcile gap. Tests: `tests/fluent-bit_tenant_log_routing_test.yaml` mocks the namespaces with `kubernetesProvider` and pins which get a rule and an output, which are skipped (unlabelled, empty label, label equal to `global.target`), and that a routed record carries the target tenant. Three of its cases fail against the previous revision of this branch. `tests/fluent-bit_tenant_log_routing_offline_test.yaml` pins that an offline render routes nothing. `internal/controller/tenantlogrouting/reconciler_test.go` covers first sight, no-op while labels are unchanged (including after a manual reconcile), a tenant turning monitoring on, a routed namespace being deleted, and a missing release. Refs #2194 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> No entry of the trigger map matches the diff: it touches `packages/system/monitoring-agents` (values, a template helper, tests), a new reconciler in `internal/controller`, its registration in `cmd/cozystack-controller`, and a comment in the cozystack-controller ClusterRole. The one open question is `cozystack/website`: `content/en/docs/next/operations/services/monitoring/logs.md` describes log collection in general terms and does not say where a tenant's container logs land, so the behaviour change above could deserve a sentence there. I have left every box empty for a maintainer to decide. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(monitoring): container logs from a tenant that runs its own monitoring stack now go to that tenant's VictoriaLogs instead of the root one. The node fluent-bit routes them by the namespace.cozystack.io/monitoring label, and cozystack-controller re-renders the node agents when that label changes on any namespace. ``` |
||
|
|
6ab3e0be78 |
feat(postgres): allow preloading an allowlist of shipped modules (#3506)
## What this PR does - Adds `postgresql.sharedPreloadLibraries`, a closed enum of four module names that already ship in the CloudNativePG PostgreSQL image, rendered onto the CNPG `Cluster` field of the same meaning. - Leaves the `shared_preload_libraries` entry in the `postgresql.parameters` blocklist exactly as it is. - Adds `tests/shared_preload_libraries_test.yaml`: absent by default, single and multiple libraries, no interference with `parameters`, schema rejection of a non-allowlisted name and of an absolute path, and the parameter blocklist still firing when the allowlist is in use. - Adds `hack/check-postgres-preload-allowlist.bats`, which renders the chart with `--skip-schema-validation` to exercise the template-side allowlist that helm-unittest never reaches, and checks that it matches the schema enum. `shared_preload_libraries` is blocked in `postgresql.parameters`, and rightly so: the parameter is free text and accepts a path, so it can point at a writable location such as the PGDATA volume. The block does not make every preloadable module unreachable, though. CloudNativePG carries a managed extensions list (`ManagedExtensions` in `pkg/postgres/configuration.go`, checked on v1.30.0, the operator version in this repo): as soon as a parameter under the `pg_stat_statements.`, `auto_explain.` or `pgaudit.` prefix is set, the operator adds that library to `shared_preload_libraries` itself, and for `pg_stat_statements` and `pgaudit` it also runs `CREATE EXTENSION`. The chart already passes such keys through, so `postgresql.parameters."pg_stat_statements.track": all` works on main today. What this PR adds is narrower than a missing capability. `pg_prewarm` has no route at all, and its autoprewarm worker only runs when the module is preloaded. For the other three, the field is an explicit switch that does not require setting one of their tuning parameters, even to its default value, just to get the library loaded. The two routes combine rather than override: the operator merges this list with the libraries it preloads on its own, so removing a module from `sharedPreloadLibraries` does not unload it while a parameter under its prefix is still set. The field description says so. The distinction this PR draws is between a library *name* and a library *path*. Nothing needs to be injected to preload these four: `pg_stat_statements.so`, `auto_explain.so`, `pgaudit.so` and `pg_prewarm.so` are already in `$libdir` in the image, and that directory is on the read-only root filesystem, so it cannot be written to. `dynamic_library_path` stays blocked, so `$libdir` remains the only search path. A bare name from a fixed enum therefore has no way to reach code that is not already in the image, which is the property the blanket block was protecting. The enum in `values.schema.json` is the primary guard and rejects a bad value before rendering starts. The template carries the same list as a backstop for renders that bypass schema validation, using the existing fail-fast `SECURITY:` idiom from the parameter blocklist a few lines below it. helm-unittest validates the schema on every render, so it never reaches that backstop; the bats suite renders with `--skip-schema-validation` instead. I checked that it fails when the template condition is replaced with `false`, and when the template list gains a name the enum does not have. I verified the mechanism end to end on a live CloudNativePG 1.27.3 cluster rather than assuming it: with `spec.postgresql.shared_preload_libraries` set, the operator writes `shared_preload_libraries = 'pg_stat_statements'` into `postgresql.conf`, `SHOW shared_preload_libraries` reports it, and `SELECT ... FROM pg_stat_statements` returns rows. Adding a second module to the list took effect too, and the CNPG instance manager restarts PostgreSQL in place for it, with no pod restart. That restart covers every instance, the primary included, so writes pause briefly. The CRD documents the field as additive to the operator's own defaults, so this does not clobber anything CNPG needs to preload itself. Scope note: the four names are the modules in the image that are useful to a tenant and safe to preload. Anything requiring a background worker or a shipped-but-privileged module was deliberately left out, and the enum is the single place to extend if that changes. ### Screenshots No UI change. ### Downstream repositories `packages/apps/postgres/values.schema.json` gains a field, which per the trigger map reaches `cozystack/terraform-provider-cozystack`: its postgres resource carries a hand-maintained schema and its own expand/flatten pair, with no codegen from `values.schema.json`, so `sharedPreloadLibraries` will be invisible to Terraform users until it is added there by hand. The enum matters twice over there, because the provider restates enums as `stringvalidator.OneOf` lists. I searched that repository and found no open PR or issue covering it (the only open PR is a dependabot Go module bump). I have not filed the follow-up, so I am leaving the box unticked rather than claiming work that does not exist. The same gap applies to `users.<name>.createRole` in #3505; one follow-up there can cover both fields. I walked the rest of the map against the diff and found nothing else: no package is added, renamed or removed, no existing version enum changes, no default changes, `release.prefix` and the output Secret and Service names are untouched, and no CRD outside the regenerated `packages/system/postgres-rd/cozyrds/postgres.yaml` is affected. The `README.md` parameter table is regenerated here and postgres is already in the website generator's app list, so the website path stays automated. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: not filed yet, see the note above - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(postgres): add `postgresql.sharedPreloadLibraries`, a closed allowlist of modules shipped in the PostgreSQL image (`pg_stat_statements`, `auto_explain`, `pgaudit`, `pg_prewarm`) that are preloaded at server start. It is the first way to preload `pg_prewarm`, and lets the other three be preloaded without setting one of their parameters, which CloudNativePG already treats as a request to preload them. The `shared_preload_libraries` parameter stays blocked: only bare names from the allowlist are accepted, never a path. Changing the list restarts PostgreSQL on every instance, the primary included. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional PostgreSQL shared preload library configuration for `pg_stat_statements`, `auto_explain`, `pgaudit`, and `pg_prewarm`. * Preserves the configured library order. CloudNativePG may automatically preload the first three libraries when their related parameters are set; `pg_prewarm` must be enabled through this list. * Changing the list restarts every PostgreSQL instance, including the primary. * **Validation** * Rejects unsupported modules, absolute paths, and attempts to set preload libraries through free-form parameters. * **Documentation** * Added configuration guidance covering supported libraries and restart behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
924d28cf3d |
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Closes the gpu-operator half of #3563 — gap 3 (the operator cannot validate a driver it did not install) and gap 4 (nothing routes CUDA libraries to a tenant pod). Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for the CUDA libraries. Three changes, each closing one silent failure: | Change | Without it | |---|---| | `driver.enabled: false`, `toolkit.enabled: false` | the driver container loads its own module and clashes with the extension's; the toolkit container rewrites `/etc/containerd/config.toml`, which Talos owns and which already registers the `nvidia` runtime with `enable_cdi = true` and `cdi_spec_dirs = ['/run/cdi']` | | `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`, `DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks for a driver-container install, finds none, never writes its sentinel, and the device plugin stays blocked behind `toolkit-validation` — the loop on `Attempting to validate a driver container installation` (**gap 3**) | | `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`, `NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`, `NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no `runtimeClassName` and did not need one on Ubuntu, where the toolkit made `nvidia` the default containerd runtime. Talos does not, so CDI is the only route: the pod gets the device and no CUDA libraries (**gap 4**) | Every value is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. No DaemonSet, no new image, no new package — the fix is values. ### The removed default The previous default fed `NVreg_NvLinkDisable=1` to the driver container through the `nvidia-kernel-module-params` ConfigMap. With that container off the ConfigMap is mounted by nothing, so this removes the default. On Talos the parameter belongs to the worker machine config, which the `kernelModules` field of the worker pool writes: that field is top-level in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by #3571, and its automatic set for a pool holding an `nvidia.com/*` GPU already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart keeps the ConfigMap feature for an operator who re-enables the driver container through `valuesOverride`. A test pins the absence so the dead default cannot come back and offer a second, inert source for the same setting. ### Relationship to #3571 Independent files, no conflict, but only useful together: this makes `driver.enabled: false` correct, and #3571 is what actually loads the modules the extension ships. Merged alone, this changes a GPU node group from "operator fights the OS" to "operator stays out of the way and the modules still are not loaded" — better, but not yet working. #3571 alone leaves the operator fighting for the driver. Either order is safe; both are needed for a GPU node group to work. ### What is verified, and what is not Verified here: the rendered HelmRelease values, that `valuesOverride` still wins over each new key, and that the removed default is gone. The individual values are the ones running in production today, applied by hand through `addons.gpuOperator.valuesOverride`. Not verified here: this exact default exercised end-to-end as the chart's own output. That needs a Talos GPU node group whose modules load, which needs #3571 released. I would rather say so than imply a green e2e that does not exist — happy to hold this as draft until I have run a single-node L40S cluster against the chart default, and I will report the result on this PR either way. ### Not covered HAMi. When `addons.hami.enabled` the operator's device plugin stays off and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring on Talos, which this default does not attempt. Flagging rather than guessing — the production cluster this was derived from runs HAMi disabled, so I have no validated configuration for that path. Worth its own issue if a maintainer confirms the shape. Also not in scope, and unchanged from the merge base: `talos.installerRepository` and `talos.version` reach the worker machine config unvalidated (being tracked separately by @lexfrei), and the `nvidia-operator-validator` chain for the *management* cluster's own passthrough/vgpu variants, which never load the host driver at all. ### Screenshots No UI changes. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/643 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Configuration** * GPU workloads on Talos workers now use host-provided NVIDIA drivers and tooling by default; the GPU Operator’s driver and toolkit are disabled. * Device discovery uses CDI annotations and UUID-based device IDs. When HAMi is enabled, the Operator’s device plugin is disabled. * Host-root validation no longer creates `/dev/char` symlinks. * Driver kernel-module configuration is no longer created by default. * **Documentation** * Clarified when kernel-module configuration applies and what settings are needed to re-enable the driver. * Noted that `valuesOverride` replaces lists wholesale. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d466fd96b3 |
fix(rabbitmq): bound the topology operator Secret cache (#4130)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Fixes #4128. `messaging-topology-operator` 1.14.2 caches **every Secret in the cluster** — its `ClusterRole` grants `secrets: list, watch` cluster-wide with no namespace or label restriction — and the vendored manifest limits it to `500Mi`. On a cluster that has grown enough Secrets it OOMKills and never recovers, because the initial `LIST` alone no longer fits. While it is down, no topology CR reconciles for any tenant. Measured on the affected cluster: 8 211 Secrets across 291 namespaces, 293 MB of Secret payload (87 % of it Helm release history), against 21 topology CRs and 1 `RabbitmqCluster`. Working set sat flat at 331–335 Mi for 30+ hours, crossed 500Mi, then crashlooped 154 times. The operator was caching 8 211 Secrets in order to read 2. Upstream fixed exactly this, and it is not a bigger limit — their default is still `512Mi`: - `329aa9e4e` (2026-06-30, released in v1.20.0) gates the Secret informer on `rabbitmq.com/topology-operator: "true"`, to bound "memory usage on large clusters". - `0850c3ae8` (2026-07-22, released in v1.20.1) bypasses the cache for `Service` reads. v1.20.2 is the current patch release carrying both; v1.19.x and earlier carry neither. This PR refreshes the vendored manifest to v1.20.2 through the package's existing `make update` pipeline (fetch, drop the `Namespace` object, rewrite `rabbitmq-system` → `cozy-rabbitmq-operator`). **The bump alone would break every install**, which is why the app chart moves with it. The app chart's credentials Secret is referenced by `User.spec.importCredentialsSecret`, and from 1.20 the admission webhook rejects a user-provided Secret without the label: ``` secret %q must have label rabbitmq.com/topology-operator=true to be used by the Topology Operator ``` Verified against the 1.20.2 source, two things deliberately do **not** change: - The `RabbitmqCluster` `default-user` Secret needs no label — `rabbitmqclient/cluster_reference.go` reads it through `apiReader.Get`, uncached. - The `Status.Binding` contract is unchanged since 1.14.2, so the vendored `cluster-operator` 2.9.0 stays compatible and is left alone to keep this PR focused. The v1.20.2 manifest is self-contained: it ships its own `Issuer` and both `Certificate` objects (webhook + metrics), and adds probes, a `securityContext` that satisfies `restricted` PodSecurity, a defaulting `MutatingWebhookConfiguration` and secure metrics. Same 13 CRDs, none added or removed. ### Tests - `packages/apps/rabbitmq/tests/user_credentials_test.yaml` renders two users and asserts that each `User` imports its own `<release>-<user>-credentials` Secret and that every credentials Secret carries `rabbitmq.com/topology-operator: "true"`. Deleting the label line from the template fails it. - The rabbitmq e2e suite now creates its cluster with one user and asserts `User/rabbitmq-test-app` reaches `Ready`, so the webhook and the User reconcile run against the real 1.20.2 operator. Its `catch` gains a describe of that `User` and the topology operator's logs. ### Upgrade path Existing installs carry unlabeled credentials Secrets until each tenant's `rabbitmq-*` HelmRelease re-renders. No migration is added: in 1.20.2 the User controller watches Secrets and maps an import Secret back to its Users through the `.spec.importCredentialsSecret.name` field index (`secretToUserRequests`), so relabelling the Secret requeues the User immediately rather than on the next interval, and nothing is removed from the broker in between. The import path no longer writes `<user>-user-credentials`, and nothing in this repo reads the ones 1.14.2 left behind. ### Verification - `make generate` in both touched packages — no diff beyond the intentional changes - `helm unittest packages/apps/rabbitmq` — 103/103, and the new suite fails with the label removed ### Screenshots No UI changes. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> Walked the trigger map against the diff, file by file. `packages/apps/rabbitmq/templates/rabbitmq.yaml` adds one label to a rendered Secret: no package added or renamed, no `values.schema.json` field, no version enum, no `values.yaml` default, no `release.prefix`, and the Secret's *name* is unchanged, so the provider's literal-name lookups are unaffected. `packages/system/rabbitmq-operator/templates/messaging-topology-operator.yml` is a vendored upstream manifest for an already-present component: no package layout change, no `cozyrds` rename, no namespace rename, no installer or platform values key. `packages/apps/rabbitmq/tests/` and `hack/e2e-chainsaw/rabbitmq/` are test-only and appear nowhere in the map. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(rabbitmq): update messaging-topology-operator to 1.20.2, which bounds its Secret cache to Secrets labeled `rabbitmq.com/topology-operator=true` instead of caching every Secret in the cluster. Before this, the operator's memory grew with the cluster's total Secret count and OOMKilled against its 500Mi limit on large clusters, stopping reconciliation of all Queue, Exchange, Binding, User, Permission, Vhost and Policy resources for every tenant. The managed RabbitMQ chart now labels the credentials Secret it creates, so existing users need no action. If you manage topology resources by hand and point `connectionSecret`, `importCredentialsSecret`, `uriSecret` or `upstreamSecret` at your own Secret, add the label `rabbitmq.com/topology-operator: "true"` to it or the admission webhook will reject the resource. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configuration options for resource cleanup, queue types, connection and channel limits, shovel queue arguments, and federation settings. * Added metrics support, health checks, leader-election configuration, and queue webhook capabilities. * Improved compatibility of generated credential Secrets with the RabbitMQ Topology Operator. * **Bug Fixes** * Improved validation and documentation for topology resources, including requiring an exchange for topic permissions. * Updated operator components and Kubernetes resource configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3c657a4cc1 |
feat(vpc): expose allowSubnets per subnet
Every VPC subnet is rendered private, so kube-ovn drops any packet whose source lies outside the subnet's own CIDR. A tenant that runs a VPN or router VM on a subnet and points a static route at it gets the forward path from the existing routes value, but traffic arriving from the remote network is then dropped by the private ACL. Subnet is cluster-scoped, so the only workaround so far has been an operator patching spec.allowSubnets by hand: something the tenant cannot do, that no application value records, and that is lost whenever the subnet is recreated. Declaring it per subnet keeps the widening in the tenant's own VPC and scoped to the one subnet that needs it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
6c7d22edab |
fix(linstor): build piraeus-server's protobuf code on an arm64 builder
linstor-server's build always downloads the linux-x86_64 protoc, and the builder stage runs on the build platform. The release now builds the arm64 images on a native arm64 runner, where that protoc cannot execute and generateJava fails. A patch picks the protoc build for the host's architecture. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
35e3593c85 |
fix(seaweedfs): describe the writes-failing alert by what it measures
The rule fires when the master fails a volume pick in every 5-minute window for 15 minutes. That proves recurring refusals, not that no upload gets through: writes into other buckets can succeed in between, and a steady stream of new buckets, each failing a few picks while its first volumes grow, produces the same pattern. The master publishes no count of successful picks to tell the two apart, so the rule stays and its description and the operations page now say what it has seen. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e23c6af59b |
fix(postgres-operator): keep the WAL backlog window across a recreated pod
The scraped series carries the pod IP in its instance label, so a recreated primary starts a new series, and min_over_time over it saw only the samples since the restart. The rule could then page ten minutes after the pod came back with a queue that was still draining, not after the 30 minutes its description promises. Grouping by pod name first keeps one series per pod across the restart, and the description now says what happens to a pod name that is newer than the window. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9b56e70ca6 |
feat(mongodb): support the system-bucket backup flow (useSystemBucket) (#3967)
## What this PR does MongoDB was the only `cozy-default` app with **no `useSystemBucket` flow**: its backup storage was always static on the `PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*` values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant could not back up to the platform system bucket out of the box the way Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default MongoDB backup-restore works OOTB. PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR fields** (not secret refs), and the platform bucket name/endpoint are only known at BackupJob time — so, unlike ClickHouse's runtime `secretKeyRef` env, the chart cannot render them. The implementation mirrors the CNPG driver's pattern: - **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket` flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret. Helm never owns `spec.backup.storages`, so a Flux re-render can't fight the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors Postgres, and `useSystemBucket: true` with `backup.enabled: false` is refused both at render time and by an `x-kubernetes-validations` rule on `backup` in the schema (the same mechanism as the `storageClass` immutability rule). Note on the schema rule: the aggregated apiserver does not evaluate CEL on application-backed resources yet (#2657), so today it is a UI/schema-level marker and the render-time refusal is what stops a Flux release; it also trips the API review gate (“validation rule added”), so this PR needs an API-owner approval on top of the code review. - **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates (`endpointUrl` already scheme'd by the `backupstrategy-controller.endpoint` helper, `credentialsSecret: cozy-backups-creds`, `prefix: <namespace>/<application>`). - **Driver** (`psmdbstrategy_controller.go`): SSA-injects `spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB` at BackupJob time under a dedicated field manager (`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is **gated on the app's `backup.useSystemBucket` flag** (a legacy app leaves it false and is never touched; a useSystemBucket app is re-injected once per mint (once per BackupJob, plus once per storage-race re-mint), before its operator CR is created, so a later coordinate change catches up on the next job while a streaming dump is never patched under). The backup CR is minted only once the operator reports the storage pushed into pbm (`status.observedGeneration` caught up and `PBMReady` True): pbm writes to what its own config says, not to the spec, so a CR minted before that push would land at the previous coordinates while its Backup recorded the new ones; the job waits at `PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup deadline. Once a job has its CR, none of the mint-time reads or holds can fail it: the strategy and app reads the artifact still needs are bounded by the 24h running ceiling instead, and the cluster is not read at all **and on the release having rendered that flag**: the app CR is projected from the HelmRelease's desired values, so the flag reads true before helm-controller renders the revision, while the tenant's own storage, scheduled tasks and PITR stream from the previous render still sit on the cluster. As long as the cluster carries `spec.backup.tasks` or `spec.backup.pitr`, the driver does not inject and holds the BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it at the deadline), so a wedged legacy revision never has its nightly dump and oplog redirected into the shared bucket. The backup snapshot falls back to the strategy coordinates when the operator status echo is empty; that fallback is gated on the CR's own `percona.com/delete-backup` finalizer (the flow the CR was minted on), not on the app flag re-read at completion, so a legacy backup never records the platform bucket and a flag flipped mid-dump does not change the artifact. The restore path rebuilds `backupSource` from it (the `cozy-backups-creds` reference is projected into the restore namespace before dispatch). - **Retention/cleanup**: on the `useSystemBucket` flow the driver owns the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is gated out, and an on-demand backup is not covered by task retention). It stamps the operator's `percona.com/delete-backup` finalizer on each backup it mints **against a storage that carries the platform coordinates** — the strategy's bucket and `credentialsSecret`, read back from the live cluster after the injection — never on the flag alone: a strategy without an `s3` block, or a storage that points anywhere else, holds the job (`MongoDBStrategyHasNoS3` / `PerconaServerMongoDBStorageForeign`) instead of minting a finalized dump against a bucket the platform does not own. On Cozystack `Backup` deletion (by hand today; by platform retention once it lands, since a `Plan` only creates BackupJobs and never deletes a `Backup`) the driver deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer frees the object — the finalizer's presence is the ownership marker cleanup reads back. That reading is unambiguous because cleanup resolves the CR by the name recorded in `driverMetadata` (the one the driver minted), not by scanning: the psmdb operator does stamp `percona.com/delete-backup` on its own `cron-*` task-minted backups, but those are never in scope. A legacy backup keeps the pre-existing hands-off contract, decided from the snapshot before any read of the operator CR. A terminating namespace or the `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips the driver's finalizer and releases; a `Get` of the operator CR that fails with anything other than NotFound/NoMatchError still requeues, and the operator's own `release-lock` finalizer can hold the CR while the operator is down, so those two are what can keep a namespace in `Terminating`. `schedule` and `retentionPolicy` are inert on this flow (there is no engine-side retention and a `Plan` has no retention field yet); platform-level retention is tracked in [#4192](https://github.com/cozystack/cozystack/pull/4192) and teardown GC of orphaned backups in [#3966](https://github.com/cozystack/cozystack/issues/3966), so until those land bound `cozy-backups` with an S3 lifecycle policy. - **Bounds on the waits**: the injection race with the operator's cluster cache (an `unable to get storage` error on a freshly injected storage) is retried by deleting the errored CR and minting a fresh one, with the retry count kept on the BackupJob (`backups.cozystack.io/mongodb-storage-race-retries`) because the deleted CR is reaped before the next poll; after three retries the operator's error is terminal. A system-bucket dump that has started streaming is not cancelled on the 30-minute deadline, but fails after 24 hours in `running` (`BackupAbandoned` warning naming the operator CR, which is left in place as the handle to whatever pbm still writes). The deadline cancel of a not-yet-started CR pins its delete to the resourceVersion the live read answered for. - **RBAC**: `patch` on `perconaservermongodbs` for the injection (server-side apply), and `create`/`delete`/`patch` on `perconaservermongodbbackups` — `delete` to prune the shared-bucket archive on `Backup` deletion, `patch` to strip the driver's finalizer when releasing a wedged cleanup. Coverage: Go unit tests over `internal/backupcontroller` — the injection gate keyed off the app flag (legacy cluster never patched) and off the applied render (a cluster still carrying tasks or pitr is held, untouched), the SSA-injection precondition with the whole entry (per-application prefix included) asserted on the cluster, strategy templating per application, the finalizer refused against a storage without platform coordinates (strategy without `s3` over a legacy entry, foreign coordinates on the merged entry), the retry budget driven with the operator simulated reaping the deleted CR between polls, the restore snapshot fallback and credential swap across all four legacy/system flow combinations, the finalizer-owned cleanup with its escape hatches (skip annotation before any read, `NoMatchError` treated as gone, terminating-namespace release using the pre-read CR, a failed strip surfaced as an Event, a legacy CR never deleted, a legacy snapshot released without a read, a system-bucket snapshot reaching the prune), the deadline cancelling a not-yet-running operator backup only on a positive live read (including no live client at all, and a CR that moved since the read), and the running ceiling. Plus helm-unittest on both charts (chart skips the creds Secret and omits `spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps them on the legacy flow; the strategy renders the S3 coordinates and the `insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore e2e against the platform bucket is a follow-up (needs the system bucket wired in CI, same status as the Postgres/ClickHouse useSystemBucket e2e). One item to confirm in e2e: the psmdb operator accepting `spec.backup.enabled=true` with an empty `storages` between install and the first BackupJob. ### Screenshots Not applicable (no UI changes). ### Downstream repositories The tenant-facing schema surface is one new field, `spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD (regenerated), plus a validation rule on `spec.backup`. `terraform-provider-cozystack` hand-models MongoDB and may want to expose it, but it is a new optional boolean that defaults false (no drift for existing users), and the chart `README.md` regenerates on release. Leaving every box unticked per the trigger-map guidance so a maintainer decides on the provider follow-up. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added opt-in system bucket mode for MongoDB backups. * Automatically injects platform bucket storage without application-managed S3 credentials. * Supports endpoint, region, path prefix, addressing, and TLS settings. * Manages shared-bucket backup artifacts when system bucket backups are deleted. * **Bug Fixes** * Improved restore credential selection to avoid cross-bucket mismatches. * Preserved backup metadata when storage details are unavailable. * **Documentation** * Documented configuration, scheduling, retention, PITR limitations, and restore guidance. * **Validation** * Prevented incompatible bootstrap and system bucket restore configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
e510d49b3d |
feat(monitoring): re-render node agents when a monitoring label changes
The node fluent-bit builds its tenant routes from a lookup of namespace labels, and helm-controller upgrades a release only when its chart or values digest changes. A tenant turning monitoring on therefore got no route, and a removed one kept its route, until an unrelated change upgraded monitoring-agents. The reconciler digests the label across all namespaces and requests a forced upgrade of the release when the digest moves. It digests every labelled namespace rather than repeating the chart's selection, so it cannot drift from the chart; the cost is an occasional upgrade that renders the same config. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
cdff5ed380 |
feat(monitoring): route tenant logs to the tenant's own VictoriaLogs
A tenant that runs its own monitoring stack gets a VictoriaLogs instance, but the node fluent-bit sends every container log to global.target, so that instance stays empty. The tenant chart already names the stack a namespace reports to in namespace.cozystack.io/monitoring, the label the per-tenant vmagent selects on, so the log path keys off the same label. The routed record leaves the default outputs (keep false): a tenant with its own monitoring holds its logs, the root store no longer does. The re-tagged record re-enters the filter chain, where the Match * "Add tenant" cannot overwrite the key, so a per-target Set labels it with the tenant it is sent to. One rewrite_tag carries every rule, keeping a single emitter however many tenants are routed. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
874590e200 |
test(keycloak): pin organizationsEnabled rendering in configure-kk
Cover both paths of the new value: the chart default renders spec.organizationsEnabled: false (no behaviour change), and setting organizationsEnabled: true renders through to the ClusterKeycloakRealm. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
adb2918daf |
feat(keycloak): expose organizationsEnabled realm toggle
The keycloak-configure chart hardcoded the realm with the Organizations feature off, with no value to enable it. Operators who flip organizationsEnabled in the Keycloak admin console see it reverted on the next ClusterKeycloakRealm reconcile, since the EDP operator pushes the chart-rendered spec back to the realm. Expose organizationsEnabled in values.yaml (default false, no behaviour change) and map it to ClusterKeycloakRealm.spec.organizationsEnabled so the feature can be enabled declaratively and survive reconciles. Assisted-by: LLM Signed-off-by: mattia-eleuteri <mattia@hidora.io> |
||
|
|
f7c04b7a39 |
feat(kubernetes): expose kernel modules for Talos worker pools
A Talos system extension installs a kernel module but does not load it; loading is machine.kernel.modules' job. The worker machine config had no such block and no values key could add one, so a GPU pool on Talos workers was unusable as shipped: the extension supplied the NVIDIA module, nothing loaded it, and the failure was silent end to end. The VM held the PCI device, the node advertised no GPU, and no component logged an error. The field is three-state, and the states are distinguishable because it carries no default: absent lets the chart choose (a pool holding at least one nvidia.com/* GPU gets nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset, in that order, since Talos loads the list in sequence and the last three depend on the first; any other pool gets nothing), a non-empty list replaces that choice outright, and an explicit [] opts out even on a GPU pool. A default of [] would collapse absent into opted-out for every pool and make the automatic set unreachable. The automatic nvidia entry carries NVreg_NvLinkDisable=1. Individual GPUs are passed into worker VMs without their NVSwitches, and on non-Talos workers the gpu-operator driver container applies that parameter; on Talos that container has to be off, so nothing else would. Names and parameters reach the reconcile Job's unquoted heredoc, so they are validated at render and the emitted list is rebuilt from those two fields alone: the schema does not prune unknown keys, and any other key passed through toYaml would reach the shell unchecked. A pool that resolves to no modules renders the machine config it rendered before, byte for byte, so its content-hash Job name does not rotate. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
669fb2416a |
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for CUDA libraries to reach a tenant pod. Three things change, each fixing one silent failure: driver.enabled and toolkit.enabled go false. The driver container would load its own module and clash with the extension's; the toolkit container would rewrite /etc/containerd/config.toml, which Talos owns and which already registers the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi']. validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and disables /dev/char symlink creation. The driver validator otherwise looks for a driver container install, finds none, never writes its sentinel, and leaves the device plugin blocked behind toolkit-validation — the loop on "Attempting to validate a driver container installation". devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need one on Ubuntu, where the toolkit made nvidia the default containerd runtime; Talos does not, so CDI is the only route for the CUDA libraries. DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set and tested alongside it: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of them breaks device resolution on its own. The previous default fed NVreg_NvLinkDisable=1 to the driver container through the nvidia-kernel-module-params ConfigMap. With that container off the ConfigMap is mounted by nothing, so the default is removed. On Talos a module parameter belongs to the worker machine config, which the kernelModules field of the KubernetesNodes app (packages/apps/kubernetes-nodes, #3571) writes. The wrapper chart keeps the ConfigMap for an operator who re-enables the driver container. Every value here is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. Not covered: HAMi. When addons.hami.enabled the operator's device plugin stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent CDI wiring on Talos that this default does not attempt. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
bb3ddee233 |
fix(rabbitmq): bound the topology operator Secret cache
messaging-topology-operator 1.14.2 caches every Secret in the cluster: its ClusterRole grants secrets list/watch cluster-wide with no namespace or label restriction, so the controller-runtime cache holds the whole set. Against the vendored 500Mi limit this OOMKills on any cluster that has grown enough Secrets, and it never recovers, because the initial LIST alone no longer fits. While it is down no topology CR reconciles, for any tenant. Observed on a cluster with 8211 Secrets across 291 namespaces (293 MB, 87% of it Helm release history): a flat 333Mi working set for days, then the ceiling, then a permanent crashloop. The operator was caching 8211 Secrets to read 2. Upstream fixed this by gating the Secret informer on a label, released in v1.20.0, and by bypassing the cache for Service reads, released in v1.20.1. Refresh the vendored manifest to v1.20.2, the current patch release carrying both, through the package's existing make update pipeline. The bump alone would break every install: the app chart's credentials Secret is referenced by User.spec.importCredentialsSecret, and from 1.20 the admission webhook rejects a user-provided Secret that does not carry rabbitmq.com/topology-operator=true. Label it in the same change, pin the label in the chart's unit tests, and give the e2e RabbitMQ a user so the webhook and the User reconcile are exercised against the real operator. The RabbitmqCluster default-user Secret needs no label, it is read uncached via apiReader, and the Status.Binding contract is unchanged since 1.14.2, so the vendored cluster-operator 2.9.0 stays compatible and does not move here. Refs #4128 Signed-off-by: Mattia Eleuteri <mattia@hidora.io> Assisted-by: LLM |
||
|
|
25cdd51881 |
feat(postgres): allow preloading an allowlist of shipped modules
shared_preload_libraries is blocked in postgresql.parameters, and rightly so: the parameter is free text and accepts a path, so it can point at a writable location such as the PGDATA volume. What the block leaves is an indirect route. CloudNativePG preloads pg_stat_statements, auto_explain and pgaudit by itself as soon as a parameter under their prefix is set, such as pg_stat_statements.track, so enabling one of them today means setting one of its tuning parameters, even to its default value. pg_prewarm has no route at all, and its autoprewarm worker only runs when the module is preloaded. postgresql.sharedPreloadLibraries is the explicit switch. It is a closed enum of four module names that already ship in the CloudNativePG PostgreSQL image, rendered onto the CNPG Cluster field of the same meaning, which the operator merges with the libraries it preloads on its own. A bare name can only resolve inside $libdir, which sits on the read-only image filesystem, and dynamic_library_path stays blocked, so no path can be introduced. The blocklist on postgresql.parameters is left untouched. The template repeats the enum as a backstop for renders that skip schema validation. helm-unittest always validates the schema and never reaches it, so a bats suite renders with --skip-schema-validation to keep the backstop, and its agreement with the enum, under test. Verified on a live CloudNativePG 1.27.3 cluster: with the field set, the operator writes shared_preload_libraries into postgresql.conf, SHOW reports the modules, and pg_stat_statements returns rows. Changing the list restarts PostgreSQL on every instance, the primary included, through the instance manager and without restarting the pod. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
292189c7fa |
feat(seaweedfs): alert before an instance stops taking writes
A SeaweedFS instance that runs out of room fails quietly. A volume server below its minFreeSpacePercent reserve turns every volume it holds read-only, and one out of slots cannot host the new volume a bucket needs once its volumes are full; either way uploads fail, backups archiving into the instance fail in turn, and their WAL piles up on the Postgres volume. Nothing pages on the way: the generic PVC alert waits for 3% free, below the 5% reserve and far below a server that ran out of slots with half its disk free, and its early warning needs the disk to keep filling, which stops once writes fail. The rules read what the servers themselves publish, the space left above the reserve, the slots left and the master's failed volume picks, so they follow the reserve and the size limit an instance is configured with instead of restating them. A new bucket's first writes fail a few picks while its volumes grow, so only failures that go on for 15 minutes page. Every selector is pinned to the release namespace, because the root vmalert evaluates each instance's rules against every namespace. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1e1cddb0a0 |
feat(postgres-operator): alert when the WAL archive backlog does not drain
LastFailedArchiveTime compares the last failed archival with the last successful one, so it resolves as soon as any segment gets through. A cluster whose archiving fails intermittently, or runs slower than it writes WAL, clears that alert every time one upload succeeds while the queue of segments waiting on the Postgres volume keeps growing, until the volume is full and the database stops. That alert is also only a warning. A healthy archiver empties archive_status within seconds of each WAL switch. A queue that has not been empty once in 30 minutes means point in time recovery is falling behind whatever the archiver is reporting, so that is what pages, as critical. Idle clusters and replicas have an empty queue and stay quiet. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8b861668b8 |
ci(release): build pre-release images for amd64 and arm64 and gate releases on it (#4505)
## What this PR does Pre-release builds now publish every image as an amd64 and arm64 index, and a release fails if anything it ships is single-arch. Main, release lines and pull requests keep building amd64 only. It builds on #4498, which made every image buildable for both architectures. Part of #1961. **This PR is on hold until everything under "Merge after" has landed.** Merged earlier, the new gate fails the next rc on the images those PRs fix. ### How a pre-release is built A pre-release tag gets a second job, for arm64, next to the existing release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`. `prepare-release` waits for it, and if the arm64 job fails the rc fails with it. There is no amd64-only fallback. Stable tags do not run it: promotion copies the rc digests with `skopeo copy --multi-arch all`, so a stable release inherits the indexes. After the amd64 build, the stitch joins each amd64 image with its arm64 twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag of the amd64 image moves to the index, the component-versioned ones included. The script then rewrites the digests in the tree and republishes the packages artifact and the installer chart pinned on them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an old digest survives anywhere outside `charts/`. ### The gate A new check fails the rc when a pinned digest in the tree is not an index with both linux/amd64 and linux/arm64. It lives in `hack/verify-multiarch.sh`. It checks first-party and third-party refs alike. Images that are amd64 by nature go into `hack/multiarch-allowlist`, one repository per line, each with a mandatory reason. Today that is only the e2e sandbox. An entry that matches nothing is reported as stale. The static gate only sees digest-pinned refs in the tree. A tag-only ref, an image from a vendored chart default, or one an operator starts at runtime is invisible to it. So the rc e2e also audits every image its nodes pulled, with the same check and the same allowlist. Neither covers a package the e2e never installs. ### Nightly A new nightly workflow builds main for arm64. It keeps the arm64 build cache warm, since the rc job only reads it. It also prints the single-arch refs that would block the next rc. Nothing it builds is published. ### Smaller changes - `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so the arm64 build keeps a cache of its own. - `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the amd64-only e2e sandbox. - The kamaji provider image is pushed under the build's `IMAGE_TAG` like every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent builds overwrote each other's component tag. Closes #4503. - `hack/nightly-mirror.sh` verifies a copy against the raw manifest digest. Plain `skopeo inspect` resolves an index to one platform's child. - matchbox is rebuilt for both platforms in the amd64 release job. The arm64 leg skips the talos package, so there is no arm64 half to stitch, and its Dockerfile only copies files, so no QEMU is needed. Releases then network-boot arm64 machines too. Closes #4524. The stitch also rewrites the kamaji ref inside `files/components.gz`, recompressed with `gzip -n` so the bytes are reproducible, so the kamaji control-plane provider ships multi-arch too. keda and kuberture now name the repository next to their pinned digest, so the gate can resolve them; their rendered manifests do not change. ### Merge after - #4552 builds the Talos installer, matchbox, Harbor and the Velero KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator, kilo and migration-controller to the root `build:` list. It replaces #4507, #4515, #4525 and #4528. - #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy to their multi-arch releases, which are already published. The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and #4522 here, and the multi-arch PRs in ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy. ### Verification The bats suites pass: stitch, verify-multiarch, build-matrix, common-envs, nightly-mirror and the release contracts. The first three also pass under `hack/cozytest.sh` with dash. Each new test was red before its implementation, and the verify-multiarch and stitch tests each have a mutation that turns them red. I ran `verify-multiarch --report` against main. It lists the first-party refs the stitch will fix, plus the third-party and special cases above. The workflows have not run yet. The arm64 job's tools and duration, the stitch against real registries, and the audit are checked for the first time on the nightly and on the next rc. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Nothing under `hack/` is moved or renamed, and no make target changes its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The satellite repositories listed under "Merge after" are prerequisites, not follow-ups this change forces on them. ### Release note ```release-note ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Release candidates now build amd64 and arm64 images in parallel and combine eligible images into verified multi-architecture indexes before publication. * A nightly arm64 image build is available for testing and supplies a build cache for release-candidate builds. * **Bug Fixes** * End-to-end checks audit whether pulled images support both architectures. Audit failures block prerelease checks, while stable releases continue with a warning. * End-to-end test artifacts now include collected image references and multi-architecture audit results. * **Documentation** * Updated release and image guidance covers multi-architecture builds, verification checks, and troubleshooting, including arm64 build and stitching failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d9ed566326 |
fix(migration): restore the class reclaim policy on imported volumes (#4480)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The VM import handoff sets the transferred `PersistentVolume` to `Retain` so the data survives the swap from the Forklift-populated claim to the VMDisk's claim, and then never gives the policy back. Every imported disk therefore ends up with `persistentVolumeReclaimPolicy: Retain`, while its StorageClass says `Delete` and every VMDisk created directly carries `Delete`. When the tenant later deletes an imported VMDisk through the normal path, its DataVolume and claim go away, but the volume stays `Released` with its backing storage: an orphaned, billed volume that no tenant can see or remove. #### How it was measured On a lab cluster with a LINSTOR-backed `replicated` class (`reclaimPolicy: Delete`): a 16 GiB VM imported from vSphere produced a VMDisk whose PV was `Retain`, while every non-imported VMDisk's PV in the cluster was `Delete`. Deleting the VMInstance and then the VMDisk removed the DataVolume and the claim and left the PV `Released`, still `Retain`, with its LINSTOR resource in place. Patching that PV back to `Delete` made the CSI provisioner delete both the PV and the LINSTOR resource, which confirms the reclaim policy is the only thing keeping it. #### The fix `Retain` is now held only for the window it protects. A new step, `settleReclaimPolicy`, restores the policy the volume's StorageClass declares (`Delete` when the class names none or no longer exists) once the handoff is provably over, and until then reports "not yet" and touches nothing: - the replacement claim is `Bound` and the PV is `Bound` with a `claimRef` naming that claim by UID, as the PV controller reports it rather than as the controller wrote it, so no release is in flight; - the transferred claim is gone; - the claim's one and only owner is the VMDisk's DataVolume, as a controller reference whose UID matches the live DataVolume (CDI's adoption), so the only deletion that can reach the volume is the VMDisk's own; - the VMDisk exists and carries this task's output markers. `fulfill` runs this step after adopting every disk and before creating the VMInstance and deleting the scaffolding VM, so a VM is reported `Succeeded` only once its volumes carry their class's policy, and an import that fails after this point still leaves disks that clean up like any other. The step is a compare-and-set, so repeated passes and restarts are no-ops once settled. It also runs on the path that recovers outputs whose status was never written, for tasks caught mid-handoff by an upgrade. The comment that called permanent `Retain` "part of the contract" and the matching paragraph in `packages/system/migration-controller/README.md` are updated; the README also says how to fix volumes left on `Retain` by imports completed before this change, since those tasks are terminal and are not reconciled again. In practice a VM now stays in `Creating` for one more requeue (about 20 s) after the swap, while the PV controller rebinds and CDI adopts the claim. #### Tests - `TestSettleRestoresTheClassReclaimPolicy`: a finished handoff gets the class's policy back: `Delete`, `Retain` when the class asks for it, and `Delete` when the class names none or is gone. - `TestSettleIsIdempotent`: a second pass over a settled volume reports settled and does not rewrite the PV. - `TestSettleKeepsRetainWhileTheHandoffIsOpen`: thirteen cases, each breaking one condition (claim not created by this import, claim not bound, PV still `Released`, `claimRef` still on the old claim, `claimRef` on the new claim with a stale UID, transferred claim still present, claim not yet adopted by CDI, an extra owner, a DataVolume owner that is not the controller, an owner reference to a previous DataVolume, DataVolume missing, VMDisk missing, VMDisk not created by this import). Every one keeps `Retain`. - `TestFulfillRestoresTheReclaimPolicyBeforeFinishing`: behavioural regression over `fulfill`. The first pass swaps the volume, keeps `Retain`, creates no VMInstance and keeps the scaffolding VM. After simulating the PV controller and CDI, the second pass restores `Delete`, creates the VMInstance and removes the scaffolding VM. With the new step turned into a no-op, this test fails. - `TestFulfillSettlesReclaimedOutputs`: the recovery path for outputs whose status was never written. With the handoff still open the volume keeps `Retain`; once it is finished the policy goes back to `Delete`. - The existing `adoptVolume` tests still assert `Retain` during the swap. `go test ./internal/migrationcontroller/...` and `make test-controllers` pass, `go vet` is clean, and `make generate` leaves the tree unchanged. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `internal/migrationcontroller/`, `packages/system/migration-controller/README.md` and a comment in `packages/system/migration-controller/templates/rbac.yaml`, which match no entry of the trigger map; the website's VMware migration page does not describe the reclaim policy, so it needs no change. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(migration): disks imported from VMware no longer keep a Retain reclaim policy after the import. The volume gets its StorageClass policy back once it is bound to its VMDisk, so deleting an imported VMDisk deletes its volume instead of leaving an orphaned Released PersistentVolume. Volumes imported before this fix keep Retain and have to be patched back by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Imported volumes are protected during claim transfers, then returned to their StorageClass reclaim policy after the replacement claim and volume are safely adopted. If the class or policy is unavailable, the policy defaults to `Delete`. * VM imports wait for volume adoption and reclaim-policy settlement before completing, helping prevent premature volume deletion and incomplete imports. Previously unrecorded handoff outputs are also settled before being returned. * **Documentation** * Clarified reclaim-policy behavior during imports and how to restore cleanup behavior for volumes from older imports. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
717d091970 |
fix(backups, postgres): start a Postgres restore from the backup it names (#4443)
## What this PR does A Postgres `RestoreJob` now starts from the base backup its `Backup` refers to, and the PITR documentation describes the window the recovery actually has. **Starting from the named backup.** Without `recoveryTarget.backupID` the barman-cloud plugin picks the base backup itself (`internal/cnpgi/restore/restore.go` in plugin-barman-cloud, then `FindBackupInfo` in barman-cloud `pkg/catalog`): the newest backup of the catalog when no `recoveryTime` is set, otherwise the newest one whose end is not after `recoveryTime`, on any timeline since no `targetTLI` is passed. A `RestoreJob` naming an older `Backup` therefore started from another one, and on an archive that holds sibling timelines — left by in-place restores that archived back under the source `serverName`, before restored clusters got a fresh `newServerName` — it could start from another branch and bring back that branch's data. `FindBackupInfo` gives `backupID` precedence over time-based selection, and CNPG's webhook does not count it among the mutually exclusive targets, so it combines with `targetTime`. - on restore the CNPG driver reads barman's `backupId` and the backup's `stoppedAt` from the `Backup`'s `cnpg.io/Backup` and sets the new chart value `bootstrap.backupID`, rendered as `bootstrap.recovery.recoveryTarget.backupID`; - it does not pin for a `recoveryTime` earlier than one second past `stoppedAt`: the named backup cannot serve an instant before it is consistent, and `stoppedAt` is truncated to the second while the catalog compares barman's end time to the microsecond. Those requests keep the plugin's time-based choice, as today; - it does not pin either when the `cnpg.io/Backup` is gone. plugin-barman-cloud's retention deletes it together with the base backup it describes (`deleteBackupsNotInCatalog`), and the driver deletes one only with its artifact, so a missing one means the catalog has lost that base backup: pinning it would purge the target and then fail the recovery. The plugin's choice recovers to the same end of the archive, and the `RestoreJob` gets a `BaseBackupGone` warning event. A base backup deleted from the object store by other means is still pinned for as long as its `cnpg.io/Backup` stays, and that restore fails after the purge; plugin-barman-cloud deletes such a `cnpg.io/Backup` on its next maintenance pass (`deleteBackupsNotInCatalog`, every `retentionPolicyIntervalSeconds`, retention policy or not), but only from the primary of the cluster that took it, under the same UID. Telling it apart earlier would need reading the barman catalog, which the driver does not do; the documentation says so, and that deleting the `cnpg.io/Backup` by hand is the way out; - a re-restore into the same target clears a `backupID` left by a previous one. **Documentation** (`docs/operations/backup-classes.md`, PostgreSQL PITR): - the upper bound of the window is the last transaction (commit or abort) in the archived WAL, not the newest segment: recovery ends just before the first transaction after the target, and a segment switch or a base backup is not one. On a database at rest every instant after its last write fails with `recovery ended before configured recovery target was reached`, including "at/before the most recent completed backup", which the page recommended as the safe choice; the state as of the last write is obtained by omitting `recoveryTime`; - aiming at a backup's `stoppedAt` misses that backup (the plugin keeps only a backup whose end is not after the target), falling back to an older one or to `no target backup found`; the page now says to aim one second past it; - when a base backup removed from the bucket outside retention is still pinned, and how to unpin it; - how to read the upper bound from the archive, what pinning the base backup changes, and a section on archives with several timelines: PostgreSQL follows `recovery_target_timeline: latest` from the backup's timeline through the consecutively numbered history files it can find, so a missing `.history` confines the replay to the backup's own timeline and the bound is the last transaction on that timeline, while a targeted timeline that does not descend from the backup stops recovery with `requested timeline N is not a child of this server's history`. **Tests.** `TestCNPGRestoreBackupID` (the pinning rule), `TestReconcileCNPGRestore_PinsTheRequestedBackup` (drives a restore through the purge and reads back the patched `Postgres` app, including a `cnpg.io/Backup` removed by retention, with and without `recoveryTime`), `TestBuildPostgresAppRestorePatch_ReplacesBackupID`, a deep-copy test for the new `stoppedAt` field, and three `helm unittest` cases for `recoveryTarget`. With the rule disabled, 4 of the rule cases and 3 of the reconcile cases fail; with the template change removed, 2 chart cases fail; the retention case fails against the first version of this PR, which pinned an id recorded on the artifact; the deep-copy test fails once `StoppedAt` is dropped from the generated `DeepCopyInto`. `go test ./internal/backupcontroller/...` and `helm unittest packages/apps/postgres` (139/139, after rebasing on main) pass. Not run: the e2e suite (`hack/e2e-chainsaw`), and a restore against a live cluster with this driver. ### Screenshots No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/716. `content/en/docs/next/operations/services/backup-classes.md` (and `v1.6/`) carries a copy of the corrected "The recoverable window" and "Discovering the earliest / latest restorable time" sections; the window and `stoppedAt` corrections apply to released versions too, the `backupID` part only once this is released. - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: none expected. The diff adds `bootstrap.backupID` to the postgres `values.schema.json`, but the provider does not manage the `bootstrap` block (`internal/provider/postgresql_schema.go`: "bootstrap blocks use server defaults"), and the `RestoreJob` CRD is unchanged. Left unticked for a maintainer to confirm. - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(backups): a Postgres RestoreJob now starts from the base backup its Backup refers to (recoveryTarget.backupID) instead of whichever backup the barman-cloud plugin picks, which on an archive with several timelines could restore another branch's data; a recoveryTime before that backup's end keeps the previous time-based choice. The PITR documentation now bounds the recoverable window by the last archived transaction rather than the newest WAL segment. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Restores can target a specific base backup and optionally replay to a requested recovery time. If no backup is specified, the newest eligible backup is selected automatically. * **Bug Fixes** * Restore handling warns when the referenced base backup is unavailable and allows the plugin to select a backup from the catalog. * **Documentation** * Clarified recovery-window boundaries, timestamp precision, idle databases, timeline branches, and behavior when a requested target cannot be reached or predates the earliest base backup. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c740031453 |
Merge remote-tracking branch 'origin/main' into ci/build-arm64-leg
Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
497dd6fc4f |
fix(backups): start a Postgres restore from the backup it names
Without recoveryTarget.backupID the barman-cloud plugin chooses the base backup itself: the newest one in the catalog when no recoveryTime is set, and the newest one ending by recoveryTime otherwise, on any timeline since no targetTLI is passed. A RestoreJob naming an older Backup therefore started from another one, and on an archive holding sibling timelines - left by in-place restores that archived back under the source serverName - from another branch, bringing back that branch's data. The restore reads barman's backupId and the backup's stoppedAt from the live cnpg.io/Backup behind the Backup, and passes the id through the chart as bootstrap.recovery.recoveryTarget.backupID. The plugin gives backupID precedence over time-based selection, and CNPG does not count it among the mutually exclusive targets, so it combines with targetTime. Two cases keep the plugin's choice. A recoveryTime earlier than one second past stoppedAt: the named backup cannot serve an instant before it is consistent, and stoppedAt is truncated to the second while barman compares its end time to the microsecond. And a cnpg.io/Backup that is gone: plugin-barman-cloud's retention deletes it together with the base backup it describes, and the driver deletes one only with its artifact, so pinning its id would purge the target and then fail the recovery on every retry. That restore records a BaseBackupGone warning event instead. A re-restore into the same target clears a backupID left by a previous one. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
c6a4933cf6 |
test(migration): cover the recovery settle and the handoff guards
Recovering outputs that no status recorded goes through fulfill's vm == nil branch, which settles the reclaim policy of volumes an older controller left on Retain. Nothing drove that branch, so the write path could be removed with the suite still green. A fulfill test now runs it with the handoff open and finished. The guards that stop the controller from rewriting the policy of a volume it did not produce, or before CDI controls the claim, get one case each: a claim or VMDisk marked by another import, a non-controller DataVolume owner reference, and a claimRef naming the replacement claim under a stale UID. The storageclasses RBAC comment now also names the reclaim policy read. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
6555ae16b8 |
fix(mongodb): start a backup only once the operator has pushed the storage into pbm
The wait after a storage injection watched the cluster's spec, which is not what pbm writes to. At the pinned operator version the backup reconciler hands pbm only a storage name and records status.s3 from the spec entry it read, while the storage pbm uses is whatever the cluster reconciler last pushed into pbm's own config; that push flips PBMReady False and back to True and the reconcile stamps observedGeneration when it ends. A backup started between the apply and that push dumped to the previous coordinates while its Backup recorded the new ones, so both its restore and its prune pointed at the wrong bucket. The wait was also no wait at all: the status write that named it re-queued the job at once, and the live read always showed the apply. The mint now waits for observedGeneration to reach the cluster's generation and for PBMReady to be True, and fails at the backup deadline if the operator never gets there. The strategy, app and cluster reads still ran ahead of the operator CR lookup and were bounded by the backup deadline, so a strategy absent for one reconcile 40 minutes into a streaming dump failed the job and left the finalized CR unreachable. The CR is resolved first now; with one in hand the cluster is not read at all, and the reads the artifact needs are bounded by the running ceiling, past which the job fails naming the CR left in place. The settle check treated a field the entry lacked as agreement, so a strategy that newly set forcePathStyle minted in the same pass as the apply that changed addressing. A missing field now counts as its zero value, which keeps an entry the operator rewrote without an empty region settled while catching a value the apply adds; the storage type is compared outright. The runbook still said the finalizer keeps a Plan from accumulating objects, the RBAC comment still said the timeout cancel runs on either flow, and four wait messages read a duration as a delay. All corrected; the runbook also describes the pbm wait. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |