mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
4d7f9e485ab46e401bde15e7f3c09adebbbfc657
495
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
cdc4b88a90 |
fix(ci): close e2e suite selection gaps in select-e2e and select-install (#4634)
## What this PR does The e2e test selector picked the wrong Chainsaw suites for some diffs. In one case it lost an escalation, in another it ran the full suite where a subset was enough, and in two more it missed a suite or ran nothing. The fixes are in `hack/select-e2e.sh`, with the `.yml` part also in `hack/select-install.sh`. - A changed file in a directory under `hack/e2e-chainsaw/` that holds no suite escalated only when it was the whole diff. Next to any path that selected a suite, the escalation was dropped with nothing on stderr. The directory name is now checked against the suite list right where the path is classified, like the `hack/e2e-apps/` rule already did. The empty-selection backstop can't be reached anymore. I kept it as a guard, and the doc says its line has no test. - A seaweedfs change ran the full suite, because `cozystack.seaweedfs-application` mapped to a `seaweedfs` suite that does not exist. Now it selects the suites that use tenant-root's object storage: bucket, the backup round-trips that write through the in-cluster `seaweedfs-s3` endpoint, and harbor, which keeps its registry in a BucketClaim served by the seaweedfs COSI driver. The issue says this mapping would break the round-trip test. It doesn't, since that test only walks suite to source to suite, and `vm-disk-application` already maps to `vminstance` the same way. The test takes the expected set from the suites' own yaml. - A suite written as `chainsaw-test.yml` was invisible to both selectors. Chainsaw v0.2.15 loads that name when there is no `chainsaw-test.yaml` (`pkg/discovery/load.go`), so such a suite runs but never gets selected. Both scripts list it now. `select-install.sh --validate` also passed when `find` failed, and now it fails. - An edit under `examples/backups/_lib/` was read as an app named `_lib` and selected nothing. It escalates now, same as `hack/e2e-chainsaw/_lib/`. - The selector comments quoted bats file counts and escalation-branch counts. All of them were stale, so I removed them. The doc half of #4120 was already fixed on main. Each fix has a test that fails on main. I also mutated each fix: its own test goes red and no other test does. Three related gaps outside these lines are tracked in #4630: the `packages/core/testing/Makefile` comment that names only `chainsaw-test.yaml`, suites built from numbered step files, and a file placed directly in `examples/backups/`. Two things worth knowing before merge: - Any file except the `.disabled` one in a parked suite's directory, like `hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml`, now always runs the full suite. That is the safe direction. If it gets expensive, those directories could be made inert. - The `examples/backups/_lib/` rule and its doc line point at a directory another PR adds. If this PR merges first, the rule just has nothing to match yet. Closes #3816, closes #3710, closes #4283, closes #4120, closes #4205, closes #4626. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): e2e test selection no longer drops an escalation for a non-suite path under hack/e2e-chainsaw/, selects the suites that use seaweedfs instead of the full run, sees suites written as chainsaw-test.yml, and escalates edits to examples/backups/_lib/. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * End-to-end test selection now recognizes suites configured with either supported YAML filename and reports discovery failures instead of silently proceeding. * Changes to shared backup walkthrough helpers and unrecognized test-suite directories now trigger broader test coverage. * Storage-related changes in SeaweedFS now select all relevant test suites. * **Documentation** * Updated testing guidance to describe suite discovery, file-change matching, and when the full test suite is selected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
031ded71ee |
feat(seaweedfs): alert before an instance stops taking writes (#4573)
## What this PR does A SeaweedFS instance that runs out of room stops taking writes without anything paging. A volume server below its `minFreeSpacePercent` reserve turns every volume it holds read-only; before that, it can run out of slots, and a bucket whose volumes are all full then cannot get a new one, even with half the disk free. Backups archiving into the instance fail in turn, and a Postgres cluster doing continuous WAL archiving keeps the segments on its own volume until that fills too. The generic `KubePersistentVolumeFillingUp` alert does not catch this: it pages below 3% free, while a volume server stalls at its 5% reserve or, when slots run out, with the disk still half empty; and its early warning needs the disk to keep filling, which stops once writes fail. This PR adds per-instance rules to the SeaweedFS system chart, on signals the servers already publish and the platform already scrapes, rendered under the same `monitoring.enabled` gate as the chart's ServiceMonitors: - `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less than 10% of its disk left above the reserve. - `SeaweedFSVolumeServerFull` (warning): a volume server cannot host another volume, because it is at the reserve or has no slot left. - `SeaweedFSWritesFailing` (critical): the master has failed to place a write in every 5-minute window for 15 minutes. Writes may still land in between; the usual cause is a bucket with no writable volume left that the master cannot grow. A new bucket's first writes fail a few picks while its volumes grow, which the 15-minute hold does not reach unless new buckets keep arriving. Every expression is pinned to the release namespace, because the root vmalert evaluates each instance's rules against every namespace's metrics. It also adds `docs/operations/seaweedfs-capacity-for-backups.md`: how to size an instance for continuous WAL archiving, what the rules mean, and how to grow an instance. The expressions were validated live rather than by unit tests, since the repository has no rule-evaluation tooling: on a test cluster, a standalone SeaweedFS 4.45 on a 1 GiB volume was scraped by the platform's vmagent and the rules rendered from this chart were evaluated by the root vmalert. All three loaded healthy and each reached `firing`: `SeaweedFSVolumeServerDiskLow` with 5.7% of the disk left above the reserve, `SeaweedFSVolumeServerFull` both at the reserve and on slot exhaustion, which stopped one bucket at 50% disk use, and `SeaweedFSWritesFailing` under continuous failing writes, resolving once they stopped. A new bucket's first successful writes on a free disk did fail a few picks, which is why that rule holds for 15 minutes. On the healthy platform instance all three stay quiet. helm-unittest pins what a render can break: the namespace selector of each rule, the shared label set of the two arms of the full-server rule, the gate, the critical rule's hold, and that alert templating reaches vmalert unrendered; each assertion was checked to fail when its line is mutated. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches a system chart template, its test and a document under `docs/operations/`; no app values, platform values, ApplicationDefinition, tooling or `hack/` path from the trigger map. ### Release note ```release-note feat(seaweedfs): alert when a SeaweedFS volume server nears its free-space reserve, cannot host another volume, or keeps failing to place writes, and document how to size an instance for continuous WAL archiving. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added namespace-scoped SeaweedFS monitoring alerts for low or exhausted disk capacity, unavailable volume slots, and write-placement failures. Capacity alerts use a 10% disk threshold, and alerts trigger after 15 minutes; write failures are marked critical. - **Documentation** - Added guidance on estimating SeaweedFS storage needs for Postgres backups, interpreting capacity alerts, and expanding storage when supported by the StorageClass. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
08011cef09 |
fix(ci): escalate an edit to the shared backup walkthrough helpers
examples/backups/<app>/ selects that app's suite, and a directory whose name is no suite selects nothing. A helpers directory shared by every walkthrough, examples/backups/_lib/, falls into the second case: an edit to it alone produces an empty selection, which both lanes read as "skip Chainsaw", so a change reaching every backup round-trip would run none. It now escalates to the full suite like hack/e2e-chainsaw/_lib/. The set of suites that run a walkthrough is spread across the suites' own scripts, and a derived subset that misses one would fail open. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a6ec30d56f |
fix(ci): discover Chainsaw suites written as chainsaw-test.yml
Chainsaw loads chainsaw-test.yml when a directory has no chainsaw-test.yaml, but select-e2e and select-install enumerated the .yaml name only. A suite spelled that way ran whenever the whole directory was handed to chainsaw, yet no escalation listed it and no edit to it could select it, so it silently dropped out of every narrowed run. select-install also sent find's errors to /dev/null inside a pipeline that reported sort's status, so a failed listing validated no suite at all and passed. It is now captured and fails validation. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
e070bc7d1d |
fix(ci): escalate a per-suite Chainsaw path that names no suite
A path under hack/e2e-chainsaw/<dir>/ put <dir> into the selection without checking that it is a suite, leaving the final intersection to drop it. Alone the path escalated through the empty-selection backstop, but beside any path that contributed a suite the escalation vanished and the run narrowed to that suite with nothing on stderr. Shared material next to _lib/ or a suite nested deeper than discovery looks then went untested in exactly the pull requests that also touch an app. The directory name is now membership-tested where the path is classified, as the hack/e2e-apps/ arm already does, so the decision is a property of the path and names it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9ac67d76a2 |
test(tests): preserve Bats failure reports during fixture cleanup (#3848)
## What this PR does Test-level `EXIT` cleanup overrides the Bats TAP handler, so a failed assertion can disappear from test results. This moves cleanup in eight unit files after their assertions. Failed tests keep scratch files for inspection. Prerequisite for [the Bats runner change](https://github.com/cozystack/cozystack/pull/3849). Existing subshell traps in the OpenAPI and CAPK tests remain supported. Closes #3623 ### Validation `make unit-tests test-controllers -j4 --output-sync=target` passed. The converted files passed under Bats 1.14.0 and `dash`. A deliberately failed assertion produces `not ok` after this change, and the trap audit rejects an added test-level handler. ### Downstream repositories Test cleanup and its documentation only. No downstream consumer changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` |
||
|
Andrey Kolkov
|
f6a615f693 |
fix(backups): scrub legacy plaintext passwords from CNPG Backup snapshots (#4558)
## What this PR does A CNPG `Backup` taken while the Postgres app still accepted `users[].password` copied that password into `status.underlyingResources`, where anyone with `get backups` in the namespace can read it. #4078 stopped new snapshots from carrying the field but left every existing `Backup` as it was (#4179). This adds `LegacyPasswordScrubber`, a leader-elected runnable in the backup strategy controller that, on start, lists `Backup` objects and removes `users.*.password` from every snapshot whose `kind`/`apiVersion` is the CNPG snapshot's. - **Why on every leader start and not a platform migration.** The migration hook runs pre-upgrade, while the old controller is still leader, so a `Backup` the old replica writes during the rollout would keep its password and the migration never re-runs. With `--leader-elect` on, the old replicas stop writing before the new leader's scrub runs, and a `Backup` brought back from an object-level export is cleaned on the next start too. On a clean cluster the whole pass is one List from the manager cache, which `BackupReconciler` already populates. - **Write shape.** The patch is a merge patch computed from the diff, so the request body only nulls the `password` keys and cannot overwrite status another writer changed since the List. The snapshot is handled as generic JSON (numbers kept as `json.Number`), so fields the current snapshot type does not know survive. - **Failure handling.** A `Backup` that cannot be decoded or patched is logged and skipped, and a failed List is logged too; `Start` always returns nil, because a returned error would stop the manager and take every backup down over one object. - **Restore compatibility.** Restore already ignored the key; a test decodes a scrubbed snapshot through `unmarshalCNPGBackupSnapshot` to pin that. - RBAC: `list` and `patch` on `backups.cozystack.io/backups` are already granted to the controller, so the chart is unchanged. - `docs/operations/backup-classes.md` no longer tells operators to delete pre-upgrade `Backup` objects, and says what the scrub does not reach (audit logs, etcd backups, external exports of the objects, Helm history of the init-script Secret), so a password once set through `users[].password` is still treated as exposed until rotated. The runnable carries a `TODO(#4179)` to remove it once upgrading from a release that accepted `users[].password` is no longer supported. Verification: `go test ./internal/backupcontroller/...` and `go build ./cmd/backupstrategy-controller/` are green. The new tests cover scrubbing with everything else kept (including an unknown field with a large integer), leaving non-CNPG, other-apiVersion, clean and snapshot-less Backups unpatched, and continuing past a Backup that fails to decode or patch. Disabling the delete, or returning on the first patch error, makes the tests fail. Not exercised on a live cluster yet. Fixes #4179 ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map against the diff: it touches only `internal/backupcontroller`, `cmd/backupstrategy-controller` and the in-repo `docs/operations/backup-classes.md`, with no package, values, schema, CRD, variant or asset change. The website carries no "delete pre-upgrade Backups" workaround this would make obsolete; its stale `users[name].password` rows are the #4078 follow-up already noted there. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup strategy controller now removes plaintext `users[].password` values left in `status.underlyingResources` of CNPG `Backup` objects taken before managed Postgres stopped accepting user passwords. It runs on every controller start and needs no action. It does not reach copies outside the cluster (audit logs, etcd backups, exported `Backup` objects), so a password once set through `users[].password` should still be treated as exposed until it is rotated. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Security** * At controller startup, plaintext user passwords are removed from matching CNPG backup snapshots. Other snapshot data is preserved. * This cleanup does not affect external copies or remove older init-script manifests from Helm history. Treat passwords previously set through `users[].password` as exposed until rotated. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Marian Koreniuk
|
7500a9fbae |
test(kubernetes): add the Proxmox e2e suite, parked disabled
Upstream CI has no hypervisor, so this suite cannot run there; it ships
`.disabled` and is run by a lane that owns a Proxmox stand.
Named for what the chart actually creates. The resource definition prefixes the
release, so a Kubernetes CR called k8s-e2e produces
ProxmoxCluster/kubernetes-k8s-e2e in the tenant's own namespace, and the admin
kubeconfig Secret is kubernetes-k8s-e2e-admin-kubeconfig.
Stand-specific values are variables rather than hardcoded. Every one of them is
a bare ${COZY_PVE_*} that run-proxmox.sh expands with envsubst, supplying the
defaults itself — GNU envsubst leaves a ${VAR:-default} in the text exactly as
it is, set or not, and that reaches the apiserver as a string where the schema
wants an integer — and refusing a fixture with a variable still in it. The
StorageClass and the PVC probe read the same COZY_PROXMOX_STORAGE, so the volume
lands where the probe looks. The runner reads the tenant kubeconfig's .conf key,
the address a runner outside the management cluster can reach.
The stand's resource pool comes in through COZY_PVE_POOL, empty by default.
hack/proxmox-e2e-fixture_test.bats pins the shape of the fixture and the render.
The PVC probe reads the storage again after the claim is deleted: a deleted
claim whose disk is still there is the leak the pre-delete hook's Step 1b
exists to prevent. keep_tenant is honoured by the suite's own teardown, not
only by the workflow's. The workflow carries no fallback for the stand's
variables: an unset one fails the job before a key is sent anywhere. The
schedule runs only where vars.PROXMOX_NIGHTLY is true: the job targets a
self-hosted runner with the `proxmox` label, and a scheduled run on a repository
without one would sit in the queue until GitHub cancels it.
The suite also asserts the half of the deployment that lives in the management
cluster — the CSI controller and the cloud-controller-manager — because that is
the property the pair is designed around and a tenant-only assertion would pass
with the token in the wrong place.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
Alexey Artamonov
|
35e3593c85 |
fix(seaweedfs): describe the writes-failing alert by what it measures
The rule fires when the master fails a volume pick in every 5-minute window for 15 minutes. That proves recurring refusals, not that no upload gets through: writes into other buckets can succeed in between, and a steady stream of new buckets, each failing a few picks while its first volumes grow, produces the same pattern. The master publishes no count of successful picks to tell the two apart, so the rule stays and its description and the operations page now say what it has seen. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
9b56e70ca6 |
feat(mongodb): support the system-bucket backup flow (useSystemBucket) (#3967)
## What this PR does MongoDB was the only `cozy-default` app with **no `useSystemBucket` flow**: its backup storage was always static on the `PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*` values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant could not back up to the platform system bucket out of the box the way Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default MongoDB backup-restore works OOTB. PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR fields** (not secret refs), and the platform bucket name/endpoint are only known at BackupJob time — so, unlike ClickHouse's runtime `secretKeyRef` env, the chart cannot render them. The implementation mirrors the CNPG driver's pattern: - **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket` flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret. Helm never owns `spec.backup.storages`, so a Flux re-render can't fight the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors Postgres, and `useSystemBucket: true` with `backup.enabled: false` is refused both at render time and by an `x-kubernetes-validations` rule on `backup` in the schema (the same mechanism as the `storageClass` immutability rule). Note on the schema rule: the aggregated apiserver does not evaluate CEL on application-backed resources yet (#2657), so today it is a UI/schema-level marker and the render-time refusal is what stops a Flux release; it also trips the API review gate (“validation rule added”), so this PR needs an API-owner approval on top of the code review. - **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates (`endpointUrl` already scheme'd by the `backupstrategy-controller.endpoint` helper, `credentialsSecret: cozy-backups-creds`, `prefix: <namespace>/<application>`). - **Driver** (`psmdbstrategy_controller.go`): SSA-injects `spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB` at BackupJob time under a dedicated field manager (`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is **gated on the app's `backup.useSystemBucket` flag** (a legacy app leaves it false and is never touched; a useSystemBucket app is re-injected once per mint (once per BackupJob, plus once per storage-race re-mint), before its operator CR is created, so a later coordinate change catches up on the next job while a streaming dump is never patched under). The backup CR is minted only once the operator reports the storage pushed into pbm (`status.observedGeneration` caught up and `PBMReady` True): pbm writes to what its own config says, not to the spec, so a CR minted before that push would land at the previous coordinates while its Backup recorded the new ones; the job waits at `PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup deadline. Once a job has its CR, none of the mint-time reads or holds can fail it: the strategy and app reads the artifact still needs are bounded by the 24h running ceiling instead, and the cluster is not read at all **and on the release having rendered that flag**: the app CR is projected from the HelmRelease's desired values, so the flag reads true before helm-controller renders the revision, while the tenant's own storage, scheduled tasks and PITR stream from the previous render still sit on the cluster. As long as the cluster carries `spec.backup.tasks` or `spec.backup.pitr`, the driver does not inject and holds the BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it at the deadline), so a wedged legacy revision never has its nightly dump and oplog redirected into the shared bucket. The backup snapshot falls back to the strategy coordinates when the operator status echo is empty; that fallback is gated on the CR's own `percona.com/delete-backup` finalizer (the flow the CR was minted on), not on the app flag re-read at completion, so a legacy backup never records the platform bucket and a flag flipped mid-dump does not change the artifact. The restore path rebuilds `backupSource` from it (the `cozy-backups-creds` reference is projected into the restore namespace before dispatch). - **Retention/cleanup**: on the `useSystemBucket` flow the driver owns the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is gated out, and an on-demand backup is not covered by task retention). It stamps the operator's `percona.com/delete-backup` finalizer on each backup it mints **against a storage that carries the platform coordinates** — the strategy's bucket and `credentialsSecret`, read back from the live cluster after the injection — never on the flag alone: a strategy without an `s3` block, or a storage that points anywhere else, holds the job (`MongoDBStrategyHasNoS3` / `PerconaServerMongoDBStorageForeign`) instead of minting a finalized dump against a bucket the platform does not own. On Cozystack `Backup` deletion (by hand today; by platform retention once it lands, since a `Plan` only creates BackupJobs and never deletes a `Backup`) the driver deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer frees the object — the finalizer's presence is the ownership marker cleanup reads back. That reading is unambiguous because cleanup resolves the CR by the name recorded in `driverMetadata` (the one the driver minted), not by scanning: the psmdb operator does stamp `percona.com/delete-backup` on its own `cron-*` task-minted backups, but those are never in scope. A legacy backup keeps the pre-existing hands-off contract, decided from the snapshot before any read of the operator CR. A terminating namespace or the `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips the driver's finalizer and releases; a `Get` of the operator CR that fails with anything other than NotFound/NoMatchError still requeues, and the operator's own `release-lock` finalizer can hold the CR while the operator is down, so those two are what can keep a namespace in `Terminating`. `schedule` and `retentionPolicy` are inert on this flow (there is no engine-side retention and a `Plan` has no retention field yet); platform-level retention is tracked in [#4192](https://github.com/cozystack/cozystack/pull/4192) and teardown GC of orphaned backups in [#3966](https://github.com/cozystack/cozystack/issues/3966), so until those land bound `cozy-backups` with an S3 lifecycle policy. - **Bounds on the waits**: the injection race with the operator's cluster cache (an `unable to get storage` error on a freshly injected storage) is retried by deleting the errored CR and minting a fresh one, with the retry count kept on the BackupJob (`backups.cozystack.io/mongodb-storage-race-retries`) because the deleted CR is reaped before the next poll; after three retries the operator's error is terminal. A system-bucket dump that has started streaming is not cancelled on the 30-minute deadline, but fails after 24 hours in `running` (`BackupAbandoned` warning naming the operator CR, which is left in place as the handle to whatever pbm still writes). The deadline cancel of a not-yet-started CR pins its delete to the resourceVersion the live read answered for. - **RBAC**: `patch` on `perconaservermongodbs` for the injection (server-side apply), and `create`/`delete`/`patch` on `perconaservermongodbbackups` — `delete` to prune the shared-bucket archive on `Backup` deletion, `patch` to strip the driver's finalizer when releasing a wedged cleanup. Coverage: Go unit tests over `internal/backupcontroller` — the injection gate keyed off the app flag (legacy cluster never patched) and off the applied render (a cluster still carrying tasks or pitr is held, untouched), the SSA-injection precondition with the whole entry (per-application prefix included) asserted on the cluster, strategy templating per application, the finalizer refused against a storage without platform coordinates (strategy without `s3` over a legacy entry, foreign coordinates on the merged entry), the retry budget driven with the operator simulated reaping the deleted CR between polls, the restore snapshot fallback and credential swap across all four legacy/system flow combinations, the finalizer-owned cleanup with its escape hatches (skip annotation before any read, `NoMatchError` treated as gone, terminating-namespace release using the pre-read CR, a failed strip surfaced as an Event, a legacy CR never deleted, a legacy snapshot released without a read, a system-bucket snapshot reaching the prune), the deadline cancelling a not-yet-running operator backup only on a positive live read (including no live client at all, and a CR that moved since the read), and the running ceiling. Plus helm-unittest on both charts (chart skips the creds Secret and omits `spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps them on the legacy flow; the strategy renders the S3 coordinates and the `insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore e2e against the platform bucket is a follow-up (needs the system bucket wired in CI, same status as the Postgres/ClickHouse useSystemBucket e2e). One item to confirm in e2e: the psmdb operator accepting `spec.backup.enabled=true` with an empty `storages` between install and the first BackupJob. ### Screenshots Not applicable (no UI changes). ### Downstream repositories The tenant-facing schema surface is one new field, `spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD (regenerated), plus a validation rule on `spec.backup`. `terraform-provider-cozystack` hand-models MongoDB and may want to expose it, but it is a new optional boolean that defaults false (no drift for existing users), and the chart `README.md` regenerates on release. Leaving every box unticked per the trigger-map guidance so a maintainer decides on the provider follow-up. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added opt-in system bucket mode for MongoDB backups. * Automatically injects platform bucket storage without application-managed S3 credentials. * Supports endpoint, region, path prefix, addressing, and TLS settings. * Manages shared-bucket backup artifacts when system bucket backups are deleted. * **Bug Fixes** * Improved restore credential selection to avoid cross-bucket mismatches. * Preserved backup metadata when storage details are unavailable. * **Documentation** * Documented configuration, scheduling, retention, PITR limitations, and restore guidance. * **Validation** * Prevented incompatible bootstrap and system bucket restore configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
1ead6ed90d |
docs(seaweedfs): size an instance for continuous WAL archiving
The default instance holds a little under 10 GiB, while a Postgres cluster archiving WAL into it keeps every segment for the whole retention window, 30 days by default. Nothing in the tree says how to work out what a bucket needs or what happens when it does not fit, so the first sign is usually a database that has stopped. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
8b861668b8 |
ci(release): build pre-release images for amd64 and arm64 and gate releases on it (#4505)
## What this PR does Pre-release builds now publish every image as an amd64 and arm64 index, and a release fails if anything it ships is single-arch. Main, release lines and pull requests keep building amd64 only. It builds on #4498, which made every image buildable for both architectures. Part of #1961. **This PR is on hold until everything under "Merge after" has landed.** Merged earlier, the new gate fails the next rc on the images those PRs fix. ### How a pre-release is built A pre-release tag gets a second job, for arm64, next to the existing release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`. `prepare-release` waits for it, and if the arm64 job fails the rc fails with it. There is no amd64-only fallback. Stable tags do not run it: promotion copies the rc digests with `skopeo copy --multi-arch all`, so a stable release inherits the indexes. After the amd64 build, the stitch joins each amd64 image with its arm64 twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag of the amd64 image moves to the index, the component-versioned ones included. The script then rewrites the digests in the tree and republishes the packages artifact and the installer chart pinned on them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an old digest survives anywhere outside `charts/`. ### The gate A new check fails the rc when a pinned digest in the tree is not an index with both linux/amd64 and linux/arm64. It lives in `hack/verify-multiarch.sh`. It checks first-party and third-party refs alike. Images that are amd64 by nature go into `hack/multiarch-allowlist`, one repository per line, each with a mandatory reason. Today that is only the e2e sandbox. An entry that matches nothing is reported as stale. The static gate only sees digest-pinned refs in the tree. A tag-only ref, an image from a vendored chart default, or one an operator starts at runtime is invisible to it. So the rc e2e also audits every image its nodes pulled, with the same check and the same allowlist. Neither covers a package the e2e never installs. ### Nightly A new nightly workflow builds main for arm64. It keeps the arm64 build cache warm, since the rc job only reads it. It also prints the single-arch refs that would block the next rc. Nothing it builds is published. ### Smaller changes - `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so the arm64 build keeps a cache of its own. - `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the amd64-only e2e sandbox. - The kamaji provider image is pushed under the build's `IMAGE_TAG` like every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent builds overwrote each other's component tag. Closes #4503. - `hack/nightly-mirror.sh` verifies a copy against the raw manifest digest. Plain `skopeo inspect` resolves an index to one platform's child. - matchbox is rebuilt for both platforms in the amd64 release job. The arm64 leg skips the talos package, so there is no arm64 half to stitch, and its Dockerfile only copies files, so no QEMU is needed. Releases then network-boot arm64 machines too. Closes #4524. The stitch also rewrites the kamaji ref inside `files/components.gz`, recompressed with `gzip -n` so the bytes are reproducible, so the kamaji control-plane provider ships multi-arch too. keda and kuberture now name the repository next to their pinned digest, so the gate can resolve them; their rendered manifests do not change. ### Merge after - #4552 builds the Talos installer, matchbox, Harbor and the Velero KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator, kilo and migration-controller to the root `build:` list. It replaces #4507, #4515, #4525 and #4528. - #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy to their multi-arch releases, which are already published. The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and #4522 here, and the multi-arch PRs in ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy. ### Verification The bats suites pass: stitch, verify-multiarch, build-matrix, common-envs, nightly-mirror and the release contracts. The first three also pass under `hack/cozytest.sh` with dash. Each new test was red before its implementation, and the verify-multiarch and stitch tests each have a mutation that turns them red. I ran `verify-multiarch --report` against main. It lists the first-party refs the stitch will fix, plus the third-party and special cases above. The workflows have not run yet. The arm64 job's tools and duration, the stitch against real registries, and the audit are checked for the first time on the nightly and on the next rc. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Nothing under `hack/` is moved or renamed, and no make target changes its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The satellite repositories listed under "Merge after" are prerequisites, not follow-ups this change forces on them. ### Release note ```release-note ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Release candidates now build amd64 and arm64 images in parallel and combine eligible images into verified multi-architecture indexes before publication. * A nightly arm64 image build is available for testing and supplies a build cache for release-candidate builds. * **Bug Fixes** * End-to-end checks audit whether pulled images support both architectures. Audit failures block prerelease checks, while stable releases continue with a warning. * End-to-end test artifacts now include collected image references and multi-architecture audit results. * **Documentation** * Updated release and image guidance covers multi-architecture builds, verification checks, and troubleshooting, including arm64 build and stitching failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
51a2d859ac |
docs(release): describe what the release job builds and what blocks an rc
The amd64 release job builds the Talos installer and matchbox for both arches, so the pages that said CI never builds both in one job, and that build-amd64 builds amd64 only, were wrong. Every rc stitch also skips matchbox and the e2e sandbox, and the nightly report lists matchbox and the refs an in-tree rebuild stamps over; the page said every such skip and line fails the next rc. Name the ones that do not. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
71144d4fd6 |
ci(release): pin the matchbox rebuild in the release job
tags.yaml only runs on tags, so no pull request exercises it. The matchbox rebuild is the only thing that makes its ref an index, and losing the step, moving it past the tree capture or letting it switch the default builder would surface as a failed gate on the next rc. The comments that called the talos images amd64 only now say they are built for both arches in the amd64 job. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c740031453 |
Merge remote-tracking branch 'origin/main' into ci/build-arm64-leg
Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Mattia Eleuteri
|
1cefadcef6 |
docs(backups): say when a base backup gone from the bucket is still pinned
A restore leaves the base backup to the plugin only once the cnpg.io/Backup is gone. A base backup removed from the bucket by a lifecycle rule or by hand keeps its cnpg.io/Backup until the plugin's next maintenance pass on the source cluster, and indefinitely once that cluster no longer runs under the same UID. A restore in that window fails after the purge and on every retry, with nothing on the RestoreJob to unpin it; deleting the cnpg.io/Backup is the way out and does not touch the object store. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
Mattia Eleuteri
|
64e837727c |
docs(backups): say what recovery does when the latest timeline is a sibling
recovery_target_timeline latest targets the highest consecutively numbered history file, without checking that it descends from the base backup. On an archive holding sibling branches that can be another branch, and recovery then stops with "requested timeline N is not a child of this server's history" rather than following the backup's own line of descent. The page also names the second case where a restore does not pin its base backup: a cnpg.io/Backup removed by retention. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
Mattia Eleuteri
|
6999609585 |
docs(backups): bound Postgres PITR by the last archived transaction
The recoverable window was described as ending at the newest archived WAL segment, and the safe fallback as "at/before the most recent completed backup". A time-targeted recovery ends just before the first transaction after the target, and a segment switch or a backup is not one, so on a database at rest every instant after its last write fails, including the end of a later backup. Aiming at a backup's stoppedAt also misses it: stoppedAt is truncated to the second and the plugin compares barman's end time to the microsecond. Describe the bound the recovery actually has, how to read it from the archive, the one-second margin on stoppedAt, what pinning the base backup changes, and why archives with several timelines need it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
Myasnikov Daniil
|
10727c0391 |
test(tests): preserve Bats failure reports during fixture cleanup
Test-level EXIT handlers replace the handler Bats uses for TAP output, so a failed assertion can disappear from the reported test results. Cleanup must preserve those failures before the unit runner changes. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
19764c4bba |
fix(backups): scrub legacy plaintext passwords from CNPG Backup snapshots
A CNPG Backup taken while the Postgres app still accepted users[].password copied that password into status.underlyingResources, where anyone with `get backups` in the namespace can read it. Dropping the field stopped new snapshots from carrying it but left every existing Backup exposed. The backup strategy controller now removes users.*.password from CNPG snapshots when its leader starts. It runs on every start rather than once, so a Backup written by a pre-upgrade replica during the rollout, or brought back from an object-level export, is still cleaned; on a clean cluster it is a single cached List. The patch is a diff-computed merge patch that only nulls the password keys, and a Backup that cannot be decoded or patched is logged and skipped instead of stopping the manager. Fixes #4179 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
6555ae16b8 |
fix(mongodb): start a backup only once the operator has pushed the storage into pbm
The wait after a storage injection watched the cluster's spec, which is not what pbm writes to. At the pinned operator version the backup reconciler hands pbm only a storage name and records status.s3 from the spec entry it read, while the storage pbm uses is whatever the cluster reconciler last pushed into pbm's own config; that push flips PBMReady False and back to True and the reconcile stamps observedGeneration when it ends. A backup started between the apply and that push dumped to the previous coordinates while its Backup recorded the new ones, so both its restore and its prune pointed at the wrong bucket. The wait was also no wait at all: the status write that named it re-queued the job at once, and the live read always showed the apply. The mint now waits for observedGeneration to reach the cluster's generation and for PBMReady to be True, and fails at the backup deadline if the operator never gets there. The strategy, app and cluster reads still ran ahead of the operator CR lookup and were bounded by the backup deadline, so a strategy absent for one reconcile 40 minutes into a streaming dump failed the job and left the finalized CR unreachable. The CR is resolved first now; with one in hand the cluster is not read at all, and the reads the artifact needs are bounded by the running ceiling, past which the job fails naming the CR left in place. The settle check treated a field the entry lacked as agreement, so a strategy that newly set forcePathStyle minted in the same pass as the apply that changed addressing. A missing field now counts as its zero value, which keeps an entry the operator rewrote without an empty region settled while catching a value the apply adds; the storage type is compared outright. The runbook still said the finalizer keeps a Plan from accumulating objects, the RBAC comment still said the timeout cancel runs on either flow, and four wait messages read a duration as a delay. All corrected; the runbook also describes the pbm wait. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
841933dcec |
docs(mongodb): stop saying a Plan prunes the shared bucket
A Plan creates BackupJobs and never deletes a Backup, and PlanSpec has no retention field, so nothing on the platform bounds a MongoDB archive on the shared bucket today except an S3 lifecycle policy or a manual Backup delete. The runbook's tuning section said a Plan does, two hundred lines after the same page said a Plan carries no retention, and three driver comments named "Plan retention" as the path that reclaims the bucket. An operator reading the first sentence would skip the lifecycle policy the release note calls a prerequisite. All of them now say what deletes a Backup today, and that the finalizer frees the object whenever one is deleted, whatever does the deleting. The settle check after an apply compared only the fields the desired entry carries, so an entry written with forcePathStyle while the strategy still set it counted as settled once the strategy dropped the field, and the apply that removed it minted in the same pass. The check now treats a field the apply is about to remove as a change too. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
707e38d17d |
fix(mongodb): keep every mint-time hold off a job that already has its CR
The injection and the ownership switch were gated on the job having no operator CR yet; the precondition between them was not, and neither was the pair of holds ahead of them. Each of those fails the job at the 30-minute deadline, so a tenant disabling backups, or the platform dropping the strategy's s3 block, while a system-bucket dump streamed failed the BackupJob at 30 minutes with the CR left finalized and no Backup object to reach it, bypassing the 24-hour ceiling that exists so a shared-bucket archive is never stranded. Everything up to the mint now runs only while there is no CR; a job with one goes straight to the state switch. The settle wait after the apply was keyed on the storage key being present, so a change to the strategy coordinates rewrote an entry the operator had cached and minted in the same pass, the case the wait exists for. It is now keyed on the entry's content: only an apply that changes what the entry says costs a poll. The mint-time reads (the render's tasks and pitr, the storage entry) came from the manager's cache, which can lag a re-render back to the legacy flow by one reconcile and show the flag true with the tasks not yet visible. With a dynamic client they now come from the apiserver, the way the deadline cancel already reads the operator CR. Tests age the existing-CR case past the deadline and vary the three holds, pin the coordinate-change wait, the live read and the term that keeps the storage-race retry from swallowing an unrelated operator error. The runbook's tuning section no longer summarises the defaults as 30-day retention: only the CNPG and Velero strategies carry one. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
46a8bd7363 |
fix(mongodb): apply the storage once per job, and pin the terms that decide ownership
The system-bucket storage was applied on every reconcile of a job, not once per job: with the operator CR already minted nothing returned before the apply, so a streaming dump saw an SSA patch every poll for up to a day, and the only thing keeping that patch off a cluster that had re-acquired the chart's tasks and pitr since the mint was a predicate term no test held. The storage a CR names is fixed for its dump, so the apply now runs only before the CR exists; the legacy-render hold covers that window, and a test pins that an existing CR is never injected over. The other predicate that decides whether a Backup owns an archive compared the live entry's bucket and credentialsSecret against the strategy, and only the bucket half had a fixture. The credential is what tells a tenant's entry from the platform's when a bucket name collides, so a case varying only that term joins the foreign-bucket one. The cleanup shortcut's snapshot-kind term and the 24h running ceiling are pinned the same way, the latter as an absolute boundary pair. The generated field description said the schema rejects useSystemBucket without backup.enabled. The aggregated apiserver does not evaluate CEL on application schemas yet, so the render-time refusal is the one that holds; the description and the runbook now say so. The runbook's restore paragraph still described the unconditional credential swap this branch narrowed, exempted MariaDB from a retention warning its shipped strategy does not earn, and pointed at a Plan retention field that does not exist; the example README and a dispatcher test's comment still carried the pre-branch claim that the driver never injects and never deletes. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
a6bb0556fc |
fix(mongodb): own archives by where the storage points, and bound the open loops
The prune finalizer on a driver-minted PerconaServerMongoDBBackup means the platform owns the archive: cleanup deletes the operator CR on that marker and the operator removes the object from storage. It was stamped from the app's useSystemBucket flag, which says nothing about where the storage the CR names actually points. With a strategy carrying no s3 block nothing was injected, the presence-only precondition passed on a legacy tenant-bucket entry, and the CR went out finalized against the tenant's own bucket; deleting the Cozystack Backup then deleted the tenant's archive. The finalizer now follows the storage's coordinates as read back from the live cluster after the injection, and a strategy without s3 coordinates, or a storage pointing anywhere else, holds the job instead of minting. The injection keyed on the same desired value. The app CR is projected from the HelmRelease's spec.values, so the flag reads true before helm-controller renders the revision, while the tenant's own storage, scheduled tasks and PITR stream stay on the cluster. Injecting over that entry redirected the tenant's nightly dump and oplog into the shared bucket under platform credentials. A cluster still carrying tasks or pitr is now left alone and the job held until the release renders the flag, and the schema carries a rule marking useSystemBucket without backup.enabled invalid, the pair the chart already refuses at render. The storage-race retry counted surviving operator CRs against its budget, but an errored CR is reaped the moment it is deleted (the operator drops delete-backup without pruning on a non-ready CR), so the count never left one and the job minted a CR per poll for the whole deadline. The count now lives on the BackupJob and survives the reap. A system-bucket dump in running had no bound at all: a wedged pbm agent left the job Running forever and a nightly Plan stacked another beside it. It now fails after 24 hours with the operator CR left in place as the handle to whatever pbm still writes. The deadline cancel pins its delete to the resourceVersion the live read answered for, and the artifact-retry failure names the window that was actually in force. Tests pin the guards that were exercised only in the harmless direction: a system-bucket snapshot reaching the prune, the resolved prefix reaching the cluster, a missing live client deferring the cancel, and the finalizer refused against foreign coordinates. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Aleksei Sviridkin
|
cb035f8eac |
docs(agents): stop calling the image-ref known gaps unfixed
Two of the gaps listed there are handled in the tree now, and each entry already says whether it is. The heading sentence still claimed all of them were open. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
79854b16fb |
Merge remote-tracking branch 'origin/fix/build-list-orphan-images' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2f08832392 |
Merge remote-tracking branch 'origin/feat/talos-arm64' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
6d759cc73e |
feat(build): build package images for amd64 and arm64 (#4498)
## What this PR does
Package images can now be built for amd64 and arm64 in one `make image`,
and a published build is multi-arch unless `PLATFORM` says otherwise.
This is the build half of arm64 support from #1961 and #903. CI still
publishes amd64 only until it gets a native arm64 leg, which comes in a
separate PR.
Builder stages now run on the build platform and compile for the target
one, so Go never runs under emulation. Under emulation it is slow and it
crashes: the migration-controller Dockerfile already records a "bad
pointer in Go heap" from exactly that. On top of this, `PLATFORM`
defaults to `linux/amd64,linux/arm64`. A build on an arm64 workstation
can't publish an arm64-only digest any more. That happened once already,
the cilium pin was arm64-only until
|
||
|
cozystack-ci[bot]
|
59faddc98a |
docs: add changelog for v1.6.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
5924b3fd34 |
docs: describe how a pre-release is built and gated for arm64
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
144d080342 |
feat(build): gate a release on every pinned image being multi-arch
A release pins one ref per image for clusters of either architecture. A ref that resolves to a single-platform manifest, or to an index missing amd64 or arm64, installs on one and fails to pull on the other, and nothing before an arm64 install would notice. hack/verify-multiarch.sh resolves every ref hack/lib/image-refs.sh enumerates, third-party ones included since an arm64 node pulls them too, and fails naming each ref that is not an index carrying both linux/amd64 and linux/arm64. A ref that cannot be resolved fails as well, rather than passing unchecked. A rate limit or a 5xx is the one answer retried, a bounded number of times, and a ref still refused fails as rate-limited rather than as single-arch. --report prints the same list without failing, for a tree that is not being released. Images that cannot be built for both architectures go in hack/multiarch-allowlist by repository, each with a reason, so a bump needs no edit; they are still listed, and an entry that matches no single-arch ref is named as stale. --refs-file checks a list of refs instead of a tree, a tag-only ref resolved by its tag, with the same allowlist and output. It is for images the tree does not pin by digest, such as the ones an e2e run's nodes actually pulled. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7753394bfe |
build: rebuild every package that has an image target
flux-plunger, keycloak-operator, kilo and migration-controller have image targets but were missing from the root build list, which the build matrix, the main build and release preparation all iterate. No CI path rebuilt them, so their pins stayed on images pushed by hand while Dockerfile, patch and base-image changes never shipped, and the multi-arch pre-release build would find single-arch pins it cannot replace. migration-controller shipped a bare :latest with no digest. Add the four packages, and a test that derives the set from the package Makefiles so a new image target cannot be left out again. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f02963a2c7 |
fix(keycloak-operator): keep the image host inside repository
The image target stamped only `tag`, leaving the committed `registry: ghcr.io` in place, so a build against the CI registry would render the public host with a digest that exists only in the build registry. Stamping the build host into `registry` instead would split the source registry across two keys, where the nightly mirror's substring host rewrite cannot reach it and the published tree would keep pointing at the private registry. Set `registry: ""` and keep the whole host in `repository`, stamped by the image target. The upstream chart joins the two keys, so the rendered image string is unchanged, and the ref now has the same shape as every other stamped reference the mirror and promotion already handle. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
769d3b3897 |
fix(build): push the kamaji provider image under the build's IMAGE_TAG
The capi-providers-cpprovider Makefile assigned IMAGE_TAG itself, which beats the value CI puts in the environment. Every PR, main and release line build therefore pushed the same component-version tag and overwrote each other's, and its second tag was built from an undefined TAG variable and ended in a bare "-". The image now goes through image-tags like every other package: the build-unique IMAGE_TAG always, the component version only under PUBLISH_VERSIONED=1. The stamped refs keep the component version as their tag, so the promote, retag and mirror tooling see the same refs as before. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9a7094427e |
feat(build): stitch amd64 and arm64 builds into multi-arch indexes
The arm64 images are built on native arm64 runners, apart from the amd64 build, so each build pushes a single-platform image. The stitch joins every first-party image with its -arm64 twin into one index, moves every tag this build pushed for it to that index, and repins the tree on the index digest, so one ref serves both architectures. Most refs in the tree name only the build's IMAGE_TAG, while under PUBLISH_VERSIONED=1 the build also pushes the component version, so the refs alone cannot say which tags to move. The image-tags macro therefore appends every tag it pushes to PUSHED_TAGS_LOG when that is set, and the stitch moves exactly those, plus IMAGE_TAG and the tag a ref carries. It reads no other tag: another build's tag that sits on the same amd64 bytes is not this build's to move, and scanning every tag of every repository would cost thousands of registry reads per release. It reads refs through hack/lib/image-refs.sh like every other consumer, rewrites only the files that enumeration names, and then searches the whole tree outside charts/, gzip payloads included, for any old digest it re-pointed. A ref whose digest is also pinned where the rewrite cannot reach is skipped instead of half rewritten. The kamaji control-plane provider's chart ships its ref only inside files/components.gz, built from a plain copy with gzip -n, which no text search sees. The stitch rewrites both copies, recompressing with gzip -n so the result is what the package's image target would have written, rather than leaving that image amd64 only. An image with no arm64 twin (talos, the e2e sandbox), or a pair that is not one amd64 and one arm64 image, is skipped and named. A registry read answered with a rate limit or a 5xx is retried a bounded number of times; any other registry error fails the run at once rather than reading as a missing arm64 image. On a rerun after a stitch that failed partway, the build tag can already sit on an index of the pinned image and an arm64 image, not necessarily this run's if the arm64 leg rebuilt it. The stitch repins on that index instead of skipping the image, and copies it to any of the build's other tags still left on the amd64 image. A tag moved to anything else fails the run, since the tree then no longer matches the registry. MATRIX_ARCH=arm64 makes hack/build-matrix.sh leave out the e2e sandbox, which pins linux/amd64: an arm64 leg building it would push a second amd64 image under its -arm64 tag. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
6907c22302 |
docs(agents): describe the talos package as multi-arch
The build notes said the talos package pins linux/amd64, which no longer holds and would steer an agent into re-adding the pin. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
084ac47237 |
feat(build): build published images for amd64 and arm64 by default
PLATFORM defaulted to empty, so buildx built for the host alone. On an arm64 workstation `make image` published an arm64-only digest, which amd64 nodes then fail to pull; the cilium pin shipped exactly that until it was rebuilt by hand. A build now targets linux/amd64 and linux/arm64 unless PLATFORM is set, and the digest a package stamps is the index digest. LOAD=1 keeps building for the host only. The classic docker image store cannot load a multi-platform index, and loading is the documented local workflow. This changes local pushes: a `make image` on the default docker driver without the containerd image store now fails with "Multi-platform build is not supported for the docker driver", where it used to build for the host. Such a build needs a docker-container builder (BUILDER), or LOAD=1 or an explicit PLATFORM for a single architecture. talos and testing pin linux/amd64 with `override`, placed before the include because common-envs.mk expands BUILDX_ARGS immediately. The Talos assets are amd64: matchbox bakes the kernel and initramfs into its image, and the e2e sandbox boots the disk under qemu-system-x86_64. An arm64 variant of either would still serve or boot amd64 files. CI pins linux/amd64 on every image build. Nothing there sets up QEMU, fork builds export a single-platform archive, and tags.yaml builds on the default docker driver, so the default would break or emulate those jobs. Native arm64 builds belong in a leg of their own. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
368d00b5d4 |
test(e2e): remove unused monitoring suites and audit BATS runners (#4019)
## What this PR does I removed the two monitoring OIDC BATS suites that had no runner after the Chainsaw migration, and brought the runner audit and wrapper OIDC test from [#3778](https://github.com/cozystack/cozystack/pull/3778). Also added checks for audience mapper and `OIDC_MODE=System`. The named `KeycloakRealmGroup` test documents coverage the document-count and kind assertions already provided. Runner audit reports unused BATS suites, missing files in runner paths, and unresolved Make variables (including `$(E2E_SUITE)`). Live Monitoring API admission, Grafana operator reconciliation and mode transitions are not tested here. ### Validation - `make -j4 unit-tests SHELL=/usr/bin/dash` passed. - Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart tests passed. - Runner audit: 12 cases passed under dash and Bats 1.14.0. The full shell syntax and BATS sweep passed under dash too. - Three chart mutations and seven runner mutations failed as expected. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note NONE ``` |
||
|
scooby87
|
03e2da5805 |
feat(postgres,mariadb)!: drop plaintext user passwords (#4078)
## What this PR does Removes the plaintext `users[].password` field from the `apps/postgres` and `apps/mariadb` charts (and the mariadb `root` account), so managed-database passwords can no longer be pinned in tenant values or committed to git. Passwords are always chart-generated into the `<release>-credentials` Secret and preserved across reconciles via `lookup`, matching the model `redis`/`valkey` already use. This closes the "User passwords" item of the portal upstream epic (aenix-org/cozyportal#1060). The generated password never lands in the init-script Secret or the SQL it runs: the postgres init-job reads each user's password from the mounted `<release>-credentials` Secret at run time and applies it through a psql variable (`ALTER ROLE … WITH PASSWORD :'pw'`), so it does not linger in the init-script manifest a tenant can read. It does live in cleartext `stringData` in the `<release>-credentials` Secret itself, which is a chart manifest and therefore kept in Helm release history for `MaxHistory` revisions like any chart-generated secret — that residue is exactly why rotation cannot be built on the chart (see below). A `password` or a `passwordRotation` key left over in a tenant's values is accepted-but-inert and draws an admission warning, so an operator editing it is told it has no effect rather than getting a silent 200. Backup/restore into a copy keeps the target's chart-managed credentials working: the CNPG driver clears `bootstrap.enabled` once recovery is healthy so the init-job re-applies the generated passwords onto the recovered roles, and the MariaDB driver excludes `mysql.global_priv` from the dump (`ignoreGlobalPriv`) for backups captured from this release onward, so restoring one of those into a copy does not clobber the target's chart-managed users or `root`. Excluding the grant table has a flip side, and it applies to every MariaDB restore, not only a to-copy one: only accounts declared through `User` CRs (the chart's `users` map) are captured, so an account created out of band directly in MySQL is in no backup and a restore brings back its data and `mysql.db` grants but not the account itself. This is also a backup-side fix only: a restore from a backup taken **before** this change still replays the source's `mysql.global_priv` and overwrites the target's users and `root`, and `root` then needs an out-of-band reset (the operator applies `rootPasswordSecretKeyRef` only at datadir bootstrap, so there is no in-band path). See `docs/operations/backup-classes.md` for the convergence window, the out-of-band-account limitation, and this old-backup caveat. Password **rotation** is deliberately **not** in this PR. A chart-rendered credentials Secret keeps its cleartext in Helm release history for `MaxHistory` revisions, so a rotation built on it cannot revoke a leaked password; rotation is being reworked as a controller that owns the Secret — design proposal in cozystack/community#72. Migration is seamless: existing releases keep their current passwords on upgrade (the value already lives in `<release>-credentials`, which `lookup` re-reads and preserves); a `password` still present in a tenant's values is ignored, so no existing HelmRelease fails validation. Verification: `helm unittest` passes for both charts (postgres 87, mariadb 126, including render-time username guards that reject any name k8s would later reject as an invalid Secret key or User resource name); `go build ./...`, `go vet`, the backup-controller tests and the apps-apiserver tests are green; `make generate` was run with the CI-pinned `cozyvalues-gen` v1.6.0 so `values.schema.json`, the READMEs, the API types, and the `cozyrds` ApplicationDefinitions are all in sync (the branch was rebased onto `main`, which reconciled cleanly with the read-replica autoscaling feature that landed there). ### Screenshots N/A — no UI change. ### Downstream repositories - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: affected — `content/en/docs/next/cozystack-api/go-types.md` is hand-written and still shows `Password:` in the Go examples, and the managed-app reference pages regenerate from these chart READMEs, so both document `users[name].password` until refreshed (not yet filed). - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: affected — the provider hand-maintains the postgres/mariadb schemas and per-field defaults; this PR removes `users[].password`. A follow-up issue/PR is needed there (not yet filed). `terraform-provider-cozystack` and `website` are reached by this diff; both boxes are left unticked because the follow-ups are not yet filed. Same-repo future work this change suggests: the controller-owned password rotation (design in cozystack/community#72), and extending the "passwords only in the Secret" model to the other engines that still accept plaintext (`clickhouse`, `mongodb`, `nats`, `opensearch`, `rabbitmq`, `vpn`). ### Release note ```release-note feat(postgres,mariadb)!: passwords for managed PostgreSQL and MariaDB users can no longer be set via `users[].password` — they are always auto-generated and stored only in the `<release>-credentials` Secret, never rendered into the init-script or the SQL it runs. Existing releases keep their current passwords on upgrade; a `password` (or a `passwordRotation`) left in values is ignored with an admission warning. Password rotation is deferred to a follow-up controller design (cozystack/community#72). MariaDB backups now exclude the `mysql.global_priv` grant table (`ignoreGlobalPriv`), so a restore recreates only accounts declared through the chart's `users` map — an account created out of band directly in MySQL is not captured and must be recreated after a restore. ``` |
||
|
myasnikovdaniil
|
0cb37dba9e |
docs(release): add changelog for v1.6.3 (#4077)
This PR adds the changelog for release `v1.6.3`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.3.md`.
|
||
|
Aleksei Sviridkin
|
564f540f46 |
fix(kubernetes): render the OIDC UID header only from v1.33 (#4407)
## What this PR does Stacked on #4398; review the diff against `fix/kamaji-pin-hash-reset`. The tenant Kubernetes chart now renders `--requestheader-uid-headers=X-Remote-Uid` only from an effective Kubernetes version of v1.33, where `RemoteRequestHeaderUID` is beta and on by default. It no longer renders a `--feature-gates` term of its own on any version. On v1.32 it used to add a term enabling that gate, which is alpha there, whenever it could read the operator's entries and they left the gate unset; no release tag contains that path, so dropping it now changes nothing a user has run. The chart still reads `--emulated-version`, because emulation moves the effective version: `--emulated-version=1.32` on a v1.35 tenant puts the gate back to alpha-and-off, and the header flag is not rendered there. It still reads the operator's `--feature-gates` entries for an opt-out (`RemoteRequestHeaderUID=false`, or `AllBeta=false` with no explicit `=true`), so the header flag never lands next to a disabled gate. `AllAlpha` is no longer read, since it says nothing about a beta gate. An unquoted `--emulated-version=value` entry whose kube version has a non-zero patch component now fails the render and names the entry. The rule mirrors upstream rather than the regex in #4396: `toVersionMap` parses each value with `version.Parse` and refuses only `ver.Patch() != 0` (`staging/src/k8s.io/component-base/compatibility/registry.go` lines 320-326 at v1.35.0, the same code in `staging/src/k8s.io/apiserver/pkg/util/version/registry.go` at v1.31.0). `version.Parse` takes `^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$` (`versionMatchRE` in `k8s.io/apimachinery/pkg/util/version/version.go`, identical at v0.31.1 and v0.35.0). So `1.32`, `v1.32`, `1.32-rc`, `1.32.0` and `1.32.0-rc` all start an apiserver emulating 1.32, and `1.32.4` and `1.32.4-rc` do not. A `^[0-9]+\.[0-9]+$` check would have refused the first four. The chart extracts the numeric run with the same pattern, reads the minor from it, and fails only on a non-zero third component. A quoted value is still left unread. Two deliberate narrowings: - The chart no longer refuses a mix of bare and `kube:`-prefixed `--feature-gates` terms. Once the chart writes no gate term, the spelling of the operator's terms no longer concerns anything it renders. The apiserver still refuses the mix (`component-base/compatibility/registry.go:405` at v1.35.0), and the chart now passes it through like the other fatal gate values it already leaves to the apiserver: unknown names, unparseable values, quoted terms. - On v1.32 the chart now neither renders nor judges the UID flags, as it already did on v1.31. An operator's own `--requestheader-uid-headers` entry without the gate reaches an apiserver that refuses it, which is how every released chart version behaved. A cluster built from `main` that already runs a v1.32 tenant with OIDC on loses the chart's `--requestheader-uid-headers` and gate term on the next render; no release carried them. That holds only while no release is cut from `main` before this merges; a release that carries the v1.32 gate term first would turn this into a breaking change for v1.32 tenants, whose migration is to write both flags into `extraArgs` themselves. An element such as `kube=1.32=x` is now split at its first `=` only, the way `toVersionMap` splits it with `SplitN(compVer, "=", 2)`. The chart used to skip that element, which the apiserver reads as 1.32, and so rendered the header flag against an emulated 1.32. That gap predates this branch. Rebased onto #4398 at `fb2e57c3b`, which reads a two-element `--feature-gates` and `--emulated-version` separately. That behaviour carries over unchanged under the v1.33 floor: behind a hidden `--emulated-version` only an explicit `RemoteRequestHeaderUID=false` is refused beside a header entry, while a blanket `AllBeta=false` waits for a reading, because a lower emulated version can make the gate alpha again; nothing is refused as an opt-out behind a hidden `--feature-gates` value; the empty-element refusal applies behind both; and behind either one `X-Remote-Uid` is added only beside a uid-headers list of the operator's. The README breaking-changes note from #4398 now says the chart refuses neither the mixed gate spelling nor two kube emulated versions. In `docs/oidc-tenant.md` the paragraphs that list refusals now name the managed-flag move and refusal from #4398 and no longer carry a count, and the two-element paragraph now says what the chart renders behind a hidden value. Tests: `helm unittest` goes from 308 on #4398 to 298. 26 cases are deleted. 25 of them pinned the v1.32 gate term, `AllAlpha` handling or the mixed-spelling refusal. One emulated a version below 1.32, which the below-floor emulation case now covers. v1.32 cases that tested parsing rather than the gate are retargeted to v1.34. Sixteen cases are added: the patch-level refusal, the version parsing below the floor and above it (on v1.35 emulating `1.33-rc`, `v1.33`, `1.33.0` and `kube=1.33=x`, where only a correct minor renders the header), the first-`=` split, a pin that a mixed gate spelling now passes through, emulation landing exactly on 1.33, and a v1.32 opt-out beside a header entry passing through untouched. Coverage notes, not applied: no case pins that the patch-level refusal stays off with `spec.oidc.mode: None`, where the block that holds it does not run. The case "splits a component element at its first equals sign only" pins that the kube element of `kube=1.32=x` is read. It does not pin the split itself, because the version pattern ignores what follows the numbers either way. The `extraArgs` field description still calls the field the escape hatch for apiserver flags unrelated to OIDC. It is unchanged here, and it stays accurate for the v1.32 advice in the guide, because the UID flags belong to the aggregation layer rather than to OIDC. Optional, not applied: on v1.32 the chart could refuse an operator's `--requestheader-uid-headers` entry that comes without the gate, since the gate is off by default there and the apiserver will not start. The same skip also drops two refusals the branch history had at 1.32, a header entry beside `RemoteRequestHeaderUID=false` and a header list ending in a comma; both now reach an apiserver that refuses them. It stays out so that v1.32 matches v1.31 and the released charts, where the chart judges none of these flags. Not applied: for a value with a zero-prefixed major such as `01.32.4`, the patch refusal suggests `01.32`. The apiserver refuses both values over the leading zero, so the configuration fails either way and only the suggested fix is off. Why the patch refusal stays while the mixed-spelling refusal goes: the patch refusal is about `--emulated-version`, which the chart reads itself to decide what to render. The mixed spelling concerns `--feature-gates`, which the chart no longer writes to. Not split out: the `--emulated-version` parsing and the patch-level refusal share the fix commit with the floor move, because both change how the same value is read. The change also removes a comment from `hack/e2e-chainsaw/_lib/run-kubernetes.sh` that became false, so select-e2e runs the full suite for this PR. Default path: renders of the parent and this branch through `tests/values/common.yaml` and `tests/values/oidc-system.yaml` at v1.31 to v1.35 are byte-identical, once the randomly generated Kamaji token fields are masked, except for OIDC System at v1.32, which loses exactly the two UID lines. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The only field that changes is the description of `version`, which returns to its earlier wording. The website's kubernetes reference page is regenerated from the README on the next tag, and the Terraform provider keeps its own copy of that description, which already reads "Kubernetes major.minor version to deploy.". Closes #4373 Closes #4396 ### Release note ```release-note fix(kubernetes): the tenant chart renders the aggregation-layer UID header flag with OIDC on only from an effective Kubernetes v1.33, and never renders a --feature-gates term of its own. v1.32 tenants render no UID flags, which matches every released version, so nothing regresses. This supersedes the v1.32 clause of the #3539 release note. An unquoted --emulated-version entry with a patch-level kube version such as 1.32.4, which the kube-apiserver refuses, now fails the render and names the entry. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * OIDC-enabled clusters on Kubernetes 1.33 and later now receive the UID request-header setting without an added feature-gate flag. * Chart rendering now checks Kubernetes and Talos version compatibility and prevents rendering when worker deployments do not meet ownership or retention requirements. * **Bug Fixes** * Emulated Kubernetes versions with non-zero patch components now fail with a clear error; supported version formats are recognized more consistently. * **Documentation** * Updated version and OIDC guidance to reflect current chart behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
339cffe70f |
feat(kamaji)!: update to 26.9.4-edge and read every apiServer extraArgs entry (#4398)
## What this PR does BREAKING CHANGE: a Kubernetes application whose `controlPlane.apiServer.extraArgs` names a flag the tenant control plane now owns fails to render until that entry is removed, except that an `--enable-admission-plugins=` entry is moved into the new `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into the new `controlPlane.kubelet.preferredAddressTypes`. Closes #3541. Updates the Kamaji build this repo ships and rewrites the part of the tenant Kubernetes chart whose correctness rested on the old one. The new pin is `26.9.4-edge`, which is what `make update` picks. Two upstream changes reach us. The tenant control plane no longer folds the kube-apiserver argument list into a map keyed by flag name, so entries supplied through the control plane spec arrive at the apiserver verbatim, duplicates included. And the reset that rewrites those arguments from scratch keys on a hash of the list rather than on its length, so swapping one flag for another no longer strands the old one on the running Deployment. The OIDC block in the tenant chart read only the last entry of each flag. That was right while earlier entries were dropped before the apiserver saw them, and it is wrong now. An operator whose `extraArgs` carried `--feature-gates=RemoteRequestHeaderUID=false` followed by an unrelated `--feature-gates` entry got `--requestheader-uid-headers=X-Remote-Uid` rendered next to a gate the apiserver has switched off, and the control plane would not start. A blanket `AllBeta=false` has the same shape, and so does an `--emulated-version` entry naming kube followed by one naming another component. The chart now reads every entry with the accumulation each flag performs. Occurrences of `--requestheader-uid-headers` append into one list, so where a list of the operator's own omits `X-Remote-Uid` the chart adds its own element beside it instead of refusing to render: the union is what the apiserver validates, and their header keeps working. Feature-gate terms collect per component and are applied in one pass, so the chart carries only its own term, placed last where it wins. Naming the kube component both bare and `kube:`-prefixed now fails the render, because the apiserver refuses that configuration and nothing the chart renders repairs it. `--emulated-version` accumulates too, and two entries naming kube stop the apiserver outright, so the chart renders nothing rather than pick a version that will not exist. The same upstream change narrows what `extraArgs` can do, and the chart now refuses the part that stopped working. Kamaji 26.9.4-edge sets a list of kube-apiserver flags itself and drops any `extraArgs` entry that names one before the apiserver sees the list, where `26.3.6-edge` merged `extraArgs` last and let the entry win. A tenant with `--enable-admission-plugins=NodeRestriction,AlwaysPullImages` loses both plugins on the first reconcile after the upgrade, with nothing to say so. The drop happens on the running control plane whatever the chart renders, so the chart cannot prevent it; what it can do is refuse to render such an entry, in every `oidc.mode`, and name it. The list mirrors `buildKubeAPIServerCommand` at the pinned tag for the control plane this chart renders (an etcd DataStore, konnectivity on, no `dataStoreOverrides`), and the `controlPlane.apiServer.extraArgs` description in the chart README carries it. A flag the control plane only defaults, such as `--authorization-mode` or `--requestheader-group-headers`, is still yours to set. `--enable-admission-plugins` gets a replacement: the new `controlPlane.apiServer.admissionControllers` passes a list through to the KamajiControlPlane `spec.admissionControllers`, which the control plane renders into that flag, and the application API checks each name against the enum the KamajiControlPlane CRD declares. The list is rendered only when non-empty, so a tenant that sets nothing renders exactly as before, and a non-empty list replaces the control plane's default set, as the `extraArgs` entry did under the previous build. `--kubelet-preferred-address-types` gets the same treatment: the list the chart used to hardcode on the KamajiControlPlane is now `controlPlane.kubelet.preferredAddressTypes`, which renders that list when left empty. Neither entry is refused where it can be moved: an `--enable-admission-plugins=` or `--kubelet-preferred-address-types=` entry is moved into its field while the field is empty, on every render, and left out of the rendered `extraArgs`, so a tenant carrying one keeps its setting across the Kamaji update with no edit and no ordering to get right. The entry is refused beside a non-empty field, when it names nothing, and in the two-entry spelling. A separate defect in the same package is fixed here. The TenantControlPlane schema reaches existing clusters through the `kamaji-crds` subchart's templated CRDs, which every release upgrade applies; `crds/` holds the copies a fresh install lays down first. Directory-based renders lost that schema entirely, `helm template`, `lint` and `package` alike, because the parent `.helmignore` matched the subchart's `hack/` directory at any depth and that is where the templates keep the schema they read with `.Files.Get`. Helm honours `.helmignore` for a directory and skips it for an archive, so the cluster never saw the difference and no local check could catch a schema defect. The patterns are anchored to the chart root and a unit test asserts the rendered CRDs carry their versions. The bump is not cleanly reversible, and that is the one thing to know before rolling it out. The args annotation holds a checksum once the new build has written it, and the old build reads that annotation with `strconv.Atoi`. The parse fails and `resetKubeAPIServerFlags` returns early, ahead of both the args reset and the annotation write that would put a number back. So after a rollback to `26.3.6-edge` the count-keyed reset is disabled on every TenantControlPlane: a later removal of an `extraArgs` entry no longer clears the flag from the running Deployment, and it stays that way until someone writes a number into `kube-apiserver.kamaji.clastix.io/args` by hand. Findings from a live upgrade on a dev cluster, none of them caused by the diff. The DNS-service-IP check moved out of a CEL rule on the CRD into the admission webhook (`internal/webhook/handlers/tcp_dns.go`), so it now depends on that webhook being reachable where the apiserver used to enforce it by itself. The manager's readiness probe targets port 8081 while the webhook serves on 9443, so a Ready pod does not prove 9443 accepts connections; with `failurePolicy: Fail` on the TenantControlPlane webhook, that window rejects writes rather than letting them through unvalidated. The readiness probe this build adds to kube-scheduler and kube-controller-manager survives a rollback, since the old build neither sets nor clears those two; that one is benign. Seven limits worth naming rather than leaving to be found. The mixed-spelling refusal quotes the entry carrying each spelling, so one entry holding both, `--feature-gates=Foo=true,kube:Bar=true`, is quoted twice in its own message; the refusal is right and the message reads oddly. The unit test that pins the rendered CRD schemas asserts on a count, so a CRD missing from the render altogether fails on an empty comparison rather than naming the CRD that went missing. There is one window during the upgrade itself: the chart used to fold an operator's own `--feature-gates` terms into the entry it rendered, because the old build would otherwise drop them, and it now renders only its own term. Nothing orders a tenant release after the Kamaji operator has rolled, so a `v1.32` tenant that sets gate terms of its own with OIDC on can lose them until the new operator reconciles it, which it then does on its own. The same window takes a `--requestheader-uid-headers` list of the operator's own that lacks `X-Remote-Uid`: the chart now renders its own entry beside it instead of refusing, and the old operator keeps one value per flag name, the chart's, so the operator's header is dropped until the new operator reconciles. Which entry an error message quotes is pinned only where one entry is eligible. Three variables choose between equally eligible entries, the gate dictionary taking the last setting for a name and the uid-headers ones the first, and no test distinguishes first from last when two entries qualify, so a future edit could make a message name the wrong entry of the two without a test noticing. What the chart renders and what it refuses is pinned exhaustively; it is the provenance in three message strings that is not. The block is deliberately inconsistent about two shapes that stop the apiserver equally hard: two `--emulated-version` entries naming kube make it withhold silently, while a kube-prefixed and a bare gate spelling in the same configuration make it refuse the render. The refusal is right where the apiserver rejects the configuration whatever the chart does; the withholding is right where the chart cannot read a version it must render against. They are two rules, not one, and a third that covers both is fair to ask for. Moving the refusal out from under those conditions so it fires everywhere is the obvious third rule and it is not free: the apiserver refuses the mix on `v1.31` too, but from `k8s.io/apiserver/pkg/util/version` rather than from the component-base `componentGlobalsRegistry.Set()` the message names, so hoisting it means rewriting that sentence as well. Reading a flag and its value written as two entries would be better than withholding for it, and is a change of behaviour rather than a correction, so it is not here. And two sentences about `--emulated-version` in `docs/oidc-tenant.md` sit thirty lines apart and read as a retraction in sequence, though each is true at its own layer: `toVersionMap` accepts an element naming a foreign component, and the registry rejects it later. What this leaves open, by name. The guard for a valueless flag at the end of `extraArgs` covers `--requestheader-uid-headers`, `--feature-gates` and `--emulated-version` only; any other flag that takes a value swallows the appended `--authentication-config` the same way, which is an input mistake the chart does not look for. The managed-flag refusal matches names as written, as Kamaji does: an underscore spelling such as `--secure_port` passes both, and the apiserver normalises it to `--secure-port`, where the operator's value wins as the later one, the same as before this change. Rendering `--authentication-config` ahead of the operator's entries would close that class for OIDC, since only the UID flags would then follow them; it changes the rendered order, and so rolls every OIDC tenant that has `extraArgs` of its own, and belongs in a separate change after the OIDC floor move. One pre-existing comment needs checking: the note above `extraArgs` in `templates/cluster.yaml` says that omitting the field would leave the live array untouched under JSON merge patch, which reads against Helm's `createPatch`, where a key the previous manifest carried and the new one does not is patched to null; what that comment was guarding against is still to be verified. The KamajiControlPlane CRD accepts only the plugin names it lists, so a plugin outside that list that the apiserver leaves off by default can no longer be enabled; that limit is Kamaji's, since it owns the flag and restricts the field. A name moved out of an `extraArgs` entry is not checked against that list by the chart, so one outside it fails when helm-controller applies the KamajiControlPlane, with an error that names it, rather than at render; the chart keeps no copy of the list to check it with. A repeated address type in a moved entry is dropped, keeping the first place it appears, because the field is a set. With `oidc.mode: None` the chart does not read `extraArgs` for the OIDC flags at all, and two shapes regress with the bump there: a kube feature-gate term written both bare and `kube:`-prefixed across entries, and two `--emulated-version` entries naming kube. The old build kept one entry per flag, the new one passes both, and the apiserver refuses both at startup. Next to a flag and value written as two entries the chart keeps the refusals that hold whatever the hidden value is, but it does not read positions: a hidden `--feature-gates` term keeps the opt-out refusal off even when it sits before the explicit opt-out and so cannot undo it. What the live run could not exercise: the konnectivity datapath, because the stand had no tenant workers, and `--requestheader-uid-headers`, which comes from the tenant Kubernetes chart and no image swap reaches. Both are covered by render and unit tests only. One shape the old collapse was quietly repairing, with OIDC on, needs a guard now that it is gone. A bare `--requestheader-uid-headers` in `extraArgs` used to be overwritten by the entry of the same name the chart renders, because the collapse kept one value per flag name; the same held for a bare `--feature-gates` on `v1.32`, where the chart renders a gate entry too. A bare `--emulated-version`, or a bare `--feature-gates` on any other version, had no entry of the chart's to be overwritten by and was emitted bare in sorted position, where it took the next flag as its value, so it never worked. Without it those entries reach the apiserver as written, and the apiserver gives a valueless long flag the next argument whatever that argument looks like. What follows the operator's last entry is the `--authentication-config` the chart appends, so the flag swallows it and the AuthenticationConfiguration is never applied. On `--requestheader-uid-headers` the path is accepted as a header name and the control plane comes up with OIDC silently off, failing every login for that tenant with nothing in the cluster to say why; the other two reject it and the apiserver exits. The chart refuses to render that shape and says which entry to fix. Two things in the image build fall out of the bump. The datastore deletion fix carried here as a patch landed upstream verbatim, so its patch no longer applies, and one failing patch fails the whole apply of the directory. The Go module overlay is obsolete for the opposite reason: this tag already carries `golang.org/x/net`, `golang.org/x/crypto` and `go.opentelemetry.io/otel/sdk` above the versions the overlay forced, so keeping it would downgrade all three. What I checked: `helm unittest` passes at 307 cases for the package, with the OIDC suite carrying every shape above, and the image builder stage builds end to end at the new tag with the one remaining patch. Kamaji's own support for Kubernetes 1.36 and 1.37 does not move the tenant version enum here; that is a separate update of the `kubernetes` package. No image reference in `packages/system/kamaji/values.yaml` is touched, since CI stamps that digest. ### Screenshots No UI changes. ### Downstream repositories I walked the trigger map file by file against the diff. The chart templates, the vendored Kamaji chart, the image build and the docs reach nothing downstream. The one generated file that reaches downstream is `packages/apps/kubernetes/values.schema.json`, which gains `controlPlane.apiServer.admissionControllers` and `controlPlane.kubelet.preferredAddressTypes` and a changed `extraArgs` description; both new fields are optional with defaults, so a stored Kubernetes object needs no change, no field is removed or renamed and no version enum moves. The new fields are the one trigger the map fires on: the Terraform provider keeps a hand-written schema. `packages/apps/kubernetes/README.md` feeds the website's managed-apps reference pages, which a bot regenerates on a stable tag. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: model `controlPlane.apiServer.admissionControllers` and `controlPlane.kubelet.preferredAddressTypes` on the kubernetes resource (schema, model, expand/flatten). - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kamaji): update Kamaji to 26.9.4-edge. Tenant kube-apiserver arguments are now derived from a hash of the control plane's extraArgs instead of from its length, so on the first reconcile after the upgrade every tenant control plane rewrites those arguments from scratch and rolls once. The rebuilt list also orders them differently, Kamaji's own flags sorted first and the operator's kept verbatim after them, so the roll happens whether or not the tenant sets any extraArgs. A tenant with OIDC enabled takes a second roll when the chart's own flags land. Rolling the Kamaji image back to an earlier build does not restore the old behaviour: the older code reads that annotation as a number, gives up when it finds a checksum, and from then on never clears a flag you remove from extraArgs, on any TenantControlPlane, until the annotation is set back to a number by hand. The tenant Kubernetes chart reads every extraArgs entry the way the apiserver accumulates them: a --requestheader-uid-headers list that omits X-Remote-Uid now gets the chart's own element beside it instead of failing the render, a feature gate set in an earlier entry is honoured, and naming the kube feature-gates component both bare and kube:-prefixed fails the render, because the apiserver refuses that configuration outright. Writing --feature-gates or --emulated-version as a flag and a value in two separate entries now keeps the chart from rendering UID flags on its own, because an entry the chart cannot read may hide an opt-out; a --requestheader-uid-headers list of your own still gets X-Remote-Uid beside it, and write --flag=value to get the rest back. Action required for tenants that pass their own apiserver flags: Kamaji now sets a list of flags itself and drops a controlPlane.apiServer.extraArgs entry naming one, where the previous build let that entry win, and the Kubernetes application now refuses to render such an entry, except for the two it moves into a field. A tenant carrying one of the others stops reconciling after the upgrade, and no later change to that tenant applies, until the entry is removed; the entry no longer has an effect either way. The list is in the description of controlPlane.apiServer.extraArgs. An --enable-admission-plugins=value entry is moved into the new controlPlane.apiServer.admissionControllers, a list that replaces the control plane's default set as the entry did, and a --kubelet-preferred-address-types=value entry into the new controlPlane.kubelet.preferredAddressTypes, while that field is empty; nothing has to be edited for them. Before upgrading, list the affected entries with: kubectl get kuberneteses.apps.cozystack.io --all-namespaces --output json | jq --raw-output '["--advertise-address","--client-ca-file","--enable-admission-plugins","--service-cluster-ip-range","--kubelet-client-certificate","--kubelet-client-key","--kubelet-preferred-address-types","--proxy-client-cert-file","--proxy-client-key-file","--requestheader-allowed-names","--requestheader-client-ca-file","--secure-port","--service-account-key-file","--service-account-signing-key-file","--tls-cert-file","--tls-private-key-file","--etcd-servers","--etcd-cafile","--etcd-certfile","--etcd-keyfile","--etcd-prefix","--etcd-compaction-interval","--egress-selector-config-file"] as $owned | .items[] | .metadata as $m | (.spec.controlPlane.apiServer.extraArgs // [])[] | select((split("=")[0]) as $f | $owned | index($f)) | "\($m.namespace)/\($m.name): \(.)"' Entries it lists for --enable-admission-plugins or --kubelet-preferred-address-types need a change only when written as two entries, when they name nothing, or next to a non-empty field of their own. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Configure tenant control-plane admission controllers and kubelet preferred address types. - Choose a cert-manager issuer for webhook certificates; self-signed certificates remain the default. - **Bug Fixes** - Improved OIDC argument handling for repeated, split, quoted, and conflicting settings, with clearer render-time errors. - DataStores are recognized as unused when no tenant control planes reference them. - **Validation** - DataStore endpoints are checked for length, count, and IPv6 address formatting. - Chart-managed tenant control-plane arguments are rejected when supplied through `extraArgs`. - **Documentation** - Updated Kubernetes OIDC and certificate issuer guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
ce74fdeea9 |
docs(backups): describe restored credentials and pre-upgrade residue
Say that a restored copy converges its application passwords shortly after recovery, that MariaDB backups leave out the grant table, and that dropping the password field does not rewrite copies made before the upgrade: old Backup snapshots and Helm revisions still hold the password that stays live. The credentials projector comment counts only the engines that still take a password from values. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
a92e89e969 |
docs(kubernetes): describe the OIDC UID header from v1.33
The tenant OIDC guide, the version field description and the e2e helper comment still described a --feature-gates term rendered on v1.32. Left as they were, they would tell an operator that the chart switches that alpha gate on, which it no longer does. The guide now says what a v1.32 tenant has to write itself to get the UID, and which --emulated-version values fail the render, so an operator who meets that refusal finds it explained. The breaking-changes note in the README said the chart refuses the mixed gate spelling while OIDC is on; with that refusal gone it would promise a render error that no longer comes. The version description returns to its earlier wording in values.yaml and in the update-versions.sh template, because that template rewrites the line on every version bump and would otherwise bring the old text back. make generate carries it into the schema, README, API types and ApplicationDefinition. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4e9f83ac0d |
fix(kubernetes)!: move or refuse extraArgs flags the control plane owns
From 26.9.4-edge the tenant control plane sets a list of kube-apiserver flags itself and drops any extraArgs entry naming one before the apiserver sees the list; 26.3.6-edge let such an entry win. An existing tenant that set, for example, its own admission plugins through extraArgs would lose them on the first reconcile after the upgrade with nothing to say so, and a flag written with its value in a separate entry leaves that value behind as a positional argument the apiserver refuses. Two of those flags have a KamajiControlPlane field the control plane renders them from. An --enable-admission-plugins=value or --kubelet-preferred-address-types=value entry is moved into that field while the field is empty, and left out of the rendered extraArgs, so the tenant keeps its setting across the upgrade without an edit. The move is computed from the values on every render; the last entry wins, as it did while the control plane kept one entry per flag name, and a repeated address type is dropped because that field is a set. The post-upgrade Job that patches extraArgs onto the KamajiControlPlane reads the same list. Every other entry naming an owned flag is refused at render time in every OIDC mode, since the chart cannot stop the drop and the refusal is where the operator learns of it. So are the two moved flags beside a non-empty field, with a value that names nothing or is quoted, and in the two-entry spelling. The list mirrors the Kamaji builder for the shape of control plane this chart renders, and the extraArgs description names the same flags. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
88264216b7 |
docs(kubernetes): restate the OIDC UID flags for the merged reading
The migration note told operators to move the extraArgs entry count in the same edit so the running control plane would rewrite its apiserver arguments; the control plane compares a hash of the list now, so any change already forces the rebuild and no second edit is owed in either direction. The paragraphs describing a value that survives only as the last entry of its flag, and an empty value reaching the apiserver as a bare flag, describe a build that is no longer shipped. What replaces them is the accumulation each flag actually performs: a uid-headers list the apiserver validates as the union of every entry, feature gates settled across entries with the last term winning and one spelling of the kube component throughout, and an emulated version that two entries cannot both name. One contract the earlier text sold is narrowed to what the chart keeps. The render checks run only where the chart renders something, so the three readings it refuses to guess at switch them off as well, and a shape that would otherwise be refused passes through to a control plane that does not start. A flag and value written as two entries is a narrower case, and the page says which checks hold next to one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
c55ab268e2 |
test(monitoring): remove orphaned OIDC BATS suites
The two pre-Chainsaw monitoring OIDC suites have had no invocation site since the app suite migrated to Chainsaw. Remove them instead of carrying dead tests that force the selector to run every live suite. Keep the existing selector and capture guards for the historical path shape, and update their comments now that the directory is empty. Retain a named chart assertion for the no-KeycloakRealmGroup ownership invariant; the existing count and kind assertions already enforce the same behavior. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Timofei Larkin
|
180030be72 |
fix(api): refuse only what a name declaration actually breaks
Three cases refused creates of a kind, or were documented as refused, without the declaration being at fault. A schema that is not a JSON object was reported as an invalid x-cozystack-name, although such a schema has no declaration to blame, and the spec schema's own loader already reports it. It now declares nothing, so creates keep working and a kind with a legacy cap keeps it. Any unknown key inside a declaration took the kind offline, so a generator adding a title or an example would do the same. Keywords that constrain nothing, the JSON Schema annotations and x- extensions, are now ignored. A constraint keyword the server does not implement is still refused, because accepting it would drop it silently. The docs said a declaration no name satisfies is refused. That holds for lengths only: a pattern is checked to compile, not to match anything. The docs now say so. The parser moves to pkg/apis/apps/validation so the API review gate can reuse it rather than keep a copy of its rules. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
146e06e75e |
feat(api): take Application name limits from the application's own schema (#4367)
## What this PR does
The aggregated API server capped two kinds' names by knowing what their
charts render. A `Kubernetes` cluster name was held to 32 characters
because every worker pool is a `KubernetesNodes` release named
`kubernetes-nodes-<name>-<pool>`, and the default `md0` pool has to fit
the 53-character Helm release name limit. A `Kafka` name was held to 44
because the KRaft controller pod `kafka-<name>-c-<hash>-<id>` has to fit
the 63-character DNS-1123 label limit. The arithmetic is right, but it
is the charts' knowledge living in the platform: the next application
with a sub-resource budget needs another `if r.kindName == ...` branch,
and an application served from outside this repository — the whole point
of `ApplicationDefinition` — cannot get one at all.
An application now declares the limits on its own name in its own
schema, and the server applies whatever it finds there for whatever kind
the definition describes. The chart's `values.yaml` carries a `## @name`
annotation, `cozyvalues-gen` v1.7.0 publishes it at the root of
`values.schema.json` as `x-cozystack-name`, and `hack/update-crd.sh`
carries it into the `ApplicationDefinition`:
```yaml
## @name {string} - Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.
## @maxLength 32
```
```json
"x-cozystack-name": {
"type": "string",
"description": "Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-<name>-<pool>`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.",
"maxLength": 32
}
```
`maxLength`, `minLength` and `pattern` are honoured; `pattern` is
unanchored, as JSON Schema specifies. A declaration only tightens:
`prefix + name` must still fit the 53-character Helm limit. A rejected
name is answered with the chart's own description, or with "limit
declared by the <Kind> application" when the chart gives none.
The schema is untrusted input — definitions from other repositories are
written by hand — so a declaration that cannot be enforced as written
(not an object, a type error, an unknown keyword such as a misspelled
`maxLenght`, `maxLength` below 1, a negative `minLength`, `minLength`
above `maxLength` or above the Helm budget, a pattern that is not RE2)
refuses every create of that kind with `configuration error: the <Kind>
ApplicationDefinition declares an invalid x-cozystack-name: …`, and logs
the same at start-up. Dropping the declaration would admit names the
chart cannot render; applying it would reject every name and blame the
user. Other kinds are unaffected, and existing objects of the affected
kind can still be read, updated and deleted.
An upgrade brings `cozystack-api` up before `kubernetes-rd` and
`kafka-rd` re-render (`kubernetes-rd` depends on `cozystack-engine`), so
the new server can briefly read a definition that predates its
declaration. For exactly that case — a definition that declares nothing
— the two previous caps stay in `legacyNameCaps` as a floor, to be
removed once no supported upgrade starts from such a definition. That is
the only per-kind knowledge left in the name checks.
Also in this PR:
- The `pre-commit` job pins `cozyvalues-gen` v1.7.0, the first release
that understands `@name`
([cozystack/cozyvalues-gen#33](https://github.com/cozystack/cozyvalues-gen/pull/33)).
- The API review gate now diffs the root `x-cozystack-name` of every
definition as `metadata.name`, so a chart that lowers its cap or adds a
pattern is reported like any other tightened field. On this PR it
reports the Kubernetes and Kafka caps as added; their effective limits
are unchanged.
- The migration 56 comment that pointed at the removed
`maxKubernetesClusterName` now points at the chart's declaration.
Not in scope, because none of it is a static limit a chart can declare:
the `Tenant` name pattern in `pkg/apis/apps/validation`, the computed
tenant-namespace length check (it depends on the parent namespace at
request time), the tenant quota hierarchy, and the Phase 2 removed-field
warning for `Kubernetes`.
### How it is tested
`TestNameCapComesFromTheShippedSchema` reads the shipped `kubernetes`
and `kafka` ApplicationDefinitions off disk, builds storage through
`NewREST` as the server does, and checks the cap and the chart's
explanation; it then rewrites `maxLength` inside the shipped schema and
checks the limit moves with it, which a cap held in code would not.
`TestInvalidNameSchemaFailsClosed` covers each malformed declaration
through the same path, `TestLegacyNameCapsCoverUndeclaredDefinitions`
pins the upgrade floor and that a declaration replaces it, and
`TestClassifyNameDeclaration` covers the gate.
### Downstream repositories
Two repositories are affected and neither follow-up is filed yet, so no
box is ticked.
- `cozystack/website` — `content/en/docs/next/development.md` documents
the `values.yaml` annotations and needs `@name`;
`content/en/docs/next/cozystack-api/application-definitions.md`
describes `openAPISchema` and needs `x-cozystack-name`.
- `cozystack/external-apps-example` — it pins no `cozyvalues-gen`
version and regenerates against whatever is current; nothing in it
declares a name constraint, so it is unaffected until it chooses to.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(api): Application name limits now come from the application's own schema instead of being hardcoded per kind in the aggregated API server. A chart declares them with a `## @name` annotation in `values.yaml`, which cozyvalues-gen v1.7.0 publishes as `x-cozystack-name`; the API server enforces `maxLength`, `minLength` and `pattern` on `metadata.name` for any kind, answers a rejected name with the chart's own explanation, and refuses creates of a kind whose declaration it cannot enforce with a configuration error. The Kubernetes (32) and Kafka (44) name caps are unchanged in effect and are now declared by their charts.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Applications can declare minimum and maximum name lengths and
formatting patterns.
* Kubernetes cluster names are limited to 32 characters, and Kafka
cluster names to 44 characters, to meet generated resource naming
limits.
* Names that violate declared constraints are rejected with an
explanatory message. Tighter name constraints are also reported as
breaking changes.
* Declared maximum lengths cannot exceed the platform’s existing Helm
release-name budget.
* **Documentation**
* Added guidance on declaring and enforcing chart-specific resource-name
limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|