mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
6297ebd70865dfd8a5f084521715a7720c7b8971
727
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e978ed7f2f |
fix(postgres): stop backup jobs from fighting over the WAL archive destination (#5221)
## What this PR does Part of #5202. A second PR will restore the last good destination when a job that changed it fails. The ObjectStore a Postgres Cluster archives its WAL to is shared by every backup of the application, yet every running BackupJob server-side applied its own strategy's destination onto it every five seconds with forced ownership. Two BackupJobs of one application with different BackupClasses flipped the destination back and forth until the healthy backup failed too, and a webhook or apiserver blip during that apply failed the BackupJob at once. - A BackupJob writes the destination and the Cluster plugin only before its `cnpg.io/Backup` exists. Afterwards it only reads the live Cluster, and a failed read is returned for a controller retry instead of failing the running backup. - Before writing, a BackupJob waits, with a `Ready=False` / `WaitingForOtherBackup` condition naming the other job, while another running BackupJob of the application has a `cnpg.io/Backup` in flight. The wait is not capped, for the same reason as in #5211. The queue reads the Cluster's own Backups from the apiserver; with leader election one reconcile runs at a time, so only a leader handover can still let two jobs through. An atomic per-application claim is #4559. - Before writing, a BackupJob also checks the apiserver for a restore of its application and holds while one runs, including a job released from the queue: a write landing during the restore would put the restored Cluster back on the source's WAL prefix. - Write errors that cannot succeed on retry (invalid, bad request, forbidden) still fail the job at once. Any other is retried with a `ClusterPluginApplyRetrying` condition for the backup deadline, counted from the first failure since the job last waited, and recorded in a new `DestinationApplied` condition. - The pass that creates the `cnpg.io/Backup` records on it the destination, endpoint, credentials Secret and serverName it wrote (`backups.cozystack.io/cnpg-written-destination`), and the artifact and its size probe are built from that record, so a strategy edited, or a Cluster re-rendered, while the backup runs does not redirect the artifact. A Backup without the record falls back to the strategy as before. - The plugin rollout deadline and the instance check count from the write that attached the plugin, not from the job's start. The apply that attaches it also stamps the time on the Cluster (`backups.cozystack.io/barman-plugin-attached-at`), so a lost BackupJob status write after the attach does not fall back to an older time, and a job that starts while another job's recent attach is still rolling waits for that rollout too. A stamp a few minutes ahead of the controller's clock is waited out; one further ahead is treated as corrupt. Tests drive the reconciler through a running backup that must not rewrite the destination, the queue and its release, a restore starting while a job waits, each class of write error, a recovered write, a failed Cluster read while the Backup runs, lost status writes around the attach, another job's rollout, clock drift on the attach stamp, the recorded destination and serverName, and the rollout deadline. Each guard was checked by mutation. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(postgres): Postgres BackupJobs of the same application no longer overwrite each other's WAL archive destination: a job writes it only before its backup starts and waits while another backup or a restore of the application is in flight. A transient error while writing it is retried instead of failing the job at once, and the backup artifact names the destination and serverName the backup was actually written to. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Reliability** * CNPG backups wait while another backup or restore is using the shared destination, and retry temporary destination-write failures while observing the backup deadline. * Backup artifacts and size checks use the destination recorded when the backup was created, even if destination settings later change. * Backup processing handles plugin rollout timing and destination-hold release more consistently. * **Restore** * Restore operations account for shared WAL archive destination conflicts and guard against incompatible backup selections. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
583f18437d |
test(clickhouse): give the credentials-wait BackupJob a UID
The test pins that a backup held for late bucket credentials keeps its archive name, and a run's archive name now carries part of the BackupJob's UID, so the fixture needs one for the name it expects to be the name the controller mints. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
dae76d186c |
Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for late bucket credentials, against the controller's archive helper. That helper now names a run's archive with part of the BackupJob's UID, so the merged test has to give its BackupJob one. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5c6fc10a23 |
test(postgres): pin three guards no test caught when changed
A job whose own Backup exists skips the wait for another job's plugin rollout, and the size probe keeps the strategy's Secret key names while it takes the Secret from the record, adding a credentials block only when the strategy has none. Each of these could be changed with the suite still green: dropping the own-Backup check, or forcing the missing-credentials branch either way. Cases for a job with a running Backup next to a recent foreign attach, a strategy without credentials and a strategy with custom key names now fail on each change. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
238b3ee491 |
fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but only the `s3_lifecycle` job of its plugin worker applies it, and that job is scheduled by the admin server. Cozystack deployed neither, so a lifecycle rule returned 200 and no object ever expired, on the platform backup bucket and on tenant buckets alike. That also made the documented way to bound backup storage a no-op. This PR adds the admin server and a worker to Simple and MultiZone SeaweedFS instances, behind a new `lifecycle.enabled` value that is **off by default**. A Client instance is unchanged, because its rules are enforced where the instance it connects to runs. The worker registers for `s3_lifecycle` only, so vacuum, balancing and erasure coding are not started as a side effect. The admin keeps its job state on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default when that is empty; `volume.storageClass` is not reused, because in MultiZone it can name a class local to one zone or node. The admin UI is protected by a generated password, because a tenant namespace admits traffic from the whole cluster. The dashboard Role lists the two new WorkloadMonitors. Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart renders no `[grpc.s3]` section, so the s3 gateway served its internal gRPC in plaintext while the worker dials it with mTLS, and every lifecycle delete failed the TLS handshake. The section is backported from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart patch, applied with `--forward` so the next `make update` stops at the point where the patch must be dropped instead of silently reversing it. The s3 pods are restarted once so they load the section. This also repairs IAM cache propagation between s3 replicas, which failed in the same way, and it applies whether or not lifecycle is enabled. The data-plane release now retries a failed install or upgrade in place (`RetryOnFailure`) instead of rolling it back. With the previous default remediation and unlimited retries, an upgrade that waited out its timeout, for example on a tenant quota with no room for the admin volume, rolled back and retried forever, restarting the s3 pods twice per cycle. The worker's hostname anti-affinity is relaxed to a preference so a rollout finishes on a single-node cluster. #### Why lifecycle is off by default The daily walk that SeaweedFS uses to expire objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when no identity is given; it does not re-check the object's age either. A key overwritten between the walk's listing and the delete loses its new content. This path is unchanged from 4.45 through 4.52 and on upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`, `identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup buckets write unique keys and are barely exposed; a tenant bucket that rewrites a fixed key is. On top of that, turning enforcement on would start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so nothing changes for existing instances until `lifecycle.enabled` is set. Making it the default again is left to a separate PR that carries object identity on the walk path; upstream report: https://github.com/seaweedfs/seaweedfs/issues/12451. #### Turning lifecycle on and off When `lifecycle.enabled` is set, rules that have been stored but ignored start applying within minutes, and objects they have already expired are deleted on that first pass, so review existing bucket lifecycle rules first. Set `lifecycle.storageClass` before enabling, and prefer a replicated class: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. Moving the volume means disabling lifecycle, deleting the `admin-data-seaweedfs-admin-0` PVC, changing the class and enabling lifecycle again. Each instance gains two small pods and a 1Gi PVC; when the tenant quota has no room for them, the data-plane release keeps retrying in place until it has. When `lifecycle.enabled` is turned off again, the parent release removes the admin credentials Secret while the data-plane release removes the admin StatefulSet only on its own reconcile, shortly after. An admin pod that restarts in that window fails to start on the missing Secret until the StatefulSet is gone; nothing is lost, and the order is left as is. The admin PVC stays until the instance is deleted, and a later re-enable starts the admin with a new password. Verified on a test cluster with the 4.45 chart: without the backport, the worker found the expired objects and every delete failed with `tls: first record does not look like a TLS handshake`; with this branch and lifecycle enabled, a rule with a past expiration date removed the matching objects and left the others, both on a manual run and on the first scheduled pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up needed, not opened yet: `operations/services/backup-classes.md` (v1.7 and next) advises an S3 lifecycle policy, which the bundled SeaweedFS applies only once `lifecycle.enabled` is set; the SeaweedFS reference page gains `lifecycle.enabled` and `lifecycle.storageClass` from the regenerated README on the next release - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The terraform provider does not model the SeaweedFS application, so the new values do not reach it. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added optional, daily S3 bucket lifecycle-rule processing for supported SeaweedFS configurations. It is disabled by default; Client instances rely on lifecycle rules configured on the connected SeaweedFS. * Added configurable storage for lifecycle job state and dashboard visibility for lifecycle components. * Enabled the S3 gRPC server to use its internal TLS identity when security is configured. * **Bug Fixes** * Improved storage-class handling so numeric-looking names remain strings and unset classes are omitted where appropriate. * Clarified backup cleanup guidance, including when lifecycle rules apply and how to reclaim storage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3badbdef74 |
fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
1d22013646 |
fix(postgres): hold a backup until the plugin sees its new destination
The plugin-barman-cloud sidecar reads the ObjectStore through its own cache, kept for ten seconds (DefaultTTLSeconds in plugin-barman-cloud v0.13.0) and invalidated only by the sidecar's own writes, and WAL archiving keeps it warm. The pass that changed the destination created the cnpg.io/Backup a moment later, so the base backup could be taken against the previous destination while the artifact names the new one, and a restore from it finds nothing. On a live cluster with the shipped plugin, a Backup created two to four seconds after switching the destination landed in the old bucket in two of three tries; with a fifteen second wait it landed in the new one every time. The apply that changes the ObjectStore configuration now stamps the time on the ObjectStore itself, so the record cannot be lost apart from the change, and the job creates its Backup only once that change is older than the cache can be. A change is judged on the configuration a server-side dry run of the same apply returns, so values the apiserver defaults or another field manager owns do not read as a change on every pass. That wait leaves a job with the destination written and no Backup yet, which the queue, looking only for Backups in flight, did not see: a second job with another destination wrote its own, the first took that as a change and stamped again, and the two restarted each other's wait forever. Every apply of the ObjectStore therefore also names the job that made it, which only a job without a Backup of its own and let through by the queue does, and the queue holds every other job while that one runs and has no Backup yet. Once it has one, that Backup holds the destination for as long as CloudNativePG runs it, as before, so a job stuck after its Backup finished does not hold every later backup of the application. This also covers a job waiting for the rollout of the plugin it attached, during which a second job used to switch the destination as well, including when its own write left the configuration as a restore had re-rendered it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5a95bcdd64 |
fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024 bytes, and the legacy s3PathOverride prefix has no length limit. The full UID made the name 37 characters longer than before, so a prefix that fit the old name pushed the manifest key past the cap and failed the backup. The run's name now drops "clickhouse-" and the "T" and "Z" of the timestamp and spends the room on the first 12 hex digits of the UID, so it is exactly as long as the old name for every application. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9978771858 |
feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does Adds `observability-proxy`, a new system component that lets a tenant user read the logs of their own namespace from VictoriaLogs with the same Kubernetes credentials they already use for the console, and with Kubernetes RBAC deciding what they can read. This is the first part of the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94); this PR implements the namespace logs root only. Metrics and the per-application roots follow in separate PRs. Why a proxy: today the log stores are shared per root tenant and have no notion of a caller, so tenants cannot be given direct read access without seeing other tenants' logs. The proxy authenticates the caller, authorizes the request from its path, and rebuilds the upstream query so that its scope cannot be widened by the client. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. What is included: - `cmd/observability-proxy` and `internal/observabilityproxy`: the proxy. - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>` for the VictoriaLogs read endpoints the console needs (`query`, `hits`, `stats_query_range`, `tail`, `field_values`, `stream_field_values`, `facets`). Anything else under the root answers `404`. - Authentication by `TokenReview`, authorization by one `SubjectAccessReview` built from the path (`list` on `cozystack.io/logs` in the namespace), both cached briefly. - The upstream request is rebuilt from an allowlist of parameters; the namespace scope is always forced as `extra_stream_filters`. Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL `options(...)` block, and tenant headers such as `AccountID`/`ProjectID` are not forwarded or are refused. - Non-canonical paths (`..`, `//`, encoded separators) are refused with `400` before any authorization is built, because the console nginx matches on the normalized path but forwards the raw one. - Time window, step, bucket count and result limits are clamped; per-user rate, concurrency and open-tail limits answer `429`. Invalid time, step and limit parameters answer `400`. - The store is resolved from the namespace's `namespace.cozystack.io/monitoring` label. - Live `tail` streams are passed through unbuffered, and the upstream tail request disables compression so headers reach the client before the first line. - `packages/system/observability-proxy`: chart and image build, wired into the root `make build`. - Platform wiring: a new `cozystack.observability-proxy` PackageSource, enabled by default on the full variants and opt-in on the slim ones, the same way as `monitoring-agents`. - RBAC: an aggregated `cozy:observability:view` ClusterRole grants `list` on `cozystack.io/logs` to tenant `view` and above. No API server serves this resource; the rule exists only for the proxy's `SubjectAccessReview`. The proxy's own ClusterRole is limited to the calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch` on namespaces). - Console nginx: when the proxy is not installed or cannot be reached, the observability roots answer `404` with a Kubernetes `Status` body, which the console shows as not available on this cluster; the proxy's own `502`/`504` for an unreachable store pass through unchanged. - Console: one regex `location` in the console nginx routes the four observability roots to the proxy ahead of `/apis`. The upstream is resolved at request time, so the console starts even when the proxy is not installed; those paths then answer `404` with a Kubernetes `Status` body. The console Deployment now receives the cluster domain for that resolution. - E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/` with two tenants, covering `401`/`403`/`200`, scope-widening attempts, path normalization and a long-lived `tail`. #### How it was tested - `go test -race` and `go vet` for the proxy packages. - helm-unittest for `observability-proxy`, `dashboard`, `tenant` and `core/platform`; bats tests for the console nginx routing. - The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs as the root store and refuses to run when a real one exists. Its cases were replayed by a script against the real store on a test cluster. - Verified on a test cluster: - `401` without a token and with an invalid token, `403` for another tenant's namespace, `200` for the caller's own. - Client `extra_stream_filters`/`extra_filters` (GET and POST), an `AccountID` header and another tenant's stream named inside the query do not widen the scope; `options(...)` answers `400`; a path with `..` pointing at another tenant answers `400`; an endpoint outside the allowlist answers `404`. - Through the full browser chain (OIDC login, gatekeeper, gateway, console nginx), a single `tail` stayed open for more than two minutes without reconnecting, and requests from the same session to other tenants' roots answered `403`. - Not verified: the `429` limits under real load. `golangci-lint` has not been run locally. #### Known limitations - The proxy is reachable only through the console route; tenant workloads have no egress rule to it. Every request takes its concurrency or tail slot before the body is read; the token and access reviews and the body read run under `--request-timeout`, so a stalled kube-apiserver answers `503` rather than holding the request; and writes to the client carry a deadline (renewed per line for tails). `--request-timeout=0` disables those bounds and should not be used in production. - The first lines a newly started pod writes can be missing from an already open `tail`, although they are in the store. The likely cause is that Fluent Bit picks up a new log file later than the VictoriaLogs live-tail window; later lines of the same pod arrive. A history query shows them. - The per-application roots (`.../<plural>/<name>/log`, `.../metrics`) and the namespace metrics root answer `404` in this PR. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/787 (docs land once the implementation PRs merge) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: `cozystack/website` is reached. A new platform component (`packages/core/platform/sources/observability-proxy.yaml`) triggers `content/en/docs/next/guides/platform-stack/_index.md` and `operations/configuration/licenses.md`; adding it to the system bundle (`packages/core/platform/templates/bundles/system.yaml`) may also need a line in `operations/configuration/variants.md`. A user-facing page for the logs API and the new tenant `view` permission is not required by the trigger map but would be useful. Tracked in cozystack/website#787. Other repositories: no trigger matched. `packages/core/platform/values.yaml` is not changed (only a test fixture), no package under `packages/apps/` gains a values field, and no CRD changes. The root `Makefile` `build` target builds one more image; this does not touch `hack/` or the make targets `ccp` relies on, so `ccp` is not considered affected, but a reviewer may want to confirm. ### Release note ```release-note feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added an observability proxy for tenant-scoped log access through the console, including live log streaming. Requests are authenticated, access-controlled, and subject to configurable rate and concurrency limits. * Added platform packaging and deployment support, with console routing for observability requests. * **Tests** * Added automated coverage for access control, tenant scoping, query handling, streaming, proxy behavior, and deployment configuration. * **Documentation** * Updated end-to-end test guidance with host requirements for running the observability proxy suite. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8ba0380043 |
docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches S3, among them the create_remote, restore_remote and list calls the strategy makes. The status and action-log endpoints the strategy polls read in-memory state and do not reload it, so the mounted file reaches the sidecar on the calls that need credentials rather than on every request. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3b4db69f8e |
feat(backups): record the RDB snapshot location and size for Redis backups (#5046)
## What this PR does Builds on the shared S3 probe from #5223 (MariaDB and CNPG), which is on `main`. Refs #4316. Backups made by the Redis strategy carry no `Backup.status.artifact` at all: no URI and no size. This covers both the Redis and Valkey apps, which share the driver. The snapshot is a single object per backup, `<namespace>/<app>/<backup>.rdb` (`redisObjectKey`). The driver already records that key in `driverMetadata`. The default strategy uploads the object using the S3 coordinates from the backup credentials Secret, which the controller projects into the application namespace (`endpoint`, `bucketName`, `region` and the access keys). When a backup completes, the driver now: 1. reads that projected Secret (`CredentialsConfig.TargetSecretName`) in the application namespace. The projector stores its `endpoint` without a scheme, and the strategy script reaches it over https; an endpoint with a path prefix is refused by minio-go, which leaves the artifact unset; 2. reads the TLS trust from the pod spec of the completed backup Job, in the order the script applies it: `ca.crt`, then `tls.crt`, from the Secret mounted as the `s3-ca` volume (`backupStorage.endpointCASecretName`). Only when there is no `s3-ca` volume does it honour `S3_INSECURE_SKIP_VERIFY=true`, and only when every container of the pod sets it, since the probe cannot tell which container uploaded. The Secret is read at probe time; the default strategy mounts `s3-ca` only when an endpoint CA is configured, so when the volume yields no CA now (the Secret is gone or lacks both keys, or the volume is not a Secret), the probe leaves the artifact unset rather than trust the system roots or skip verification where the Job may have verified. The CA is trusted on its own, as `curl --cacert` does, not on top of the system roots; 3. runs `StatObject` on the recorded key; a response without `Content-Length`, which minio-go reports as size -1, counts as unmeasured; 4. records `status.artifact.uri = s3://<bucket>/<key>` and `status.artifact.sizeBytes` = the object's size. An empty object is recorded too, but `sizeBytes` is `omitempty`, so on the Backup it reads like an unmeasured size; since a valid RDB snapshot always carries a header, a Warning `EmptySnapshot` Event on the BackupJob flags it as a broken backup. The artifact is set **only when the object is found**. A custom strategy that writes elsewhere would otherwise get a URI pointing at an object that does not exist. On any failure (Secret missing or incomplete, endpoint invalid, CA unreadable, object not found, TLS error) the artifact stays absent and the backup still succeeds. The full error is logged at info, and once the Backup is created a Normal `ArtifactNotMeasured` Event on the BackupJob gives a fixed reason, also when a failed existence check skipped the probe; the Event lives in the application namespace, so it never carries the raw error, which may name the platform's endpoint and bucket. The probe shares the 10s timeout of the MariaDB and CNPG probes. The retry check is shared with those drivers too: a `Backup` that already exists is returned without probing, a failed existence check creates the Backup without probing, a `Create` collision is resolved by reading past the cache, and an existing Backup that cannot be read back is retried with `Ready=False` instead of failing the BackupJob. The size is useful because an RDB snapshot is the compressed dataset. It shows growth, and a snapshot that is suddenly tiny is a broken backup. Kafka is deliberately left out. Its backup is a topic-metadata export of a few kilobytes, so the size would say nothing about the data and might suggest the data itself was backed up. Tests: - the stat uses the projected credentials; - every path that leaves the artifact unset: object missing, Secret missing, bucket, endpoint, access key or secret key missing, no target Secret configured, a size the endpoint did not report; - trust resolution: `ca.crt` wins over `tls.crt` and skip-verify; `items` remapping of the volume, and a key the `items` do not project is never trusted; the Redis CA volume mounted next to `s3-ca` is not taken as the S3 CA; an `s3-ca` volume that yields no CA (Secret gone, Secret without the keys, a ConfigMap source) is refused with and without skip-verify; skip-verify applies only on an exact `true` of that variable in every container, the last entry of a repeated variable winning; - the real minio client against a TLS stub: its certificate is trusted only through the `s3-ca` volume or skip-verify, a CA that did not sign it is refused even with skip-verify set on the Job, and the shared client enforces a configured CA with and without the system roots; the client sends a signed `HEAD` and parses `Content-Length`; - the Events: an unmeasured artifact or a failed existence check raises `ArtifactNotMeasured` without the raw error; an empty snapshot is recorded and raises a Warning `EmptySnapshot`; a measured artifact, a rejected `Create` and a `Create` collision raise none; - a reconcile test pinning that the completed Job's pod spec reaches the probe; a Backup that already exists is returned without a probe or a `Create`; the `Create` collision is read back from the API server past the cache. The suite fails if any guard of the trust lookup (volume name, Secret source, variable name, every-container rule, refusal of an unreadable CA), the CA enforcement in the transport, the negative-size check, the Events, the reason-only message, their place after `Create`, the empty-snapshot warning, the existing-Backup check, the wiring, any term of the Secret check, or the API-server read-back is removed. ### Screenshots N/A — no UI changes. ### Downstream repositories I walked the trigger map against the diff. The change is Go-only in `internal/backupcontroller`, with no API, CRD, chart, value or schema change. Existing `Backup` fields are now filled for one more driver. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(backups): Redis and Valkey backups now record `status.artifact` with the `s3://` location and the size of the RDB snapshot. If the snapshot cannot be found with the platform backup credentials, the artifact stays unset and a Normal `ArtifactNotMeasured` Event on the BackupJob says why. An empty snapshot raises a Warning `EmptySnapshot` Event. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Redis backups now include artifact location and size when they can be measured. * Backups are still created when artifact measurement fails, with status reporting that the artifact was not measured. * **Bug Fixes** * Improved handling of existing backups and temporary read-back failures to support reliable reconciliation. * Improved compatibility with S3 endpoints using custom certificate authorities or configured TLS verification settings. * Zero-size snapshots are now reported with a warning. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
59a11c1788 |
refactor(postgres): judge an attach record's clock drift in one place
The job that attached the plugin and a job waiting on another job's attach each parsed the Cluster's attach record and applied the clock-drift margin on their own, so the two could drift apart on the next change. Both now take the record from one parser that also refuses one too far ahead, and differ only in what they do with a record they cannot use. The margin is now pinned for the waiting job from both sides: a record a minute ahead still holds it for the rollout. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
456473dec1 |
fix(mariadb): fail backup and restore jobs whose operator Job failed (#5207)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does When the Job behind a mariadb-operator `Backup` or `Restore` fails, the operator sets `Complete=True` with reason `JobFailed` (`SetCompleteWithJob` in mariadb-operator 25.10.2). The MariaDB driver read `Complete=True` alone as success, so a backup whose dump or upload failed ended `Succeeded` and produced a `Ready` Cozystack `Backup` pointing at an archive that does not exist, and a failed restore ended `Succeeded` too. A scheduled `Plan` kept producing such backups. The driver now treats `Complete=True` as success only with reason `JobComplete`, and fails the BackupJob or RestoreJob on `JobFailed` whatever the status, without creating a `Backup`. The failure message names the operator object and carries the Job's own `Failed` reason and message, since the operator's condition message is only `Failed` and the Job's pod, with its logs, is already deleted when the Job fails. A Job that is missing or cannot be read is named in the message instead. Any other `Complete` state keeps waiting under the existing deadline, as before; that includes `Complete=False` with reason `Failed`, which the operator sets with its own cause (`Error creating Job`, `Backup not complete`) for a failure outside the Job and then retries, and the deadline failure now carries that message. Regression tests drive the backup and restore reconcilers through every `Complete` status/reason combination, an absent `Complete` condition, and expired deadlines for an absent condition, an unknown reason and reason `Failed`. They assert the job phase and message as persisted in the API rather than on the in-memory object, a requeue while the run is in progress, and, for backups, that no `Backup` is created on failure. Separate tests cover the failure message for a Job that cannot be read and for a Job without a `Failed` condition. Backups that already exist are not touched. A `Ready` MariaDB `Backup` created by a failed run before this fix may have no archive of its own. Restoring from it can fail or select another dump from the shared prefix. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(mariadb): a MariaDB BackupJob or RestoreJob whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive. Backups created by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Backup and restore operations now report failed jobs as failures, including details from the underlying job when available. * Operations are marked successful only when the job completes successfully. Other outcomes continue to be monitored until the existing deadline, after which failures include available condition details. * Failed backups no longer create backup artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
1093f390ae |
feat(backups): warn on the BackupJob when the Redis snapshot is empty
A valid RDB snapshot always carries a header, so a 0-byte object means the backup is broken. sizeBytes is omitempty, though, so such a Backup read exactly like one whose size was never measured. The controller still records the artifact, and now raises a Warning EmptySnapshot Event on the BackupJob once the Backup is created. A test now also pins that a retry over an existing Backup returns it without calling Create. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
2013e624ca |
fix(backups): raise ArtifactNotMeasured only for a Backup this pass created
The Event was recorded before the Backup was created, so a Create that then failed left the BackupJob Failed next to an Event saying the backup succeeded, and a Create that collided with a Backup another pass had made reported on that pass's Backup. The Event is now recorded once Create succeeds. It also covers the one unmeasured path that had none: a failed existence check, which skips the probe. A test now also pins that a CA Secret key the s3-ca volume's items do not project is never trusted. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
6f3d26c491 |
feat(backups): tell on the BackupJob why a Redis artifact was not recorded
A Backup without status.artifact looked the same whether a custom strategy wrote elsewhere, the credentials Secret was unusable or the endpoint CA was gone; only the controller log told them apart. The probe now returns why it gave up, and the controller records it as a Normal ArtifactNotMeasured Event on the BackupJob. The Event lives in the application namespace, so it carries a fixed reason only; the raw error, which may name the platform's endpoint and bucket, stays in the log. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
d61a5054a1 |
fix(backups): refuse to probe a Redis snapshot behind a CA that is gone
The default strategy mounts the s3-ca volume only when the admin configured an endpoint CA, and the script then trusts that CA alone. When the volume yields no CA by the time the probe runs, because the Secret was deleted or emptied after the Job or the volume has another source, the probe used to fall back to the system roots, trusting more than the Job may have. It now leaves the artifact unset instead, with or without skip-verify. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
43bed8f6d9 |
fix(backups): keep the Redis probe verifying TLS unless every container skips
The probe read S3_INSECURE_SKIP_VERIFY from any container of the backup Job, so a custom strategy whose sidecar sets it while the uploading container verifies made the probe skip verification the upload did not. The probe cannot tell which container uploaded, so it now skips only when every container sets the variable to true, the last entry winning as in the kubelet. An s3-ca volume of a source other than a Secret yields no CA the probe can read; it is now treated like a CA Secret that is gone, so it never turns into skip-verify. Tests pin the guards that decide the probe's trust: the Redis CA volume mounted next to the S3 one, a decoy variable set to true, a CA that did not sign the endpoint, and the AlreadyExists read-back past the cache. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
e03986e8d0 |
fix(backups): treat an S3 object without a reported size as unmeasured
minio-go returns size -1 when a HEAD answer carries no Content-Length, which would land in Backup.status.artifact.sizeBytes as -1. The probe now reports such an object as unmeasured, so the field stays unset. The shared client's trust settings also gain a test against an endpoint whose certificate the configured CA did not sign, with the CA alone and on top of the system roots: nothing pinned that a configured CA is actually enforced. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
631f6cfc0d |
fix(backups): mirror the Redis script's CA trust exactly
The script passes the endpoint CA as curl --cacert, which replaces the default trust store, while the probe trusted the system roots on top of it. Trust only the CA, as the script does. A mounted CA Secret that is still there but no longer carries ca.crt or tls.crt let the probe skip TLS verification when the strategy opts into it, although the Job may have verified against a key since removed. Treat a volume that yields no CA now like a vanished Secret and refuse to skip verification in its place. Pin the probe timeout and the reconcile's retry of an unreadable Backup for Redis with tests. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
50b1a2e5ee |
test(backups): pin the Redis retry check
Cover the two retry-check paths the Redis driver shares with MariaDB and CNPG: a failed existence check creates the Backup without probing the bucket, and an existing Backup that cannot be read back is returned as errBackupNotReadBack so the reconcile is retried. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
bfea468b6c |
fix(backups): never skip TLS verification in place of a vanished CA
The Redis probe reads the CA Secret when it runs, not when the backup Job did. If the pod mounted a CA Secret that has since been deleted and the strategy also opts into S3_INSECURE_SKIP_VERIFY, the probe skipped verification against an endpoint the Job may have verified. Refuse to probe in that case. Without skip-verify a missing CA Secret still falls back to the system roots, as the script does. The endpoint comment claimed a bare host; it can carry a path, which the shared client now refuses. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
2ff547348d |
fix(backups): probe the Redis snapshot with the trust its Job used
The default Redis strategy script verifies the S3 endpoint against ca.crt, then tls.crt, from the Secret mounted as the s3-ca volume (backupStorage.endpointCASecretName), and only without either honours S3_INSECURE_SKIP_VERIFY=true. The probe always verified against the system roots, so on a cluster with a private CA, such as a self-signed SeaweedFS, every Redis and Valkey backup succeeded with no artifact. Read the same settings from the pod spec of the completed backup Job, in the order the script applies them, and pass them to the S3 connection. The shared transport gains an InsecureSkipVerify switch that is set only from that explicit opt-in. The projector always stores the endpoint as a bare host, so the branch that parsed a scheme out of it was unreachable; drop it and its test. Cover the endpoint and access-key terms of the Secret completeness check. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
c8cdefc665 |
feat(backups): record the RDB snapshot location and size for Redis backups
Backups made by the Redis strategy, which also serves Valkey, carried no Backup.status.artifact at all: neither a URI nor a size. The snapshot is one object per backup, <namespace>/<app>/<backup>.rdb, whose key the driver already records, and the default strategy uploads it with the S3 coordinates of the backup credentials Secret the controller projects into the application namespace. On completion the driver now stats that object with the same coordinates and records its s3:// URI and size. The artifact is set only when the object is found, so a custom strategy that writes elsewhere gets no URI pointing at a missing object. A Backup that already exists is returned without probing, as for the MariaDB and CNPG drivers. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
5635d9d815 |
test(postgres): pin that a rollout dated by the attach record runs out
The deadline tests dated every rollout by the job's own condition, so a change that let a rollout dated by the Cluster's attach record wait forever would have passed them. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a27de054ca |
fix(postgres): wait for the rollout another backup job's attach started
Only the job that attaches the barman-cloud plugin waited for the instance rollout it causes. A second job of the same application that started meanwhile found the plugin already there, had no rollout of its own to wait for and created its cnpg.io/Backup at once, against instances CNPG was still recreating, and such a Backup fails for good. The queue does not hold it either, since the attaching job has no Backup yet. The attach record on the Cluster says when the plugin was attached, so a job without a rollout of its own now waits for instances newer than it while the attach is recent enough for that rollout to be running. An older attach has had its rollout judged by the job that made it. The record belongs to that job, so one this job cannot read, or one further ahead than controller clocks drift, is skipped rather than failing it; the latter would otherwise hold every later job of the application for as long as it lies ahead. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d37701a327 |
fix(postgres): tolerate an attach record slightly ahead of the clock
An attach record in the future failed the backup job at once, because it would hold every instance as older than the plugin. A record a few minutes ahead is also what a leadership change between controllers whose clocks differ leaves behind after an attach that worked, and failing then throws away a backup that would have run. Within that margin the job now keeps waiting: the instances predate the record, so the rollout check holds the backup, and the deadline counted from the record is not spent by the wait. Only a record further ahead is taken as corrupt and fails the job. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c69787ff49 |
perf(postgres): list only the Cluster's own backups for the queue
Every pass of a backup job waiting for another one reads the cnpg.io/Backups from the apiserver, and the read covered every Backup in the namespace, so a namespace with a long backup history paid for it on each poll of each waiting job. The queue only looks at Backups this controller created for the Cluster, which have carried the cluster and owning-job labels since the driver was introduced, so it now asks the apiserver for those alone. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9a8025ca14 |
fix(postgres): record the serverName a backup was archived under
The artifact took its destination from the record written when the cnpg.io/Backup was created, but its serverName, the archive prefix under that destination, from the live Cluster when the backup completed. A restore that re-renders the Cluster in between gives it another serverName, and the artifact then pairs the original destination with a prefix the backup was never written to, so a later restore searches the wrong archive. The record now carries the serverName too; a record written before it did still falls back to the live Cluster. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
b4551eb379 |
fix(postgres): read restores from the apiserver before writing the destination
A backup job that is about to write the WAL archive destination first checks for a restore of its application, because a write landing during the restore puts the restored Cluster back on the source's WAL prefix and the restore fails with an expected-empty-archive error after it has purged the target. That check read RestoreJobs from the informer cache, which can still miss one created a moment ago. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8a631311a7 |
fix(postgres): fail a backup job whose plugin attach record is in the future
The rollout check takes the later of the latch and the Cluster's attach record, and the record is never rewritten while the plugin stays attached. A record that parses but lies in the future made every instance look older than the plugin and counted the rollout deadline from that time, so the job waited past the deadline by as far as the record lay ahead, without end for one far enough out. Such a record says no more about when the plugin was attached than one that does not parse, and fails the job the same way. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
fe03589d91 |
test(postgres): cover a lost attach record on the retry of a failed attach
A status write that fails with the apiserver unavailable takes the same path as one lost to a conflict, but the attach it follows can itself be the retry of an apply that failed first, which leaves its own condition on the job. The test now runs that sequence as well, so the rollout check is held to it whichever way the record was lost. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8d92aee796 |
test(postgres): pin that an older attach record keeps the latch
The rollout check takes the later of the latch and the Cluster's attach record. A record left by an earlier attach of a plugin since removed is older than the latch when someone who keeps no record attaches the plugin again, and taking it would accept an instance recreated before the latch as one that rolled with the plugin. The record is absent not only when no backup job attached the plugin but also when the chart did, or a controller that kept no record, so the comment on the field says that. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
325d2b815c |
fix(postgres): keep the plugin attach time when its status write is lost
The rollout check took the attach time from the PluginRolledOut condition on the BackupJob, which the pass that attaches the plugin writes only after the Cluster apply went through. When that status write is lost to a conflict or a controller restart, the next pass finds the plugin attached and falls back to the latch written before the attach. An instance recreated between the latch and the attach then counts as one that rolled with the plugin, and the cnpg.io/Backup is created against an instance that has no plugin sidecar; a latch older than the deadline also fails a rollout that has only just begun. The apply that attaches the plugin now records the time on the Cluster itself, so the record cannot be lost apart from the attach. The rollout check and its deadline use the later of the two times. A record that does not parse fails the job rather than guess which instances rolled. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e5ccde82e2 |
fix(postgres): read cnpg Backups from the apiserver before writing the destination
The queue that keeps two BackupJobs of one application from writing the WAL archive destination at once listed cnpg.io/Backups from the informer cache. The controller runs one reconcile at a time under leader election, so a Backup created by one job's pass already exists when the next pass starts, but the cache can still miss it. The next job then took the queue as empty, rewrote the destination under the running backup and left that backup's recorded destination naming a place its WAL no longer went. A job could likewise miss its own fresh Backup, write the destination again and create a second Backup. The queue now lists Backups from the apiserver, and a job that finds no Backup of its own in the cache checks the apiserver before it writes or creates one. Only a leader handover, when two processes reconcile for a moment, can still let two jobs through. The pass that creates the Backup no longer looks it up a second time right before the create. Nothing can create it in between, and once the lookup can reach the apiserver, a failed second lookup would come after the destination write and fail the job instead of retrying. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a6af14ce74 |
fix(postgres): build a backup's artifact from the destination it was written to
The WAL archive destination is written by the pass that creates the cnpg.io/Backup and not again while it runs, yet the artifact took the destination, the endpoint and the credentials Secret from the strategy rendered when the Backup completed. A strategy or BackupClass edited while the backup ran gave an artifact that pointed somewhere nothing had been written, and a restore from it found no data. The pass that writes the destination now records it on the cnpg.io/Backup it creates, and the artifact is built from that record. The record lives on the Backup rather than on the BackupJob because a tenant can edit its BackupJobs but has no access to postgresql.cnpg.io objects: a record a tenant could rewrite would point the artifact, and a restore from it, which takes its S3 keys from the strategy, at an endpoint of the tenant's choosing. The artifact size probe measures the same place: the recorded location and credentials Secret, with the Secret keys and the endpoint CA, which the record does not carry, from the strategy. Should those no longer fit after an edit, the probe reports no size instead of measuring somewhere else. A record that cannot be read, or that names no destination path, creates no artifact and fails the job, since the strategy may no longer match the write. A Backup without a record was created by a controller that kept none, and its artifact is built from the strategy as before. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
59cbf4b28e |
docs(observability-proxy): say what the request timeout bounds
The flag now also bounds the body read and the token and access reviews, and 0 lifts all of those at once; an operator raising it for slow store queries needs to know that. The store comment named a design document the repository does not have. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
d2ef5e031f |
docs(observability-proxy): describe the proxy as it stands
The package and serve comments described a delivery plan rather than the code: the command named metrics it does not serve, and serve gave a plan as the reason to refuse unknown roots before authenticating. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
4eff1617e5 |
fix(observability-proxy): bound the token and access reviews
Both reviews ran on the request context, which net/http cancels for a dropped client only once the body is read. A POST whose body was still unread therefore waited on a stalled kube-apiserver past every request limit. The reviews now share the request timeout and answer 503 when it passes. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
258c5359b6 |
fix(postgres): restart the plugin rollout whenever a backup job attaches it
Only the first attach after the latch restarted the rollout. In the platform flow the chart renders no spec.plugins, so an in-place restore recreates the Cluster without the plugin, and a backup job held through that restore attaches it again once released. The rollout condition then still carried its earlier start, the restored instances were newer than it and ready, and the Backup was created in the same pass, before CloudNativePG had rolled anything; such a Backup fails with "requested plugin is not available" or "instance manager was restarted". Every write of the job that finds the plugin missing now starts the rollout over. A plugin attached by another writer is still timed from the latch. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
32ca678077 |
fix(postgres): give a held backup job its plugin rollout deadline back
A job that latched the barman-cloud plugin rollout and then waited for another backup of the application, or for a restore of it, had that wait charged to the rollout deadline. Released after more than the deadline, it failed on the first instance that was not ready yet with "rollout did not finish", although no rollout of its own had been running while it waited. The hold is now recorded in a DestinationWriteHeld condition, and its release time restarts the rollout deadline when it is later than the rollout's start. The instance pods are still measured against the rollout's start: one that rolled while the job was held carries the plugin and counts. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8061e2dbd9 |
test(postgres): describe what the backup deadline test still guards
The deadline no longer applies to a failed cnpg.io/Backup, which now fails its BackupJob at once; it only bounds the wait for the barman-cloud plugin rollout, and the comment still claimed the former. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7efe33e87d |
test(postgres): check the stored phase of a backup job per cnpg Backup phase
The phase test read the BackupJob it passed to the reconciler, which holds the phase in memory whether or not the status write happened. A completed Backup could then leave the stored BackupJob Running, so the Plan would never see the run finish, and no test in the package noticed. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
93c3ee62c4 |
test(postgres): pin which holder a queued backup job names
When two jobs of the application hold the queue, the job waiting behind them names one of them in its Ready message, and the sort is what keeps that name stable however the cache orders their Backups. Without it the message can change from one poll to the next, and each change writes the BackupJob status again; nothing failed when the sort was removed. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a02bff7fad |
docs(postgres): say what the CNPG backup deadlines are counted from
The rollout deadline starts at the latch when another writer attached the plugin and at the write otherwise, and a wait for a restore resets the destination retry budget just as a wait for another backup does. The comments on the deadline and on the two conditions still described only the attach and the queue. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
66268d8508 |
refactor(postgres): drop a Ready reset every later path overwrites
After a successful destination write each path of the CNPG driver sets or removes Ready later in the same pass, and writes the status after it, so clearing a stale WaitingForOtherBackup or ClusterPluginApplyRetrying reason there could never outlive the pass, and no test could pin it. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
61d64a69f1 |
test(postgres): pin the requeue and the read-only pass of a running backup
Three guards of the CNPG driver could be reverted with the package tests still green. The controller watches only BackupJobs, so a job waiting in the queue or retrying the destination write is woken by nothing but its own requeue, and the tests dropped the result. A job whose Backup runs must not consult the queue, or two jobs that passed it together wait on each other forever, and must patch neither the ObjectStore nor the Cluster plugin, yet only the ObjectStore content was checked. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
567f07d708 |
fix(postgres): time the plugin rollout from the write that attached it
The PluginRolledOut latch is recorded before the Cluster write that attaches the barman-cloud plugin, and its time was used both as the start of the rollout deadline and as the age every instance pod must exceed to count as rolled. Now that a failing write is retried for up to the backup deadline, the attach can come long after the latch: the deadline was then spent before the rollout began, and a pod recreated between the latch and the attach, without the plugin sidecar, passed as rolled, so a Backup was created that CloudNativePG fails for good. The latch now carries its own reason, and the write that finds the plugin missing and attaches it re-stamps the condition, once, with the time it was attempted. A plugin someone else attached after the latch, another backup job or a restore, keeps the latch time: the rollout it started can be over by the time this job writes, and its instances would otherwise never count as rolled. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
570fcec725 |
fix(postgres): hold a queued backup job while its application is restored
The restore gate in Reconcile looks only at a BackupJob that has not started, and the CNPG driver stamps StartedAt on its first pass. A job that then waits behind another backup of the application, for as long as that backup runs, writes the WAL archive destination and the Cluster plugin once released without looking for a restore that began in the meantime. A write landing while the restore re-renders the Cluster can put the restored Cluster back on the source's WAL prefix, where recovery fails with "Expected empty archive". Every pass that is about to write the destination now looks for a restore of the application and holds the job the way the gate does. A job whose Backup already runs writes nothing and is not held. The hold resets the retry budget of the destination write, as the wait for another backup does, so time spent behind a restore does not fail the job on the first error after it. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
9cc9a3b48d |
fix(postgres): stop backup jobs from fighting over the WAL archive destination
The ObjectStore a Postgres Cluster archives its WAL to is shared by every backup of the app, yet each running BackupJob server-side applied its own strategy's destination onto it every five seconds with forced ownership. Two BackupJobs of one app with different BackupClasses flipped the destination back and forth until the healthy backup failed too, and a webhook or apiserver blip during that apply failed the BackupJob at once. A BackupJob now writes the destination only before its cnpg.io/Backup exists, and waits, with a WaitingForOtherBackup condition, while another running BackupJob of the app has a Backup in flight. The check reads the informer cache, so jobs released together, or one created while another's Backup is not yet cached, can still both proceed; an atomic per-application claim is a separate change. Write errors that cannot succeed on retry (invalid, bad request, forbidden) still fail the job at once. Any other is retried for the backup deadline counted from its first failure since the job last left the queue, recorded in a DestinationApplied condition. Once the Backup exists the Cluster is only read, and a failed read is returned for a controller retry instead of failing the running backup. The plugin rollout deadline counts from the attach rather than from the job's start, so queueing before the attach no longer shortens it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |