mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
2cf80c02a4c2a38b25faed766fd7f5caf2a214cd
219
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
583f18437d |
test(clickhouse): give the credentials-wait BackupJob a UID
The test pins that a backup held for late bucket credentials keeps its archive name, and a run's archive name now carries part of the BackupJob's UID, so the fixture needs one for the name it expects to be the name the controller mints. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
dae76d186c |
Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for late bucket credentials, against the controller's archive helper. That helper now names a run's archive with part of the BackupJob's UID, so the merged test has to give its BackupJob one. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
238b3ee491 |
fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but only the `s3_lifecycle` job of its plugin worker applies it, and that job is scheduled by the admin server. Cozystack deployed neither, so a lifecycle rule returned 200 and no object ever expired, on the platform backup bucket and on tenant buckets alike. That also made the documented way to bound backup storage a no-op. This PR adds the admin server and a worker to Simple and MultiZone SeaweedFS instances, behind a new `lifecycle.enabled` value that is **off by default**. A Client instance is unchanged, because its rules are enforced where the instance it connects to runs. The worker registers for `s3_lifecycle` only, so vacuum, balancing and erasure coding are not started as a side effect. The admin keeps its job state on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default when that is empty; `volume.storageClass` is not reused, because in MultiZone it can name a class local to one zone or node. The admin UI is protected by a generated password, because a tenant namespace admits traffic from the whole cluster. The dashboard Role lists the two new WorkloadMonitors. Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart renders no `[grpc.s3]` section, so the s3 gateway served its internal gRPC in plaintext while the worker dials it with mTLS, and every lifecycle delete failed the TLS handshake. The section is backported from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart patch, applied with `--forward` so the next `make update` stops at the point where the patch must be dropped instead of silently reversing it. The s3 pods are restarted once so they load the section. This also repairs IAM cache propagation between s3 replicas, which failed in the same way, and it applies whether or not lifecycle is enabled. The data-plane release now retries a failed install or upgrade in place (`RetryOnFailure`) instead of rolling it back. With the previous default remediation and unlimited retries, an upgrade that waited out its timeout, for example on a tenant quota with no room for the admin volume, rolled back and retried forever, restarting the s3 pods twice per cycle. The worker's hostname anti-affinity is relaxed to a preference so a rollout finishes on a single-node cluster. #### Why lifecycle is off by default The daily walk that SeaweedFS uses to expire objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when no identity is given; it does not re-check the object's age either. A key overwritten between the walk's listing and the delete loses its new content. This path is unchanged from 4.45 through 4.52 and on upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`, `identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup buckets write unique keys and are barely exposed; a tenant bucket that rewrites a fixed key is. On top of that, turning enforcement on would start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so nothing changes for existing instances until `lifecycle.enabled` is set. Making it the default again is left to a separate PR that carries object identity on the walk path; upstream report: https://github.com/seaweedfs/seaweedfs/issues/12451. #### Turning lifecycle on and off When `lifecycle.enabled` is set, rules that have been stored but ignored start applying within minutes, and objects they have already expired are deleted on that first pass, so review existing bucket lifecycle rules first. Set `lifecycle.storageClass` before enabling, and prefer a replicated class: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. Moving the volume means disabling lifecycle, deleting the `admin-data-seaweedfs-admin-0` PVC, changing the class and enabling lifecycle again. Each instance gains two small pods and a 1Gi PVC; when the tenant quota has no room for them, the data-plane release keeps retrying in place until it has. When `lifecycle.enabled` is turned off again, the parent release removes the admin credentials Secret while the data-plane release removes the admin StatefulSet only on its own reconcile, shortly after. An admin pod that restarts in that window fails to start on the missing Secret until the StatefulSet is gone; nothing is lost, and the order is left as is. The admin PVC stays until the instance is deleted, and a later re-enable starts the admin with a new password. Verified on a test cluster with the 4.45 chart: without the backport, the worker found the expired objects and every delete failed with `tls: first record does not look like a TLS handshake`; with this branch and lifecycle enabled, a rule with a past expiration date removed the matching objects and left the others, both on a manual run and on the first scheduled pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up needed, not opened yet: `operations/services/backup-classes.md` (v1.7 and next) advises an S3 lifecycle policy, which the bundled SeaweedFS applies only once `lifecycle.enabled` is set; the SeaweedFS reference page gains `lifecycle.enabled` and `lifecycle.storageClass` from the regenerated README on the next release - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The terraform provider does not model the SeaweedFS application, so the new values do not reach it. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added optional, daily S3 bucket lifecycle-rule processing for supported SeaweedFS configurations. It is disabled by default; Client instances rely on lifecycle rules configured on the connected SeaweedFS. * Added configurable storage for lifecycle job state and dashboard visibility for lifecycle components. * Enabled the S3 gRPC server to use its internal TLS identity when security is configured. * **Bug Fixes** * Improved storage-class handling so numeric-looking names remain strings and unset classes are omitted where appropriate. * Clarified backup cleanup guidance, including when lifecycle rules apply and how to reclaim storage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3badbdef74 |
fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5a95bcdd64 |
fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024 bytes, and the legacy s3PathOverride prefix has no length limit. The full UID made the name 37 characters longer than before, so a prefix that fit the old name pushed the manifest key past the cap and failed the backup. The run's name now drops "clickhouse-" and the "T" and "Z" of the timestamp and spends the room on the first 12 hex digits of the UID, so it is exactly as long as the old name for every application. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8ba0380043 |
docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches S3, among them the create_remote, restore_remote and list calls the strategy makes. The status and action-log endpoints the strategy polls read in-memory state and do not reload it, so the mounted file reaches the sidecar on the calls that need credentials rather than on every request. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3b4db69f8e |
feat(backups): record the RDB snapshot location and size for Redis backups (#5046)
## What this PR does Builds on the shared S3 probe from #5223 (MariaDB and CNPG), which is on `main`. Refs #4316. Backups made by the Redis strategy carry no `Backup.status.artifact` at all: no URI and no size. This covers both the Redis and Valkey apps, which share the driver. The snapshot is a single object per backup, `<namespace>/<app>/<backup>.rdb` (`redisObjectKey`). The driver already records that key in `driverMetadata`. The default strategy uploads the object using the S3 coordinates from the backup credentials Secret, which the controller projects into the application namespace (`endpoint`, `bucketName`, `region` and the access keys). When a backup completes, the driver now: 1. reads that projected Secret (`CredentialsConfig.TargetSecretName`) in the application namespace. The projector stores its `endpoint` without a scheme, and the strategy script reaches it over https; an endpoint with a path prefix is refused by minio-go, which leaves the artifact unset; 2. reads the TLS trust from the pod spec of the completed backup Job, in the order the script applies it: `ca.crt`, then `tls.crt`, from the Secret mounted as the `s3-ca` volume (`backupStorage.endpointCASecretName`). Only when there is no `s3-ca` volume does it honour `S3_INSECURE_SKIP_VERIFY=true`, and only when every container of the pod sets it, since the probe cannot tell which container uploaded. The Secret is read at probe time; the default strategy mounts `s3-ca` only when an endpoint CA is configured, so when the volume yields no CA now (the Secret is gone or lacks both keys, or the volume is not a Secret), the probe leaves the artifact unset rather than trust the system roots or skip verification where the Job may have verified. The CA is trusted on its own, as `curl --cacert` does, not on top of the system roots; 3. runs `StatObject` on the recorded key; a response without `Content-Length`, which minio-go reports as size -1, counts as unmeasured; 4. records `status.artifact.uri = s3://<bucket>/<key>` and `status.artifact.sizeBytes` = the object's size. An empty object is recorded too, but `sizeBytes` is `omitempty`, so on the Backup it reads like an unmeasured size; since a valid RDB snapshot always carries a header, a Warning `EmptySnapshot` Event on the BackupJob flags it as a broken backup. The artifact is set **only when the object is found**. A custom strategy that writes elsewhere would otherwise get a URI pointing at an object that does not exist. On any failure (Secret missing or incomplete, endpoint invalid, CA unreadable, object not found, TLS error) the artifact stays absent and the backup still succeeds. The full error is logged at info, and once the Backup is created a Normal `ArtifactNotMeasured` Event on the BackupJob gives a fixed reason, also when a failed existence check skipped the probe; the Event lives in the application namespace, so it never carries the raw error, which may name the platform's endpoint and bucket. The probe shares the 10s timeout of the MariaDB and CNPG probes. The retry check is shared with those drivers too: a `Backup` that already exists is returned without probing, a failed existence check creates the Backup without probing, a `Create` collision is resolved by reading past the cache, and an existing Backup that cannot be read back is retried with `Ready=False` instead of failing the BackupJob. The size is useful because an RDB snapshot is the compressed dataset. It shows growth, and a snapshot that is suddenly tiny is a broken backup. Kafka is deliberately left out. Its backup is a topic-metadata export of a few kilobytes, so the size would say nothing about the data and might suggest the data itself was backed up. Tests: - the stat uses the projected credentials; - every path that leaves the artifact unset: object missing, Secret missing, bucket, endpoint, access key or secret key missing, no target Secret configured, a size the endpoint did not report; - trust resolution: `ca.crt` wins over `tls.crt` and skip-verify; `items` remapping of the volume, and a key the `items` do not project is never trusted; the Redis CA volume mounted next to `s3-ca` is not taken as the S3 CA; an `s3-ca` volume that yields no CA (Secret gone, Secret without the keys, a ConfigMap source) is refused with and without skip-verify; skip-verify applies only on an exact `true` of that variable in every container, the last entry of a repeated variable winning; - the real minio client against a TLS stub: its certificate is trusted only through the `s3-ca` volume or skip-verify, a CA that did not sign it is refused even with skip-verify set on the Job, and the shared client enforces a configured CA with and without the system roots; the client sends a signed `HEAD` and parses `Content-Length`; - the Events: an unmeasured artifact or a failed existence check raises `ArtifactNotMeasured` without the raw error; an empty snapshot is recorded and raises a Warning `EmptySnapshot`; a measured artifact, a rejected `Create` and a `Create` collision raise none; - a reconcile test pinning that the completed Job's pod spec reaches the probe; a Backup that already exists is returned without a probe or a `Create`; the `Create` collision is read back from the API server past the cache. The suite fails if any guard of the trust lookup (volume name, Secret source, variable name, every-container rule, refusal of an unreadable CA), the CA enforcement in the transport, the negative-size check, the Events, the reason-only message, their place after `Create`, the empty-snapshot warning, the existing-Backup check, the wiring, any term of the Secret check, or the API-server read-back is removed. ### Screenshots N/A — no UI changes. ### Downstream repositories I walked the trigger map against the diff. The change is Go-only in `internal/backupcontroller`, with no API, CRD, chart, value or schema change. Existing `Backup` fields are now filled for one more driver. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(backups): Redis and Valkey backups now record `status.artifact` with the `s3://` location and the size of the RDB snapshot. If the snapshot cannot be found with the platform backup credentials, the artifact stays unset and a Normal `ArtifactNotMeasured` Event on the BackupJob says why. An empty snapshot raises a Warning `EmptySnapshot` Event. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Redis backups now include artifact location and size when they can be measured. * Backups are still created when artifact measurement fails, with status reporting that the artifact was not measured. * **Bug Fixes** * Improved handling of existing backups and temporary read-back failures to support reliable reconciliation. * Improved compatibility with S3 endpoints using custom certificate authorities or configured TLS verification settings. * Zero-size snapshots are now reported with a warning. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
456473dec1 |
fix(mariadb): fail backup and restore jobs whose operator Job failed (#5207)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does When the Job behind a mariadb-operator `Backup` or `Restore` fails, the operator sets `Complete=True` with reason `JobFailed` (`SetCompleteWithJob` in mariadb-operator 25.10.2). The MariaDB driver read `Complete=True` alone as success, so a backup whose dump or upload failed ended `Succeeded` and produced a `Ready` Cozystack `Backup` pointing at an archive that does not exist, and a failed restore ended `Succeeded` too. A scheduled `Plan` kept producing such backups. The driver now treats `Complete=True` as success only with reason `JobComplete`, and fails the BackupJob or RestoreJob on `JobFailed` whatever the status, without creating a `Backup`. The failure message names the operator object and carries the Job's own `Failed` reason and message, since the operator's condition message is only `Failed` and the Job's pod, with its logs, is already deleted when the Job fails. A Job that is missing or cannot be read is named in the message instead. Any other `Complete` state keeps waiting under the existing deadline, as before; that includes `Complete=False` with reason `Failed`, which the operator sets with its own cause (`Error creating Job`, `Backup not complete`) for a failure outside the Job and then retries, and the deadline failure now carries that message. Regression tests drive the backup and restore reconcilers through every `Complete` status/reason combination, an absent `Complete` condition, and expired deadlines for an absent condition, an unknown reason and reason `Failed`. They assert the job phase and message as persisted in the API rather than on the in-memory object, a requeue while the run is in progress, and, for backups, that no `Backup` is created on failure. Separate tests cover the failure message for a Job that cannot be read and for a Job without a `Failed` condition. Backups that already exist are not touched. A `Ready` MariaDB `Backup` created by a failed run before this fix may have no archive of its own. Restoring from it can fail or select another dump from the shared prefix. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(mariadb): a MariaDB BackupJob or RestoreJob whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive. Backups created by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Backup and restore operations now report failed jobs as failures, including details from the underlying job when available. * Operations are marked successful only when the job completes successfully. Other outcomes continue to be monitored until the existing deadline, after which failures include available condition details. * Failed backups no longer create backup artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
1093f390ae |
feat(backups): warn on the BackupJob when the Redis snapshot is empty
A valid RDB snapshot always carries a header, so a 0-byte object means the backup is broken. sizeBytes is omitempty, though, so such a Backup read exactly like one whose size was never measured. The controller still records the artifact, and now raises a Warning EmptySnapshot Event on the BackupJob once the Backup is created. A test now also pins that a retry over an existing Backup returns it without calling Create. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
2013e624ca |
fix(backups): raise ArtifactNotMeasured only for a Backup this pass created
The Event was recorded before the Backup was created, so a Create that then failed left the BackupJob Failed next to an Event saying the backup succeeded, and a Create that collided with a Backup another pass had made reported on that pass's Backup. The Event is now recorded once Create succeeds. It also covers the one unmeasured path that had none: a failed existence check, which skips the probe. A test now also pins that a CA Secret key the s3-ca volume's items do not project is never trusted. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
6f3d26c491 |
feat(backups): tell on the BackupJob why a Redis artifact was not recorded
A Backup without status.artifact looked the same whether a custom strategy wrote elsewhere, the credentials Secret was unusable or the endpoint CA was gone; only the controller log told them apart. The probe now returns why it gave up, and the controller records it as a Normal ArtifactNotMeasured Event on the BackupJob. The Event lives in the application namespace, so it carries a fixed reason only; the raw error, which may name the platform's endpoint and bucket, stays in the log. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
d61a5054a1 |
fix(backups): refuse to probe a Redis snapshot behind a CA that is gone
The default strategy mounts the s3-ca volume only when the admin configured an endpoint CA, and the script then trusts that CA alone. When the volume yields no CA by the time the probe runs, because the Secret was deleted or emptied after the Job or the volume has another source, the probe used to fall back to the system roots, trusting more than the Job may have. It now leaves the artifact unset instead, with or without skip-verify. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
43bed8f6d9 |
fix(backups): keep the Redis probe verifying TLS unless every container skips
The probe read S3_INSECURE_SKIP_VERIFY from any container of the backup Job, so a custom strategy whose sidecar sets it while the uploading container verifies made the probe skip verification the upload did not. The probe cannot tell which container uploaded, so it now skips only when every container sets the variable to true, the last entry winning as in the kubelet. An s3-ca volume of a source other than a Secret yields no CA the probe can read; it is now treated like a CA Secret that is gone, so it never turns into skip-verify. Tests pin the guards that decide the probe's trust: the Redis CA volume mounted next to the S3 one, a decoy variable set to true, a CA that did not sign the endpoint, and the AlreadyExists read-back past the cache. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
e03986e8d0 |
fix(backups): treat an S3 object without a reported size as unmeasured
minio-go returns size -1 when a HEAD answer carries no Content-Length, which would land in Backup.status.artifact.sizeBytes as -1. The probe now reports such an object as unmeasured, so the field stays unset. The shared client's trust settings also gain a test against an endpoint whose certificate the configured CA did not sign, with the CA alone and on top of the system roots: nothing pinned that a configured CA is actually enforced. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
631f6cfc0d |
fix(backups): mirror the Redis script's CA trust exactly
The script passes the endpoint CA as curl --cacert, which replaces the default trust store, while the probe trusted the system roots on top of it. Trust only the CA, as the script does. A mounted CA Secret that is still there but no longer carries ca.crt or tls.crt let the probe skip TLS verification when the strategy opts into it, although the Job may have verified against a key since removed. Treat a volume that yields no CA now like a vanished Secret and refuse to skip verification in its place. Pin the probe timeout and the reconcile's retry of an unreadable Backup for Redis with tests. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
50b1a2e5ee |
test(backups): pin the Redis retry check
Cover the two retry-check paths the Redis driver shares with MariaDB and CNPG: a failed existence check creates the Backup without probing the bucket, and an existing Backup that cannot be read back is returned as errBackupNotReadBack so the reconcile is retried. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
bfea468b6c |
fix(backups): never skip TLS verification in place of a vanished CA
The Redis probe reads the CA Secret when it runs, not when the backup Job did. If the pod mounted a CA Secret that has since been deleted and the strategy also opts into S3_INSECURE_SKIP_VERIFY, the probe skipped verification against an endpoint the Job may have verified. Refuse to probe in that case. Without skip-verify a missing CA Secret still falls back to the system roots, as the script does. The endpoint comment claimed a bare host; it can carry a path, which the shared client now refuses. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
2ff547348d |
fix(backups): probe the Redis snapshot with the trust its Job used
The default Redis strategy script verifies the S3 endpoint against ca.crt, then tls.crt, from the Secret mounted as the s3-ca volume (backupStorage.endpointCASecretName), and only without either honours S3_INSECURE_SKIP_VERIFY=true. The probe always verified against the system roots, so on a cluster with a private CA, such as a self-signed SeaweedFS, every Redis and Valkey backup succeeded with no artifact. Read the same settings from the pod spec of the completed backup Job, in the order the script applies them, and pass them to the S3 connection. The shared transport gains an InsecureSkipVerify switch that is set only from that explicit opt-in. The projector always stores the endpoint as a bare host, so the branch that parsed a scheme out of it was unreachable; drop it and its test. Cover the endpoint and access-key terms of the Secret completeness check. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
c8cdefc665 |
feat(backups): record the RDB snapshot location and size for Redis backups
Backups made by the Redis strategy, which also serves Valkey, carried no Backup.status.artifact at all: neither a URI nor a size. The snapshot is one object per backup, <namespace>/<app>/<backup>.rdb, whose key the driver already records, and the default strategy uploads it with the S3 coordinates of the backup credentials Secret the controller projects into the application namespace. On completion the driver now stats that object with the same coordinates and records its s3:// URI and size. The artifact is set only when the object is found, so a custom strategy that writes elsewhere gets no URI pointing at a missing object. A Backup that already exists is returned without probing, as for the MariaDB and CNPG drivers. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
10e0ef9ece |
fix(postgres): fail a backup job as soon as its cnpg Backup fails (#5211)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Part of #5201. The remaining part, a bound on a Backup that stays `pending`, is in #5209. A Postgres BackupJob whose `cnpg.io/Backup` had failed stayed `Running` for 30 minutes with reason `CNPGBackupTransientFailure`, on the premise that CloudNativePG may move a Backup out of the failed phase. It does not: in CloudNativePG 1.30.0 the backup controller returns early on any Backup already `failed` or `completed`, so the wait only delayed the verdict. The BackupJob now fails as soon as the Backup is `failed`, with the error CloudNativePG recorded. When that error is only the plugin's exit status, the message points to the `plugin-barman-cloud` sidecar log, where the storage error (`AccessDenied`, a connect timeout) is. A Backup in any other phase keeps the BackupJob waiting as before, but the BackupJob now carries a `Ready=False` / `WaitingForCNPGBackup` condition naming the Backup and its phase instead of no condition at all. These phases stay uncapped on purpose: CloudNativePG keeps a healthy Backup `pending` while another backup of the same Cluster runs and while it restarts one on another instance, so a fixed deadline here would fail healthy backups, in particular scheduled ones that queue behind a long previous run. #5209 tracks bounding a Backup that is stuck rather than queued, together with the Plan concurrency that makes queueing common. A table-driven test drives the reconciler through each Backup phase and asserts the job phase, the message, the persisted condition, that an unchanged phase does not rewrite the BackupJob status, and that only a completed Backup produces a Cozystack `Backup`. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(postgres): a Postgres BackupJob now fails as soon as its CloudNativePG Backup fails, with the recorded error, instead of after 30 minutes, and a BackupJob waiting on a CloudNativePG Backup reports a WaitingForCNPGBackup condition with the Backup's phase. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * CNPG backups that enter a failed state now fail their BackupJob immediately, with the recorded error shown when available. * BackupJobs waiting for a CNPG backup to finish remain in a waiting state, even after the backup deadline; the deadline applies while waiting for the backup to be created. * Waiting status now includes a clear reason, and failure messages provide a sidecar-log hint when relevant. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
2046649b4c |
fix(clickhouse): match a backup Job's archive by its whole name
The archive a backup Job carries was found as a substring of its Pod spec. The name without the BackupJob's UID starts the name with it, and a strategy may append to either, so a Job that wrote a longer archive could get the shorter name recorded on its Backup, and a restore of that Backup would read another archive. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
fc15ed7171 |
fix(clickhouse): keep the archive name of a backup running across the upgrade
A backup Job created before archive names carried the BackupJob's UID writes its archive under the name without it, and the Job is not re-rendered when the controller is upgraded. Matching only the new name left that Backup with no archive recorded, so its restore fell back to guessing by timestamp. The name without the UID is matched second, because it is a prefix of the one with it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1291e616c7 |
fix(clickhouse): give every backup run an archive of its own
An archive was named from the application and the BackupJob's start time, which has one-second resolution. Two runs of one application started in the same second, as two Plans on the same schedule are, wrote the same archive. The strategy takes a finished upload of its archive as an earlier attempt of its own run, so the later run could succeed on the other run's upload and record an archive missing any table created since. The BackupJob's UID now ends the name. It goes after the timestamp so names keep sorting chronologically, and the restore of a Backup that records no archive, which accepts only a bare timestamp after the release prefix, never picks one of these. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9e8e10c307 |
fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does Refs #5205. The S3 endpoint the platform backups use is copied twice, and neither copy is refreshed after `publishing.host` changes. The COSI sidecar writes `BucketInfo` once per `BucketAccess`, so the credentials Secret `bucket-cozy-backups-system-credentials` rendered from it keeps the old endpoint. The Velero BSL and the default Strategy CRs keep whatever endpoint the last render of `backupstrategy-controller` saw. The default objects gate only acted on missing objects, so backups kept going to the old host and nothing reported it. The gate now also forces a render when a copy is stale: - the bucket's `<bucket>-system` release, when the credentials Secret's endpoint differs from the COSI Secret it is rendered from; - `backupstrategy-controller`, when the endpoint an object was rendered with differs from the credentials Secret's. The chart stamps the rendered endpoint on the BSL and on every default Strategy as `backups.cozystack.io/endpoint`, because the endpoint itself sits at a different path in every kind. Objects without the annotation count as stale until a render stamps it; `MinForceInterval` bounds the forces. Stale credentials are forced before the BackupClass and the routed objects are read, so an absent BackupClass or an unreadable Strategy still fails the check but does not keep the old endpoint in the credentials Secret. A `BucketInfo` without an endpoint is not compared, since rendering from it would write an empty endpoint or fail the bucket release. Repairing missing objects keeps working while the credentials comparison is stale or fails, and a `systemSecretName` that is not derived from a COSI Secret (no `-credentials` suffix, or the raw COSI Secret itself) is not compared at all. RBAC is unchanged: the controller already holds cluster-wide `get` on Secrets. No new metric: nothing would consume it. What this does not do: the COSI Secret still holds the endpoint the `BucketAccess` was granted with, so after a host change the endpoint follows only once an operator deletes the `BucketAccess` and forces `bucket-cozy-backups` to recreate it (this issues new S3 keys). From there the gate replaces the remaining manual steps of the recovery described in the issue. Automatic re-grant is left for a separate issue, and the operator runbook goes to the website, hence `Refs` rather than `Fixes`. Known limitation: the credentials comparison runs only when `backupStorage.systemSecretName` is `bucket-<bucketName>-system-credentials`, the Secret the forced `bucket-<bucketName>-system` release renders. If an operator sets `bucketName` and `systemSecretName` inconsistently, the comparison is switched off rather than forcing a release that never rewrites the compared Secret, so a stale endpoint in that Secret is not repaired. Repairing a missing credentials Secret still forces the release derived from `bucketName`, as before this PR. Tested with unit tests for the gate (including the comparison normalisation shared with the chart helpers), helm-unittest for the annotation and the wiring, and a bats guard that every rendered Strategy CR and the BSL carry the annotation. On a live cluster, a server-side render of the chart produced exactly the annotation the gate derives from the real credentials Secret, so the steady state does not force anything. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: an operator runbook for changing `publishing.host` with platform backups enabled (re-grant the `BucketAccess`, then let the gate converge; restart ClickHouse pods that read the credentials from environment variables). Not filed yet, pending approval of the text. - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup strategies and storage locations now record their storage endpoint, enabling detection and repair of outdated endpoint settings after bucket access changes. * Bucket credentials and backup objects are refreshed independently when their endpoints are missing or out of date. * **Documentation** * Clarified when endpoint changes trigger updates and how suspended releases are handled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5f0da5a09a |
docs(mariadb): list the operator's Failed reason on the Complete condition
The reason list read as everything the operator writes on Complete, yet mariadb-operator also sets Complete=False with reason Failed when a run fails before or outside its Job. A reader taking the list as exhaustive would treat Failed as an unknown reason, or make it terminal, while the operator retries those failures. A table row pins that the driver keeps waiting on it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
69cea9ccbd |
fix(mariadb): keep the operator's message in deadline failures
The operator sets Complete=False with reason Failed and its own cause
in the message ("Error creating Job", "Backup not complete") when a
run fails before or outside its Job. Such a run waits out the
deadline, and the deadline failure kept only the status and reason,
so the BackupJob or RestoreJob ended without the one line that says
why.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
ee5e51ee6c |
fix(backups): keep forcing objects that still lack the endpoint annotation
An object without backups.cozystack.io/endpoint stopped counting as stale as soon as one forced render had been requested, whether or not that render succeeded. A failed upgrade left the objects unannotated and the gate silent until the controller restarted, which also hid the climbing force counter meant to report exactly that failure. Such an object now stays stale until a render stamps it; MinForceInterval already bounds the cost. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9f51b9db04 |
fix(mariadb): report the operator Job's own failure on JobFailed
The operator's JobFailed condition message is only "Failed", and with the operator's default restartPolicy OnFailure the Job controller deletes the pod once the backoff limit is spent, before the operator sets JobFailed. Pointing the reader at the Job's logs therefore sent them to logs that no longer exist. The Job, which the operator names after the Backup or Restore, keeps its Failed condition with the reason and message, so the BackupJob or RestoreJob now carries those instead; a Job that cannot be read is named in the message rather than dropped. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7e1fc7173c |
test(mariadb): cover the absent Complete condition and expired deadlines
Every first reconcile after the operator Backup or Restore is created sees no Complete condition, and the guards that keep that case from dereferencing nil could be dropped with the outcome tables staying green. The tables now also cover the deadline failure for a run that never reports a known reason, and read the job status back from the API, so a lost status write fails them instead of passing on the in-memory object. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
bd2b6c4b49 |
fix(backups): leave the credentials Secret alone when BucketInfo has no endpoint
A BucketInfo without an endpoint differs from every credentials Secret that has one, so the gate forced the bucket release on every throttle window. That render cannot converge: an empty endpoint would be written into the Secret every driver reads, and with the key absent the template's trimPrefix fails and parks the release in Failed. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ccac0e17e9 |
fix(mariadb): fail backup and restore jobs whose operator Job failed
mariadb-operator reports a failed backing Job as Complete=True with reason JobFailed, and Complete=True alone was read as success. A backup whose dump or upload failed therefore ended Succeeded and left a Ready Backup pointing at an archive that does not exist, and a scheduled Plan kept producing such Backups; a failed restore ended Succeeded too. Success now requires reason JobComplete, and JobFailed fails the job whatever the status. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
399ccd3f4a |
fix(backups): refresh stale credentials even when the routed objects are unreadable
The credentials endpoint comparison needs only the credentials Secret and the COSI Secret it is rendered from, yet its forced render was issued only after the BackupClass and every routed object had been read. A missing BackupClass or one unreadable Strategy therefore kept the old endpoint in the credentials Secret, which the projector hands to every driver, not just the one whose object failed. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ef7626a951 |
fix(backups): keep repairing missing default objects while credentials lag
The credentials comparison returned early when the credentials Secret was stale or the COSI Secret could not be read, so a missing default Strategy or BSL was no longer created on that tick, nor on any later one while the bucket release stayed suspended, kept failing, or the COSI Secret stayed unreadable. Repairing missing objects is what the gate did before the comparison existed, and it does not depend on the answer. Run the object check on every tick and force each release for what it renders, reporting a comparison error alongside instead of in place of the repair. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d0f47f7dd5 |
fix(backups): compare the credentials endpoint only for a Secret derived from COSI
backupStorage.systemSecretName may name a Secret the bucket chart does not render from a COSI Secret: the raw COSI Secret itself, which the projector accepts, or any Secret without the -credentials suffix. The COSI Secret name was then the source name unchanged, so the gate compared a Secret with itself and either reported it stale on every tick or failed the check, and stopped repairing missing default objects as well. Skip the comparison when there is no distinct COSI Secret or no endpoint key to compare. The gate compares the annotation with what the chart's endpoint helper renders, and the helper renders a present but blank endpoint as a bare https:// instead of falling back, so the gate has to compute the same value or its expectation and the chart disagree for that Secret. The gate's docs, metric help and log lines named only missing objects, so they did not cover a stale endpoint they now report. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c76080cc07 |
test(clickhouse): pin the archive name across a credentials wait
A backup's archive is named from the BackupJob's persisted StartedAt, and the restore of a Backup that records no archive picks one stamped no later than its takenAt. The credentials wait holds the Job after StartedAt is set and writes the job's status while it holds. Should a held reconcile restamp StartedAt, the Pod would write an archive the earlier reconciles never named; nothing caught that. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
86c541368d |
Merge branch 'main' into fix/clickhouse-backup-late-credentials
Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
477540b8f1 |
fix(backups): keep projecting credentials when forcePathStyle cannot be parsed
The credentials projection carries every backup driver's keys, but only the ClickHouse sidecar's file needs forcePathStyle parsed as a boolean. A value that failed the parse failed the whole projection as a malformed source Secret, a terminal reason pointing at the wrong object, so CNPG, MariaDB, etcd and the other drivers lost their backups over a parse that serves one file. The value is now left out of that file and the reason is logged at error level with the namespace; the rest of the projection proceeds. The chart refuses such a value at render time, so this covers an environment set around it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
62b4213784 |
fix(seaweedfs): keep bucket lifecycle enforcement off unless enabled
The daily walk that expires objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when none is given (walker_dispatcher.go and identityMatches, unchanged from 4.45 through 4.52). It does not re-check the object's age either, so a key overwritten between the walk's listing and the delete loses its new content. Backup buckets write unique keys and barely meet that window; a tenant bucket that rewrites a fixed key does. Turning enforcement on by default would also start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so the admin server and the worker now run only where lifecycle.enabled is set. On by default can return once the walk carries identity. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
970191dc8a |
fix(clickhouse): restore the archive a Backup was taken as (#5208)
## What this PR does A ClickHouse `RestoreJob` restored the newest archive under the app prefix, whichever `Backup` it named. Strategy pod generated the archive name with `date -u` and nothing was recorded on the `Backup`, so restore could only list the remote archives and take the last one. Restoring an older `Backup` brought back the latest data. Now the controller names the archive before it creates the backup Job (`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so retried pods write the same one), passes it to the pod as `.ArchiveName` and records it in the `Backup` `driverMetadata`. Restore takes exactly that archive and fails with its name when storage has no such archive. The name is recorded only when the Job's pod got it, so a Job left from the previous controller during an upgrade, or a custom strategy with its own names, leaves no name instead of a wrong one. A retried pod reuses the name, so it waits for an attempt the sidecar still runs (no 423) and asks for `resume`. A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after the schedule slot, so a `BackupJob` created again under a used name met the earlier run's `Backup` when its Job completed, took it as its result and left the archive it just wrote recorded nowhere. Now it fails if that `Backup` records another archive, and the message names both. A retry of the same run still gets its own `Backup` back. Old `Backup`s have no recorded name and stay restorable: the pod logs a `WARNING` and takes the newest archive stamped no later than `takenAt`, and fails when nothing qualifies. While the chart rolls out, previous controller pod can still render the new template and pass no `takenAt`, so a restore of a `Backup` without recorded name fails before it touches any table, and works when created again after the rollout. Fixes https://github.com/cozystack/cozystack/issues/5199 Tests are go unit tests for naming, recording, a reused `BackupJob` name and restore render, and `hack/altinity-backup-strategy.bats`, which runs the shipped script against a stubbed clickhouse-backup API (the restore tests go red on the old template). Checked on a live ClickHouse with two backups of 1500 and 2000 rows: restore of the older one by recorded name ends with 1500 rows, fallback for an unnamed older backup picks the same archive with the warning, a missing archive fails without touching the table, a retry over an existing local backup completes under the same name. The controller binary itself was not run on the cluster, only its rendered output in the pod. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backups now use a consistent archive name tied to the backup job, and completed backups retain that name for future restores. * Restores can select the recorded archive and backup timestamp. If no archive was recorded, the system selects the newest eligible archive for the application. * **Bug Fixes** * A restore now fails when its recorded archive is missing instead of silently selecting a different one. * Backup requests avoid starting a duplicate action when the matching action is already in progress. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
33608326f1 |
test(clickhouse): pin that an unreadable hold state keeps the job held
The credentials hold returns the error when it cannot read the strategy Job or the projected Secret, and both the backup and the restore path pass it on, so the reconcile retries instead of starting the Job. Nothing tested that: swallowing either error, in the hold or at either call site, left the suite green while letting the Job start before the credentials reached the sidecar, which is the failure the hold exists to prevent. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d7ae34dfc3 |
fix(backups): fail a ClickHouse BackupJob whose Backup name is taken
A Backup outlives the BackupJob that made it, so a BackupJob created again under a name already used meets the earlier run's Backup when its own Job completes. That happens by hand, and from a Plan reconciled again inside its starting deadline, since a Plan names each BackupJob after its schedule slot. The Altinity driver took the earlier Backup as its result: the BackupJob turned Succeeded with a BackupRef recording the earlier run's archive, the archive this run wrote was recorded on nothing, and restoring what the BackupJob reported brought back the older data. An existing Backup is now taken as this run's only when it records this run's archive, which is what an earlier reconcile of the same run leaves when its status write is lost. Any other fails the BackupJob, naming both archives, and leaves the earlier Backup untouched. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6345b939d2 |
fix(clickhouse): stop reporting a credentials wait once the strategy job runs
The hold sets Ready=False/WaitingForCredentials with a countdown, and nothing removed it once the Job was created, so a backup or restore that ran for minutes kept saying it was still waiting. The condition is dropped when the Job runs; a Ready condition set for another reason is left alone. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5f636ce624 |
fix(clickhouse): do not hold strategy jobs on a credentials stamp that cannot expire
The projector rewrites the credentials-updated-at stamp only when the projected content changes, so a stamp the hold cannot use stays put. An unreadable one failed every reconcile with no condition on the job, and one far in the future held each job again on every reconcile; in both cases every ClickHouse backup and restore in the namespace stopped for good. Such a stamp is now logged and ignored. A stamp only slightly ahead, as a skewed clock writes it, still waits, and that wait ends. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
38b00730ff |
fix(clickhouse): keep the clickhouse-backup default region when none is configured
The environment the sidecar used to read had no S3_REGION when the platform carried no region, which left clickhouse-backup on its own us-east-1. Writing an empty region into the config file replaces that default with an empty one, and requests are then signed for no region. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
27e0aa2880 |
docs(backups): say who applies a bucket lifecycle rule
The notes that send a tenant to a bucket lifecycle rule for a backup object Cozystack leaves behind read as if some policy were already in place. None is: the rule has to be set on the bucket, and it is applied only by an object store that implements lifecycle, which the bundled SeaweedFS now does once a day while lifecycle.enabled is on. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
cd91138f33 |
fix(backups): re-render the default backup objects when the S3 endpoint changes
The S3 endpoint the platform backups use is copied twice and neither copy is refreshed. The COSI sidecar writes BucketInfo once per BucketAccess, so the credentials Secret rendered from it keeps its endpoint, and the Velero BSL and the default Strategy CRs keep whatever endpoint the last render of backupstrategy-controller saw. The default objects gate only acted on objects that were missing, so after a publishing.host change every backup kept going to the old host with nothing reporting it. The gate now also forces a render when a copy is stale: the bucket's <bucket>-system release when the credentials Secret's endpoint differs from the COSI Secret it is rendered from, and backupstrategy-controller when the endpoint an object was rendered with differs from the credentials Secret's. That endpoint sits at a different path in every kind, in some only as a bare host, so the chart stamps the rendered value on each object as backups.cozystack.io/endpoint and the gate compares that. Objects rendered before the annotation existed count as stale until one forced render has gone through, which stamps it. The COSI Secret itself still holds the endpoint the BucketAccess was granted with; it follows a host change only once the BucketAccess is re-granted. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f5f50d2a77 |
fix(clickhouse): hold a strategy job until projected credentials reach the sidecar
The clickhouse-backup sidecar now reads the projected credentials from a mounted Secret, which kubelet rewrites only after its sync period, about 75 seconds after the change in practice. The first BackupJob projects the Secret and started the strategy Job a second later, so its attempts raced the volume update and the backup could still fail on a sidecar without credentials. The projector records when the projected content last changed, and the ClickHouse driver holds back creating a backup or restore Job until two minutes have passed since, with a WaitingForCredentials condition. An unchanged re-projection keeps the time, so only the first projection and a rotation cause a wait, and a Job already running is never held. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c9ce4c480f |
fix(clickhouse): let the backup sidecar pick up credentials projected after it started
With backup.useSystemBucket the clickhouse-backup sidecar took its S3 settings from optional secretKeyRefs on cozy-backups-creds. The platform projects that Secret into the namespace at the first BackupJob, usually after the ClickHouse pod started, and a container's environment is fixed at start, so the sidecar ran without credentials and every backup and restore failed until the pod was restarted by hand. Rotated keys stayed stale the same way. The projector now also writes the s3 section of a clickhouse-backup config into the Secret, and the chart mounts that key, optionally, as the sidecar's config file instead of setting S3 environment variables. clickhouse-backup reloads its config file in each API handler that reaches S3 and kubelet rewrites a mounted Secret in place, so projected and rotated credentials reach the running sidecar. The flat keys stay for the Redis strategy and for ClickHouse releases still rendered by the previous chart. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
59f0013b1f |
fix(backups): attribute a MariaDB dump only when no other run can own it
Every backup of a MariaDB application writes its dump under one shared prefix, so the probe took the dump whose file-name timestamp (the backup pod's clock) and LastModified (the object store's clock) both fall in the operator Backup's lifetime, with a minute of slack for skew. That keeps a neighbour's dump out only while this run's own dump is in the window. With the pod clock more than the slack behind, the own dump is named before the window opens, and a previous run that finished within the minute before this one was created becomes the single match: a 123-byte dump was reported as 666 bytes. A pod clock ahead or an object store clock behind does the same with the next run, and an own dump that is gone by the time it is looked up, as it can be for a restore long after the backup, leaves a neighbour as the single match with no skew at all. A dump is now attributed only when ownership is proven: - it is the one dump that either clock places in the window. A dump that one clock places there and the other does not may be this run's own with a skewed clock, so it leaves any other match unproven, the way a second dump in the window already did, and is not taken itself; - both of its clocks place it in the window; - neither clock places it in the window of another operator Backup that writes under the same bucket and prefix and has completed. This rejects a neighbour's dump when this run's own is missing, which the clocks alone cannot tell. Operator Backups still running are not counted as neighbours: their window has no end, and one stuck short of Complete (a Job whose pod never starts stays JobRunning for good) would otherwise keep every later run of the application from being sized. A dump such a run writes inside this run's window still competes with this run's own. Narrower slack does not prove ownership: a neighbour that finished within the narrower slack still matches, and a smaller skew pushes the own dump out. The Job's start and completion times are stamped by the control plane, like the operator Backup's own, so they narrow the window by seconds without removing the dependence on the two clocks a dump carries. What is left: a neighbour's dump is taken only when this run's own is missing from the listing or misplaced on both clocks and the neighbour's operator Backup no longer exists or has not completed. Every other unproven case leaves the size unset, as two runs that overlap or follow each other within the slack already did. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
560c44f71e |
fix(backups): read a MariaDB dump from where its operator Backup wrote it
The size probe took the bucket, endpoint, prefix, credentials and TLS settings from the strategy as it renders on the reconcile that creates the Backup, which is not necessarily what it rendered when the operator Backup was created: an existing operator Backup is reused untouched, and a backup can run for up to the driver's 30-minute deadline. An admin who edits the strategy in between sends the probe to the new location, where it measures the wrong prefix or nothing at all. The operator Backup's own spec.storage.s3 is where the dump went, so the probe reads that. It is also the only source a restore can use, since it targets that operator Backup long after the strategy may have changed. An operator Backup that does not store to S3 is refused before anything is listed. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |