mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
206 lines
12 KiB
YAML
206 lines
12 KiB
YAML
backupStrategyController:
|
|
image: "ghcr.io/cozystack/cozystack/backupstrategy-controller:v1.6.0@sha256:e8bf0521d1d1d8b90ebe66e9013010887033ba2653901799d57f2df47c9d879e"
|
|
# chBackupClientImage is rendered into the Altinity strategy Pod, which drives
|
|
# clickhouse-backup's HTTP API with curl + jq. rabbitmqBackupClientImage is
|
|
# rendered into the RabbitMQ strategy Pod, which calls the management API and
|
|
# moves the definitions export to S3 with curl --aws-sigv4. Both run the
|
|
# images/backup-http-client/ image (alpine + curl + jq + ca-certificates) and
|
|
# must hold the same ref: one repository pinned at two digests fails the
|
|
# release promotion (hack/image-pin-consistency.bats). They are separate keys
|
|
# so either strategy can move to its own image later.
|
|
# DO NOT hand-edit: `make image` stamps both.
|
|
chBackupClientImage: "ghcr.io/cozystack/cozystack/backup-http-client:v1.6.0"
|
|
rabbitmqBackupClientImage: "ghcr.io/cozystack/cozystack/backup-http-client:v1.6.0"
|
|
# redisBackupClientImage is rendered into the Redis strategy Pod. The
|
|
# spotahome operator ships no backup CRD, so the strategy runs a one-shot Job
|
|
# that dumps an RDB snapshot to S3 and, on restore, loads it into a throwaway
|
|
# redis to emit a faithful replay stream. It ships its own image (redis 8 +
|
|
# curl) built from images/redis-backup/; `make image` re-digests this value.
|
|
redisBackupClientImage: "ghcr.io/cozystack/cozystack/redis-backup:v1.6.0"
|
|
# redisBackupActiveDeadlineSeconds bounds one backup/restore Job so a run that
|
|
# wedges (e.g. a --rdb dump blocking on a stalled master) becomes a Failed Job
|
|
# the driver reports, rather than a Running BackupJob a cron Plan stacks behind.
|
|
# The driver promotes this to the Job's activeDeadlineSeconds, so it bounds the
|
|
# whole run across retries, not each Pod attempt separately.
|
|
redisBackupActiveDeadlineSeconds: 3600
|
|
# redisBackupResources sizes the Redis strategy Pod. Restore is the demanding
|
|
# side: the throwaway loader holds the whole dataset in memory and writes the
|
|
# RDB plus its AOF rewrite to ephemeral storage, so a large Redis needs memory
|
|
# and ephemeral-storage well above the backup's. Raise these for tenants with
|
|
# large instances; the defaults suit small-to-medium ones.
|
|
redisBackupResources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 64Mi
|
|
ephemeral-storage: 256Mi
|
|
limits:
|
|
cpu: "1"
|
|
memory: 1Gi
|
|
ephemeral-storage: 4Gi
|
|
# The Kafka metadata strategy carries no client image here: unlike the Altinity
|
|
# and RabbitMQ drivers, which need only curl (plus jq for Altinity), its driver
|
|
# runs the Kafka Admin-API CLI (kafka-topics / kafka-configs), which only the
|
|
# Strimzi broker image ships. So the controller resolves the image from
|
|
# the target Kafka broker at reconcile time (strategy-kafka-default.yaml renders
|
|
# the container image as the "{{ .ClientImage }}" placeholder), and records it on
|
|
# the Backup so cleanup can reuse it once the broker is gone. No pin to drift.
|
|
replicas: 2
|
|
debug: false
|
|
metrics:
|
|
enabled: true
|
|
bindAddress: ":8443"
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 64Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 128Mi
|
|
# backupStorage describes the platform-managed S3 bucket used by every
|
|
# default BackupClass strategy. The bucket lives in the backupStorage.namespace
|
|
# (tenant-root by default) and is provisioned through the apps.cozystack.io/Bucket CR.
|
|
# Tenants never see these coordinates directly — they reference cozy-default
|
|
# BackupClass and the backupstrategy-controller projects credentials per BackupJob.
|
|
backupStorage:
|
|
# provisionBucket controls whether to create the apps.cozystack.io/Bucket CR.
|
|
# Set to false when the cluster runs against an external S3 (no SeaweedFS)
|
|
# and the admin manages the credentials Secret manually.
|
|
provisionBucket: true
|
|
# The Bucket CR lives in tenant-root (a tenant namespace) because the
|
|
# apps.cozystack.io/Bucket chart's RBAC helper fails Helm render on any
|
|
# namespace that does not start with "tenant-" (see
|
|
# packages/library/cozy-lib/templates/_rbac.tpl). tenant-root is the
|
|
# root tenant present on every Cozystack install, so the system Bucket
|
|
# naturally lives there. Cross-namespace Secret read is granted to the
|
|
# backupstrategy-controller via cluster-scoped Secret RBAC.
|
|
namespace: tenant-root
|
|
bucketName: cozy-backups
|
|
# bucketNameOverride is an escape hatch for offline `helm template`
|
|
# renders (CI / local diffs). When set, the chart skips the live-cluster
|
|
# BucketClaim status lookup and uses this string as the S3 bucket name.
|
|
# Leave empty in production — the COSI driver assigns its own bucket
|
|
# name independent of .bucketName above, and the BucketClaim status
|
|
# is the only authoritative source.
|
|
bucketNameOverride: ""
|
|
# endpoint is the S3 endpoint URL shared by every default Strategy CR, the
|
|
# Velero BSL and the controller Deployment. For a provisioned bucket
|
|
# (provisionBucket: true, the default) it is NOT used verbatim: the
|
|
# "backupstrategy-controller.endpoint" helper derives the real endpoint from
|
|
# the COSI bucket's system credentials Secret (backupStorage.systemSecretName)
|
|
# and forces https://, because Cozystack's SeaweedFS runs with
|
|
# enableSecurity=true and serves TLS on :8333 behind the self-signed
|
|
# "SeaweedFS CA". The COSI-advertised endpoint is the external S3 ingress
|
|
# (ACME cert), which every consumer can verify — unlike the in-cluster
|
|
# service, whose CA is not plumbed to the backup operators and which the Etcd
|
|
# Strategy's S3 schema (no caCert field) cannot target at all.
|
|
#
|
|
# This value is the FALLBACK, used only when the derivation cannot run:
|
|
# - provisionBucket: false (external S3) — set it to your S3 endpoint,
|
|
# scheme included; it is used verbatim (the MariaDB/FDB strategies compute
|
|
# their secure flag from the scheme, so https:// enables TLS there — make
|
|
# the CA bundle reachable to those operators before pointing at a
|
|
# self-signed external endpoint);
|
|
# - offline `helm template` / unit renders and the pre-reconcile first
|
|
# install, where the Secret lookup returns nothing.
|
|
endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333
|
|
# region is re-projected into cozy-backups-creds on the next reconcile
|
|
# (1-minute tick for system namespaces, on-demand for tenant namespaces).
|
|
region: us-east-1
|
|
forcePathStyle: true
|
|
# systemSecretName — the human-friendly Secret produced by the Bucket
|
|
# application in the backupStorage.namespace (tenant-root by default). The Bucket release is
|
|
# wrapped with the bucket-rd "bucket-" prefix (see
|
|
# packages/system/bucket-rd/cozyrds/bucket.yaml), so the human-friendly
|
|
# Secret rendered by packages/system/bucket/templates/user-credentials.yaml
|
|
# is named <bucket-release>-<user>-credentials. The projector also
|
|
# tolerates the raw COSI Secret format (single BucketInfo JSON key) so a
|
|
# cluster that wires an external S3 Secret manually still works.
|
|
systemSecretName: bucket-cozy-backups-system-credentials
|
|
# endpointCASecretName — optional Secret (key ca.crt) in the app namespace
|
|
# that the Redis strategy Pod trusts when the S3 endpoint is https with a
|
|
# self-signed cert (e.g. SeaweedFS). Empty by default: the projected endpoint
|
|
# is always https and the platform bucket's ACME cert verifies against the
|
|
# image's system CA store unaided, so no CA override is needed. Set it only for
|
|
# a private-CA endpoint. When set, the Secret is required — a missing one fails
|
|
# the Pod loudly rather than silently skipping verification.
|
|
endpointCASecretName: ""
|
|
# insecureSkipTLSVerify disables S3 certificate verification (curl -k). Off by
|
|
# default and an explicit opt-in, never a fallback: leaving it false keeps an
|
|
# untrusted-cert endpoint failing closed instead of silently insecure. Prefer
|
|
# endpointCASecretName for a self-signed endpoint.
|
|
insecureSkipTLSVerify: false
|
|
# systemNamespaces — namespaces where the controller eagerly projects
|
|
# cozy-backups-creds so long-lived static references (Velero BSL, FDB
|
|
# operator deployments) keep finding the Secret without on-demand
|
|
# rendering. Tenants are projected lazily during BackupJob reconcile.
|
|
systemNamespaces:
|
|
- cozy-velero
|
|
# reconcileDefaultObjects enables the controller's DefaultObjectsGate.
|
|
#
|
|
# The Strategy CRs and the Velero BSL below are Helm-templated behind a
|
|
# `lookup` of the BucketClaim THIS chart creates, because the S3 bucket
|
|
# name is assigned by the COSI driver (bucket-<claim-UID>) and cannot be
|
|
# computed at render time. On a fresh install the lookup is empty and those
|
|
# templates render nothing — and helm-controller does not re-render a
|
|
# release whose chart and values did not change (drift detection is off on
|
|
# operator-generated HelmReleases), so the skip is PERMANENT. Clusters have
|
|
# run for months with only the BackupClass present and no Strategy CRs at
|
|
# all; recovery needed a hand-stamped reconcile.fluxcd.io/forceAt +
|
|
# requestedAt, because a plain reconcile request does not re-render.
|
|
#
|
|
# The systemSecretName Secret above sits one release earlier in the same
|
|
# trap: packages/system/bucket/templates/user-credentials.yaml renders it
|
|
# behind a `lookup` of the COSI Secret, and skips it just as permanently.
|
|
# Without it the projector has no source at all, so nothing downstream can
|
|
# resolve — and neither can this gate, which reads the bucket name from it.
|
|
#
|
|
# With this enabled the controller repairs both, in order: while the
|
|
# credentials Secret is missing it forces the bucket's <bucket>-system
|
|
# release; once the bucket name resolves it checks that every object
|
|
# cozy-default routes to exists and forces THIS release when any is
|
|
# missing. It repairs a stale S3 endpoint the same way: when the
|
|
# credentials Secret's endpoint differs from the COSI Secret it is rendered
|
|
# from it forces the bucket's <bucket>-system release (only while
|
|
# systemSecretName is bucket-<bucketName>-system-credentials, the one
|
|
# Secret that release renders), and when the
|
|
# backups.cozystack.io/endpoint annotation on the Velero BSL or a default
|
|
# Strategy differs from the credentials Secret's endpoint, or is absent, it
|
|
# forces THIS release. The COSI Secret itself keeps the endpoint it was granted with, so
|
|
# after a publishing.host change that endpoint follows only once the
|
|
# BucketAccess is re-granted. Both are throttled, independently, and a no-op
|
|
# in the steady state. A suspended release is skipped rather than
|
|
# re-stamped forever. The gate does NOT create the objects itself: their
|
|
# bodies are values-driven and Helm stays their single owner.
|
|
#
|
|
# Turn this off only if you manage the default Strategy CRs out of band and
|
|
# accept that a missed install-time render is never repaired. Watch
|
|
# cozystack_backup_default_objects_missing (paired with
|
|
# cozystack_backup_default_objects_check_errors_total, which marks the
|
|
# gauge stale) if you do.
|
|
reconcileDefaultObjects: true
|
|
# velero — BSL rendering. The cozy-default BackupStorageLocation is
|
|
# created in cozy-velero by templates/velero-bsl.yaml so endpoint/bucket/
|
|
# region come from the same backupStorage block used by Strategy CRs and
|
|
# the projector, eliminating drift.
|
|
#
|
|
# bslEnabled defaults to TRUE because cozystack.velero is a hard
|
|
# dependsOn for this PackageSource (see
|
|
# packages/core/platform/sources/backupstrategy-controller.yaml) — the
|
|
# Velero CRDs are guaranteed to exist by the time this chart installs.
|
|
# The BSL template additionally gates itself on a populated BucketClaim
|
|
# status via the bucketName helper, so on a bootstrap install the BSL
|
|
# materialises only after the platform Bucket reconciles AND the
|
|
# DefaultObjectsGate has forced a real Helm upgrade (an interval reconcile
|
|
# alone does not re-render). Setting this
|
|
# to false is an explicit opt-out for clusters that disable VM backups
|
|
# entirely and want to avoid the cluster-default BSL. In that case the
|
|
# cozy-default-velero-vminstance / -vmdisk Strategy CRs are gated off the
|
|
# same flag and are NOT rendered either, so the DefaultObjectsGate does
|
|
# not expect them (it would otherwise force a Helm upgrade forever against
|
|
# a render that can never produce them). VM BackupJobs against cozy-default
|
|
# still fail, because the BackupClass keeps routing VMInstance / VMDisk to
|
|
# those absent strategies.
|
|
velero:
|
|
bslEnabled: true
|
|
namespace: cozy-velero
|