Files
Andrey Kolkov 0d918e1f91 fix(backups): resolve Kafka backup client image from the target broker
The Kafka metadata strategy pinned kafkaBackupClientImage to an upstream
Strimzi tag with no producer: nothing stamped it and Renovate does not
manage helm-values, so it drifted from the broker version the operator
provisions. A dedicated small client image (the sibling curl-only drivers'
follow-up) is not viable here because the driver runs the Kafka Admin-API
CLI, which only the broker image ships.

Resolve the client image from a live broker pod of the target Strimzi Kafka
at reconcile time instead, so the CLI always matches the broker (any version
the operator picks, not only the chart default), and record it on the Backup
so cleanup can reuse it once the source cluster is gone. The chart no longer
pins the image; the strategy template renders it as the .ClientImage
placeholder the controller fills.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00

239 lines
14 KiB
YAML

backupStrategyController:
image: "ghcr.io/cozystack/cozystack/backupstrategy-controller:v1.6.0@sha256:e8bf0521d1d1d8b90ebe66e9013010887033ba2653901799d57f2df47c9d879e"
# chBackupClientImage is the image rendered into the Altinity strategy
# Pod: it drives clickhouse-backup's HTTP API via curl + jq.
#
# WORKAROUND — platform-migrations is the wrong image for this job and
# should be replaced by a dedicated one (see the TODO below). It is
# reused here only because it happens to bundle curl + jq, and this
# release could not absorb a new image. What that reuse actually costs:
# - the strategy script needs exactly curl, jq, sleep and date, yet
# this image is 349 MB carrying kubectl, helm, git, cozyhr and the
# etcd-migrate binary into a tenant-adjacent backup Pod;
# - its ENTRYPOINT is run-migrations.sh, so only the `command:`
# override in strategy-altinity-default.yaml stands between this Pod
# and running platform migrations with kubectl and helm on the PATH;
# - it couples the backup client to migrations: bumping
# ETCD_OPERATOR_VERSION or editing any migration script re-digests
# this image, forcing this pin to move for reasons that have nothing
# to do with backups. That coupling is what broke — see below.
#
# TODO: replace with a dedicated ch-backup-client image built by THIS
# package's own Makefile (alpine + curl + jq + ca-certificates measures
# 23 MB, 15x smaller). That also removes the cross-package stamp
# entirely, because the pin would then be produced by the package that
# owns it. The often-cited objection — "a second tag release CI must
# re-digest each cut" — is not the real cost: every other first-party
# image already pays it, automatically. The real defect was a pin with
# NO producer, which a package-owned image cannot reproduce.
#
# DO NOT hand-edit: packages/core/platform/Makefile stamps this key with
# the same ref it writes to .migrations.image there, so the two stay in
# lockstep by construction. They previously did not — this key had no
# producer at all and froze at v1.4.0-rc.2 while the other advanced to
# v1.5.0. Two digests under one repository means a promotion tries to
# retag both to the same stable tag, and the write-once guard in
# hack/promote-retag.sh fails the release. hack/image-pin-consistency.bats
# asserts no repository is pinned at more than one digest.
chBackupClientImage: "ghcr.io/cozystack/cozystack/platform-migrations:v1.6.0@sha256:6777042e3e9c6bad76e7d96fb64baa115061975f39061cb3ede84e21d0e2213f"
# rabbitmqBackupClientImage is the image rendered into the RabbitMQ Job
# strategy (strategy-rabbitmq-default.yaml): its script needs only curl +
# ca-certificates to call the RabbitMQ management API and push the
# definitions export to S3 (curl --aws-sigv4). It is a distinct key from
# chBackupClientImage so the RabbitMQ strategy carries its own upgrade
# cadence, but it reuses the same platform-migrations image for now — hence
# it is stamped in lockstep with the two keys above (packages/core/platform/
# Makefile), so the repository stays pinned at a single digest
# (hack/image-pin-consistency.bats). Same TODO applies: replace with a
# dedicated alpine + curl + ca-certificates image built by this package's
# Makefile, which also lets this pin move on its own.
# DO NOT hand-edit: stamped by packages/core/platform/Makefile.
rabbitmqBackupClientImage: "ghcr.io/cozystack/cozystack/platform-migrations:v1.6.0@sha256:6777042e3e9c6bad76e7d96fb64baa115061975f39061cb3ede84e21d0e2213f"
# redisBackupClientImage is rendered into the Redis strategy Pod. The
# spotahome operator ships no backup CRD, so the strategy runs a one-shot Job
# that dumps an RDB snapshot to S3 and, on restore, loads it into a throwaway
# redis to emit a faithful replay stream. It ships its own image (redis 8 +
# curl) built from images/redis-backup/; `make image` re-digests this value.
redisBackupClientImage: "ghcr.io/cozystack/cozystack/redis-backup:v1.6.0"
# redisBackupActiveDeadlineSeconds bounds one backup/restore Job so a run that
# wedges (e.g. a --rdb dump blocking on a stalled master) becomes a Failed Job
# the driver reports, rather than a Running BackupJob a cron Plan stacks behind.
# The driver promotes this to the Job's activeDeadlineSeconds, so it bounds the
# whole run across retries, not each Pod attempt separately.
redisBackupActiveDeadlineSeconds: 3600
# redisBackupResources sizes the Redis strategy Pod. Restore is the demanding
# side: the throwaway loader holds the whole dataset in memory and writes the
# RDB plus its AOF rewrite to ephemeral storage, so a large Redis needs memory
# and ephemeral-storage well above the backup's. Raise these for tenants with
# large instances; the defaults suit small-to-medium ones.
redisBackupResources:
requests:
cpu: 10m
memory: 64Mi
ephemeral-storage: 256Mi
limits:
cpu: "1"
memory: 1Gi
ephemeral-storage: 4Gi
# The Kafka metadata strategy carries no client image here: unlike the Altinity
# and RabbitMQ drivers (curl-only, so a shared platform-migrations image works),
# its driver runs the Kafka Admin-API CLI (kafka-topics / kafka-configs), which
# only the Strimzi broker image ships. So the controller resolves the image from
# the target Kafka broker at reconcile time (strategy-kafka-default.yaml renders
# the container image as the "{{ .ClientImage }}" placeholder), and records it on
# the Backup so cleanup can reuse it once the broker is gone. No pin to drift.
replicas: 2
debug: false
metrics:
enabled: true
bindAddress: ":8443"
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
# backupStorage describes the platform-managed S3 bucket used by every
# default BackupClass strategy. The bucket lives in the backupStorage.namespace
# (tenant-root by default) and is provisioned through the apps.cozystack.io/Bucket CR.
# Tenants never see these coordinates directly — they reference cozy-default
# BackupClass and the backupstrategy-controller projects credentials per BackupJob.
backupStorage:
# provisionBucket controls whether to create the apps.cozystack.io/Bucket CR.
# Set to false when the cluster runs against an external S3 (no SeaweedFS)
# and the admin manages the credentials Secret manually.
provisionBucket: true
# The Bucket CR lives in tenant-root (a tenant namespace) because the
# apps.cozystack.io/Bucket chart's RBAC helper fails Helm render on any
# namespace that does not start with "tenant-" (see
# packages/library/cozy-lib/templates/_rbac.tpl). tenant-root is the
# root tenant present on every Cozystack install, so the system Bucket
# naturally lives there. Cross-namespace Secret read is granted to the
# backupstrategy-controller via cluster-scoped Secret RBAC.
namespace: tenant-root
bucketName: cozy-backups
# bucketNameOverride is an escape hatch for offline `helm template`
# renders (CI / local diffs). When set, the chart skips the live-cluster
# BucketClaim status lookup and uses this string as the S3 bucket name.
# Leave empty in production — the COSI driver assigns its own bucket
# name independent of .bucketName above, and the BucketClaim status
# is the only authoritative source.
bucketNameOverride: ""
# endpoint is the S3 endpoint URL shared by every default Strategy CR, the
# Velero BSL and the controller Deployment. For a provisioned bucket
# (provisionBucket: true, the default) it is NOT used verbatim: the
# "backupstrategy-controller.endpoint" helper derives the real endpoint from
# the COSI bucket's system credentials Secret (backupStorage.systemSecretName)
# and forces https://, because Cozystack's SeaweedFS runs with
# enableSecurity=true and serves TLS on :8333 behind the self-signed
# "SeaweedFS CA". The COSI-advertised endpoint is the external S3 ingress
# (ACME cert), which every consumer can verify — unlike the in-cluster
# service, whose CA is not plumbed to the backup operators and which the Etcd
# Strategy's S3 schema (no caCert field) cannot target at all.
#
# This value is the FALLBACK, used only when the derivation cannot run:
# - provisionBucket: false (external S3) — set it to your S3 endpoint,
# scheme included; it is used verbatim (the MariaDB/FDB strategies compute
# their secure flag from the scheme, so https:// enables TLS there — make
# the CA bundle reachable to those operators before pointing at a
# self-signed external endpoint);
# - offline `helm template` / unit renders and the pre-reconcile first
# install, where the Secret lookup returns nothing.
endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333
# region is re-projected into cozy-backups-creds on the next reconcile
# (1-minute tick for system namespaces, on-demand for tenant namespaces).
# However, charts that consume the region via secretKeyRef in env vars —
# the chart-emitted ClickHouse sidecar on `useSystemBucket: true` is the
# only such consumer today — only pick up the new value on Pod restart.
# Plan a rollout of those Pods when you change this value.
region: us-east-1
forcePathStyle: true
# systemSecretName — the human-friendly Secret produced by the Bucket
# application in the backupStorage.namespace (tenant-root by default). The Bucket release is
# wrapped with the bucket-rd "bucket-" prefix (see
# packages/system/bucket-rd/cozyrds/bucket.yaml), so the human-friendly
# Secret rendered by packages/system/bucket/templates/user-credentials.yaml
# is named <bucket-release>-<user>-credentials. The projector also
# tolerates the raw COSI Secret format (single BucketInfo JSON key) so a
# cluster that wires an external S3 Secret manually still works.
systemSecretName: bucket-cozy-backups-system-credentials
# endpointCASecretName — optional Secret (key ca.crt) in the app namespace
# that the Redis strategy Pod trusts when the S3 endpoint is https with a
# self-signed cert (e.g. SeaweedFS). Empty by default: the projected endpoint
# is always https and the platform bucket's ACME cert verifies against the
# image's system CA store unaided, so no CA override is needed. Set it only for
# a private-CA endpoint. When set, the Secret is required — a missing one fails
# the Pod loudly rather than silently skipping verification.
endpointCASecretName: ""
# insecureSkipTLSVerify disables S3 certificate verification (curl -k). Off by
# default and an explicit opt-in, never a fallback: leaving it false keeps an
# untrusted-cert endpoint failing closed instead of silently insecure. Prefer
# endpointCASecretName for a self-signed endpoint.
insecureSkipTLSVerify: false
# systemNamespaces — namespaces where the controller eagerly projects
# cozy-backups-creds so long-lived static references (Velero BSL, FDB
# operator deployments) keep finding the Secret without on-demand
# rendering. Tenants are projected lazily during BackupJob reconcile.
systemNamespaces:
- cozy-velero
# reconcileDefaultObjects enables the controller's DefaultObjectsGate.
#
# The Strategy CRs and the Velero BSL below are Helm-templated behind a
# `lookup` of the BucketClaim THIS chart creates, because the S3 bucket
# name is assigned by the COSI driver (bucket-<claim-UID>) and cannot be
# computed at render time. On a fresh install the lookup is empty and those
# templates render nothing — and helm-controller does not re-render a
# release whose chart and values did not change (drift detection is off on
# operator-generated HelmReleases), so the skip is PERMANENT. Clusters have
# run for months with only the BackupClass present and no Strategy CRs at
# all; recovery needed a hand-stamped reconcile.fluxcd.io/forceAt +
# requestedAt, because a plain reconcile request does not re-render.
#
# The systemSecretName Secret above sits one release earlier in the same
# trap: packages/system/bucket/templates/user-credentials.yaml renders it
# behind a `lookup` of the COSI Secret, and skips it just as permanently.
# Without it the projector has no source at all, so nothing downstream can
# resolve — and neither can this gate, which reads the bucket name from it.
#
# With this enabled the controller repairs both, in order: while the
# credentials Secret is missing it forces the bucket's <bucket>-system
# release; once the bucket name resolves it checks that every object
# cozy-default routes to exists and forces THIS release when any is
# missing. Both are throttled, independently, and a no-op in the steady
# state. A suspended release is skipped rather than re-stamped forever. The
# gate does NOT create the objects itself: their bodies are values-driven
# and Helm stays their single owner.
#
# Turn this off only if you manage the default Strategy CRs out of band and
# accept that a missed install-time render is never repaired. Watch
# cozystack_backup_default_objects_missing (paired with
# cozystack_backup_default_objects_check_errors_total, which marks the
# gauge stale) if you do.
reconcileDefaultObjects: true
# velero — BSL rendering. The cozy-default BackupStorageLocation is
# created in cozy-velero by templates/velero-bsl.yaml so endpoint/bucket/
# region come from the same backupStorage block used by Strategy CRs and
# the projector, eliminating drift.
#
# bslEnabled defaults to TRUE because cozystack.velero is a hard
# dependsOn for this PackageSource (see
# packages/core/platform/sources/backupstrategy-controller.yaml) — the
# Velero CRDs are guaranteed to exist by the time this chart installs.
# The BSL template additionally gates itself on a populated BucketClaim
# status via the bucketName helper, so on a bootstrap install the BSL
# materialises only after the platform Bucket reconciles AND the
# DefaultObjectsGate has forced a real Helm upgrade (an interval reconcile
# alone does not re-render). Setting this
# to false is an explicit opt-out for clusters that disable VM backups
# entirely and want to avoid the cluster-default BSL. In that case the
# cozy-default-velero-vminstance / -vmdisk Strategy CRs are gated off the
# same flag and are NOT rendered either, so the DefaultObjectsGate does
# not expect them (it would otherwise force a Helm upgrade forever against
# a render that can never produce them). VM BackupJobs against cozy-default
# still fail, because the BackupClass keeps routing VMInstance / VMDisk to
# those absent strategies.
velero:
bslEnabled: true
namespace: cozy-velero