Addresses the four findings on #3524.
**The `fail` in the bucket chart (MAJOR).** The <bucket>-<user>-credentials
Secret was made to `fail` the render while its COSI lookup was empty, so the
release would retry and converge instead of skipping the Secret permanently.
Helm cannot render a partial set, so that fail aborts the WHOLE
<bucket>-system release: the other users' Secrets, and the bucket UI
Deployment/Service/Ingress/HTTPRoute with them. One declared user whose
BucketAccess never provisions takes down every other user of that bucket,
and on an already-installed release parks it in Failed, blocking every later
upgrade.
**No per-bucket escape (MAJOR).** Rather than add one, remove the need for
it. The gate already forces the release that renders the lookup-gated
Strategy CRs; the credentials Secret is the same trap one release earlier,
and the gate reads that Secret to resolve the bucket name in the first
place. So while it is absent, or carries no bucket name, force the bucket's
<bucket>-system release instead of doing nothing. That cannot deadlock its
own precondition: the BucketClaim and the BucketAccess whose COSI Secret the
lookup reads are rendered unconditionally by the parent release, not by the
one being forced.
The two releases are throttled independently — they are forced in sequence
on a bootstrap, so a shared timestamp would delay the second by a full
MinForceInterval for no reason. Coordinates come from the chart under
provisionBucket, and are omitted on external S3 where the Secret is
admin-managed and no release renders it. No RBAC delta: patch on
helmreleases is already cluster-scoped.
The chart goes back to a partial render, with the comment now explaining why
the skip is not self-healing and why failing is the wrong lever, and a test
that renders the UI with every user unresolvable so a reintroduced `fail`
fails the suite. Tenant buckets keep their pre-existing behaviour;
generalising the repair to every bucket needs a controller that owns them
and is tracked separately.
**Stale gauge (MINOR).** cozystack_backup_default_objects_missing was only
written on the happy path, so the state the gate exists to catch reported 0
and the documented alert never fired. It is now written on the unresolved
path too — the credentials Secret counts as one of the objects the default
backups depend on, which is what it always was. On an API error the gauge is
deliberately left alone rather than flapping, so a new
cozystack_backup_default_objects_check_errors_total marks it stale, and the
doc pairs the two instead of overstating the gauge.
**Suspended releases (MINOR).** forceHelmRelease now does a point Get and
skips a release with spec.suspend: true. helm-controller ignores forceAt and
requestedAt while suspended (`cozyhr suspend` sets exactly that), so the gate
was re-stamping every MinForceInterval for the whole suspension and climbing
the force counter — which the runbook attributes to a render that is not
producing the objects, the wrong diagnosis. The skip is logged and not
counted, so a climbing counter keeps its documented meaning.
Reported-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-By: Claude <noreply@anthropic.com>
(cherry picked from commit f859b710a4)
29 KiB
Backup Classes
Cozystack ships a single platform-managed BackupClass named cozy-default. It is provisioned automatically when the backupstrategy-controller package is installed and references the system-managed bucket provisioned through the apps.cozystack.io/Bucket CR cozy-backups in the tenant-root namespace (the real S3 bucket name is the COSI-assigned one from BucketClaim.status.bucketName).
Tenants reference cozy-default from BackupJob, Plan, and RestoreJob resources — they do not supply S3 credentials, endpoints, or paths. The platform projects the system-managed credentials Secret into the tenant namespace per BackupJob (or, for long-lived references like Velero's BackupStorageLocation, into a fixed list of system namespaces on a periodic tick), and the default strategy templates encode <namespace>/<application> into every S3 path so two tenants with the same application name never collide.
Supported applications
Bound by cozy-default (work out-of-the-box)
| Application Kind | Driver | Strategy CR |
|---|---|---|
apps.cozystack.io/Postgres |
CloudNativePG (barman) | strategy.backups.cozystack.io/CNPG cozy-default-cnpg |
apps.cozystack.io/MariaDB |
mariadb-operator dump | strategy.backups.cozystack.io/MariaDB cozy-default-mariadb |
apps.cozystack.io/ClickHouse |
Altinity clickhouse-backup sidecar |
strategy.backups.cozystack.io/Altinity cozy-default-altinity |
apps.cozystack.io/Etcd |
etcd-operator snapshot | strategy.backups.cozystack.io/Etcd cozy-default-etcd |
apps.cozystack.io/VMInstance |
Velero + kubevirt-velero-plugin | strategy.backups.cozystack.io/Velero cozy-default-velero-vminstance |
apps.cozystack.io/VMDisk |
Velero | strategy.backups.cozystack.io/Velero cozy-default-velero-vmdisk |
Shipped but NOT bound (admin opt-in required)
| Application Kind | Driver | Strategy CR |
|---|---|---|
apps.cozystack.io/FoundationDB |
FoundationDB operator backup_agent | strategy.backups.cozystack.io/FoundationDB cozy-default-foundationdb |
The FoundationDB strategy CR is rendered by the chart so admins can reference it from a custom BackupClass once the operator-side plumbing (mounting cozy-backups-creds into the cozy-foundationdb-operator Deployment) is wired manually. See "FoundationDB caveat" below.
Endpoint format per driver
Different operators expect different endpoint shapes; the strategy templates rendered by backupstrategy-controller resolve one S3 endpoint (via the backupstrategy-controller.endpoint helper) and adapt it to each consumer's contract. For a provisioned bucket (provisionBucket: true, the default) the endpoint is derived from the COSI bucket's system-credentials Secret (backupStorage.systemSecretName) and forced to https:// — the external S3 ingress with an ACME cert, the only endpoint the backup operators can verify. SeaweedFS's in-cluster S3 serves TLS on :8333 behind the self-signed "SeaweedFS CA", and the Etcd Strategy S3 schema has no caCert field, so the in-cluster endpoint cannot be targeted directly. The backupStorage.endpoint chart value (a full URL like http://seaweedfs-s3.tenant-root.svc:8333) is the fallback, used for external S3 (provisionBucket: false) and for offline helm template/pre-reconcile renders where the Secret lookup returns nothing. The resolved endpoint is adapted per consumer:
| Driver | Strategy template field | Form |
|---|---|---|
| CNPG (Postgres) | barmanObjectStore.endpointURL |
full URL (scheme preserved) |
| Etcd | destination.s3.endpoint |
full URL (scheme preserved) |
| MariaDB | storage.s3.endpoint |
bare host:port (scheme stripped); tls.enabled derived from the scheme |
| FoundationDB | blobStoreConfiguration.accountName + urlParameters.secure_connection |
bare host:port + derived secure flag |
| Velero | BackupStorageLocation.spec.config.s3Url |
full URL (scheme preserved) |
| ClickHouse sidecar | S3_ENDPOINT env |
bare host:port (from projected Secret) |
The projected cozy-backups-creds.endpoint key is stripped of scheme so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL receive the resolved endpoint described above — derived from the COSI system Secret (forced https://) for a provisioned bucket, or the backupStorage.endpoint fallback for external S3.
VM-driven (Velero) backups land in the same cozy-backups bucket under the velero/ prefix. A BackupStorageLocation named cozy-default is shipped by the backupstrategy-controller chart (packages/system/backupstrategy-controller/templates/velero-bsl.yaml) so endpoint/bucket/region come from the same backupStorage values block used by Strategy CRs and the projector.
FoundationDB caveat
The strategy CR cozy-default-foundationdb is shipped, but it is not bound by cozy-default yet. Restore runs fdbrestore from inside the cozy-foundationdb-operator Deployment, which does not yet mount cozy-backups-creds. Until the operator deployment is updated to mount the projected Secret, FDB platform-default restore silently fails — admins who need it today should keep using a per-app Bucket plus a custom BackupClass, or wire the credentials file into the operator deployment themselves.
Cleanup gotcha (zombie backup_agent). Unlike CNPG/MariaDB/Altinity (one-shot operator-side Backup CRs), the FoundationDB driver creates an apps.foundationdb.org/FoundationDBBackup CR that drives a long-lived backup_agent Deployment streaming continuously to S3. Deleting a Cozystack Backup (e.g. via retention sweeping) does NOT stop that Deployment — the agent keeps writing until the next BackupJob's stopOtherFoundationDBBackups call swaps it out, until an admin invokes examples/backups/foundationdb/cleanup.sh, or until the operator-side CR is deleted by hand. If a tenant deletes their last Cozystack Backup and never submits another BackupJob, the agent pods will continue running indefinitely and accumulate S3 PUTs. This is intentional today (the driver has no RBAC verb to stop the operator-side CR on Cozystack-Backup deletion) but admins should be aware of it.
ClickHouse: opt-in to the system bucket
The clickhouse-backup sidecar runs inside the ClickHouse Pod itself, so the Helm chart is what wires its S3 credentials. Existing tenants on the legacy backup.s3* values continue to work unchanged. To switch a release onto the platform bucket, set:
backup:
enabled: true
useSystemBucket: true
When useSystemBucket: true:
- The chart-emitted
<release>-backup-s3Secret is no longer rendered. - The sidecar consumes
cozy-backups-creds(projected by the platform). S3_PATHis set to<namespace>/<release>so two tenants with the same ClickHouse release name never share a prefix.
s3Region, s3Bucket, endpoint, s3AccessKey, s3SecretKey, and s3CredentialsSecret are ignored in this mode.
Inspecting the defaults
kubectl get backupclasses
kubectl get backupclass cozy-default -o yaml
kubectl -n tenant-root get bucket cozy-backups
kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
kubectl -n cozy-velero get backupstoragelocation cozy-default
The bucket lives in tenant-root and is provisioned through the apps.cozystack.io/Bucket CR. The system-managed credentials Secret never leaves that namespace. The backupstrategy-controller projects a copy under the name cozy-backups-creds into a tenant namespace right before each BackupJob or RestoreJob runs, and refreshes the same Secret in cozy-velero (and any other namespace listed in backupStorage.systemNamespaces) on a 1-minute tick. The projected Secret carries multiple key formats so each driver finds what it needs in one place:
| Key | Consumer |
|---|---|
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY |
CNPG, MariaDB, Etcd |
accessKey / secretKey (plus bucketName, endpoint, region) |
ClickHouse sidecar |
cloud |
Velero (AWS credentials file format) |
blob_credentials.json |
FoundationDB backup_agent |
Bootstrap window
On a fresh-cluster install, the Velero BackupStorageLocation cozy-default is rendered before the credentials projector has had a chance to copy cozy-backups-creds into cozy-velero. The BSL reports Unavailable until the projector's first synchronous round completes (which runs as soon as the backupstrategy-controller acquires leadership — in practice moments after the Pod becomes Ready, typically tens of seconds after helm install returns, not minutes). Velero rejects new Backup AND Restore requests against storageLocation: cozy-default during that window. Plan VM backup automation accordingly, or wait for the BSL to become ready before submitting backups: kubectl -n cozy-velero wait backupstoragelocation cozy-default --for=jsonpath='{.status.phase}'=Available --timeout=5m.
Note on controller restarts. The BSL flickers Unavailable on every backupstrategy-controller pod restart while the projector replays its first synchronous round. The window is short (single-digit seconds) but operators who alert on BSL availability should suppress alerts during the controller's kube_pod_container_status_restarts_total{container=backupstrategy-controller} events or use a longer evaluation window than the projector tick (60s).
Cozy-default Bucket bootstrap
cozy-default ships an apps.cozystack.io/Bucket cozy-backups CR in tenant-root, which the bucket-application chart turns into a BucketClaim; the COSI driver then assigns the real S3 bucket name (bucket-<claim-UID>, so it cannot be computed in advance) and writes it to the BucketClaim's .status.bucketName. The strategy templates and the Velero BSL all read that real bucket name (Helm lookup against the BucketClaim). On a fresh install the BucketClaim takes a reconcile cycle to populate its status — until it does, the strategy templates render empty and only the Bucket CR + BackupClass are present in the cluster.
That skip does not repair itself on a reconcile. helm-controller re-renders a release only when its chart or values change; the interval reconcile is a no-op for a healthy release, and drift detection is off on operator-generated HelmReleases. A cluster that lost the race at install time therefore keeps BackupClass cozy-default with no Strategy CRs and no Velero BSL indefinitely — which also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration.
The same trap sits one release earlier, on the Secret everything else depends on. bucket-cozy-backups-system-credentials is rendered by the bucket-cozy-backups-system release behind a lookup of the COSI Secret, and is skipped just as permanently when that lookup is empty. Without it the credentials projector has no source at all, so no strategy, no Velero and no v1.6.0 etcd migration can resolve — and the gate cannot even determine the bucket name, which it reads from that Secret.
Convergence is driven instead by the controller's default-objects gate (backupStorage.reconcileDefaultObjects, on by default), which repairs both, in order. While the credentials Secret is absent or carries no bucket name, the gate stamps reconcile.fluxcd.io/forceAt + requestedAt on the bucket's bucket-<name>-system HelmRelease. Once the bucket name resolves it checks that every object cozy-default routes to exists and stamps the same pair on the backupstrategy-controller HelmRelease, forcing the real Helm upgrade that re-runs the lookups. The two are throttled independently (one force per 5 minutes each), so the second is not delayed by the first. Expect the objects within a minute or two of the bucket becoming ready.
The gate does not force a suspended HelmRelease — helm-controller ignores both annotations while spec.suspend is true, so it logs the skip and leaves the force counter alone. A release left suspended (cozyhr suspend) is therefore never repaired until it is resumed.
Watch it with:
kubectl -n cozy-backup-controller logs deploy/backupstrategy-controller | grep default-objects-gate
Manual recovery on an affected cluster
Only needed on a cluster running a version without the gate (or with reconcileDefaultObjects: false). A plain flux reconcile helmrelease does nothing here — it does not re-render. You need a forced upgrade, and both annotations: forceAt is what makes helm-controller run a real Helm upgrade, and it is only honoured together with requestedAt.
First confirm the bucket name is actually resolvable — forcing before that just re-runs the same empty lookup:
kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}'
Then force the two releases, in this order:
# 1. The credentials Secret. It is rendered by the -system release, not by
# bucket-cozy-backups — easy to miss, and the projector (hence every
# strategy and Velero) has no source without it.
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
# 2. The Strategy CRs and the Velero BSL.
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
kubectl get $(kubectl get crd -o name | grep strategy.backups.cozystack.io) 2>/dev/null
kubectl -n cozy-velero get backupstoragelocation cozy-default
The race is nested: step 2's lookup resolves from the BucketClaim status, but the projector — and the v1.6.0 etcd migration — read the Secret from step 1, so a cluster missing both needs both.
Observability
The credentials projector emits two Prometheus counters labelled by namespace (and reason for failures):
cozystack_backup_credentials_projection_successes_totalcozystack_backup_credentials_projection_failures_total
Alert on rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0 or absent_over_time(cozystack_backup_credentials_projection_successes_total[10m]) to catch a stale BSL credential or a malformed source Secret without log scraping.
The default-objects gate emits three more:
cozystack_backup_default_objects_missing{backupclass="cozy-default"}— how many of the objects the platform default backups depend on are absent: the credentials Secret, the Strategy CRscozy-defaultroutes to, and the Velero BSL. This is the alert that would have caught the missing Strategy CRs: it is non-zero whatever the HelmRelease'sReadycondition says. Alert onmin_over_time(cozystack_backup_default_objects_missing[15m]) > 0.cozystack_backup_default_objects_check_errors_total{backupclass="cozy-default"}— checks that could not reach a conclusion (an API error reading the source Secret, the BackupClass, or one of the routed objects). The gauge above is deliberately not written on those ticks, so that it does not flap on a transient API error — which means that while this counter climbs, the gauge is stale and a0on it proves nothing. Pair the two:min_over_time(cozystack_backup_default_objects_missing[15m]) > 0 or rate(cozystack_backup_default_objects_check_errors_total[15m]) > 0.cozystack_backup_default_objects_force_reconciles_total{namespace,name}— forced Helm upgrades issued, labelled by the release forced (this chart's own, or the bucket's-systemrelease). A counter that keeps climbing means the forced render is not producing the objects (a missing CRD, for instance), which is a different problem from the install-time race. A suspended release is skipped before the patch and is not counted here, so a paused release cannot masquerade as a render that keeps failing — look for theskipped forcing a suspended HelmReleaselog line instead.
Admin overrides for cozy-default
cozy-default is rendered by the backupstrategy-controller chart and owned by Flux's helm-controller. Direct kubectl edit backupclass cozy-default is overwritten on the next helm reconcile — the same applies to its companion strategy.backups.cozystack.io/* CRs (cozy-default-cnpg, cozy-default-etcd, cozy-default-mariadb, cozy-default-altinity, cozy-default-foundationdb, the two cozy-default-velero-*). The supported override path is the backupStorage block on the platform component of the cozystack.cozystack-platform Package CR:
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
name: cozystack.cozystack-platform
spec:
components:
platform:
values:
backupStorage:
provisionBucket: true # default; set false for external S3
bucketName: cozy-backups # apps.cozystack.io/Bucket release name
endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333
region: us-east-1
forcePathStyle: true
systemSecretName: bucket-cozy-backups-system-credentials
systemNamespaces:
- cozy-velero
The platform chart forwards this block into the child Package cozystack.backupstrategy-controller as components.backupstrategy-controller.values.backupStorage (packages/core/platform/templates/bundles/system.yaml), from where the cozystack operator merges it into the backupstrategy-controller HelmRelease over the chart defaults. Two paths that look plausible do not work: spec.components.backupstrategy-controller on the cozystack.cozystack-platform Package is silently ignored (the only component under that PackageSource is platform), and patching the child Package cozystack.backupstrategy-controller directly is reverted whenever the platform helm-reconcile re-renders it.
| Knob | Effect |
|---|---|
provisionBucket |
Toggle creation of the in-cluster apps.cozystack.io/Bucket CR. Set false for external S3 (see Disabling the platform-managed bucket). |
bucketName |
Two modes. With provisionBucket: true (default): K8s name of the Bucket CR + lookup key for the COSI BucketClaim — the actual S3 bucket name is the COSI-assigned UUID, surfaced through BucketClaim.status.bucketName. With provisionBucket: false: taken verbatim as the real S3 bucket name and baked into every strategy CR + the Velero BSL. |
namespace |
Namespace the Bucket CR (and its system-credentials Secret) lives in — tenant-root by default. Must be a tenant namespace (tenant-*): the Bucket chart's RBAC helper fails the Helm render for any other prefix. |
bucketNameOverride |
Escape hatch for offline helm template renders — bypasses the live-cluster BucketClaim lookup. Leave empty in production. |
endpoint |
Fallback S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced https://) instead; this value is used only for external S3 (provisionBucket: false) and offline renders. For external S3, switching it to https:// enables TLS in the MariaDB/FoundationDB strategies — ensure the CA bundle is reachable to the relevant operator/driver Pods first. |
region |
Re-projected into cozy-backups-creds on the next reconcile. Pod-restart required for chart-emitted clients consuming the region via env (ClickHouse sidecar today). |
forcePathStyle |
Path-style addressing; SeaweedFS S3 requires it, AWS S3 typically doesn't. |
systemSecretName |
Name of the human-friendly Secret produced by the Bucket app (or pre-created manually for external S3). The projector also accepts the raw COSI Secret format. |
systemNamespaces |
Namespaces where the controller eagerly projects cozy-backups-creds (Velero BSL, FDB operator). Tenants are projected lazily during BackupJob reconcile. |
When the override needs to go beyond storage coordinates — different retention, different driver→Kind binding, multi-region split — create a sibling BackupClass with a unique name (anything but cozy-default). Sibling BackupClasses live outside the chart, are admin-owned, and Flux will not touch them. Tenants opt in by setting backupClassName: <your-class> on their BackupJobs.
Tuning via a custom BackupClass
The defaults aim at a reasonable middle (30-day retention, gzip compression where applicable). To override for a specific tenant or workload, create your own BackupClass pointing at the same strategy CRs but with tweaked parameters, or a fresh strategy CR. Common knobs:
- CNPG strategy:
barmanObjectStore.retentionPolicy,data.compression,wal.compression. - MariaDB strategy:
compression,maxRetention,databases[]. - Altinity strategy: tune the
clickhouse-backupsidecar viabackup.*values on the ClickHouse release; the strategy Pod is a thin HTTP client. - FoundationDB strategy:
snapshotPeriodSeconds,agentCount,urlParameters[]. - Velero strategy (VMInstance / VMDisk):
ttl,includedResources[],excludedResources[]. - Etcd strategy: today the strategy is path-only; combine with
Plan.spec.retentionPolicyfor trim cadence.
The system-managed credentials Secret is the only way for in-cluster strategies to reach cozy-backups. Do not embed access keys in BackupClass.parameters — the security model relies on Secret references, and parameters end up in Backup.status.underlyingResources, which tenants can read.
Disabling the platform-managed bucket
If a deployment runs against an external S3 (no SeaweedFS), set backupStorage.provisionBucket: false via the platform component override described above and create the source credentials Secret in tenant-root manually (flat-key format: accessKey / secretKey / endpoint / bucketName; or the raw COSI BucketInfo JSON). In the same backupStorage block, update endpoint, region, and bucketName: with provisionBucket: false the strategies and the Velero BSL take bucketName verbatim as the real S3 bucket name (no COSI lookup), so it must name the actual bucket on the external S3 — the bucketName key inside the Secret alone is not enough. The Velero BackupStorageLocation picks the same values up automatically (the chart renders it from the same backupStorage block), so no separate BSL configuration is needed. Note that disabling the cluster-default BSL itself (the chart's velero.bslEnabled value) is not carried by the backupStorage override path — the platform Package forwards only the backupStorage block.
Upgrade notes from chart-managed backups
Postgres backups now use the CloudNativePG Barman Cloud plugin, not native
spec.backup.barmanObjectStore.Postgres backups were migrated off the deprecated native
barmanObjectStore(removed in CNPG 1.29) onto the Barman Cloud plugin. Thebarman-cloud-*binaries the native path needs are absent from thestandardimage variantkeycloak-dbpins;apps/postgrespins bare (system-flavor) tags that still ship them — its breakage was the operator/CRD skew below, not missing binaries — but the system flavor is deprecated upstream, so both move to the plugin. The chart now rendersspec.pluginson thecnpg.io/Clusterplus abarmancloud.cnpg.io/ObjectStoreCR that carries the S3 configuration, and the CNPG backup driver SSA-applies anObjectStoreand patchesspec.plugins(instead ofspec.backup.barmanObjectStore) for the platform flow;Backup/ScheduledBackupusemethod: plugin. Onhelm upgradea legacy chart-managed Cluster'sspec.backup.barmanObjectStoreis replaced byspec.plugins+ anObjectStore. This only works once theplugin-barman-cloudoperator is installed and co-located with the CNPG operator incozy-postgres-operator(shipped by thepostgres-operatorchart) and its cert-manager dependency is present — otherwise the Cluster references a plugin that is not registered and backups / WAL archiving stall.Mixed-version window: BackupJobs fail until the app re-renders with the new chart. A
BackupJobthat fires while aClusterstill carries the pre-upgrade chart's nativespec.backup.barmanObjectStoreis rejected by the CNPG admission webhook (Cannot enable a WAL archiver plugin when barmanObjectStore is configured) — the driver cannot attachspec.pluginsalongside the native config. This resolves itself once Flux re-renders the Postgres app with the new chart (which replacesbarmanObjectStorewithspec.plugins); re-run the BackupJob afterward. Upgrading only the operators/driver without the app charts leaves everybackup.enabled: truePostgres in this state permanently.Postgres
backup.enabled: truewith placeholder credentials no longer wires backups on upgrade.The pre-v1.5 defaults for
backup.s3AccessKey/backup.s3SecretKeyinpackages/apps/postgres/values.yamlwere the literal"<your-access-key>"/"<your-secret-key>"placeholders, so the Postgres chart still rendered a backup block on thecnpg.io/Cluster(with junk credentials, WAL archiving failing at runtime). Starting with v1.5 those defaults are empty strings and the chart NO LONGER renders any backup wiring when the placeholders are unmodified. Tenants on the legacy chart-managed flow who relied on those placeholders seespec.pluginsand theObjectStoredisappear from the liveClusteronhelm upgrade. Action — pick one:
- Move to the platform flow (recommended). Set
backup.useSystemBucket: true; the chart leavesspec.pluginsunset and the CNPG backup driver SSA-applies theObjectStoreand patchesspec.pluginsonto the liveClusterat first BackupJob time. No tenant-side keys required.- Stay on the legacy chart-managed flow. Supply real
backup.s3AccessKey/backup.s3SecretKey(or a pre-existingbackup.s3CredentialsSecret.name); the chart rendersspec.plugins+ theObjectStorefrom those coordinates.Two platform-level changes ride along with this migration (they are required for the plugin to run):
- CloudNativePG operator bumped to 1.28.1 (
postgres-operatorchart). A minor operator upgrade for the whole fleet; the CNPG CRDs move with it. This also fixes a latent bug where the operator image was newer than its bundled CRDs (status.instanceID.sessionIDpruned), which made the operator believe the instance manager restarted and fail every backup cluster-wide withinstance manager was restarted during backup. The vendored chart ships the CNPG CRDs as ordinary release manifests (gated bycrds.create), so a normalhelm upgradeupdates them together with the operator — thehelm.sh/resource-policy: keepannotation only prevents their deletion, not updates. The historical skew came from theimage.tagpin outrunning the vendored chart's CRDs, which this PR removes; no out-of-band CRD apply is needed unless the HelmRelease itself is held back.- LINSTOR scheduler admission webhook must not strip native-sidecar fields. Older
linstor-schedulerextender builds (≤ v0.3.5's predecessors) decoded every pod through a stale vendored Kubernetes API and re-encoded it, strippinginitContainers[].restartPolicycluster-wide — which breaks the plugin's restartable barman sidecar ("startupProbe forbidden without restartPolicy=Always"). The platform already ships the fixed webhook (linstor-schedulerre-vendored to upstream chart 0.3.1 / extender v0.3.6, which stopped stripping unknown pod fields), so no action is needed on an up-to-date platform; just do not hold thelinstor-schedulerpackage back on an older version when rolling out the plugin.The same
useSystemBucketopt-in applies to ClickHouse — see ClickHouse: opt-in to the system bucket. WhenuseSystemBucket: trueis set on ClickHouse, the legacy<release>-backupCronJob, credential Secret, and backup script are no longer rendered (they are mutually exclusive with the platform flow); migrate scheduled backups to abackups.cozystack.io/Planagainstcozy-default.
Tenant workflow
Tenants only ever see the BackupClass name. Typical apply:
apiVersion: backups.cozystack.io/v1alpha1
kind: BackupJob
metadata:
name: ad-hoc
namespace: tenant-acme
spec:
backupClassName: cozy-default
applicationRef:
apiGroup: apps.cozystack.io
kind: Postgres
name: orders-db