fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-12 16:15:26 +03:00
commit 3badbdef74
23 files changed
+2347 -120

No files matched your search

@@ -0,0 +1,263 @@
#!/usr/bin/env bats
# The clickhouse-3-system-bucket-roundtrip Test edits the host CoreDNS Corefile
# and must put it back. Its restore runs from an EXIT trap under `set +e`, so a
# restore that fails without saying so leaves the rewrite in a cluster-wide
# object and still reports a green run; the next run then refuses to start.
# These run the restore against a stubbed kubectl.
load test_helper
LIB=hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh
HELPERS=examples/backups/clickhouse/00-helpers.sh
setup() {
strict_setup
STUB=$(mktemp -d)
# A Corefile carrying the Test's marked block, as the run step leaves it.
jq -n '{apiVersion: "v1", kind: "ConfigMap",
metadata: {name: "coredns", namespace: "kube-system", resourceVersion: "7"},
data: {Corefile: ".:53 {\n # === BEGIN e2e-clickhouse-system-bucket ===\n rewrite name exact s3.example.org root-ingress-controller.tenant-root.svc.cozy.local\n # === END e2e-clickhouse-system-bucket ===\n errors\n}\n"}}' \
>"$STUB/coredns.json"
# The secret lookup follows kubectl's contract: a name together with a
# selector is refused, and a selector-only lookup prints the names in
# OWNED_SECRETS, the Secrets carrying the owner label.
cat >"$STUB/kubectl" <<'EOF'
#!/bin/bash
echo "$*" >>"$STUB/calls"
case "$*" in
*"get configmap coredns -o json"*) cat "$STUB/coredns.json" ;;
replace*)
cp "${@: -1}" "$STUB/replaced.json"
if [ -n "${REPLACE_FAILS:-}" ]; then
echo 'Error from server (Conflict): Operation cannot be fulfilled on configmaps "coredns": the object has been modified' >&2
exit 1
fi
;;
*"get secret"*)
# Positionals are "get", "secret" and then any names.
words=0 selector=
while [ $# -gt 0 ]; do
case "$1" in
-l|--selector) selector=$2; shift 2 ;;
-n|--namespace|-o|--output) shift 2 ;;
-*) shift ;;
*) words=$((words + 1)); shift ;;
esac
done
if [ -n "${LOOKUP_FAILS:-}" ]; then
echo 'Unable to connect to the server: dial tcp: i/o timeout' >&2
exit 1
fi
if [ -n "$selector" ] && [ "$words" -gt 2 ]; then
echo 'error: name cannot be provided when a selector is specified' >&2
exit 1
fi
[ "$selector" = cozystack.io/e2e-owner=clickhouse-system-bucket ] || exit 0
for s in ${OWNED_SECRETS:-}; do echo "secret/$s"; done
;;
*"delete secret "*)
if [ -n "${DELETE_FAILS:-}" ]; then
echo 'Error from server (Forbidden): secrets "seaweedfs-s3-ingress-tls" is forbidden' >&2
exit 1
fi
;;
*) echo "unexpected kubectl $*" >&2; exit 2 ;;
esac
EOF
chmod +x "$STUB/kubectl"
export STUB
}
restore() {
PATH="$STUB:$PATH" NAMESPACE=tenant-root bash -c '
set -euo pipefail
source "$1"
source "$2"
set +e
restore_shared_state
' _ "$HELPERS" "$LIB"
}
@test "a CoreDNS restore the API server refuses fails the restore" {
export REPLACE_FAILS=1
err=$(mktemp)
if restore 2>"$err"; then
echo "restore_shared_state returned 0 although kubectl replace was refused" >&2
cat "$err" >&2
false
fi
if grep -q 'Removed the s3.example.org rewrite' "$err"; then
echo "restore_shared_state reported the rewrite removed although kubectl replace was refused:" >&2
cat "$err" >&2
false
fi
}
@test "a CoreDNS restore sends the Corefile back without the marked block" {
restore
jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null || {
echo "restored Corefile:" >&2
jq -r '.data.Corefile' "$STUB/replaced.json" >&2
false
}
# The resourceVersion read travels with the replace, so a concurrent
# writer makes it fail rather than be overwritten.
[ "$(jq -r '.metadata.resourceVersion' "$STUB/replaced.json")" = 7 ]
}
@test "the restore deletes the Ingress TLS Secret the Test created" {
OWNED_SECRETS=seaweedfs-s3-ingress-tls restore
grep -qE '^-n tenant-root delete secret seaweedfs-s3-ingress-tls( |$)' "$STUB/calls" || {
echo "restore_shared_state did not delete the Secret carrying the owner label; kubectl calls:" >&2
cat "$STUB/calls" >&2
false
}
}
@test "the restore leaves an Ingress TLS Secret without the owner label alone" {
for owned in '' e2e-other seaweedfs-s3-ingress-tls-old; do
: >"$STUB/calls"
OWNED_SECRETS=$owned restore
if grep -q 'delete secret' "$STUB/calls"; then
echo "restore_shared_state deleted a Secret although the labelled ones are '$owned'; kubectl calls:" >&2
cat "$STUB/calls" >&2
false
fi
done
}
@test "a Secret delete the API server refuses fails the restore" {
err=$(mktemp)
if OWNED_SECRETS=seaweedfs-s3-ingress-tls DELETE_FAILS=1 restore 2>"$err"; then
echo "restore_shared_state returned 0 although kubectl delete secret was refused" >&2
cat "$err" >&2
false
fi
grep -qE '^-n tenant-root delete secret seaweedfs-s3-ingress-tls( |$)' "$STUB/calls" || {
echo "restore_shared_state failed before it reached the Secret delete; kubectl calls:" >&2
cat "$STUB/calls" "$err" >&2
false
}
}
@test "a Secret lookup that fails fails the restore" {
err=$(mktemp)
if LOOKUP_FAILS=1 restore 2>"$err"; then
echo "restore_shared_state returned 0 although it could not tell whether the Secret is left over" >&2
cat "$err" >&2
false
fi
grep -q 'could not list the Secrets labelled' "$err" || {
echo "restore_shared_state failed without naming the Secret lookup:" >&2
cat "$err" >&2
false
}
}
@test "the run step's finally restores the shared objects" {
# When the step ceiling fires, Chainsaw kills the shell and the EXIT trap
# of system-bucket-roundtrip.sh never runs; the step's finally still does.
# Chainsaw runs a script's content with sh from the suite's directory.
suite=hack/e2e-chainsaw/clickhouse
finally=$(yq 'select(.metadata.name == "clickhouse-3-system-bucket-roundtrip")
| .spec.steps[] | select(.name == "run") | .finally[]
| select(.description == "restore the host CoreDNS and the S3 Ingress TLS Secret") | .script.content' \
"$suite/chainsaw-test.yaml")
[ -n "$finally" ] || { echo "the run step has no finally restoring the shared objects" >&2; false; }
(cd "$suite" && PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$finally")
[ -f "$STUB/replaced.json" ] || {
echo "the run step's finally did not put the Corefile back" >&2
false
}
jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null
}
# The Altinity driver names the Backup after the BackupJob, and a Backup left
# behind under that name records an earlier run's archive, which fails the next
# run's BackupJob. It is reclaimed by name, whether or not the BackupJob
# recorded it in status.backupRef.
@test "the round-trip reclaims its Backup by name before and after the run" {
suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml
test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]'
calls="$STUB/calls"
# Nothing is left over from an earlier run, and no BackupJob recorded a
# backupRef.
cat >"$STUB/kubectl" <<'STUBEOF'
#!/bin/bash
echo "$*" >>"$STUB/calls"
case "$*" in
*" get secret "*) exit 1 ;;
esac
STUBEOF
for selector in \
'select(.name == "preconditions") | .try[].script.content' \
'select(.name == "run") | .finally[] | select(.description == "system-bucket round-trip cleanup") | .script.content'; do
: >"$calls"
script=$(yq "$test | $selector" "$suite")
[ -n "$script" ] || { echo "no script at: $selector" >&2; false; }
PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script"
grep -qE '^-n tenant-root delete backup sysbucket-backup( |$)' "$calls" || {
echo "the script at '$selector' does not delete Backup sysbucket-backup by name; kubectl calls:" >&2
cat "$calls" >&2
false
}
done
}
# cert-manager issues the Secret of a Certificate that still exists again as
# soon as it is deleted. Chainsaw deletes the Certificate it applied, and waits
# for it, before it runs the cleanup ops of the same step, so the Secret is
# deleted there and nowhere earlier.
@test "the Certificate's Secret is deleted after Chainsaw deletes the Certificate" {
test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]'
suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml
applies=$(yq "$test | select(.name == \"create-certificate-and-probe\") | .try[] | select(has(\"apply\")) | .apply.file" "$suite")
[ "$applies" = system-bucket-tls.yaml ] || {
echo "step create-certificate-and-probe no longer applies system-bucket-tls.yaml alone: $applies" >&2
false
}
deletes=$(yq -o=json -I=0 "$test | select(.name == \"create-certificate-and-probe\") | .cleanup" "$suite")
[ "$deletes" = '[{"delete":{"ref":{"apiVersion":"v1","kind":"Secret","name":"e2e-system-bucket-s3-tls"}}}]' ] || {
echo "the cleanup of step create-certificate-and-probe is not the one delete of Secret e2e-system-bucket-s3-tls: $deletes" >&2
false
}
# The preconditions run before the apply, when Chainsaw holds no Certificate.
if yq "$test | select(.name != \"preconditions\") | (.try[], .finally[]?) | .script.content // \"\"" "$suite" | grep -qE 'delete +(secret +e2e-system-bucket-s3-tls|certificate +e2e-system-bucket-s3)( |$)'; then
echo "a step script deletes the Certificate or its Secret while Chainsaw still holds the Certificate" >&2
false
fi
}
# Chainsaw registers an applied object for deletion only when its apply created
# it; an object left by a run killed past its cleanup is patched and kept. The
# Certificate would then outlive every later run, re-issuing its Secret after
# each step cleanup, and a probe Pod whose `sleep` had ended never turns Ready
# again, so the preconditions remove all three before the apply.
@test "the preconditions remove the TLS objects and probe Pod a killed run left" {
suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml
test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]'
calls="$STUB/calls"
cat >"$STUB/kubectl" <<'STUBEOF'
#!/bin/bash
echo "$*" >>"$STUB/calls"
case "$*" in
*" get secret "*) exit 1 ;;
esac
STUBEOF
: >"$calls"
script=$(yq "$test | select(.name == \"preconditions\") | .try[].script.content" "$suite")
PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script"
line_of() { grep -nE "^-n tenant-root delete $1 --ignore-not-found --wait=true( |$)" "$calls" | head -n1 | cut -d: -f1; }
cert=$(line_of 'certificate e2e-system-bucket-s3')
secret=$(line_of 'secret e2e-system-bucket-s3-tls')
pod=$(line_of 'pod e2e-system-bucket-probe')
[ -n "$cert" ] && [ -n "$secret" ] && [ -n "$pod" ] || {
echo "the preconditions do not delete, and wait for, the Certificate, its Secret and the probe Pod; kubectl calls:" >&2
cat "$calls" >&2
false
}
[ "$cert" -lt "$secret" ] || {
echo "the preconditions delete Secret e2e-system-bucket-s3-tls before its Certificate, which cert-manager then issues again" >&2
false
}
}
+223 -2
View File
@@ -91,7 +91,7 @@ spec:
status:
replicas: 1
---
# ClickHouse Altinity backup/restore E2E — the two backup Tests below sit in
# ClickHouse Altinity backup/restore E2E — the three backup Tests below sit in
# this suite alongside the plain `clickhouse` Test above. Chainsaw port of the
# intended hack/e2e-apps/backup-clickhouse.bats (cozystack#2600), shaped like the
# etcd/mariadb/postgres suites:
@@ -107,8 +107,15 @@ spec:
# driving the validated example scripts
# (examples/backups/clickhouse). Also runs in
# CI, in tenant-root.
# clickhouse-3-system-bucket-roundtrip
# — backup and in-place restore through the
# shipped cozy-default BackupClass and the
# platform system bucket, for a release
# whose Pod started before its credentials
# were projected. Runs in CI, in tenant-root.
#
# Why the split: the two Tests answer different questions at different cost. The
# Why the split: the first two Tests answer different questions at different
# cost, and the third covers the system-bucket path neither of them takes. The
# contracts Test needs no object store, so it stays in the default tenant-test
# namespace next to the other app suites and is the seconds-long guard on the
# chart wiring the Altinity strategy depends on — the sidecar container/port and
@@ -377,3 +384,217 @@ spec:
set -eu
cd ../../..
NAMESPACE="$NAMESPACE" examples/backups/clickhouse/cleanup.sh
---
# Backup and in-place restore through the shipped cozy-default BackupClass and
# the platform system bucket, for a ClickHouse whose Pod started before the
# platform projected cozy-backups-creds into its namespace. The sidecar has to
# pick those credentials up while running; until it could, the first BackupJob
# of such a release failed with "no EC2 IMDS role found" and kept failing until
# the Pod was restarted.
#
# Unlike clickhouse-2-backup-roundtrip this targets the system bucket as
# advertised, https://s3.example.org, so system-bucket-roundtrip.sh makes that
# name reachable from tenant-root for the duration of its own step and undoes
# it on exit: see its header for the two shared objects it touches and the
# cert-manager behaviour that relies on. Runs in tenant-root, where the system
# bucket's SeaweedFS and its Ingress controller sit.
#
# The precondition step deletes tenant-root's projected cozy-backups-creds,
# which any BackupJob that ran in tenant-root before this Test leaves there:
# the projector runs for each BackupJob whatever its strategy, and Chainsaw
# does not fix the order of the Tests in a suite, so the earlier one may come
# from this suite or another. It writes the Secret again on the next
# BackupJob, which is the moment under test.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: clickhouse-3-system-bucket-roundtrip
spec:
namespace: tenant-root
catch:
- events: {}
- describe:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
name: clickhouse-sysbucket
- describe:
apiVersion: cert-manager.io/v1
kind: Certificate
name: e2e-system-bucket-s3
- describe:
apiVersion: cert-manager.io/v1
kind: Certificate
name: seaweedfs-s3-ingress-tls
- describe:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: ingress-seaweedfs-s3
- describe:
apiVersion: backups.cozystack.io/v1alpha1
kind: BackupJob
name: sysbucket-backup
- describe:
apiVersion: backups.cozystack.io/v1alpha1
kind: RestoreJob
name: sysbucket-restore
- podLogs:
selector: app.kubernetes.io/instance=clickhouse-sysbucket
container: clickhouse-backup
tail: 200
- podLogs:
selector: app=clickhouse-sysbucket-keeper
tail: 100
steps:
- name: preconditions
try:
# The deletes below wait up to 270s in all (30+30+60+30+30+30+60), so a
# stuck one is named by its kubectl --timeout before the 5m ceiling kills
# the shell with no object named. A new delete needs its own --timeout.
- script:
timeout: 5m
content: |
set -eu
if kubectl -n "$NAMESPACE" get secret seaweedfs-s3-ingress-tls -o name >/dev/null 2>&1; then
echo "Secret $NAMESPACE/seaweedfs-s3-ingress-tls already exists; this Test would have to replace it" >&2
exit 1
fi
if kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF 'BEGIN e2e-clickhouse-system-bucket'; then
echo "kube-system/coredns still carries the e2e-clickhouse-system-bucket block from an earlier run" >&2
exit 1
fi
# Fixed names in a persistent namespace: a leftover Succeeded job
# would satisfy the waits in the run step without a new attempt, and
# a leftover Backup of the BackupJob's name, recording an earlier
# run's archive, fails the new BackupJob.
kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s
kubectl -n "$NAMESPACE" delete secret cozy-backups-creds --ignore-not-found --wait=true --timeout=30s
# Chainsaw reclaims only what its apply created, so objects a killed
# run left would be patched and kept. The Certificate goes before its
# Secret, which cert-manager issues again while the Certificate exists.
kubectl -n "$NAMESPACE" delete certificate e2e-system-bucket-s3 --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete pod e2e-system-bucket-probe --ignore-not-found --wait=true --timeout=60s
check:
($error == null): true
- name: create-certificate-and-probe
try:
- apply:
file: system-bucket-tls.yaml
- assert:
timeout: 2m
resource:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: e2e-system-bucket-s3
status:
(conditions[?type == 'Ready']):
- status: "True"
- assert:
timeout: 3m
resource:
apiVersion: v1
kind: Pod
metadata:
name: e2e-system-bucket-probe
status:
(conditions[?type == 'Ready']):
- status: "True"
# The Certificate's Secret carries no owner reference, so it outlives the
# Certificate, and cert-manager issues it again while the Certificate
# exists. Chainsaw runs a step's cleanup ops after this step's cleaner has
# deleted the Certificate and waited for it.
cleanup:
- delete:
ref:
apiVersion: v1
kind: Secret
name: e2e-system-bucket-s3-tls
- name: create-clickhouse
try:
- apply:
file: clickhouse-system-bucket.yaml
# 11m: above the 600s install timeout cozystack-api gives this release;
# see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
- assert:
timeout: 11m
resource:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: clickhouse-sysbucket
status:
(conditions[?type == 'Ready']):
- status: "True"
- assert:
timeout: 5m
resource:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: chi-clickhouse-sysbucket-clickhouse-0-0
status:
readyReplicas: 1
# The Pod is running with no projected credentials: the state the first
# BackupJob of a fresh release meets.
- error:
timeout: 30s
resource:
apiVersion: v1
kind: Secret
metadata:
name: cozy-backups-creds
- name: run
try:
# Inner budgets of system-bucket-roundtrip.sh:
# 280s probe: a 240s deadline for CoreDNS picking up the rewrite and
# the Ingress serving the Test certificate, plus one last 30s
# attempt and its 10s pause
# 480s BackupJob to Succeeded, including the driver's 2m hold
# 480s RestoreJob to Succeeded; no hold by then, the stamp is
# already older than the backup's own 2m wait
# 90s the EXIT trap: Corefile replace plus a 60s Secret delete
# 1330s (22m10s)
# 25m leaves 2m50s for the SQL and kubectl calls between those waits, so
# an inner wait is what fires and names what stalled.
- script:
timeout: 25m
content: |
set -eu
cd ../../..
NAMESPACE="$NAMESPACE" hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh
check:
($error == null): true
finally:
# The script's EXIT trap puts the Corefile and the Ingress TLS Secret
# back, but a step ceiling kills the shell before the trap runs. A no-op
# when the trap already did it.
- description: restore the host CoreDNS and the S3 Ingress TLS Secret
script:
timeout: 2m
content: |
set -eu
cd ../../..
# Chainsaw runs this under sh; the helpers are bash.
NAMESPACE="$NAMESPACE" bash -euo pipefail -c '
source examples/backups/clickhouse/00-helpers.sh
source hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh
restore_shared_state
'
# Chainsaw did not apply these: the jobs come from the script and the
# Backup from the controller, and deleting a BackupJob reclaims no Backup.
- description: system-bucket round-trip cleanup
script:
timeout: 3m
content: |
set -eu
kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s
# The driver names the Backup after the BackupJob; by name it is
# found even when status.backupRef was never written.
kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s
@@ -8,8 +8,8 @@
# backup.enabled=true with placeholder S3 coordinates is deliberate: the
# sidecar runs `clickhouse-backup server`, which starts its HTTP API without
# validating S3 (ALLOW_EMPTY_BACKUPS=true, lazy connect — the same reason the
# platform useSystemBucket flow marks the S3 secret keys optional so the Pod
# starts before any bucket is projected). So the Pod becomes Ready and the
# platform useSystemBucket flow can start the Pod before any bucket is
# projected). So the Pod becomes Ready and the
# chart<->strategy contract is assertable with no reachable object store. The
# real S3 backup/restore round-trip lives in clickhouse-2-backup-roundtrip,
# which runs in tenant-root against the in-cluster SeaweedFS.
@@ -0,0 +1,30 @@
# ClickHouse on the platform system bucket for the
# clickhouse-3-system-bucket-roundtrip Test. No S3 coordinates are given: the
# sidecar reads them from the clickhouse-backup.yml key the platform projects
# into cozy-backups-creds at the first BackupJob, after this Pod has started.
# backup.endpointCA trusts the SeaweedFS CA that signs the certificate the Test
# serves for s3.example.org.
apiVersion: apps.cozystack.io/v1alpha1
kind: ClickHouse
metadata:
name: sysbucket
spec:
size: 5Gi
logStorageSize: 1Gi
shards: 1
replicas: 1
# The round-trip runs SQL against this server; at nano's 128Mi the first
# query OOM-kills it, so it takes the preset clickhouse-2 runs SQL on.
resourcesPreset: small
resources: {}
backup:
enabled: true
useSystemBucket: true
endpointCA:
name: e2e-system-bucket-s3-tls
key: ca.crt
clickhouseKeeper:
enabled: true
replicas: 1
size: 1Gi
resourcesPreset: micro
+174
View File
@@ -0,0 +1,174 @@
#!/bin/bash
# Backup and in-place restore of the `sysbucket` ClickHouse through the
# cozy-default BackupClass, against the platform system bucket. Run from the
# repository root by the clickhouse-3-system-bucket-roundtrip Test, which has
# already started the release with no cozy-backups-creds in its namespace.
#
# The system bucket is addressed as https://s3.example.org, the S3 Ingress host
# the COSI driver advertises. In CI that name does not resolve and its ACME
# certificate is never issued, so for the duration of this script, and only of
# this script, two shared objects are changed and the EXIT trap below puts both
# back:
#
# - the S3 Ingress TLS Secret seaweedfs-s3-ingress-tls is created from the
# Test's own certificate (SeaweedFS CA, s3.example.org). It relies on
# cert-manager leaving a Secret's data alone while the Certificate that
# names it has no completed issuance, verified with cert-manager v1.20.2:
# a fresh ACME order would replace this certificate, and ingress-nginx
# would then serve one the sidecar does not trust, which fails the probe
# below rather than the backup.
# - the host CoreDNS Corefile gets one marked `rewrite name exact` line for
# s3.example.org. The host CoreDNS is Talos-managed and imports no snippet
# files, so the block goes into the live Corefile the way ouroboros writes
# its own; ouroboros is off in CI and no other suite uses this name.
set -euo pipefail
source examples/backups/clickhouse/00-helpers.sh
source hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh
: "${NAMESPACE:?}"
CLICKHOUSE_NAME=sysbucket
BACKUPJOB_NAME=sysbucket-backup
RESTOREJOB_NAME=sysbucket-restore
CERT_SECRET=e2e-system-bucket-s3-tls
PROBE_POD=e2e-system-bucket-probe
DNS_TARGET="root-ingress-controller.${NAMESPACE}.svc.cozy.local"
# Kubelet refreshes a ConfigMap volume within about a minute and CoreDNS's
# reload plugin polls every 30s; the rest is the 30s answer cache.
PROBE_TIMEOUT=240
# Each covers the driver's 2m hold after the credentials change plus the
# strategy Job itself.
JOB_TIMEOUT=480
workdir=$(mktemp -d)
on_exit() {
local rc=$?
set +e
restore_shared_state || rc=1
rm -rf "$workdir"
exit "$rc"
}
if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -o name >/dev/null 2>&1; then
log_error "Secret ${NAMESPACE}/${INGRESS_SECRET} already exists; this Test serves its own certificate under that name and will not overwrite one it did not create"
exit 1
fi
if corefile_has_block; then
log_error "kube-system/coredns already carries the ${S3_HOST} rewrite block, left by an earlier run; remove it by hand"
exit 1
fi
trap on_exit EXIT
log_step "Serving the Test certificate for ${S3_HOST} through the platform S3 Ingress"
kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o json \
| jq --arg name "$INGRESS_SECRET" '{apiVersion, kind, type,
metadata: {name: $name, labels: {"cozystack.io/e2e-owner": "clickhouse-system-bucket"}},
data: {"tls.crt": .data["tls.crt"], "tls.key": .data["tls.key"], "ca.crt": .data["ca.crt"]}}' \
| kubectl -n "$NAMESPACE" create -f -
log_step "Rewriting ${S3_HOST} to ${DNS_TARGET} in the host CoreDNS"
kubectl -n kube-system get configmap coredns -o json >"$workdir/coredns.json"
jq -e '.data.Corefile | startswith(".:53 {\n")' "$workdir/coredns.json" >/dev/null \
|| { log_error "kube-system/coredns Corefile does not open with '.:53 {'; refusing to edit it"; exit 1; }
# The resourceVersion read above travels with the replace, so a concurrent
# writer makes it fail instead of being overwritten.
jq --arg block "${DNS_BEGIN}"$'\n'" rewrite name exact ${S3_HOST} ${DNS_TARGET}"$'\n'"${DNS_END}" \
'.data.Corefile |= sub("^\\.:53 \\{\n"; ".:53 {\n" + $block + "\n")' \
"$workdir/coredns.json" >"$workdir/coredns-new.json"
kubectl replace -f "$workdir/coredns-new.json" >/dev/null
log_step "Checking that ${S3_HOST} serves the Test certificate from inside ${NAMESPACE}"
kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.ca\.crt}' | base64 -d >"$workdir/ca.crt"
want=$(kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -fingerprint -sha256)
deadline=$((SECONDS + PROBE_TIMEOUT))
while true; do
# --cacert makes curl verify both the chain and the host name; %{certs}
# prints the chain it was served, leaf first.
if served=$(timeout 30 kubectl -n "$NAMESPACE" exec -i "$PROBE_POD" -- sh -c \
"cat >/tmp/ca.crt && curl -sS -o /dev/null --max-time 10 --cacert /tmp/ca.crt -w '%{certs}' https://${S3_HOST}/" \
<"$workdir/ca.crt" 2>"$workdir/probe.err"); then
got=$(printf '%s\n' "$served" | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p;/END CERTIFICATE/q' \
| openssl x509 -noout -fingerprint -sha256)
if [ "$got" = "$want" ]; then
log_success "${S3_HOST} resolves in ${NAMESPACE} and serves the Test certificate"
break
fi
echo "served ${got}, want ${want}" >"$workdir/probe.err"
fi
if [ "$SECONDS" -ge "$deadline" ]; then
log_error "${S3_HOST} did not serve the Test certificate within ${PROBE_TIMEOUT}s: $(cat "$workdir/probe.err")"
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' >&2 || true
kubectl -n "$NAMESPACE" describe ingress ingress-seaweedfs-s3 >&2 || true
exit 1
fi
# TODO(e2e-replace-fixed-timeouts): no condition reports that the host
# CoreDNS answers with the rewrite and the Ingress serves the Test
# certificate, as seen from this namespace; poll until one does.
sleep 10
done
# A restarted Pod reads its credentials afresh at start, which is what the
# old environment-based sidecar needed; the round-trip proves the fix only if
# the sidecar that started without them is the one that serves both jobs.
sidecar_identity() {
kubectl -n "$NAMESPACE" get pod "chi-clickhouse-${CLICKHOUSE_NAME}-clickhouse-0-0-0" \
-o jsonpath='{.metadata.uid} {.status.containerStatuses[?(@.name=="clickhouse-backup")].restartCount}'
}
sidecar_before=$(sidecar_identity)
[[ "$sidecar_before" =~ ^[0-9a-f-]+\ [0-9]+$ ]] \
|| { log_error "could not read the ClickHouse Pod uid and clickhouse-backup restart count (got '${sidecar_before}')"; exit 1; }
# Restore takes the archive this run's Backup recorded; a row no earlier run
# wrote is what shows the data restored is that archive's.
sentinel="before-backup-$(date -u +%Y%m%dT%H%M%SZ)-${RANDOM}"
log_step "Writing the sentinel row ${sentinel}"
clickhouse_query "$CLICKHOUSE_NAME" "CREATE TABLE IF NOT EXISTS default.sentinel (id UInt32, name String) ENGINE = MergeTree ORDER BY id"
clickhouse_query "$CLICKHOUSE_NAME" "INSERT INTO default.sentinel VALUES (1, '${sentinel}')"
log_step "Backing up through cozy-default"
kubectl apply -f - <<EOF
apiVersion: backups.cozystack.io/v1alpha1
kind: BackupJob
metadata:
name: ${BACKUPJOB_NAME}
namespace: ${NAMESPACE}
spec:
applicationRef:
apiGroup: apps.cozystack.io
kind: ClickHouse
name: ${CLICKHOUSE_NAME}
backupClassName: cozy-default
EOF
wait_for_field backupjob "$BACKUPJOB_NAME" '{.status.phase}' Succeeded "$NAMESPACE" "$JOB_TIMEOUT" Failed
backup_ref=$(kubectl -n "$NAMESPACE" get backupjob "$BACKUPJOB_NAME" -o jsonpath='{.status.backupRef.name}')
[ -n "$backup_ref" ] || { log_error "BackupJob ${BACKUPJOB_NAME} succeeded with an empty backupRef"; exit 1; }
# The sidecar started before this Secret existed; a backup that succeeded
# against it is the behaviour under test, and the key it read is this one.
endpoint=$(kubectl -n "$NAMESPACE" get secret cozy-backups-creds -o jsonpath='{.data.clickhouse-backup\.yml}' | base64 -d \
| sed -n 's/^ *endpoint: *//p')
[ "$endpoint" = "https://${S3_HOST}" ] \
|| { log_error "cozy-backups-creds clickhouse-backup.yml endpoint is '${endpoint}', expected https://${S3_HOST}"; exit 1; }
log_step "Restoring in place through cozy-default"
clickhouse_query "$CLICKHOUSE_NAME" "DROP TABLE IF EXISTS default.sentinel SYNC"
kubectl apply -f - <<EOF
apiVersion: backups.cozystack.io/v1alpha1
kind: RestoreJob
metadata:
name: ${RESTOREJOB_NAME}
namespace: ${NAMESPACE}
spec:
backupRef:
name: ${backup_ref}
EOF
wait_for_field restorejob "$RESTOREJOB_NAME" '{.status.phase}' Succeeded "$NAMESPACE" "$JOB_TIMEOUT" Failed
count=$(clickhouse_query "$CLICKHOUSE_NAME" "SELECT count() FROM default.sentinel WHERE name = '${sentinel}'" | tr -d '[:space:]')
[ "$count" = 1 ] || { log_error "after restore default.sentinel holds ${count} row(s) named ${sentinel}; expected 1"; exit 1; }
sidecar_after=$(sidecar_identity)
[ "$sidecar_after" = "$sidecar_before" ] \
|| { log_error "the ClickHouse Pod or its clickhouse-backup container restarted during the round-trip (uid restarts: '${sidecar_before}' then '${sidecar_after}')"; exit 1; }
log_success "System-bucket round-trip verified: sentinel row ${sentinel} restored"
@@ -0,0 +1,60 @@
#!/bin/bash
# The two shared objects system-bucket-roundtrip.sh changes for the duration
# of its run, and how to put them back. Sourced by that script, whose EXIT trap
# restores them, and by the finally of its Chainsaw step, which restores them
# when the step ceiling killed the script before the trap could run. Needs
# NAMESPACE and log_success/log_error from examples/backups/clickhouse/00-helpers.sh.
S3_HOST=s3.example.org
INGRESS_SECRET=seaweedfs-s3-ingress-tls
OWNER_LABEL=cozystack.io/e2e-owner=clickhouse-system-bucket
DNS_BEGIN=' # === BEGIN e2e-clickhouse-system-bucket ==='
DNS_END=' # === END e2e-clickhouse-system-bucket ==='
corefile_has_block() {
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF -- "$DNS_BEGIN"
}
remove_dns_block() {
local dir
dir=$(mktemp -d)
kubectl -n kube-system get configmap coredns -o json >"$dir/coredns.json" || { rm -rf "$dir"; return 1; }
local begins
begins=$(jq -r '.data.Corefile' "$dir/coredns.json" | grep -cF -- "$DNS_BEGIN" || true)
if [ "$begins" = 0 ]; then
rm -rf "$dir"
return 0
fi
if [ "$begins" != 1 ] || ! jq -r '.data.Corefile' "$dir/coredns.json" | grep -qF -- "$DNS_END"; then
log_error "kube-system/coredns holds ${begins} '${DNS_BEGIN# *}' markers or no END marker; reconcile the Corefile by hand"
rm -rf "$dir"
return 1
fi
jq --arg b "$DNS_BEGIN" --arg e "$DNS_END" \
'.data.Corefile |= (split("\n") | (index($b)) as $i | (index($e)) as $j | .[:$i] + .[$j+1:] | join("\n"))' \
"$dir/coredns.json" >"$dir/coredns-clean.json"
# Callers run this with errexit off, so a refused replace has to return.
if ! kubectl replace -f "$dir/coredns-clean.json" >/dev/null; then
log_error "could not remove the ${S3_HOST} rewrite from kube-system/coredns; remove the marked block by hand"
rm -rf "$dir"
return 1
fi
log_success "Removed the ${S3_HOST} rewrite from kube-system/coredns"
rm -rf "$dir"
}
# Returns non-zero when either object could not be put back.
restore_shared_state() {
local rc=0
remove_dns_block || rc=1
# kubectl refuses a name together with a selector, so the label is
# looked up on its own and the name matched in what comes back.
local owned
if ! owned=$(kubectl -n "$NAMESPACE" get secret -l "$OWNER_LABEL" -o name); then
log_error "could not list the Secrets labelled ${OWNER_LABEL} in ${NAMESPACE}; delete ${INGRESS_SECRET} by hand if it carries that label"
rc=1
elif printf '%s\n' "$owned" | grep -qxF "secret/$INGRESS_SECRET"; then
kubectl -n "$NAMESPACE" delete secret "$INGRESS_SECRET" --wait=true --timeout=60s || rc=1
fi
return "$rc"
}
@@ -0,0 +1,42 @@
# Certificate and probe Pod for the clickhouse-3-system-bucket-roundtrip Test.
#
# The certificate is signed by the SeaweedFS CA already present in tenant-root,
# so its Secret carries that CA as ca.crt: the ClickHouse release trusts it via
# backup.endpointCA, and the round-trip script copies the same key pair under
# the name the platform's S3 Ingress serves for s3.example.org.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: e2e-system-bucket-s3
spec:
secretName: e2e-system-bucket-s3-tls
dnsNames:
- s3.example.org
issuerRef:
kind: Issuer
name: seaweedfs-ca-issuer
---
# Resolves and fetches s3.example.org from inside tenant-root, where the
# ClickHouse sidecar runs, so the script can prove the DNS rewrite and the
# served certificate before it spends minutes on a BackupJob. The image is the
# one hack/e2e-install-cozystack.bats pre-pulls; it carries curl but no openssl.
apiVersion: v1
kind: Pod
metadata:
name: e2e-system-bucket-probe
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 0
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: probe
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47
command: [sleep, "3600"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
@@ -85,6 +85,76 @@ func validateAltinityApplicationRef(ref corev1.TypedLocalObjectReference) error
return nil
}
// altinityCredentialsPropagation bounds how long kubelet takes to rewrite
// the mounted cozy-backups-creds in the ClickHouse Pod after it changes:
// its sync period (a minute by default) plus the Secret cache. About 75s was
// measured; the margin covers the rest.
const altinityCredentialsPropagation = 2 * time.Minute
// credentialsPropagationWait returns how long to hold back creating the
// strategy Job named jobName in namespace, so that the clickhouse-backup
// sidecar, which reads the projected credentials from a mounted file, sees
// their latest content. A Job that already exists, or a namespace without
// projected credentials, does not wait.
func credentialsPropagationWait(ctx context.Context, c client.Client, secretName, namespace, jobName string) (time.Duration, error) {
if secretName == "" {
return 0, nil
}
if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: jobName}, &batchv1.Job{}); err == nil {
return 0, nil
} else if !apierrors.IsNotFound(err) {
return 0, err
}
secret := &corev1.Secret{}
if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: secretName}, secret); err != nil {
if apierrors.IsNotFound(err) {
return 0, nil
}
return 0, err
}
raw := secret.Annotations[credentialsUpdatedAtAnnotation]
if raw == "" {
return 0, nil
}
// The projector rewrites the stamp only when the content changes, so a
// stamp this hold cannot use stays as it is: holding on it would block
// every Job in the namespace for good.
updatedAt, err := time.Parse(time.RFC3339, raw)
if err != nil {
getLogger(ctx).Error(err, "ignoring unreadable credentials stamp", "secret", namespace+"/"+secretName, "annotation", credentialsUpdatedAtAnnotation)
return 0, nil
}
if updatedAt.After(time.Now().Add(altinityCredentialsPropagation)) {
getLogger(ctx).Error(nil, "ignoring a credentials stamp further ahead than clock skew explains", "secret", namespace+"/"+secretName, "annotation", credentialsUpdatedAtAnnotation, "value", raw)
return 0, nil
}
return time.Until(updatedAt.Add(altinityCredentialsPropagation)), nil
}
// clearCredentialsWait drops the Ready condition credentialsWaitCondition
// left behind, once the job no longer waits. It reports whether it did.
func clearCredentialsWait(conditions *[]metav1.Condition) bool {
ready := apimeta.FindStatusCondition(*conditions, "Ready")
if ready == nil || ready.Reason != credentialsWaitReason {
return false
}
return apimeta.RemoveStatusCondition(conditions, "Ready")
}
const credentialsWaitReason = "WaitingForCredentials"
// credentialsWaitCondition is the Ready condition of a job held back by
// credentialsPropagationWait.
func credentialsWaitCondition(secretName string, wait time.Duration) metav1.Condition {
return metav1.Condition{
Type: "Ready",
Status: metav1.ConditionFalse,
Reason: credentialsWaitReason,
Message: fmt.Sprintf("waiting %s for the updated Secret %s to reach the ClickHouse backup sidecar",
wait.Round(time.Second), secretName),
}
}
// jobNameForBackupJob returns the deterministic batch/v1.Job name used for
// the backup pod that drives a BackupJob. Idempotency relies on this being
// pure: a retried reconcile must hit the same Job name.
@@ -265,6 +335,18 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to template Altinity strategy: %v", err))
}
wait, err := credentialsPropagationWait(ctx, r.Client, r.CredentialsConfig.TargetSecretName, j.Namespace, jobNameForBackupJob(j))
if err != nil {
return ctrl.Result{}, err
}
if wait > 0 {
apimeta.SetStatusCondition(&j.Status.Conditions, credentialsWaitCondition(r.CredentialsConfig.TargetSecretName, wait))
if err := r.Status().Update(ctx, j); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: wait}, nil
}
batchJob, err := r.ensureAltinityJob(ctx, j, j.Namespace, jobNameForBackupJob(j),
altinityModeBackup,
map[string]string{
@@ -312,6 +394,11 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv
return r.markBackupJobFailed(ctx, j, message)
default:
if clearCredentialsWait(&j.Status.Conditions) {
if err := r.Status().Update(ctx, j); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{RequeueAfter: altinityPollInterval}, nil
}
}
@@ -681,6 +768,18 @@ func (r *RestoreJobReconciler) reconcileAltinityRestore(ctx context.Context, res
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to template Altinity strategy: %v", err))
}
wait, err := credentialsPropagationWait(ctx, r.Client, r.CredentialsConfig.TargetSecretName, targetNamespace, jobNameForRestoreJob(restoreJob))
if err != nil {
return ctrl.Result{}, err
}
if wait > 0 {
apimeta.SetStatusCondition(&restoreJob.Status.Conditions, credentialsWaitCondition(r.CredentialsConfig.TargetSecretName, wait))
if err := r.Status().Update(ctx, restoreJob); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: wait}, nil
}
batchJob, err := r.ensureAltinityRestoreJob(ctx, restoreJob, targetNamespace, jobNameForRestoreJob(restoreJob),
altinityModeRestore,
map[string]string{
@@ -717,6 +816,11 @@ func (r *RestoreJobReconciler) reconcileAltinityRestore(ctx context.Context, res
return r.markRestoreJobFailed(ctx, restoreJob, message)
default:
if clearCredentialsWait(&restoreJob.Status.Conditions) {
if err := r.Status().Update(ctx, restoreJob); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{RequeueAfter: altinityPollInterval}, nil
}
}
@@ -3,12 +3,14 @@ package backupcontroller
import (
"context"
"errors"
"strings"
"testing"
"time"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
@@ -19,6 +21,7 @@ import (
"k8s.io/client-go/tools/record"
"sigs.k8s.io/controller-runtime/pkg/client"
clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake"
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1"
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
@@ -1032,3 +1035,539 @@ func TestReconcileAltinityRestore_RendersTheBackupsArchive(t *testing.T) {
})
}
}
func projectedCredentials(namespace string, updatedAt time.Time) *corev1.Secret {
return &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Namespace: namespace,
Name: "cozy-backups-creds",
Annotations: map[string]string{credentialsUpdatedAtAnnotation: updatedAt.UTC().Format(time.RFC3339)},
},
}
}
// The ClickHouse sidecar reads the projected credentials from a mounted
// file that kubelet rewrites only some time after the Secret changes, so a
// strategy Job started right after a projection would reach a sidecar with
// no or stale credentials.
func TestReconcileAltinity_WaitsForProjectedCredentialsToReachTheSidecar(t *testing.T) {
cases := []struct {
name string
secret *corev1.Secret
jobExists bool
wantWait bool
}{
{name: "credentials changed just now", secret: projectedCredentials("tenant-test", time.Now()), wantWait: true},
{name: "credentials changed long ago", secret: projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))},
{name: "no projected credentials", secret: nil},
{name: "the Job already runs", secret: projectedCredentials("tenant-test", time.Now()), jobExists: true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
now := metav1.Now()
backupJob := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
BackupClassName: "clickhouse-backup",
},
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now},
}
objs := []client.Object{backupJob, newAltinityStrategy("clickhouse-strategy")}
if tc.secret != nil {
objs = append(objs, tc.secret)
}
if tc.jobExists {
objs = append(objs, &batchv1.Job{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}})
}
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), clientfake.NewClientBuilder().WithObjects(objs...))
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
resolved := &ResolvedBackupConfig{
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
}
ctx := context.Background()
j := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
t.Fatalf("get BackupJob: %v", err)
}
res, err := r.reconcileAltinity(ctx, j, resolved)
if err != nil {
t.Fatalf("reconcileAltinity: %v", err)
}
jobs := &batchv1.JobList{}
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
t.Fatalf("list Jobs: %v", err)
}
ready := meta.FindStatusCondition(j.Status.Conditions, "Ready")
if tc.wantWait {
if len(jobs.Items) != 0 {
t.Errorf("a strategy Job was created %d time(s) before the credentials could reach the sidecar", len(jobs.Items))
}
if res.RequeueAfter <= 0 || res.RequeueAfter > altinityCredentialsPropagation {
t.Errorf("RequeueAfter = %v, want within (0, %v]", res.RequeueAfter, altinityCredentialsPropagation)
}
if ready == nil || ready.Reason != "WaitingForCredentials" {
t.Errorf("Ready = %+v, want WaitingForCredentials", ready)
}
return
}
if len(jobs.Items) != 1 {
t.Errorf("got %d strategy Jobs, want 1", len(jobs.Items))
}
if ready != nil && ready.Reason == "WaitingForCredentials" {
t.Errorf("Ready = %+v, the job should not wait", ready)
}
})
}
}
func TestReconcileAltinityRestore_WaitsForProjectedCredentialsToReachTheSidecar(t *testing.T) {
now := metav1.Now()
backup := &backupsv1alpha1.Backup{
ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
TakenAt: now,
},
}
restoreJob := &backupsv1alpha1.RestoreJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}},
Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning},
}
_, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
clientfake.NewClientBuilder().WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob,
projectedCredentials("tenant-test", time.Now())))
rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
ctx := context.Background()
rj := &backupsv1alpha1.RestoreJob{}
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil {
t.Fatalf("get RestoreJob: %v", err)
}
res, err := rr.reconcileAltinityRestore(ctx, rj, backup)
if err != nil {
t.Fatalf("reconcileAltinityRestore: %v", err)
}
jobs := &batchv1.JobList{}
if err := rr.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
t.Fatalf("list Jobs: %v", err)
}
if len(jobs.Items) != 0 {
t.Errorf("a restore Job was created before the credentials could reach the sidecar")
}
if res.RequeueAfter <= 0 {
t.Errorf("RequeueAfter = %v, want a wait", res.RequeueAfter)
}
if ready := meta.FindStatusCondition(rj.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForCredentials" {
t.Errorf("Ready = %+v, want WaitingForCredentials", ready)
}
}
// Without credentials projection configured there is no Secret to wait
// for; client-go refuses a Get with an empty name, so it must not be asked.
func TestReconcileAltinity_NoProjectionConfiguredDoesNotWait(t *testing.T) {
now := metav1.Now()
backupJob := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
BackupClassName: "clickhouse-backup",
},
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now},
}
builder := clientfake.NewClientBuilder().
WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy")).
WithInterceptorFuncs(interceptor.Funcs{
Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
if key.Name == "" {
return errors.New("resource name may not be empty")
}
return cl.Get(ctx, key, obj, opts...)
},
})
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder)
resolved := &ResolvedBackupConfig{
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
}
ctx := context.Background()
j := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
t.Fatalf("get BackupJob: %v", err)
}
if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil {
t.Fatalf("reconcileAltinity: %v", err)
}
jobs := &batchv1.JobList{}
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
t.Fatalf("list Jobs: %v", err)
}
if len(jobs.Items) != 1 {
t.Fatalf("got %d strategy Jobs, want 1", len(jobs.Items))
}
}
func projectedCredentialsStamped(namespace, stamp string) *corev1.Secret {
return &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Namespace: namespace,
Name: "cozy-backups-creds",
Annotations: map[string]string{credentialsUpdatedAtAnnotation: stamp},
},
}
}
func waitingForCredentials() []metav1.Condition {
return []metav1.Condition{credentialsWaitCondition("cozy-backups-creds", time.Minute)}
}
// Once the hold is over and the Job runs, a Ready condition still saying the
// job is waiting for credentials describes a state the job has left.
func TestReconcileAltinity_ClearsTheCredentialsWaitOnceTheJobRuns(t *testing.T) {
now := metav1.Now()
backupJob := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
BackupClassName: "clickhouse-backup",
},
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now, Conditions: waitingForCredentials()},
}
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
clientfake.NewClientBuilder().WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy"),
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
WithStatusSubresource(&backupsv1alpha1.BackupJob{}))
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
resolved := &ResolvedBackupConfig{
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
}
ctx := context.Background()
j := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
t.Fatalf("get BackupJob: %v", err)
}
if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil {
t.Fatalf("reconcileAltinity: %v", err)
}
jobs := &batchv1.JobList{}
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
t.Fatalf("list Jobs: %v", err)
}
if len(jobs.Items) != 1 {
t.Fatalf("got %d strategy Jobs, want 1", len(jobs.Items))
}
stored := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), stored); err != nil {
t.Fatalf("get BackupJob: %v", err)
}
if ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForCredentials" {
t.Errorf("stored Ready = %+v while the Job runs", ready)
}
}
func TestReconcileAltinityRestore_ClearsTheCredentialsWaitOnceTheJobRuns(t *testing.T) {
now := metav1.Now()
backup := &backupsv1alpha1.Backup{
ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
TakenAt: now,
},
}
restoreJob := &backupsv1alpha1.RestoreJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}},
Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning,
Conditions: waitingForCredentials()},
}
_, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
clientfake.NewClientBuilder().WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob,
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}))
rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
ctx := context.Background()
rj := &backupsv1alpha1.RestoreJob{}
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil {
t.Fatalf("get RestoreJob: %v", err)
}
if _, err := rr.reconcileAltinityRestore(ctx, rj, backup); err != nil {
t.Fatalf("reconcileAltinityRestore: %v", err)
}
stored := &backupsv1alpha1.RestoreJob{}
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), stored); err != nil {
t.Fatalf("get RestoreJob: %v", err)
}
if ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForCredentials" {
t.Errorf("stored Ready = %+v while the Job runs", ready)
}
}
// The stamp is only ever rewritten when the projected content changes, so a
// value the hold cannot use must not hold the job: an unreadable one would
// fail every reconcile, and one far in the future would wait again on each.
// A stamp a little ahead, as a skewed clock writes it, still waits, but only
// until that stamp plus the propagation time.
func TestCredentialsPropagationWait_UnusableStampsDoNotHoldTheJob(t *testing.T) {
cases := []struct {
name string
stamp string
wantMin time.Duration
wantMax time.Duration
}{
{name: "unreadable", stamp: "garbage"},
{name: "far in the future", stamp: "3000-01-01T00:00:00Z"},
{name: "slightly ahead", stamp: time.Now().Add(30 * time.Second).UTC().Format(time.RFC3339),
wantMin: altinityCredentialsPropagation, wantMax: altinityCredentialsPropagation + 31*time.Second},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
c := clientfake.NewClientBuilder().WithObjects(projectedCredentialsStamped("tenant-test", tc.stamp)).Build()
wait, err := credentialsPropagationWait(context.Background(), c, "cozy-backups-creds", "tenant-test", "job")
if err != nil {
t.Fatalf("credentialsPropagationWait: %v", err)
}
if wait < tc.wantMin || wait > tc.wantMax {
t.Errorf("wait = %v, want within [%v, %v]", wait, tc.wantMin, tc.wantMax)
}
})
}
}
func TestClearCredentialsWait_KeepsAReadyConditionItDidNotSet(t *testing.T) {
conditions := []metav1.Condition{{Type: "Ready", Status: metav1.ConditionFalse, Reason: "StrategyNotReady"}}
if clearCredentialsWait(&conditions) {
t.Errorf("clearCredentialsWait reported clearing a Ready condition with reason StrategyNotReady")
}
if ready := meta.FindStatusCondition(conditions, "Ready"); ready == nil || ready.Reason != "StrategyNotReady" {
t.Errorf("Ready = %+v, want the StrategyNotReady condition kept", ready)
}
}
// A read the hold cannot complete must not release the Job: starting it
// before the credentials reach the sidecar is the failure the hold prevents.
func TestReconcileAltinity_UnreadableHoldStateDoesNotStartTheJob(t *testing.T) {
cases := []struct {
name string
fail func(key client.ObjectKey, obj client.Object) bool
}{
{name: "credentials Secret", fail: func(key client.ObjectKey, obj client.Object) bool {
_, isSecret := obj.(*corev1.Secret)
return isSecret && key.Name == "cozy-backups-creds"
}},
{name: "strategy Job", fail: func(key client.ObjectKey, obj client.Object) bool {
_, isJob := obj.(*batchv1.Job)
return isJob
}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
now := metav1.Now()
backupJob := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
BackupClassName: "clickhouse-backup",
},
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now},
}
builder := clientfake.NewClientBuilder().
WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy"),
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
WithInterceptorFuncs(interceptor.Funcs{
Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
if tc.fail(key, obj) {
return apierrors.NewInternalError(errors.New("etcd unavailable"))
}
return cl.Get(ctx, key, obj, opts...)
},
})
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder)
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
resolved := &ResolvedBackupConfig{
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
}
ctx := context.Background()
j := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
t.Fatalf("get BackupJob: %v", err)
}
if _, err := r.reconcileAltinity(ctx, j, resolved); err == nil {
t.Errorf("reconcileAltinity returned no error although the %s could not be read", tc.name)
}
jobs := &batchv1.JobList{}
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
t.Fatalf("list Jobs: %v", err)
}
if len(jobs.Items) != 0 {
t.Errorf("a strategy Job was created although the %s could not be read", tc.name)
}
})
}
}
func TestReconcileAltinityRestore_UnreadableHoldStateDoesNotStartTheJob(t *testing.T) {
cases := []struct {
name string
fail func(key client.ObjectKey, obj client.Object) bool
}{
{name: "credentials Secret", fail: func(key client.ObjectKey, obj client.Object) bool {
_, isSecret := obj.(*corev1.Secret)
return isSecret && key.Name == "cozy-backups-creds"
}},
{name: "strategy Job", fail: func(key client.ObjectKey, obj client.Object) bool {
_, isJob := obj.(*batchv1.Job)
return isJob
}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
now := metav1.Now()
backup := &backupsv1alpha1.Backup{
ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
TakenAt: now,
},
}
restoreJob := &backupsv1alpha1.RestoreJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}},
Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning},
}
builder := clientfake.NewClientBuilder().
WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob,
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
WithInterceptorFuncs(interceptor.Funcs{
Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
if tc.fail(key, obj) {
return apierrors.NewInternalError(errors.New("etcd unavailable"))
}
return cl.Get(ctx, key, obj, opts...)
},
})
_, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder)
rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
ctx := context.Background()
rj := &backupsv1alpha1.RestoreJob{}
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil {
t.Fatalf("get RestoreJob: %v", err)
}
if _, err := rr.reconcileAltinityRestore(ctx, rj, backup); err == nil {
t.Errorf("reconcileAltinityRestore returned no error although the %s could not be read", tc.name)
}
jobs := &batchv1.JobList{}
if err := rr.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
t.Fatalf("list Jobs: %v", err)
}
if len(jobs.Items) != 0 {
t.Errorf("a restore Job was created although the %s could not be read", tc.name)
}
})
}
}
// A backup held for the credentials keeps the archive name it was given from
// its persisted StartedAt: the held reconciles write the job's status, and the
// Pod created after the hold, and the Backup recorded after it, must still
// name the same archive, which is stamped no later than the Backup's takenAt.
func TestReconcileAltinity_CredentialsWaitKeepsTheArchiveName(t *testing.T) {
startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC))
backupJob := &backupsv1alpha1.BackupJob{
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
Spec: backupsv1alpha1.BackupJobSpec{
ApplicationRef: newAltinityClickHouseRef("ch-test"),
BackupClassName: "clickhouse-backup",
},
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt, Phase: backupsv1alpha1.BackupJobPhaseRunning},
}
secret := projectedCredentials("tenant-test", time.Now())
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
clientfake.NewClientBuilder().WithObjects(backupJob, newAltinityArchiveStrategy("clickhouse-strategy"), secret))
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
resolved := &ResolvedBackupConfig{
StrategyRef: corev1.TypedLocalObjectReference{
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
Kind: strategyv1alpha1.AltinityStrategyKind,
Name: "clickhouse-strategy",
},
}
ctx := context.Background()
const want = "clickhouse-ch-test-20260304T050607Z"
reconcile := func(step string) {
t.Helper()
j := &backupsv1alpha1.BackupJob{}
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
t.Fatalf("%s: get BackupJob: %v", step, err)
}
if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil {
t.Fatalf("%s: reconcileAltinity: %v", step, err)
}
}
reconcile("held")
job := &batchv1.Job{}
if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); !apierrors.IsNotFound(err) {
t.Fatalf("the Job exists while the credentials wait should hold it: %v", err)
}
if err := r.Get(ctx, client.ObjectKeyFromObject(secret), secret); err != nil {
t.Fatalf("get projected Secret: %v", err)
}
secret.Annotations[credentialsUpdatedAtAnnotation] = time.Now().Add(-altinityCredentialsPropagation - time.Second).UTC().Format(time.RFC3339)
if err := r.Update(ctx, secret); err != nil {
t.Fatalf("age the credentials stamp: %v", err)
}
reconcile("after the wait")
if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); err != nil {
t.Fatalf("get backup Job: %v", err)
}
if got := job.Spec.Template.Spec.Containers[0].Args[1]; got != "--archive="+want {
t.Fatalf("backup Pod was rendered with %q after the wait, want --archive=%s", got, want)
}
job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}}
if err := r.Status().Update(ctx, job); err != nil {
t.Fatalf("mark backup Job complete: %v", err)
}
reconcile("completed")
created := &backupsv1alpha1.Backup{}
if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: backupJob.Name}, created); err != nil {
t.Fatalf("get Backup: %v", err)
}
if got := created.Spec.DriverMetadata[altinityArchiveMetaKey]; got != want {
t.Errorf("Backup records archive %q, want %q", got, want)
}
if stamp := created.Spec.TakenAt.UTC().Format(altinityArchiveTimeLayout); stamp < "20260304T050607Z" {
t.Errorf("Backup takenAt %s is earlier than its archive's stamp", stamp)
}
}
@@ -1,11 +1,14 @@
package backupcontroller
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"net/url"
"strconv"
"strings"
"time"
@@ -15,6 +18,8 @@ import (
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/yaml"
)
// CredentialsProjectionRequeue is the backoff between reconcile attempts
@@ -73,9 +78,8 @@ func IsTransient(err error) bool {
// BackupCredentialsConfig describes the platform-managed source Secret
// produced by the cozy-backups Bucket, the per-tenant projection target
// Secret name, and the bucket coordinates (endpoint used to format the
// FDB blob_credentials account-host key, region carried verbatim for
// drivers like clickhouse-backup that wire it through env, forcePathStyle
// projected as a flat key so the chart-emitted ClickHouse sidecar can read
// FDB blob_credentials account-host key, region carried verbatim,
// forcePathStyle projected so the chart-emitted ClickHouse sidecar can read
// a platform value it cannot see directly). Values are wired from
// environment variables in main() and the deployment chart populates them
// from .Values.backupStorage.
@@ -85,11 +89,11 @@ type BackupCredentialsConfig struct {
TargetSecretName string
Endpoint string
Region string
// ForcePathStyle is the string "true"/"false" from
// backupStorage.forcePathStyle, projected into the credentials Secret
// under the forcePathStyle key. The app charts cannot read
// backupStorage.* directly, so the ClickHouse sidecar consumes it via
// an optional secretKeyRef as S3_FORCE_PATH_STYLE.
// ForcePathStyle is backupStorage.forcePathStyle as the chart
// normalises it ("true", "false", or "" when unset), projected into the credentials Secret
// under the forcePathStyle key and into clickhouse-backup.yml. The app
// charts cannot read backupStorage.* directly, so the ClickHouse
// sidecar takes it from there.
ForcePathStyle string
}
@@ -107,10 +111,11 @@ func (c BackupCredentialsConfig) IsEnabled() bool {
// the data so every default Strategy CR can consume the same Secret:
//
// - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — CNPG, MariaDB, Etcd.
// - accessKey / secretKey + bucketName / endpoint / region — ClickHouse
// sidecar.
// - accessKey / secretKey + bucketName / endpoint / region — Redis
// strategy, and ClickHouse sidecars rendered before clickhouse-backup.yml.
// - cloud — Velero credentials file ([default] section).
// - blob_credentials.json — FoundationDB backup_agent.
// - clickhouse-backup.yml — ClickHouse sidecar, mounted as its config file.
//
// The source Secret is read in two formats:
//
@@ -156,8 +161,8 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr
if creds.endpoint == "" {
// Scheme-strip the env-var fallback to match the source-Secret path
// (parseSourceSecret strips there). The projected `endpoint` key is
// consumed verbatim as S3_ENDPOINT by the ClickHouse sidecar, which
// expects a bare host:port — see https://cozystack.io/docs/v1/operations/services/backup-classes/.
// a bare host:port and each consumer adds its own scheme — see
// https://cozystack.io/docs/v1/operations/services/backup-classes/.
creds.endpoint = stripScheme(cfg.Endpoint)
}
if creds.endpoint == "" {
@@ -177,6 +182,20 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr
}
cloud := buildVeleroCredentialsFile(creds.accessKey, creds.secretKey)
clickHouseConfig, pathStyleErr, err := buildClickHouseBackupConfig(creds, cfg.ForcePathStyle)
if pathStyleErr != nil {
// The chart refuses such a value at render time, so only an env set
// around it gets here. It concerns the ClickHouse file alone, and
// failing the projection would take every other driver's
// credentials down with it.
log.FromContext(ctx).Error(pathStyleErr, "left force_path_style out of "+clickHouseBackupConfigKey, "namespace", targetNamespace)
}
if err != nil {
return &ProjectionError{
Reason: ReasonSourceMalformed,
Message: fmt.Sprintf("build %s: %v", clickHouseBackupConfigKey, err),
}
}
blob, err := buildFDBBlobCredentials(creds.endpoint, creds.accessKey, creds.secretKey)
if err != nil {
return &ProjectionError{
@@ -231,28 +250,36 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr
if target.Data == nil {
target.Data = map[string][]byte{}
}
before := maps.Clone(target.Data)
target.Data["AWS_ACCESS_KEY_ID"] = []byte(creds.accessKey)
target.Data["AWS_SECRET_ACCESS_KEY"] = []byte(creds.secretKey)
// Mirror under the legacy bucket-controller key names so
// chart-emitted sidecars (clickhouse-backup) can consume the
// same Secret without a separate field map.
// Mirror under the legacy bucket-controller key names so the Redis
// strategy and older ClickHouse charts can consume the same Secret
// without a separate field map.
target.Data["accessKey"] = []byte(creds.accessKey)
target.Data["secretKey"] = []byte(creds.secretKey)
target.Data["cloud"] = []byte(cloud)
target.Data["blob_credentials.json"] = blob
target.Data[clickHouseBackupConfigKey] = clickHouseConfig
// Always overwrite endpoint/bucket/region (delete if empty in
// source) so a re-projection cannot leave stale values lingering
// after the bucket-controller rotates the source. Consumers
// (ClickHouse sidecar via secretKeyRef) must see exactly what the
// source carries, not a half-stale Secret.
// after the bucket-controller rotates the source. Consumers must
// see exactly what the source carries, not a half-stale Secret.
setOrDelete(target.Data, "endpoint", creds.endpoint)
setOrDelete(target.Data, "bucketName", creds.bucket)
setOrDelete(target.Data, "region", creds.region)
// forcePathStyle comes from the platform config (backupStorage.
// forcePathStyle), not the source Secret: the ClickHouse sidecar
// reads it as S3_FORCE_PATH_STYLE because the app chart cannot see
// forcePathStyle), not the source Secret: ClickHouse sidecars
// rendered before clickhouse-backup.yml read it as
// S3_FORCE_PATH_STYLE because the app chart cannot see
// backupStorage.* directly.
setOrDelete(target.Data, "forcePathStyle", cfg.ForcePathStyle)
if !maps.EqualFunc(before, target.Data, bytes.Equal) {
if target.Annotations == nil {
target.Annotations = map[string]string{}
}
target.Annotations[credentialsUpdatedAtAnnotation] = time.Now().UTC().Format(time.RFC3339)
}
return nil
}); err != nil {
// If the mutator refused to overwrite an unowned Secret, surface
@@ -414,6 +441,48 @@ func buildFDBBlobCredentials(endpoint, accessKey, secretKey string) ([]byte, err
return json.Marshal(doc)
}
// credentialsUpdatedAtAnnotation records when the projected content last
// changed. A consumer that mounts the Secret as a volume sees a change only
// once kubelet rewrites the volume, so it uses this to tell how fresh its
// copy can be.
const credentialsUpdatedAtAnnotation = "backups.cozystack.io/credentials-updated-at"
// clickHouseBackupConfigKey holds the s3 section of a clickhouse-backup
// config. The ClickHouse chart mounts it as the sidecar's config file
// instead of passing the values as environment: clickhouse-backup reloads
// the file in each API handler that reaches S3, among them the
// create_remote, restore_remote and list calls the strategy makes, and
// kubelet rewrites a mounted Secret in place, so credentials projected
// after the pod started, or rotated since, reach a running sidecar.
const clickHouseBackupConfigKey = "clickhouse-backup.yml"
// buildClickHouseBackupConfig renders that file. The projected endpoint is
// a bare host, and clickhouse-backup's S3 client rejects one without a
// scheme. A forcePathStyle that is not a boolean is left out of the file
// and reported in pathStyleErr; err is a failure to render the file at all.
func buildClickHouseBackupConfig(creds parsedCreds, forcePathStyle string) (config []byte, pathStyleErr, err error) {
s3 := map[string]any{
"bucket": creds.bucket,
"endpoint": "https://" + creds.endpoint,
"access_key": creds.accessKey,
"secret_key": creds.secretKey,
}
// An empty region in the file would override clickhouse-backup's own
// us-east-1 default instead of leaving it in place.
if creds.region != "" {
s3["region"] = creds.region
}
if forcePathStyle != "" {
if pathStyle, perr := strconv.ParseBool(forcePathStyle); perr != nil {
pathStyleErr = fmt.Errorf("forcePathStyle %q is not a boolean", forcePathStyle)
} else {
s3["force_path_style"] = pathStyle
}
}
config, err = yaml.Marshal(map[string]any{"s3": s3})
return config, pathStyleErr, err
}
// stripScheme returns the endpoint stripped of any leading scheme prefix
// (http:// or https://) and preserves the path portion if present. The
// resulting form is what `host[:port][/path]` callers (MariaDB strategy,
@@ -5,7 +5,11 @@ import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/go-logr/logr/funcr"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -13,6 +17,8 @@ import (
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
"sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/yaml"
)
func newFakeClient(objs ...client.Object) client.Client {
@@ -664,3 +670,188 @@ func TestProject_HappyPath_HTTPEndpoint(t *testing.T) {
t.Errorf("blob account host not stripped: %+v", blob.Accounts)
}
}
// The ClickHouse sidecar reads its S3 settings from this key as a mounted
// file, which kubelet refreshes in place: clickhouse-backup reloads its
// config file in each API handler that reaches S3, while its environment
// is fixed at start.
func TestProject_WritesClickHouseBackupConfig(t *testing.T) {
cases := []struct {
name string
forcePathStyle string
wantPathStyle *bool
}{
{name: "path style on", forcePathStyle: "true", wantPathStyle: ptrBool(true)},
{name: "path style off", forcePathStyle: "false", wantPathStyle: ptrBool(false)},
{name: "path style unset", forcePathStyle: ""},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
c := newFakeClient(flatSourceSecret())
ctx := context.Background()
cfg := defaultCfg()
cfg.ForcePathStyle = tc.forcePathStyle
if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
got := &corev1.Secret{}
if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil {
t.Fatalf("projected Secret not found: %v", err)
}
var doc struct {
S3 map[string]any `json:"s3"`
}
if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil {
t.Fatalf("%s is not YAML: %v\n%s", clickHouseBackupConfigKey, err, got.Data[clickHouseBackupConfigKey])
}
for key, want := range map[string]string{
"bucket": "cozy-backups",
"endpoint": "https://seaweedfs-s3.tenant-root.svc:8333",
"region": "us-east-1",
"access_key": "AK",
"secret_key": "SK",
} {
if doc.S3[key] != want {
t.Errorf("s3.%s = %v, want %q", key, doc.S3[key], want)
}
}
pathStyle, set := doc.S3["force_path_style"]
switch {
case tc.wantPathStyle == nil && set:
t.Errorf("s3.force_path_style = %v, want it left to the clickhouse-backup default", pathStyle)
case tc.wantPathStyle != nil && pathStyle != *tc.wantPathStyle:
t.Errorf("s3.force_path_style = %v, want %v", pathStyle, *tc.wantPathStyle)
}
})
}
}
// clickhouse-backup defaults the S3 region to us-east-1; an empty value in
// its config file would replace that default rather than fall back to it.
func TestProject_ClickHouseBackupConfigOmitsAnEmptyRegion(t *testing.T) {
c := newFakeClient(flatSourceSecret())
ctx := context.Background()
cfg := defaultCfg()
cfg.Region = ""
if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
got := &corev1.Secret{}
if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil {
t.Fatalf("projected Secret not found: %v", err)
}
var doc struct {
S3 map[string]any `json:"s3"`
}
if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil {
t.Fatalf("%s is not YAML: %v", clickHouseBackupConfigKey, err)
}
if region, set := doc.S3["region"]; set {
t.Errorf("s3.region = %q, want it left to the clickhouse-backup default", region)
}
}
// The parse only serves the ClickHouse file, while the same projection
// carries every other driver's credentials. An unparseable platform value
// must cost ClickHouse its path-style setting and say so, not fail the
// projection for all of them.
func TestProject_MalformedForcePathStyleSkipsOnlyTheClickHouseSetting(t *testing.T) {
c := newFakeClient(flatSourceSecret())
var logged []string
ctx := log.IntoContext(context.Background(), funcr.New(func(prefix, args string) {
logged = append(logged, args)
}, funcr.Options{}))
cfg := defaultCfg()
cfg.ForcePathStyle = "yes"
if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil {
t.Fatalf("projection failed for every driver over a ClickHouse-only setting: %v", err)
}
got := &corev1.Secret{}
if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil {
t.Fatalf("projected Secret not found: %v", err)
}
if string(got.Data["AWS_ACCESS_KEY_ID"]) != "AK" {
t.Errorf("AWS_ACCESS_KEY_ID = %q, want the projected key", got.Data["AWS_ACCESS_KEY_ID"])
}
if string(got.Data["forcePathStyle"]) != "yes" {
t.Errorf("forcePathStyle = %q, want the platform value carried verbatim", got.Data["forcePathStyle"])
}
var doc struct {
S3 map[string]any `json:"s3"`
}
if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil {
t.Fatalf("%s is not YAML: %v", clickHouseBackupConfigKey, err)
}
if doc.S3["access_key"] != "AK" {
t.Errorf("s3.access_key = %v, want the rest of the file written", doc.S3["access_key"])
}
if pathStyle, set := doc.S3["force_path_style"]; set {
t.Errorf("s3.force_path_style = %v, want it left out", pathStyle)
}
found := false
for _, line := range logged {
if strings.Contains(line, `"error"`) && strings.Contains(line, `\"yes\"`) && strings.Contains(line, "force_path_style") {
found = true
}
}
if !found {
t.Errorf("no error naming the rejected value and the skipped setting was logged; got %q", logged)
}
}
func ptrBool(b bool) *bool { return &b }
// Consumers that read the Secret through a mounted volume need to know when
// its content last changed; an unchanged re-projection must keep the time.
func TestProject_StampsWhenContentChanges(t *testing.T) {
src := flatSourceSecret()
c := newFakeClient(src)
ctx := context.Background()
key := types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}
stamp := func() string {
got := &corev1.Secret{}
if err := c.Get(ctx, key, got); err != nil {
t.Fatalf("projected Secret not found: %v", err)
}
return got.Annotations[credentialsUpdatedAtAnnotation]
}
if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil {
t.Fatalf("first projection: %v", err)
}
first := stamp()
at, err := time.Parse(time.RFC3339, first)
if err != nil || time.Since(at) > time.Minute {
t.Fatalf("first projection stamp %q is not a recent RFC3339 time: %v", first, err)
}
old := &corev1.Secret{}
if err := c.Get(ctx, key, old); err != nil {
t.Fatalf("get projected Secret: %v", err)
}
old.Annotations[credentialsUpdatedAtAnnotation] = "2000-01-01T00:00:00Z"
if err := c.Update(ctx, old); err != nil {
t.Fatalf("age the stamp: %v", err)
}
if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil {
t.Fatalf("unchanged re-projection: %v", err)
}
if got := stamp(); got != "2000-01-01T00:00:00Z" {
t.Fatalf("an unchanged re-projection moved the stamp to %q", got)
}
rotated := &corev1.Secret{}
if err := c.Get(ctx, types.NamespacedName{Namespace: src.Namespace, Name: src.Name}, rotated); err != nil {
t.Fatalf("get source Secret: %v", err)
}
rotated.Data["secretKey"] = []byte("SK2")
if err := c.Update(ctx, rotated); err != nil {
t.Fatalf("rotate source Secret: %v", err)
}
if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil {
t.Fatalf("rotated re-projection: %v", err)
}
if got := stamp(); got == "2000-01-01T00:00:00Z" {
t.Fatalf("a rotation did not move the stamp")
}
}
@@ -225,15 +225,20 @@ spec:
- name: REMOTE_STORAGE
value: s3
{{- /*
On the useSystemBucket flow every S3 setting but the
release's own S3_PATH below, the platform's
backupStorage.forcePathStyle included, comes
from the clickhouse-backup.yml key the platform projects
into cozy-backups-creds, mounted below as the config file.
clickhouse-backup reloads that file in each API handler
that reaches S3 and kubelet rewrites a mounted Secret in
place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar; environment
would be read only at start and would override the file,
so none of those is set here.
S3_FORCE_PATH_STYLE selects path-style vs vhost-style S3
addressing: SeaweedFS (the platform default) needs
path-style, while AWS public S3 needs vhost-style in many
regions. On the useSystemBucket flow the value is the
platform's backupStorage.forcePathStyle, which this app
chart cannot read directly, so it is projected into
cozy-backups-creds and consumed via an optional
secretKeyRef (optional: true so the Pod starts before the
first projection). On the legacy paths (own Secret via
addressing. On the legacy paths (own Secret via
s3CredentialsSecret.name, or a configured endpoint) we
cannot peek inside the Secret at render time, so we keep
the historical path-style default — AWS users in that mode
@@ -241,12 +246,8 @@ spec:
clickhouse-backup config.
*/}}
{{- if .Values.backup.useSystemBucket }}
- name: S3_FORCE_PATH_STYLE
valueFrom:
secretKeyRef:
name: {{ $sidecarSecretName | quote }}
key: forcePathStyle
optional: true
- name: CLICKHOUSE_BACKUP_CONFIG
value: /etc/clickhouse-backup-s3/config.yml
{{- else if or (not (eq (default "" .Values.backup.s3CredentialsSecret.name) "")) .Values.backup.endpoint }}
- name: S3_FORCE_PATH_STYLE
value: "true"
@@ -282,44 +283,18 @@ spec:
*/}}
- name: S3_PATH
value: {{ $s3Path | quote }}
{{- /*
optional: true in the useSystemBucket path so the Pod
starts even before the platform projector materialises
cozy-backups-creds in this namespace (projection runs at
first BackupJob time; the sidecar surfaces backup errors
only when its HTTP API is exercised). Without this gate
every fresh ClickHouse release with useSystemBucket=true
would hang in CreateContainerConfigError until a
BackupJob fires. Legacy path keeps the strict reference
shape — its Secret is chart-emitted and present at
install time.
*/}}
{{- $secretOptional := .Values.backup.useSystemBucket }}
{{- if not .Values.backup.useSystemBucket }}
- name: S3_BUCKET
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $bucketKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
{{- /*
clickhouse-backup's aws-sdk-go-v2 client rejects a scheme-less
S3_ENDPOINT ("was not a valid URI"). On the useSystemBucket flow
the projected cozy-backups-creds.endpoint is a bare host, so
prepend https:// (the platform's CNPG endpoint helper does the
same). Legacy / BYO-Secret endpoints are tenant-supplied with
their own scheme and are consumed verbatim.
*/}}
{{- if .Values.backup.useSystemBucket }}
- name: S3_ENDPOINT_HOST
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $endpointKey | quote }}, optional: true } }
- name: S3_ENDPOINT
value: "https://$(S3_ENDPOINT_HOST)"
{{- else }}
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $bucketKey | quote }} } }
- name: S3_ENDPOINT
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $endpointKey | quote }} } }
{{- end }}
- name: S3_REGION
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $regionKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $regionKey | quote }} } }
- name: S3_ACCESS_KEY
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $accessKeyKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $accessKeyKey | quote }} } }
- name: S3_SECRET_KEY
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $secretKeyKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $secretKeyKey | quote }} } }
{{- end }}
{{- if $endpointCAName }}
# Trust the mounted private-CA bundle for the S3 endpoint (see
# the `backup.endpointCA` field in values.yaml). SSL_CERT_DIR,
@@ -343,6 +318,11 @@ spec:
- name: backup-s3-ca
mountPath: /etc/ssl/cozy-s3-ca
readOnly: true
{{- end }}
{{- if .Values.backup.useSystemBucket }}
- name: backup-s3-config
mountPath: /etc/clickhouse-backup-s3
readOnly: true
{{- end }}
securityContext:
# clickhouse-server stores data under /var/lib/clickhouse with
@@ -366,13 +346,14 @@ spec:
runAsGroup: 0
seccompProfile:
type: RuntimeDefault
{{- if $endpointCAName }}
# Private-CA bundle for the backup sidecar, projected as ca.crt for
# SSL_CERT_DIR above. Not optional on purpose: a missing Secret holds
# the whole ClickHouse Pod in ContainerCreating (naming the Secret in
# `kubectl describe`) rather than mounting an empty trust dir and
# failing later inside clickhouse-backup with an opaque x509 error.
{{- if or $endpointCAName .Values.backup.useSystemBucket }}
volumes:
{{- if $endpointCAName }}
# Private-CA bundle for the backup sidecar, projected as ca.crt for
# SSL_CERT_DIR above. Not optional on purpose: a missing Secret holds
# the whole ClickHouse Pod in ContainerCreating (naming the Secret in
# `kubectl describe`) rather than mounting an empty trust dir and
# failing later inside clickhouse-backup with an opaque x509 error.
- name: backup-s3-ca
secret:
secretName: {{ $endpointCAName | quote }}
@@ -380,6 +361,18 @@ spec:
- key: {{ $endpointCAKey | quote }}
path: ca.crt
{{- end }}
{{- if .Values.backup.useSystemBucket }}
# Optional: the platform projects cozy-backups-creds into this
# namespace at the first BackupJob, after the Pod has started.
- name: backup-s3-config
secret:
secretName: {{ $sidecarSecretName | quote }}
optional: true
items:
- key: clickhouse-backup.yml
path: config.yml
{{- end }}
{{- end }}
{{- end }}
serviceTemplates:
- name: svc-template
+44 -34
View File
@@ -2,10 +2,12 @@ suite: backup sidecar S3 wiring (useSystemBucket vs legacy, path-override guard)
# Locks in the clickhouse-backup sidecar S3 wiring this PR introduced on the
# ClickHouseInstallation PodTemplate:
# - useSystemBucket=true points every S3_* env at the platform-projected
# cozy-backups-creds Secret with optional:true (the Pod starts before the
# first projection) and scopes S3_PATH to <namespace>/<release>; the
# chart-managed <release>-backup-s3 / <release>-backup Secrets are skipped;
# - useSystemBucket=true mounts the clickhouse-backup.yml key of the
# platform-projected cozy-backups-creds Secret as the sidecar's config
# file with optional:true (the Pod starts before the first projection),
# sets no other S3 setting in the environment, and scopes S3_PATH to
# <namespace>/<release>; the chart-managed <release>-backup-s3 /
# <release>-backup Secrets are skipped;
# - the legacy flow keeps the chart-emitted <release>-backup-s3 Secret and a
# strict (non-optional) secretKeyRef;
# - s3PathOverride + useSystemBucket fails fast at render time.
@@ -25,7 +27,7 @@ set:
_cluster: {}
tests:
- it: "useSystemBucket=true: sidecar reads cozy-backups-creds (optional) and scopes S3_PATH"
- it: "useSystemBucket=true: sidecar reads its S3 settings from the projected config file and scopes S3_PATH"
template: templates/clickhouse.yaml
documentSelector:
path: kind
@@ -45,28 +47,30 @@ tests:
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_PATH")].value
value: tenant-test/ch-test
# clickhouse-backup reloads this file in each API handler that reaches
# S3 and kubelet rewrites the mounted Secret in place, so credentials
# projected after the Pod started, or rotated since, reach the running
# sidecar.
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.name
value: cozy-backups-creds
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="CLICKHOUSE_BACKUP_CONFIG")].value
value: /etc/clickhouse-backup-s3/config.yml
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.optional
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].volumeMounts[?(@.name=="backup-s3-config")].mountPath
value: /etc/clickhouse-backup-s3
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].volumeMounts[?(@.name=="backup-s3-config")].readOnly
value: true
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_SECRET_KEY")].valueFrom.secretKeyRef.name
value: cozy-backups-creds
# The projected endpoint is a bare host; clickhouse-backup needs a URI, so
# the chart prepends https:// via env-var expansion on this flow.
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")].valueFrom.secretKeyRef.name
value: cozy-backups-creds
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")].valueFrom.secretKeyRef.key
value: endpoint
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT")].value
value: https://$(S3_ENDPOINT_HOST)
path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-config")].secret
value:
secretName: cozy-backups-creds
# The Pod starts before the first projection.
optional: true
items:
- key: clickhouse-backup.yml
path: config.yml
- it: "useSystemBucket=true: S3_FORCE_PATH_STYLE is sourced from cozy-backups-creds, not hardcoded"
- it: "useSystemBucket=true: no S3 setting is fixed in the sidecar environment"
template: templates/clickhouse.yaml
documentSelector:
path: kind
@@ -76,18 +80,21 @@ tests:
enabled: true
useSystemBucket: true
asserts:
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.name
value: cozy-backups-creds
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.key
value: forcePathStyle
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.optional
value: true
# Environment overrides the config file and is read only at start.
- notExists:
# must no longer be the hardcoded literal value: "true"
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].value
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")]
- notExists:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT")]
- notExists:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")]
- notExists:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_REGION")]
- notExists:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ACCESS_KEY")]
- notExists:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_SECRET_KEY")]
- notExists:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")]
- it: "useSystemBucket=true: chart-managed backup Secrets are skipped"
set:
@@ -240,7 +247,10 @@ tests:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="SSL_CERT_DIR")].value
value: /etc/ssl/cozy-s3-ca:/etc/ssl/certs
- equal:
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.name
path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-ca")].secret.secretName
value: my-s3-ca
- equal:
path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-config")].secret.secretName
value: cozy-backups-creds
# backup.enabled=false renders no sidecar at all, so the CA must not leak a
@@ -190,3 +190,28 @@ true
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
backupStorage.forcePathStyle as "true", "false", or "" when unset. Its
consumers read it differently: the controller and Velero parse it with
strconv.ParseBool, the etcd and MongoDB strategies write it into a boolean
field that Helm decodes as YAML 1.1 (so "yes" or "off" is a boolean there),
and the Kafka and RabbitMQ scripts compare the string with "false".
Normalising it here means a spelling cannot mean path style to some drivers
and not to others. The spellings accepted are those of strconv.ParseBool and
of YAML 1.1 booleans, a superset of the boolean spellings the etcd and
MongoDB strategies accepted; any other value fails the render.
*/}}
{{- define "backupstrategy-controller.forcePathStyle" -}}
{{- $value := .Values.backupStorage.forcePathStyle -}}
{{- if not (kindIs "invalid" $value) -}}
{{- $raw := toString $value -}}
{{- if has $raw (list "1" "t" "T" "true" "True" "TRUE" "y" "Y" "yes" "Yes" "YES" "on" "On" "ON") -}}
true
{{- else if has $raw (list "0" "f" "F" "false" "False" "FALSE" "n" "N" "no" "No" "NO" "off" "Off" "OFF") -}}
false
{{- else if ne $raw "" -}}
{{- fail (printf "backupStorage.forcePathStyle must be a boolean or unset, got %q" $raw) -}}
{{- end -}}
{{- end -}}
{{- end -}}
@@ -47,7 +47,7 @@ spec:
- name: BACKUP_STORAGE_REGION
value: {{ .Values.backupStorage.region | quote }}
- name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: {{ .Values.backupStorage.forcePathStyle | quote }}
value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
- name: BACKUP_STORAGE_SYSTEM_NAMESPACES
value: {{ .Values.backupStorage.systemNamespaces | join "," | quote }}
{{- if .Values.backupStorage.reconcileDefaultObjects }}
@@ -15,7 +15,7 @@ spec:
endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }}
key: {{ printf "{{ .Application.metadata.namespace }}/{{ .Application.metadata.name }}/" | quote }}
region: {{ .Values.backupStorage.region | quote }}
forcePathStyle: {{ .Values.backupStorage.forcePathStyle }}
forcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . }}
# The etcd-operator reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
# from this Secret by default (no explicit key fields like the
# MariaDB strategy's *SecretKeyRef). The projector writes exactly
@@ -69,7 +69,7 @@ spec:
- name: S3_REGION
value: {{ .Values.backupStorage.region | quote }}
- name: S3_FORCE_PATH_STYLE
value: {{ .Values.backupStorage.forcePathStyle | quote }}
value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
command: ["/usr/bin/bash", "-c"]
# The driver script is kept as a standalone shell file and
# embedded verbatim so hack/kafka-backup-script.bats can execute
@@ -76,6 +76,6 @@ spec:
prefix: {{ printf "{{ .Application.metadata.namespace }}/{{ .Application.metadata.name }}" | quote }}
region: {{ .Values.backupStorage.region | quote }}
credentialsSecret: cozy-backups-creds
forcePathStyle: {{ .Values.backupStorage.forcePathStyle }}
forcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . }}
insecureSkipTLSVerify: {{ .Values.backupStorage.insecureSkipTLSVerify }}
{{- end -}}
@@ -98,7 +98,7 @@ spec:
- name: S3_REGION
value: {{ .Values.backupStorage.region | quote }}
- name: S3_FORCE_PATH_STYLE
value: {{ .Values.backupStorage.forcePathStyle | quote }}
value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
command: ["/bin/sh", "-c"]
args:
- |
@@ -31,7 +31,7 @@ spec:
key: cloud
config:
region: {{ .Values.backupStorage.region | quote }}
s3ForcePathStyle: {{ .Values.backupStorage.forcePathStyle | quote }}
s3ForcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
s3Url: {{ include "backupstrategy-controller.endpoint" . | quote }}
{{- end }}
{{- end }}
@@ -54,3 +54,12 @@ tests:
name: BACKUP_STORAGE_SECRET_NAMESPACE
value: tenant-root
any: true
# A value no consumer took as a boolean is refused once, at render time; see
# tests/force_path_style_test.yaml for what each consumer receives.
- it: "forcePathStyle that is no boolean spelling fails the render and names the value"
set:
backupStorage.forcePathStyle: "maybe"
asserts:
- failedTemplate:
errorPattern: 'backupStorage\.forcePathStyle must be a boolean or unset, got "maybe"'
@@ -0,0 +1,501 @@
suite: forcePathStyle — one normalised boolean for every consumer
# backupStorage.forcePathStyle reaches drivers that read it differently: the
# controller and Velero parse it with strconv.ParseBool, the etcd and MongoDB
# strategies write it into a boolean field Helm decodes as YAML 1.1, and the
# Kafka and RabbitMQ scripts compare the string with "false". The chart accepts
# the spellings of both strconv.ParseBool and YAML 1.1, hands each consumer
# "true" or "false", and refuses anything else, so a spelling cannot mean path
# style to one driver and not to another.
templates:
- templates/deployment.yaml
- templates/strategy-etcd-default.yaml
- templates/strategy-mongodb-default.yaml
- templates/strategy-kafka-default.yaml
- templates/strategy-rabbitmq-default.yaml
- templates/velero-bsl.yaml
release:
name: t
namespace: cozy-backup-controller
set:
backupStorage.bucketNameOverride: test-bucket
tests:
- it: "forcePathStyle '1' reaches the controller as true"
set:
backupStorage.forcePathStyle: "1"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 't' reaches the controller as true"
set:
backupStorage.forcePathStyle: "t"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'T' reaches every consumer as true"
set:
backupStorage.forcePathStyle: "T"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- equal:
path: spec.template.destination.s3.forcePathStyle
value: true
template: templates/strategy-etcd-default.yaml
- equal:
path: spec.template.s3.forcePathStyle
value: true
template: templates/strategy-mongodb-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "true"
template: templates/strategy-kafka-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "true"
template: templates/strategy-rabbitmq-default.yaml
- equal:
path: spec.config.s3ForcePathStyle
value: "true"
template: templates/velero-bsl.yaml
- it: "forcePathStyle 'true' reaches the controller as true"
set:
backupStorage.forcePathStyle: "true"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'True' reaches the controller as true"
set:
backupStorage.forcePathStyle: "True"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'TRUE' reaches the controller as true"
set:
backupStorage.forcePathStyle: "TRUE"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'y' reaches the controller as true"
set:
backupStorage.forcePathStyle: "y"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'Y' reaches the controller as true"
set:
backupStorage.forcePathStyle: "Y"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'yes' reaches every consumer as true"
set:
backupStorage.forcePathStyle: "yes"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- equal:
path: spec.template.destination.s3.forcePathStyle
value: true
template: templates/strategy-etcd-default.yaml
- equal:
path: spec.template.s3.forcePathStyle
value: true
template: templates/strategy-mongodb-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "true"
template: templates/strategy-kafka-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "true"
template: templates/strategy-rabbitmq-default.yaml
- equal:
path: spec.config.s3ForcePathStyle
value: "true"
template: templates/velero-bsl.yaml
- it: "forcePathStyle 'Yes' reaches the controller as true"
set:
backupStorage.forcePathStyle: "Yes"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'YES' reaches the controller as true"
set:
backupStorage.forcePathStyle: "YES"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'on' reaches the controller as true"
set:
backupStorage.forcePathStyle: "on"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'On' reaches the controller as true"
set:
backupStorage.forcePathStyle: "On"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle 'ON' reaches the controller as true"
set:
backupStorage.forcePathStyle: "ON"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- it: "forcePathStyle '0' reaches the controller as false"
set:
backupStorage.forcePathStyle: "0"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'f' reaches the controller as false"
set:
backupStorage.forcePathStyle: "f"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'F' reaches the controller as false"
set:
backupStorage.forcePathStyle: "F"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'false' reaches the controller as false"
set:
backupStorage.forcePathStyle: "false"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'False' reaches every consumer as false"
set:
backupStorage.forcePathStyle: "False"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- equal:
path: spec.template.destination.s3.forcePathStyle
value: false
template: templates/strategy-etcd-default.yaml
- equal:
path: spec.template.s3.forcePathStyle
value: false
template: templates/strategy-mongodb-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "false"
template: templates/strategy-kafka-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "false"
template: templates/strategy-rabbitmq-default.yaml
- equal:
path: spec.config.s3ForcePathStyle
value: "false"
template: templates/velero-bsl.yaml
- it: "forcePathStyle 'FALSE' reaches the controller as false"
set:
backupStorage.forcePathStyle: "FALSE"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'n' reaches the controller as false"
set:
backupStorage.forcePathStyle: "n"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'N' reaches the controller as false"
set:
backupStorage.forcePathStyle: "N"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'no' reaches the controller as false"
set:
backupStorage.forcePathStyle: "no"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'No' reaches the controller as false"
set:
backupStorage.forcePathStyle: "No"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'NO' reaches the controller as false"
set:
backupStorage.forcePathStyle: "NO"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'off' reaches every consumer as false"
set:
backupStorage.forcePathStyle: "off"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- equal:
path: spec.template.destination.s3.forcePathStyle
value: false
template: templates/strategy-etcd-default.yaml
- equal:
path: spec.template.s3.forcePathStyle
value: false
template: templates/strategy-mongodb-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "false"
template: templates/strategy-kafka-default.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "false"
template: templates/strategy-rabbitmq-default.yaml
- equal:
path: spec.config.s3ForcePathStyle
value: "false"
template: templates/velero-bsl.yaml
- it: "forcePathStyle 'Off' reaches the controller as false"
set:
backupStorage.forcePathStyle: "Off"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle 'OFF' reaches the controller as false"
set:
backupStorage.forcePathStyle: "OFF"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "false"
template: templates/deployment.yaml
- it: "forcePathStyle as a YAML boolean (the default) reaches every consumer as true"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: "true"
template: templates/deployment.yaml
- equal:
path: spec.template.destination.s3.forcePathStyle
value: true
template: templates/strategy-etcd-default.yaml
- it: "forcePathStyle as a YAML boolean false reaches the scripts as false"
set:
backupStorage.forcePathStyle: false
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: S3_FORCE_PATH_STYLE
value: "false"
template: templates/strategy-kafka-default.yaml
- equal:
path: spec.template.s3.forcePathStyle
value: false
template: templates/strategy-mongodb-default.yaml
- it: "forcePathStyle as the number 1 reaches the boolean fields as true"
set:
backupStorage.forcePathStyle: 1
asserts:
- equal:
path: spec.template.destination.s3.forcePathStyle
value: true
template: templates/strategy-etcd-default.yaml
# Unset leaves each driver its own default, as before.
- it: "forcePathStyle null stays unset for every consumer"
set:
backupStorage.forcePathStyle: null
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: ""
template: templates/deployment.yaml
- equal:
path: spec.template.destination.s3.forcePathStyle
value: null
template: templates/strategy-etcd-default.yaml
- equal:
path: spec.template.s3.forcePathStyle
value: null
template: templates/strategy-mongodb-default.yaml
- it: "forcePathStyle empty stays unset for every consumer"
set:
backupStorage.forcePathStyle: ""
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BACKUP_STORAGE_FORCE_PATH_STYLE
value: ""
template: templates/deployment.yaml
- equal:
path: spec.template.destination.s3.forcePathStyle
value: null
template: templates/strategy-etcd-default.yaml
@@ -105,10 +105,6 @@ backupStorage:
endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333
# region is re-projected into cozy-backups-creds on the next reconcile
# (1-minute tick for system namespaces, on-demand for tenant namespaces).
# However, charts that consume the region via secretKeyRef in env vars —
# the chart-emitted ClickHouse sidecar on `useSystemBucket: true` is the
# only such consumer today — only pick up the new value on Pod restart.
# Plan a rollout of those Pods when you change this value.
region: us-east-1
forcePathStyle: true
# systemSecretName — the human-friendly Secret produced by the Bucket