mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
23 files changed
+2347
-120
No files matched your search
@@ -0,0 +1,263 @@
|
||||
#!/usr/bin/env bats
|
||||
# The clickhouse-3-system-bucket-roundtrip Test edits the host CoreDNS Corefile
|
||||
# and must put it back. Its restore runs from an EXIT trap under `set +e`, so a
|
||||
# restore that fails without saying so leaves the rewrite in a cluster-wide
|
||||
# object and still reports a green run; the next run then refuses to start.
|
||||
# These run the restore against a stubbed kubectl.
|
||||
|
||||
load test_helper
|
||||
|
||||
LIB=hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh
|
||||
HELPERS=examples/backups/clickhouse/00-helpers.sh
|
||||
|
||||
setup() {
|
||||
strict_setup
|
||||
STUB=$(mktemp -d)
|
||||
# A Corefile carrying the Test's marked block, as the run step leaves it.
|
||||
jq -n '{apiVersion: "v1", kind: "ConfigMap",
|
||||
metadata: {name: "coredns", namespace: "kube-system", resourceVersion: "7"},
|
||||
data: {Corefile: ".:53 {\n # === BEGIN e2e-clickhouse-system-bucket ===\n rewrite name exact s3.example.org root-ingress-controller.tenant-root.svc.cozy.local\n # === END e2e-clickhouse-system-bucket ===\n errors\n}\n"}}' \
|
||||
>"$STUB/coredns.json"
|
||||
# The secret lookup follows kubectl's contract: a name together with a
|
||||
# selector is refused, and a selector-only lookup prints the names in
|
||||
# OWNED_SECRETS, the Secrets carrying the owner label.
|
||||
cat >"$STUB/kubectl" <<'EOF'
|
||||
#!/bin/bash
|
||||
echo "$*" >>"$STUB/calls"
|
||||
case "$*" in
|
||||
*"get configmap coredns -o json"*) cat "$STUB/coredns.json" ;;
|
||||
replace*)
|
||||
cp "${@: -1}" "$STUB/replaced.json"
|
||||
if [ -n "${REPLACE_FAILS:-}" ]; then
|
||||
echo 'Error from server (Conflict): Operation cannot be fulfilled on configmaps "coredns": the object has been modified' >&2
|
||||
exit 1
|
||||
fi
|
||||
;;
|
||||
*"get secret"*)
|
||||
# Positionals are "get", "secret" and then any names.
|
||||
words=0 selector=
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-l|--selector) selector=$2; shift 2 ;;
|
||||
-n|--namespace|-o|--output) shift 2 ;;
|
||||
-*) shift ;;
|
||||
*) words=$((words + 1)); shift ;;
|
||||
esac
|
||||
done
|
||||
if [ -n "${LOOKUP_FAILS:-}" ]; then
|
||||
echo 'Unable to connect to the server: dial tcp: i/o timeout' >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ -n "$selector" ] && [ "$words" -gt 2 ]; then
|
||||
echo 'error: name cannot be provided when a selector is specified' >&2
|
||||
exit 1
|
||||
fi
|
||||
[ "$selector" = cozystack.io/e2e-owner=clickhouse-system-bucket ] || exit 0
|
||||
for s in ${OWNED_SECRETS:-}; do echo "secret/$s"; done
|
||||
;;
|
||||
*"delete secret "*)
|
||||
if [ -n "${DELETE_FAILS:-}" ]; then
|
||||
echo 'Error from server (Forbidden): secrets "seaweedfs-s3-ingress-tls" is forbidden' >&2
|
||||
exit 1
|
||||
fi
|
||||
;;
|
||||
*) echo "unexpected kubectl $*" >&2; exit 2 ;;
|
||||
esac
|
||||
EOF
|
||||
chmod +x "$STUB/kubectl"
|
||||
export STUB
|
||||
}
|
||||
|
||||
restore() {
|
||||
PATH="$STUB:$PATH" NAMESPACE=tenant-root bash -c '
|
||||
set -euo pipefail
|
||||
source "$1"
|
||||
source "$2"
|
||||
set +e
|
||||
restore_shared_state
|
||||
' _ "$HELPERS" "$LIB"
|
||||
}
|
||||
|
||||
@test "a CoreDNS restore the API server refuses fails the restore" {
|
||||
export REPLACE_FAILS=1
|
||||
err=$(mktemp)
|
||||
if restore 2>"$err"; then
|
||||
echo "restore_shared_state returned 0 although kubectl replace was refused" >&2
|
||||
cat "$err" >&2
|
||||
false
|
||||
fi
|
||||
if grep -q 'Removed the s3.example.org rewrite' "$err"; then
|
||||
echo "restore_shared_state reported the rewrite removed although kubectl replace was refused:" >&2
|
||||
cat "$err" >&2
|
||||
false
|
||||
fi
|
||||
}
|
||||
|
||||
@test "a CoreDNS restore sends the Corefile back without the marked block" {
|
||||
restore
|
||||
jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null || {
|
||||
echo "restored Corefile:" >&2
|
||||
jq -r '.data.Corefile' "$STUB/replaced.json" >&2
|
||||
false
|
||||
}
|
||||
# The resourceVersion read travels with the replace, so a concurrent
|
||||
# writer makes it fail rather than be overwritten.
|
||||
[ "$(jq -r '.metadata.resourceVersion' "$STUB/replaced.json")" = 7 ]
|
||||
}
|
||||
|
||||
@test "the restore deletes the Ingress TLS Secret the Test created" {
|
||||
OWNED_SECRETS=seaweedfs-s3-ingress-tls restore
|
||||
grep -qE '^-n tenant-root delete secret seaweedfs-s3-ingress-tls( |$)' "$STUB/calls" || {
|
||||
echo "restore_shared_state did not delete the Secret carrying the owner label; kubectl calls:" >&2
|
||||
cat "$STUB/calls" >&2
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
@test "the restore leaves an Ingress TLS Secret without the owner label alone" {
|
||||
for owned in '' e2e-other seaweedfs-s3-ingress-tls-old; do
|
||||
: >"$STUB/calls"
|
||||
OWNED_SECRETS=$owned restore
|
||||
if grep -q 'delete secret' "$STUB/calls"; then
|
||||
echo "restore_shared_state deleted a Secret although the labelled ones are '$owned'; kubectl calls:" >&2
|
||||
cat "$STUB/calls" >&2
|
||||
false
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
@test "a Secret delete the API server refuses fails the restore" {
|
||||
err=$(mktemp)
|
||||
if OWNED_SECRETS=seaweedfs-s3-ingress-tls DELETE_FAILS=1 restore 2>"$err"; then
|
||||
echo "restore_shared_state returned 0 although kubectl delete secret was refused" >&2
|
||||
cat "$err" >&2
|
||||
false
|
||||
fi
|
||||
grep -qE '^-n tenant-root delete secret seaweedfs-s3-ingress-tls( |$)' "$STUB/calls" || {
|
||||
echo "restore_shared_state failed before it reached the Secret delete; kubectl calls:" >&2
|
||||
cat "$STUB/calls" "$err" >&2
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
@test "a Secret lookup that fails fails the restore" {
|
||||
err=$(mktemp)
|
||||
if LOOKUP_FAILS=1 restore 2>"$err"; then
|
||||
echo "restore_shared_state returned 0 although it could not tell whether the Secret is left over" >&2
|
||||
cat "$err" >&2
|
||||
false
|
||||
fi
|
||||
grep -q 'could not list the Secrets labelled' "$err" || {
|
||||
echo "restore_shared_state failed without naming the Secret lookup:" >&2
|
||||
cat "$err" >&2
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
@test "the run step's finally restores the shared objects" {
|
||||
# When the step ceiling fires, Chainsaw kills the shell and the EXIT trap
|
||||
# of system-bucket-roundtrip.sh never runs; the step's finally still does.
|
||||
# Chainsaw runs a script's content with sh from the suite's directory.
|
||||
suite=hack/e2e-chainsaw/clickhouse
|
||||
finally=$(yq 'select(.metadata.name == "clickhouse-3-system-bucket-roundtrip")
|
||||
| .spec.steps[] | select(.name == "run") | .finally[]
|
||||
| select(.description == "restore the host CoreDNS and the S3 Ingress TLS Secret") | .script.content' \
|
||||
"$suite/chainsaw-test.yaml")
|
||||
[ -n "$finally" ] || { echo "the run step has no finally restoring the shared objects" >&2; false; }
|
||||
(cd "$suite" && PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$finally")
|
||||
[ -f "$STUB/replaced.json" ] || {
|
||||
echo "the run step's finally did not put the Corefile back" >&2
|
||||
false
|
||||
}
|
||||
jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null
|
||||
}
|
||||
|
||||
# The Altinity driver names the Backup after the BackupJob, and a Backup left
|
||||
# behind under that name records an earlier run's archive, which fails the next
|
||||
# run's BackupJob. It is reclaimed by name, whether or not the BackupJob
|
||||
# recorded it in status.backupRef.
|
||||
@test "the round-trip reclaims its Backup by name before and after the run" {
|
||||
suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml
|
||||
test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]'
|
||||
calls="$STUB/calls"
|
||||
# Nothing is left over from an earlier run, and no BackupJob recorded a
|
||||
# backupRef.
|
||||
cat >"$STUB/kubectl" <<'STUBEOF'
|
||||
#!/bin/bash
|
||||
echo "$*" >>"$STUB/calls"
|
||||
case "$*" in
|
||||
*" get secret "*) exit 1 ;;
|
||||
esac
|
||||
STUBEOF
|
||||
for selector in \
|
||||
'select(.name == "preconditions") | .try[].script.content' \
|
||||
'select(.name == "run") | .finally[] | select(.description == "system-bucket round-trip cleanup") | .script.content'; do
|
||||
: >"$calls"
|
||||
script=$(yq "$test | $selector" "$suite")
|
||||
[ -n "$script" ] || { echo "no script at: $selector" >&2; false; }
|
||||
PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script"
|
||||
grep -qE '^-n tenant-root delete backup sysbucket-backup( |$)' "$calls" || {
|
||||
echo "the script at '$selector' does not delete Backup sysbucket-backup by name; kubectl calls:" >&2
|
||||
cat "$calls" >&2
|
||||
false
|
||||
}
|
||||
done
|
||||
}
|
||||
|
||||
# cert-manager issues the Secret of a Certificate that still exists again as
|
||||
# soon as it is deleted. Chainsaw deletes the Certificate it applied, and waits
|
||||
# for it, before it runs the cleanup ops of the same step, so the Secret is
|
||||
# deleted there and nowhere earlier.
|
||||
@test "the Certificate's Secret is deleted after Chainsaw deletes the Certificate" {
|
||||
test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]'
|
||||
suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml
|
||||
applies=$(yq "$test | select(.name == \"create-certificate-and-probe\") | .try[] | select(has(\"apply\")) | .apply.file" "$suite")
|
||||
[ "$applies" = system-bucket-tls.yaml ] || {
|
||||
echo "step create-certificate-and-probe no longer applies system-bucket-tls.yaml alone: $applies" >&2
|
||||
false
|
||||
}
|
||||
deletes=$(yq -o=json -I=0 "$test | select(.name == \"create-certificate-and-probe\") | .cleanup" "$suite")
|
||||
[ "$deletes" = '[{"delete":{"ref":{"apiVersion":"v1","kind":"Secret","name":"e2e-system-bucket-s3-tls"}}}]' ] || {
|
||||
echo "the cleanup of step create-certificate-and-probe is not the one delete of Secret e2e-system-bucket-s3-tls: $deletes" >&2
|
||||
false
|
||||
}
|
||||
# The preconditions run before the apply, when Chainsaw holds no Certificate.
|
||||
if yq "$test | select(.name != \"preconditions\") | (.try[], .finally[]?) | .script.content // \"\"" "$suite" | grep -qE 'delete +(secret +e2e-system-bucket-s3-tls|certificate +e2e-system-bucket-s3)( |$)'; then
|
||||
echo "a step script deletes the Certificate or its Secret while Chainsaw still holds the Certificate" >&2
|
||||
false
|
||||
fi
|
||||
}
|
||||
|
||||
# Chainsaw registers an applied object for deletion only when its apply created
|
||||
# it; an object left by a run killed past its cleanup is patched and kept. The
|
||||
# Certificate would then outlive every later run, re-issuing its Secret after
|
||||
# each step cleanup, and a probe Pod whose `sleep` had ended never turns Ready
|
||||
# again, so the preconditions remove all three before the apply.
|
||||
@test "the preconditions remove the TLS objects and probe Pod a killed run left" {
|
||||
suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml
|
||||
test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]'
|
||||
calls="$STUB/calls"
|
||||
cat >"$STUB/kubectl" <<'STUBEOF'
|
||||
#!/bin/bash
|
||||
echo "$*" >>"$STUB/calls"
|
||||
case "$*" in
|
||||
*" get secret "*) exit 1 ;;
|
||||
esac
|
||||
STUBEOF
|
||||
: >"$calls"
|
||||
script=$(yq "$test | select(.name == \"preconditions\") | .try[].script.content" "$suite")
|
||||
PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script"
|
||||
line_of() { grep -nE "^-n tenant-root delete $1 --ignore-not-found --wait=true( |$)" "$calls" | head -n1 | cut -d: -f1; }
|
||||
cert=$(line_of 'certificate e2e-system-bucket-s3')
|
||||
secret=$(line_of 'secret e2e-system-bucket-s3-tls')
|
||||
pod=$(line_of 'pod e2e-system-bucket-probe')
|
||||
[ -n "$cert" ] && [ -n "$secret" ] && [ -n "$pod" ] || {
|
||||
echo "the preconditions do not delete, and wait for, the Certificate, its Secret and the probe Pod; kubectl calls:" >&2
|
||||
cat "$calls" >&2
|
||||
false
|
||||
}
|
||||
[ "$cert" -lt "$secret" ] || {
|
||||
echo "the preconditions delete Secret e2e-system-bucket-s3-tls before its Certificate, which cert-manager then issues again" >&2
|
||||
false
|
||||
}
|
||||
}
|
||||
@@ -91,7 +91,7 @@ spec:
|
||||
status:
|
||||
replicas: 1
|
||||
---
|
||||
# ClickHouse Altinity backup/restore E2E — the two backup Tests below sit in
|
||||
# ClickHouse Altinity backup/restore E2E — the three backup Tests below sit in
|
||||
# this suite alongside the plain `clickhouse` Test above. Chainsaw port of the
|
||||
# intended hack/e2e-apps/backup-clickhouse.bats (cozystack#2600), shaped like the
|
||||
# etcd/mariadb/postgres suites:
|
||||
@@ -107,8 +107,15 @@ spec:
|
||||
# driving the validated example scripts
|
||||
# (examples/backups/clickhouse). Also runs in
|
||||
# CI, in tenant-root.
|
||||
# clickhouse-3-system-bucket-roundtrip
|
||||
# — backup and in-place restore through the
|
||||
# shipped cozy-default BackupClass and the
|
||||
# platform system bucket, for a release
|
||||
# whose Pod started before its credentials
|
||||
# were projected. Runs in CI, in tenant-root.
|
||||
#
|
||||
# Why the split: the two Tests answer different questions at different cost. The
|
||||
# Why the split: the first two Tests answer different questions at different
|
||||
# cost, and the third covers the system-bucket path neither of them takes. The
|
||||
# contracts Test needs no object store, so it stays in the default tenant-test
|
||||
# namespace next to the other app suites and is the seconds-long guard on the
|
||||
# chart wiring the Altinity strategy depends on — the sidecar container/port and
|
||||
@@ -377,3 +384,217 @@ spec:
|
||||
set -eu
|
||||
cd ../../..
|
||||
NAMESPACE="$NAMESPACE" examples/backups/clickhouse/cleanup.sh
|
||||
---
|
||||
# Backup and in-place restore through the shipped cozy-default BackupClass and
|
||||
# the platform system bucket, for a ClickHouse whose Pod started before the
|
||||
# platform projected cozy-backups-creds into its namespace. The sidecar has to
|
||||
# pick those credentials up while running; until it could, the first BackupJob
|
||||
# of such a release failed with "no EC2 IMDS role found" and kept failing until
|
||||
# the Pod was restarted.
|
||||
#
|
||||
# Unlike clickhouse-2-backup-roundtrip this targets the system bucket as
|
||||
# advertised, https://s3.example.org, so system-bucket-roundtrip.sh makes that
|
||||
# name reachable from tenant-root for the duration of its own step and undoes
|
||||
# it on exit: see its header for the two shared objects it touches and the
|
||||
# cert-manager behaviour that relies on. Runs in tenant-root, where the system
|
||||
# bucket's SeaweedFS and its Ingress controller sit.
|
||||
#
|
||||
# The precondition step deletes tenant-root's projected cozy-backups-creds,
|
||||
# which any BackupJob that ran in tenant-root before this Test leaves there:
|
||||
# the projector runs for each BackupJob whatever its strategy, and Chainsaw
|
||||
# does not fix the order of the Tests in a suite, so the earlier one may come
|
||||
# from this suite or another. It writes the Secret again on the next
|
||||
# BackupJob, which is the moment under test.
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: clickhouse-3-system-bucket-roundtrip
|
||||
spec:
|
||||
namespace: tenant-root
|
||||
catch:
|
||||
- events: {}
|
||||
- describe:
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
name: clickhouse-sysbucket
|
||||
- describe:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
name: e2e-system-bucket-s3
|
||||
- describe:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
name: seaweedfs-s3-ingress-tls
|
||||
- describe:
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
name: ingress-seaweedfs-s3
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
name: sysbucket-backup
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: RestoreJob
|
||||
name: sysbucket-restore
|
||||
- podLogs:
|
||||
selector: app.kubernetes.io/instance=clickhouse-sysbucket
|
||||
container: clickhouse-backup
|
||||
tail: 200
|
||||
- podLogs:
|
||||
selector: app=clickhouse-sysbucket-keeper
|
||||
tail: 100
|
||||
steps:
|
||||
- name: preconditions
|
||||
try:
|
||||
# The deletes below wait up to 270s in all (30+30+60+30+30+30+60), so a
|
||||
# stuck one is named by its kubectl --timeout before the 5m ceiling kills
|
||||
# the shell with no object named. A new delete needs its own --timeout.
|
||||
- script:
|
||||
timeout: 5m
|
||||
content: |
|
||||
set -eu
|
||||
if kubectl -n "$NAMESPACE" get secret seaweedfs-s3-ingress-tls -o name >/dev/null 2>&1; then
|
||||
echo "Secret $NAMESPACE/seaweedfs-s3-ingress-tls already exists; this Test would have to replace it" >&2
|
||||
exit 1
|
||||
fi
|
||||
if kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF 'BEGIN e2e-clickhouse-system-bucket'; then
|
||||
echo "kube-system/coredns still carries the e2e-clickhouse-system-bucket block from an earlier run" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Fixed names in a persistent namespace: a leftover Succeeded job
|
||||
# would satisfy the waits in the run step without a new attempt, and
|
||||
# a leftover Backup of the BackupJob's name, recording an earlier
|
||||
# run's archive, fails the new BackupJob.
|
||||
kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s
|
||||
kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s
|
||||
kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s
|
||||
kubectl -n "$NAMESPACE" delete secret cozy-backups-creds --ignore-not-found --wait=true --timeout=30s
|
||||
# Chainsaw reclaims only what its apply created, so objects a killed
|
||||
# run left would be patched and kept. The Certificate goes before its
|
||||
# Secret, which cert-manager issues again while the Certificate exists.
|
||||
kubectl -n "$NAMESPACE" delete certificate e2e-system-bucket-s3 --ignore-not-found --wait=true --timeout=30s
|
||||
kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s
|
||||
kubectl -n "$NAMESPACE" delete pod e2e-system-bucket-probe --ignore-not-found --wait=true --timeout=60s
|
||||
check:
|
||||
($error == null): true
|
||||
|
||||
- name: create-certificate-and-probe
|
||||
try:
|
||||
- apply:
|
||||
file: system-bucket-tls.yaml
|
||||
- assert:
|
||||
timeout: 2m
|
||||
resource:
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: e2e-system-bucket-s3
|
||||
status:
|
||||
(conditions[?type == 'Ready']):
|
||||
- status: "True"
|
||||
- assert:
|
||||
timeout: 3m
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: e2e-system-bucket-probe
|
||||
status:
|
||||
(conditions[?type == 'Ready']):
|
||||
- status: "True"
|
||||
# The Certificate's Secret carries no owner reference, so it outlives the
|
||||
# Certificate, and cert-manager issues it again while the Certificate
|
||||
# exists. Chainsaw runs a step's cleanup ops after this step's cleaner has
|
||||
# deleted the Certificate and waited for it.
|
||||
cleanup:
|
||||
- delete:
|
||||
ref:
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
name: e2e-system-bucket-s3-tls
|
||||
|
||||
- name: create-clickhouse
|
||||
try:
|
||||
- apply:
|
||||
file: clickhouse-system-bucket.yaml
|
||||
# 11m: above the 600s install timeout cozystack-api gives this release;
|
||||
# see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
|
||||
- assert:
|
||||
timeout: 11m
|
||||
resource:
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: clickhouse-sysbucket
|
||||
status:
|
||||
(conditions[?type == 'Ready']):
|
||||
- status: "True"
|
||||
- assert:
|
||||
timeout: 5m
|
||||
resource:
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: chi-clickhouse-sysbucket-clickhouse-0-0
|
||||
status:
|
||||
readyReplicas: 1
|
||||
# The Pod is running with no projected credentials: the state the first
|
||||
# BackupJob of a fresh release meets.
|
||||
- error:
|
||||
timeout: 30s
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: cozy-backups-creds
|
||||
|
||||
- name: run
|
||||
try:
|
||||
# Inner budgets of system-bucket-roundtrip.sh:
|
||||
# 280s probe: a 240s deadline for CoreDNS picking up the rewrite and
|
||||
# the Ingress serving the Test certificate, plus one last 30s
|
||||
# attempt and its 10s pause
|
||||
# 480s BackupJob to Succeeded, including the driver's 2m hold
|
||||
# 480s RestoreJob to Succeeded; no hold by then, the stamp is
|
||||
# already older than the backup's own 2m wait
|
||||
# 90s the EXIT trap: Corefile replace plus a 60s Secret delete
|
||||
# 1330s (22m10s)
|
||||
# 25m leaves 2m50s for the SQL and kubectl calls between those waits, so
|
||||
# an inner wait is what fires and names what stalled.
|
||||
- script:
|
||||
timeout: 25m
|
||||
content: |
|
||||
set -eu
|
||||
cd ../../..
|
||||
NAMESPACE="$NAMESPACE" hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh
|
||||
check:
|
||||
($error == null): true
|
||||
finally:
|
||||
# The script's EXIT trap puts the Corefile and the Ingress TLS Secret
|
||||
# back, but a step ceiling kills the shell before the trap runs. A no-op
|
||||
# when the trap already did it.
|
||||
- description: restore the host CoreDNS and the S3 Ingress TLS Secret
|
||||
script:
|
||||
timeout: 2m
|
||||
content: |
|
||||
set -eu
|
||||
cd ../../..
|
||||
# Chainsaw runs this under sh; the helpers are bash.
|
||||
NAMESPACE="$NAMESPACE" bash -euo pipefail -c '
|
||||
source examples/backups/clickhouse/00-helpers.sh
|
||||
source hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh
|
||||
restore_shared_state
|
||||
'
|
||||
# Chainsaw did not apply these: the jobs come from the script and the
|
||||
# Backup from the controller, and deleting a BackupJob reclaims no Backup.
|
||||
- description: system-bucket round-trip cleanup
|
||||
script:
|
||||
timeout: 3m
|
||||
content: |
|
||||
set -eu
|
||||
kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s
|
||||
kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s
|
||||
# The driver names the Backup after the BackupJob; by name it is
|
||||
# found even when status.backupRef was never written.
|
||||
kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s
|
||||
@@ -8,8 +8,8 @@
|
||||
# backup.enabled=true with placeholder S3 coordinates is deliberate: the
|
||||
# sidecar runs `clickhouse-backup server`, which starts its HTTP API without
|
||||
# validating S3 (ALLOW_EMPTY_BACKUPS=true, lazy connect — the same reason the
|
||||
# platform useSystemBucket flow marks the S3 secret keys optional so the Pod
|
||||
# starts before any bucket is projected). So the Pod becomes Ready and the
|
||||
# platform useSystemBucket flow can start the Pod before any bucket is
|
||||
# projected). So the Pod becomes Ready and the
|
||||
# chart<->strategy contract is assertable with no reachable object store. The
|
||||
# real S3 backup/restore round-trip lives in clickhouse-2-backup-roundtrip,
|
||||
# which runs in tenant-root against the in-cluster SeaweedFS.
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# ClickHouse on the platform system bucket for the
|
||||
# clickhouse-3-system-bucket-roundtrip Test. No S3 coordinates are given: the
|
||||
# sidecar reads them from the clickhouse-backup.yml key the platform projects
|
||||
# into cozy-backups-creds at the first BackupJob, after this Pod has started.
|
||||
# backup.endpointCA trusts the SeaweedFS CA that signs the certificate the Test
|
||||
# serves for s3.example.org.
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: ClickHouse
|
||||
metadata:
|
||||
name: sysbucket
|
||||
spec:
|
||||
size: 5Gi
|
||||
logStorageSize: 1Gi
|
||||
shards: 1
|
||||
replicas: 1
|
||||
# The round-trip runs SQL against this server; at nano's 128Mi the first
|
||||
# query OOM-kills it, so it takes the preset clickhouse-2 runs SQL on.
|
||||
resourcesPreset: small
|
||||
resources: {}
|
||||
backup:
|
||||
enabled: true
|
||||
useSystemBucket: true
|
||||
endpointCA:
|
||||
name: e2e-system-bucket-s3-tls
|
||||
key: ca.crt
|
||||
clickhouseKeeper:
|
||||
enabled: true
|
||||
replicas: 1
|
||||
size: 1Gi
|
||||
resourcesPreset: micro
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
#!/bin/bash
|
||||
# Backup and in-place restore of the `sysbucket` ClickHouse through the
|
||||
# cozy-default BackupClass, against the platform system bucket. Run from the
|
||||
# repository root by the clickhouse-3-system-bucket-roundtrip Test, which has
|
||||
# already started the release with no cozy-backups-creds in its namespace.
|
||||
#
|
||||
# The system bucket is addressed as https://s3.example.org, the S3 Ingress host
|
||||
# the COSI driver advertises. In CI that name does not resolve and its ACME
|
||||
# certificate is never issued, so for the duration of this script, and only of
|
||||
# this script, two shared objects are changed and the EXIT trap below puts both
|
||||
# back:
|
||||
#
|
||||
# - the S3 Ingress TLS Secret seaweedfs-s3-ingress-tls is created from the
|
||||
# Test's own certificate (SeaweedFS CA, s3.example.org). It relies on
|
||||
# cert-manager leaving a Secret's data alone while the Certificate that
|
||||
# names it has no completed issuance, verified with cert-manager v1.20.2:
|
||||
# a fresh ACME order would replace this certificate, and ingress-nginx
|
||||
# would then serve one the sidecar does not trust, which fails the probe
|
||||
# below rather than the backup.
|
||||
# - the host CoreDNS Corefile gets one marked `rewrite name exact` line for
|
||||
# s3.example.org. The host CoreDNS is Talos-managed and imports no snippet
|
||||
# files, so the block goes into the live Corefile the way ouroboros writes
|
||||
# its own; ouroboros is off in CI and no other suite uses this name.
|
||||
set -euo pipefail
|
||||
|
||||
source examples/backups/clickhouse/00-helpers.sh
|
||||
source hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh
|
||||
|
||||
: "${NAMESPACE:?}"
|
||||
CLICKHOUSE_NAME=sysbucket
|
||||
BACKUPJOB_NAME=sysbucket-backup
|
||||
RESTOREJOB_NAME=sysbucket-restore
|
||||
CERT_SECRET=e2e-system-bucket-s3-tls
|
||||
PROBE_POD=e2e-system-bucket-probe
|
||||
DNS_TARGET="root-ingress-controller.${NAMESPACE}.svc.cozy.local"
|
||||
# Kubelet refreshes a ConfigMap volume within about a minute and CoreDNS's
|
||||
# reload plugin polls every 30s; the rest is the 30s answer cache.
|
||||
PROBE_TIMEOUT=240
|
||||
# Each covers the driver's 2m hold after the credentials change plus the
|
||||
# strategy Job itself.
|
||||
JOB_TIMEOUT=480
|
||||
|
||||
workdir=$(mktemp -d)
|
||||
|
||||
on_exit() {
|
||||
local rc=$?
|
||||
set +e
|
||||
restore_shared_state || rc=1
|
||||
rm -rf "$workdir"
|
||||
exit "$rc"
|
||||
}
|
||||
|
||||
if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -o name >/dev/null 2>&1; then
|
||||
log_error "Secret ${NAMESPACE}/${INGRESS_SECRET} already exists; this Test serves its own certificate under that name and will not overwrite one it did not create"
|
||||
exit 1
|
||||
fi
|
||||
if corefile_has_block; then
|
||||
log_error "kube-system/coredns already carries the ${S3_HOST} rewrite block, left by an earlier run; remove it by hand"
|
||||
exit 1
|
||||
fi
|
||||
trap on_exit EXIT
|
||||
|
||||
log_step "Serving the Test certificate for ${S3_HOST} through the platform S3 Ingress"
|
||||
kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o json \
|
||||
| jq --arg name "$INGRESS_SECRET" '{apiVersion, kind, type,
|
||||
metadata: {name: $name, labels: {"cozystack.io/e2e-owner": "clickhouse-system-bucket"}},
|
||||
data: {"tls.crt": .data["tls.crt"], "tls.key": .data["tls.key"], "ca.crt": .data["ca.crt"]}}' \
|
||||
| kubectl -n "$NAMESPACE" create -f -
|
||||
|
||||
log_step "Rewriting ${S3_HOST} to ${DNS_TARGET} in the host CoreDNS"
|
||||
kubectl -n kube-system get configmap coredns -o json >"$workdir/coredns.json"
|
||||
jq -e '.data.Corefile | startswith(".:53 {\n")' "$workdir/coredns.json" >/dev/null \
|
||||
|| { log_error "kube-system/coredns Corefile does not open with '.:53 {'; refusing to edit it"; exit 1; }
|
||||
# The resourceVersion read above travels with the replace, so a concurrent
|
||||
# writer makes it fail instead of being overwritten.
|
||||
jq --arg block "${DNS_BEGIN}"$'\n'" rewrite name exact ${S3_HOST} ${DNS_TARGET}"$'\n'"${DNS_END}" \
|
||||
'.data.Corefile |= sub("^\\.:53 \\{\n"; ".:53 {\n" + $block + "\n")' \
|
||||
"$workdir/coredns.json" >"$workdir/coredns-new.json"
|
||||
kubectl replace -f "$workdir/coredns-new.json" >/dev/null
|
||||
|
||||
log_step "Checking that ${S3_HOST} serves the Test certificate from inside ${NAMESPACE}"
|
||||
kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.ca\.crt}' | base64 -d >"$workdir/ca.crt"
|
||||
want=$(kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.tls\.crt}' | base64 -d \
|
||||
| openssl x509 -noout -fingerprint -sha256)
|
||||
deadline=$((SECONDS + PROBE_TIMEOUT))
|
||||
while true; do
|
||||
# --cacert makes curl verify both the chain and the host name; %{certs}
|
||||
# prints the chain it was served, leaf first.
|
||||
if served=$(timeout 30 kubectl -n "$NAMESPACE" exec -i "$PROBE_POD" -- sh -c \
|
||||
"cat >/tmp/ca.crt && curl -sS -o /dev/null --max-time 10 --cacert /tmp/ca.crt -w '%{certs}' https://${S3_HOST}/" \
|
||||
<"$workdir/ca.crt" 2>"$workdir/probe.err"); then
|
||||
got=$(printf '%s\n' "$served" | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p;/END CERTIFICATE/q' \
|
||||
| openssl x509 -noout -fingerprint -sha256)
|
||||
if [ "$got" = "$want" ]; then
|
||||
log_success "${S3_HOST} resolves in ${NAMESPACE} and serves the Test certificate"
|
||||
break
|
||||
fi
|
||||
echo "served ${got}, want ${want}" >"$workdir/probe.err"
|
||||
fi
|
||||
if [ "$SECONDS" -ge "$deadline" ]; then
|
||||
log_error "${S3_HOST} did not serve the Test certificate within ${PROBE_TIMEOUT}s: $(cat "$workdir/probe.err")"
|
||||
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' >&2 || true
|
||||
kubectl -n "$NAMESPACE" describe ingress ingress-seaweedfs-s3 >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
# TODO(e2e-replace-fixed-timeouts): no condition reports that the host
|
||||
# CoreDNS answers with the rewrite and the Ingress serves the Test
|
||||
# certificate, as seen from this namespace; poll until one does.
|
||||
sleep 10
|
||||
done
|
||||
|
||||
# A restarted Pod reads its credentials afresh at start, which is what the
|
||||
# old environment-based sidecar needed; the round-trip proves the fix only if
|
||||
# the sidecar that started without them is the one that serves both jobs.
|
||||
sidecar_identity() {
|
||||
kubectl -n "$NAMESPACE" get pod "chi-clickhouse-${CLICKHOUSE_NAME}-clickhouse-0-0-0" \
|
||||
-o jsonpath='{.metadata.uid} {.status.containerStatuses[?(@.name=="clickhouse-backup")].restartCount}'
|
||||
}
|
||||
sidecar_before=$(sidecar_identity)
|
||||
[[ "$sidecar_before" =~ ^[0-9a-f-]+\ [0-9]+$ ]] \
|
||||
|| { log_error "could not read the ClickHouse Pod uid and clickhouse-backup restart count (got '${sidecar_before}')"; exit 1; }
|
||||
|
||||
# Restore takes the archive this run's Backup recorded; a row no earlier run
|
||||
# wrote is what shows the data restored is that archive's.
|
||||
sentinel="before-backup-$(date -u +%Y%m%dT%H%M%SZ)-${RANDOM}"
|
||||
log_step "Writing the sentinel row ${sentinel}"
|
||||
clickhouse_query "$CLICKHOUSE_NAME" "CREATE TABLE IF NOT EXISTS default.sentinel (id UInt32, name String) ENGINE = MergeTree ORDER BY id"
|
||||
clickhouse_query "$CLICKHOUSE_NAME" "INSERT INTO default.sentinel VALUES (1, '${sentinel}')"
|
||||
|
||||
log_step "Backing up through cozy-default"
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
metadata:
|
||||
name: ${BACKUPJOB_NAME}
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
applicationRef:
|
||||
apiGroup: apps.cozystack.io
|
||||
kind: ClickHouse
|
||||
name: ${CLICKHOUSE_NAME}
|
||||
backupClassName: cozy-default
|
||||
EOF
|
||||
wait_for_field backupjob "$BACKUPJOB_NAME" '{.status.phase}' Succeeded "$NAMESPACE" "$JOB_TIMEOUT" Failed
|
||||
backup_ref=$(kubectl -n "$NAMESPACE" get backupjob "$BACKUPJOB_NAME" -o jsonpath='{.status.backupRef.name}')
|
||||
[ -n "$backup_ref" ] || { log_error "BackupJob ${BACKUPJOB_NAME} succeeded with an empty backupRef"; exit 1; }
|
||||
|
||||
# The sidecar started before this Secret existed; a backup that succeeded
|
||||
# against it is the behaviour under test, and the key it read is this one.
|
||||
endpoint=$(kubectl -n "$NAMESPACE" get secret cozy-backups-creds -o jsonpath='{.data.clickhouse-backup\.yml}' | base64 -d \
|
||||
| sed -n 's/^ *endpoint: *//p')
|
||||
[ "$endpoint" = "https://${S3_HOST}" ] \
|
||||
|| { log_error "cozy-backups-creds clickhouse-backup.yml endpoint is '${endpoint}', expected https://${S3_HOST}"; exit 1; }
|
||||
|
||||
log_step "Restoring in place through cozy-default"
|
||||
clickhouse_query "$CLICKHOUSE_NAME" "DROP TABLE IF EXISTS default.sentinel SYNC"
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: RestoreJob
|
||||
metadata:
|
||||
name: ${RESTOREJOB_NAME}
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
backupRef:
|
||||
name: ${backup_ref}
|
||||
EOF
|
||||
wait_for_field restorejob "$RESTOREJOB_NAME" '{.status.phase}' Succeeded "$NAMESPACE" "$JOB_TIMEOUT" Failed
|
||||
|
||||
count=$(clickhouse_query "$CLICKHOUSE_NAME" "SELECT count() FROM default.sentinel WHERE name = '${sentinel}'" | tr -d '[:space:]')
|
||||
[ "$count" = 1 ] || { log_error "after restore default.sentinel holds ${count} row(s) named ${sentinel}; expected 1"; exit 1; }
|
||||
sidecar_after=$(sidecar_identity)
|
||||
[ "$sidecar_after" = "$sidecar_before" ] \
|
||||
|| { log_error "the ClickHouse Pod or its clickhouse-backup container restarted during the round-trip (uid restarts: '${sidecar_before}' then '${sidecar_after}')"; exit 1; }
|
||||
log_success "System-bucket round-trip verified: sentinel row ${sentinel} restored"
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/bin/bash
|
||||
# The two shared objects system-bucket-roundtrip.sh changes for the duration
|
||||
# of its run, and how to put them back. Sourced by that script, whose EXIT trap
|
||||
# restores them, and by the finally of its Chainsaw step, which restores them
|
||||
# when the step ceiling killed the script before the trap could run. Needs
|
||||
# NAMESPACE and log_success/log_error from examples/backups/clickhouse/00-helpers.sh.
|
||||
|
||||
S3_HOST=s3.example.org
|
||||
INGRESS_SECRET=seaweedfs-s3-ingress-tls
|
||||
OWNER_LABEL=cozystack.io/e2e-owner=clickhouse-system-bucket
|
||||
DNS_BEGIN=' # === BEGIN e2e-clickhouse-system-bucket ==='
|
||||
DNS_END=' # === END e2e-clickhouse-system-bucket ==='
|
||||
|
||||
corefile_has_block() {
|
||||
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF -- "$DNS_BEGIN"
|
||||
}
|
||||
|
||||
remove_dns_block() {
|
||||
local dir
|
||||
dir=$(mktemp -d)
|
||||
kubectl -n kube-system get configmap coredns -o json >"$dir/coredns.json" || { rm -rf "$dir"; return 1; }
|
||||
local begins
|
||||
begins=$(jq -r '.data.Corefile' "$dir/coredns.json" | grep -cF -- "$DNS_BEGIN" || true)
|
||||
if [ "$begins" = 0 ]; then
|
||||
rm -rf "$dir"
|
||||
return 0
|
||||
fi
|
||||
if [ "$begins" != 1 ] || ! jq -r '.data.Corefile' "$dir/coredns.json" | grep -qF -- "$DNS_END"; then
|
||||
log_error "kube-system/coredns holds ${begins} '${DNS_BEGIN# *}' markers or no END marker; reconcile the Corefile by hand"
|
||||
rm -rf "$dir"
|
||||
return 1
|
||||
fi
|
||||
jq --arg b "$DNS_BEGIN" --arg e "$DNS_END" \
|
||||
'.data.Corefile |= (split("\n") | (index($b)) as $i | (index($e)) as $j | .[:$i] + .[$j+1:] | join("\n"))' \
|
||||
"$dir/coredns.json" >"$dir/coredns-clean.json"
|
||||
# Callers run this with errexit off, so a refused replace has to return.
|
||||
if ! kubectl replace -f "$dir/coredns-clean.json" >/dev/null; then
|
||||
log_error "could not remove the ${S3_HOST} rewrite from kube-system/coredns; remove the marked block by hand"
|
||||
rm -rf "$dir"
|
||||
return 1
|
||||
fi
|
||||
log_success "Removed the ${S3_HOST} rewrite from kube-system/coredns"
|
||||
rm -rf "$dir"
|
||||
}
|
||||
|
||||
# Returns non-zero when either object could not be put back.
|
||||
restore_shared_state() {
|
||||
local rc=0
|
||||
remove_dns_block || rc=1
|
||||
# kubectl refuses a name together with a selector, so the label is
|
||||
# looked up on its own and the name matched in what comes back.
|
||||
local owned
|
||||
if ! owned=$(kubectl -n "$NAMESPACE" get secret -l "$OWNER_LABEL" -o name); then
|
||||
log_error "could not list the Secrets labelled ${OWNER_LABEL} in ${NAMESPACE}; delete ${INGRESS_SECRET} by hand if it carries that label"
|
||||
rc=1
|
||||
elif printf '%s\n' "$owned" | grep -qxF "secret/$INGRESS_SECRET"; then
|
||||
kubectl -n "$NAMESPACE" delete secret "$INGRESS_SECRET" --wait=true --timeout=60s || rc=1
|
||||
fi
|
||||
return "$rc"
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
# Certificate and probe Pod for the clickhouse-3-system-bucket-roundtrip Test.
|
||||
#
|
||||
# The certificate is signed by the SeaweedFS CA already present in tenant-root,
|
||||
# so its Secret carries that CA as ca.crt: the ClickHouse release trusts it via
|
||||
# backup.endpointCA, and the round-trip script copies the same key pair under
|
||||
# the name the platform's S3 Ingress serves for s3.example.org.
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: e2e-system-bucket-s3
|
||||
spec:
|
||||
secretName: e2e-system-bucket-s3-tls
|
||||
dnsNames:
|
||||
- s3.example.org
|
||||
issuerRef:
|
||||
kind: Issuer
|
||||
name: seaweedfs-ca-issuer
|
||||
---
|
||||
# Resolves and fetches s3.example.org from inside tenant-root, where the
|
||||
# ClickHouse sidecar runs, so the script can prove the DNS rewrite and the
|
||||
# served certificate before it spends minutes on a BackupJob. The image is the
|
||||
# one hack/e2e-install-cozystack.bats pre-pulls; it carries curl but no openssl.
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: e2e-system-bucket-probe
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
terminationGracePeriodSeconds: 0
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: probe
|
||||
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47
|
||||
command: [sleep, "3600"]
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: [ALL]
|
||||
@@ -85,6 +85,76 @@ func validateAltinityApplicationRef(ref corev1.TypedLocalObjectReference) error
|
||||
return nil
|
||||
}
|
||||
|
||||
// altinityCredentialsPropagation bounds how long kubelet takes to rewrite
|
||||
// the mounted cozy-backups-creds in the ClickHouse Pod after it changes:
|
||||
// its sync period (a minute by default) plus the Secret cache. About 75s was
|
||||
// measured; the margin covers the rest.
|
||||
const altinityCredentialsPropagation = 2 * time.Minute
|
||||
|
||||
// credentialsPropagationWait returns how long to hold back creating the
|
||||
// strategy Job named jobName in namespace, so that the clickhouse-backup
|
||||
// sidecar, which reads the projected credentials from a mounted file, sees
|
||||
// their latest content. A Job that already exists, or a namespace without
|
||||
// projected credentials, does not wait.
|
||||
func credentialsPropagationWait(ctx context.Context, c client.Client, secretName, namespace, jobName string) (time.Duration, error) {
|
||||
if secretName == "" {
|
||||
return 0, nil
|
||||
}
|
||||
if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: jobName}, &batchv1.Job{}); err == nil {
|
||||
return 0, nil
|
||||
} else if !apierrors.IsNotFound(err) {
|
||||
return 0, err
|
||||
}
|
||||
secret := &corev1.Secret{}
|
||||
if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: secretName}, secret); err != nil {
|
||||
if apierrors.IsNotFound(err) {
|
||||
return 0, nil
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
raw := secret.Annotations[credentialsUpdatedAtAnnotation]
|
||||
if raw == "" {
|
||||
return 0, nil
|
||||
}
|
||||
// The projector rewrites the stamp only when the content changes, so a
|
||||
// stamp this hold cannot use stays as it is: holding on it would block
|
||||
// every Job in the namespace for good.
|
||||
updatedAt, err := time.Parse(time.RFC3339, raw)
|
||||
if err != nil {
|
||||
getLogger(ctx).Error(err, "ignoring unreadable credentials stamp", "secret", namespace+"/"+secretName, "annotation", credentialsUpdatedAtAnnotation)
|
||||
return 0, nil
|
||||
}
|
||||
if updatedAt.After(time.Now().Add(altinityCredentialsPropagation)) {
|
||||
getLogger(ctx).Error(nil, "ignoring a credentials stamp further ahead than clock skew explains", "secret", namespace+"/"+secretName, "annotation", credentialsUpdatedAtAnnotation, "value", raw)
|
||||
return 0, nil
|
||||
}
|
||||
return time.Until(updatedAt.Add(altinityCredentialsPropagation)), nil
|
||||
}
|
||||
|
||||
// clearCredentialsWait drops the Ready condition credentialsWaitCondition
|
||||
// left behind, once the job no longer waits. It reports whether it did.
|
||||
func clearCredentialsWait(conditions *[]metav1.Condition) bool {
|
||||
ready := apimeta.FindStatusCondition(*conditions, "Ready")
|
||||
if ready == nil || ready.Reason != credentialsWaitReason {
|
||||
return false
|
||||
}
|
||||
return apimeta.RemoveStatusCondition(conditions, "Ready")
|
||||
}
|
||||
|
||||
const credentialsWaitReason = "WaitingForCredentials"
|
||||
|
||||
// credentialsWaitCondition is the Ready condition of a job held back by
|
||||
// credentialsPropagationWait.
|
||||
func credentialsWaitCondition(secretName string, wait time.Duration) metav1.Condition {
|
||||
return metav1.Condition{
|
||||
Type: "Ready",
|
||||
Status: metav1.ConditionFalse,
|
||||
Reason: credentialsWaitReason,
|
||||
Message: fmt.Sprintf("waiting %s for the updated Secret %s to reach the ClickHouse backup sidecar",
|
||||
wait.Round(time.Second), secretName),
|
||||
}
|
||||
}
|
||||
|
||||
// jobNameForBackupJob returns the deterministic batch/v1.Job name used for
|
||||
// the backup pod that drives a BackupJob. Idempotency relies on this being
|
||||
// pure: a retried reconcile must hit the same Job name.
|
||||
@@ -265,6 +335,18 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv
|
||||
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to template Altinity strategy: %v", err))
|
||||
}
|
||||
|
||||
wait, err := credentialsPropagationWait(ctx, r.Client, r.CredentialsConfig.TargetSecretName, j.Namespace, jobNameForBackupJob(j))
|
||||
if err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
if wait > 0 {
|
||||
apimeta.SetStatusCondition(&j.Status.Conditions, credentialsWaitCondition(r.CredentialsConfig.TargetSecretName, wait))
|
||||
if err := r.Status().Update(ctx, j); err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
return ctrl.Result{RequeueAfter: wait}, nil
|
||||
}
|
||||
|
||||
batchJob, err := r.ensureAltinityJob(ctx, j, j.Namespace, jobNameForBackupJob(j),
|
||||
altinityModeBackup,
|
||||
map[string]string{
|
||||
@@ -312,6 +394,11 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv
|
||||
return r.markBackupJobFailed(ctx, j, message)
|
||||
|
||||
default:
|
||||
if clearCredentialsWait(&j.Status.Conditions) {
|
||||
if err := r.Status().Update(ctx, j); err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
}
|
||||
return ctrl.Result{RequeueAfter: altinityPollInterval}, nil
|
||||
}
|
||||
}
|
||||
@@ -681,6 +768,18 @@ func (r *RestoreJobReconciler) reconcileAltinityRestore(ctx context.Context, res
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to template Altinity strategy: %v", err))
|
||||
}
|
||||
|
||||
wait, err := credentialsPropagationWait(ctx, r.Client, r.CredentialsConfig.TargetSecretName, targetNamespace, jobNameForRestoreJob(restoreJob))
|
||||
if err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
if wait > 0 {
|
||||
apimeta.SetStatusCondition(&restoreJob.Status.Conditions, credentialsWaitCondition(r.CredentialsConfig.TargetSecretName, wait))
|
||||
if err := r.Status().Update(ctx, restoreJob); err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
return ctrl.Result{RequeueAfter: wait}, nil
|
||||
}
|
||||
|
||||
batchJob, err := r.ensureAltinityRestoreJob(ctx, restoreJob, targetNamespace, jobNameForRestoreJob(restoreJob),
|
||||
altinityModeRestore,
|
||||
map[string]string{
|
||||
@@ -717,6 +816,11 @@ func (r *RestoreJobReconciler) reconcileAltinityRestore(ctx context.Context, res
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, message)
|
||||
|
||||
default:
|
||||
if clearCredentialsWait(&restoreJob.Status.Conditions) {
|
||||
if err := r.Status().Update(ctx, restoreJob); err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
}
|
||||
return ctrl.Result{RequeueAfter: altinityPollInterval}, nil
|
||||
}
|
||||
}
|
||||
@@ -3,12 +3,14 @@ package backupcontroller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
batchv1 "k8s.io/api/batch/v1"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
apierrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
"k8s.io/apimachinery/pkg/api/meta"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
||||
@@ -19,6 +21,7 @@ import (
|
||||
"k8s.io/client-go/tools/record"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
|
||||
|
||||
strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1"
|
||||
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
|
||||
@@ -1032,3 +1035,539 @@ func TestReconcileAltinityRestore_RendersTheBackupsArchive(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func projectedCredentials(namespace string, updatedAt time.Time) *corev1.Secret {
|
||||
return &corev1.Secret{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Namespace: namespace,
|
||||
Name: "cozy-backups-creds",
|
||||
Annotations: map[string]string{credentialsUpdatedAtAnnotation: updatedAt.UTC().Format(time.RFC3339)},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// The ClickHouse sidecar reads the projected credentials from a mounted
|
||||
// file that kubelet rewrites only some time after the Secret changes, so a
|
||||
// strategy Job started right after a projection would reach a sidecar with
|
||||
// no or stale credentials.
|
||||
func TestReconcileAltinity_WaitsForProjectedCredentialsToReachTheSidecar(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
secret *corev1.Secret
|
||||
jobExists bool
|
||||
wantWait bool
|
||||
}{
|
||||
{name: "credentials changed just now", secret: projectedCredentials("tenant-test", time.Now()), wantWait: true},
|
||||
{name: "credentials changed long ago", secret: projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))},
|
||||
{name: "no projected credentials", secret: nil},
|
||||
{name: "the Job already runs", secret: projectedCredentials("tenant-test", time.Now()), jobExists: true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
now := metav1.Now()
|
||||
backupJob := &backupsv1alpha1.BackupJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupJobSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
BackupClassName: "clickhouse-backup",
|
||||
},
|
||||
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now},
|
||||
}
|
||||
objs := []client.Object{backupJob, newAltinityStrategy("clickhouse-strategy")}
|
||||
if tc.secret != nil {
|
||||
objs = append(objs, tc.secret)
|
||||
}
|
||||
if tc.jobExists {
|
||||
objs = append(objs, &batchv1.Job{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}})
|
||||
}
|
||||
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), clientfake.NewClientBuilder().WithObjects(objs...))
|
||||
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
|
||||
resolved := &ResolvedBackupConfig{
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
}
|
||||
ctx := context.Background()
|
||||
j := &backupsv1alpha1.BackupJob{}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
|
||||
t.Fatalf("get BackupJob: %v", err)
|
||||
}
|
||||
res, err := r.reconcileAltinity(ctx, j, resolved)
|
||||
if err != nil {
|
||||
t.Fatalf("reconcileAltinity: %v", err)
|
||||
}
|
||||
jobs := &batchv1.JobList{}
|
||||
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
|
||||
t.Fatalf("list Jobs: %v", err)
|
||||
}
|
||||
ready := meta.FindStatusCondition(j.Status.Conditions, "Ready")
|
||||
if tc.wantWait {
|
||||
if len(jobs.Items) != 0 {
|
||||
t.Errorf("a strategy Job was created %d time(s) before the credentials could reach the sidecar", len(jobs.Items))
|
||||
}
|
||||
if res.RequeueAfter <= 0 || res.RequeueAfter > altinityCredentialsPropagation {
|
||||
t.Errorf("RequeueAfter = %v, want within (0, %v]", res.RequeueAfter, altinityCredentialsPropagation)
|
||||
}
|
||||
if ready == nil || ready.Reason != "WaitingForCredentials" {
|
||||
t.Errorf("Ready = %+v, want WaitingForCredentials", ready)
|
||||
}
|
||||
return
|
||||
}
|
||||
if len(jobs.Items) != 1 {
|
||||
t.Errorf("got %d strategy Jobs, want 1", len(jobs.Items))
|
||||
}
|
||||
if ready != nil && ready.Reason == "WaitingForCredentials" {
|
||||
t.Errorf("Ready = %+v, the job should not wait", ready)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReconcileAltinityRestore_WaitsForProjectedCredentialsToReachTheSidecar(t *testing.T) {
|
||||
now := metav1.Now()
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
TakenAt: now,
|
||||
},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}},
|
||||
Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning},
|
||||
}
|
||||
_, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
|
||||
clientfake.NewClientBuilder().WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob,
|
||||
projectedCredentials("tenant-test", time.Now())))
|
||||
rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
|
||||
ctx := context.Background()
|
||||
rj := &backupsv1alpha1.RestoreJob{}
|
||||
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
res, err := rr.reconcileAltinityRestore(ctx, rj, backup)
|
||||
if err != nil {
|
||||
t.Fatalf("reconcileAltinityRestore: %v", err)
|
||||
}
|
||||
jobs := &batchv1.JobList{}
|
||||
if err := rr.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
|
||||
t.Fatalf("list Jobs: %v", err)
|
||||
}
|
||||
if len(jobs.Items) != 0 {
|
||||
t.Errorf("a restore Job was created before the credentials could reach the sidecar")
|
||||
}
|
||||
if res.RequeueAfter <= 0 {
|
||||
t.Errorf("RequeueAfter = %v, want a wait", res.RequeueAfter)
|
||||
}
|
||||
if ready := meta.FindStatusCondition(rj.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForCredentials" {
|
||||
t.Errorf("Ready = %+v, want WaitingForCredentials", ready)
|
||||
}
|
||||
}
|
||||
|
||||
// Without credentials projection configured there is no Secret to wait
|
||||
// for; client-go refuses a Get with an empty name, so it must not be asked.
|
||||
func TestReconcileAltinity_NoProjectionConfiguredDoesNotWait(t *testing.T) {
|
||||
now := metav1.Now()
|
||||
backupJob := &backupsv1alpha1.BackupJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupJobSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
BackupClassName: "clickhouse-backup",
|
||||
},
|
||||
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now},
|
||||
}
|
||||
builder := clientfake.NewClientBuilder().
|
||||
WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy")).
|
||||
WithInterceptorFuncs(interceptor.Funcs{
|
||||
Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
|
||||
if key.Name == "" {
|
||||
return errors.New("resource name may not be empty")
|
||||
}
|
||||
return cl.Get(ctx, key, obj, opts...)
|
||||
},
|
||||
})
|
||||
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder)
|
||||
resolved := &ResolvedBackupConfig{
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
}
|
||||
ctx := context.Background()
|
||||
j := &backupsv1alpha1.BackupJob{}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
|
||||
t.Fatalf("get BackupJob: %v", err)
|
||||
}
|
||||
if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil {
|
||||
t.Fatalf("reconcileAltinity: %v", err)
|
||||
}
|
||||
jobs := &batchv1.JobList{}
|
||||
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
|
||||
t.Fatalf("list Jobs: %v", err)
|
||||
}
|
||||
if len(jobs.Items) != 1 {
|
||||
t.Fatalf("got %d strategy Jobs, want 1", len(jobs.Items))
|
||||
}
|
||||
}
|
||||
|
||||
func projectedCredentialsStamped(namespace, stamp string) *corev1.Secret {
|
||||
return &corev1.Secret{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Namespace: namespace,
|
||||
Name: "cozy-backups-creds",
|
||||
Annotations: map[string]string{credentialsUpdatedAtAnnotation: stamp},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func waitingForCredentials() []metav1.Condition {
|
||||
return []metav1.Condition{credentialsWaitCondition("cozy-backups-creds", time.Minute)}
|
||||
}
|
||||
|
||||
// Once the hold is over and the Job runs, a Ready condition still saying the
|
||||
// job is waiting for credentials describes a state the job has left.
|
||||
func TestReconcileAltinity_ClearsTheCredentialsWaitOnceTheJobRuns(t *testing.T) {
|
||||
now := metav1.Now()
|
||||
backupJob := &backupsv1alpha1.BackupJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupJobSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
BackupClassName: "clickhouse-backup",
|
||||
},
|
||||
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now, Conditions: waitingForCredentials()},
|
||||
}
|
||||
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
|
||||
clientfake.NewClientBuilder().WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy"),
|
||||
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
|
||||
WithStatusSubresource(&backupsv1alpha1.BackupJob{}))
|
||||
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
|
||||
resolved := &ResolvedBackupConfig{
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
}
|
||||
ctx := context.Background()
|
||||
j := &backupsv1alpha1.BackupJob{}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
|
||||
t.Fatalf("get BackupJob: %v", err)
|
||||
}
|
||||
if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil {
|
||||
t.Fatalf("reconcileAltinity: %v", err)
|
||||
}
|
||||
jobs := &batchv1.JobList{}
|
||||
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
|
||||
t.Fatalf("list Jobs: %v", err)
|
||||
}
|
||||
if len(jobs.Items) != 1 {
|
||||
t.Fatalf("got %d strategy Jobs, want 1", len(jobs.Items))
|
||||
}
|
||||
stored := &backupsv1alpha1.BackupJob{}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), stored); err != nil {
|
||||
t.Fatalf("get BackupJob: %v", err)
|
||||
}
|
||||
if ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForCredentials" {
|
||||
t.Errorf("stored Ready = %+v while the Job runs", ready)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReconcileAltinityRestore_ClearsTheCredentialsWaitOnceTheJobRuns(t *testing.T) {
|
||||
now := metav1.Now()
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
TakenAt: now,
|
||||
},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}},
|
||||
Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning,
|
||||
Conditions: waitingForCredentials()},
|
||||
}
|
||||
_, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
|
||||
clientfake.NewClientBuilder().WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob,
|
||||
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
|
||||
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}))
|
||||
rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
|
||||
ctx := context.Background()
|
||||
rj := &backupsv1alpha1.RestoreJob{}
|
||||
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if _, err := rr.reconcileAltinityRestore(ctx, rj, backup); err != nil {
|
||||
t.Fatalf("reconcileAltinityRestore: %v", err)
|
||||
}
|
||||
stored := &backupsv1alpha1.RestoreJob{}
|
||||
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), stored); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForCredentials" {
|
||||
t.Errorf("stored Ready = %+v while the Job runs", ready)
|
||||
}
|
||||
}
|
||||
|
||||
// The stamp is only ever rewritten when the projected content changes, so a
|
||||
// value the hold cannot use must not hold the job: an unreadable one would
|
||||
// fail every reconcile, and one far in the future would wait again on each.
|
||||
// A stamp a little ahead, as a skewed clock writes it, still waits, but only
|
||||
// until that stamp plus the propagation time.
|
||||
func TestCredentialsPropagationWait_UnusableStampsDoNotHoldTheJob(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
stamp string
|
||||
wantMin time.Duration
|
||||
wantMax time.Duration
|
||||
}{
|
||||
{name: "unreadable", stamp: "garbage"},
|
||||
{name: "far in the future", stamp: "3000-01-01T00:00:00Z"},
|
||||
{name: "slightly ahead", stamp: time.Now().Add(30 * time.Second).UTC().Format(time.RFC3339),
|
||||
wantMin: altinityCredentialsPropagation, wantMax: altinityCredentialsPropagation + 31*time.Second},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
c := clientfake.NewClientBuilder().WithObjects(projectedCredentialsStamped("tenant-test", tc.stamp)).Build()
|
||||
wait, err := credentialsPropagationWait(context.Background(), c, "cozy-backups-creds", "tenant-test", "job")
|
||||
if err != nil {
|
||||
t.Fatalf("credentialsPropagationWait: %v", err)
|
||||
}
|
||||
if wait < tc.wantMin || wait > tc.wantMax {
|
||||
t.Errorf("wait = %v, want within [%v, %v]", wait, tc.wantMin, tc.wantMax)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClearCredentialsWait_KeepsAReadyConditionItDidNotSet(t *testing.T) {
|
||||
conditions := []metav1.Condition{{Type: "Ready", Status: metav1.ConditionFalse, Reason: "StrategyNotReady"}}
|
||||
if clearCredentialsWait(&conditions) {
|
||||
t.Errorf("clearCredentialsWait reported clearing a Ready condition with reason StrategyNotReady")
|
||||
}
|
||||
if ready := meta.FindStatusCondition(conditions, "Ready"); ready == nil || ready.Reason != "StrategyNotReady" {
|
||||
t.Errorf("Ready = %+v, want the StrategyNotReady condition kept", ready)
|
||||
}
|
||||
}
|
||||
|
||||
// A read the hold cannot complete must not release the Job: starting it
|
||||
// before the credentials reach the sidecar is the failure the hold prevents.
|
||||
func TestReconcileAltinity_UnreadableHoldStateDoesNotStartTheJob(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
fail func(key client.ObjectKey, obj client.Object) bool
|
||||
}{
|
||||
{name: "credentials Secret", fail: func(key client.ObjectKey, obj client.Object) bool {
|
||||
_, isSecret := obj.(*corev1.Secret)
|
||||
return isSecret && key.Name == "cozy-backups-creds"
|
||||
}},
|
||||
{name: "strategy Job", fail: func(key client.ObjectKey, obj client.Object) bool {
|
||||
_, isJob := obj.(*batchv1.Job)
|
||||
return isJob
|
||||
}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
now := metav1.Now()
|
||||
backupJob := &backupsv1alpha1.BackupJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupJobSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
BackupClassName: "clickhouse-backup",
|
||||
},
|
||||
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now},
|
||||
}
|
||||
builder := clientfake.NewClientBuilder().
|
||||
WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy"),
|
||||
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
|
||||
WithInterceptorFuncs(interceptor.Funcs{
|
||||
Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
|
||||
if tc.fail(key, obj) {
|
||||
return apierrors.NewInternalError(errors.New("etcd unavailable"))
|
||||
}
|
||||
return cl.Get(ctx, key, obj, opts...)
|
||||
},
|
||||
})
|
||||
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder)
|
||||
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
|
||||
resolved := &ResolvedBackupConfig{
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
}
|
||||
ctx := context.Background()
|
||||
j := &backupsv1alpha1.BackupJob{}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
|
||||
t.Fatalf("get BackupJob: %v", err)
|
||||
}
|
||||
if _, err := r.reconcileAltinity(ctx, j, resolved); err == nil {
|
||||
t.Errorf("reconcileAltinity returned no error although the %s could not be read", tc.name)
|
||||
}
|
||||
jobs := &batchv1.JobList{}
|
||||
if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
|
||||
t.Fatalf("list Jobs: %v", err)
|
||||
}
|
||||
if len(jobs.Items) != 0 {
|
||||
t.Errorf("a strategy Job was created although the %s could not be read", tc.name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReconcileAltinityRestore_UnreadableHoldStateDoesNotStartTheJob(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
fail func(key client.ObjectKey, obj client.Object) bool
|
||||
}{
|
||||
{name: "credentials Secret", fail: func(key client.ObjectKey, obj client.Object) bool {
|
||||
_, isSecret := obj.(*corev1.Secret)
|
||||
return isSecret && key.Name == "cozy-backups-creds"
|
||||
}},
|
||||
{name: "strategy Job", fail: func(key client.ObjectKey, obj client.Object) bool {
|
||||
_, isJob := obj.(*batchv1.Job)
|
||||
return isJob
|
||||
}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
now := metav1.Now()
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
TakenAt: now,
|
||||
},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}},
|
||||
Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning},
|
||||
}
|
||||
builder := clientfake.NewClientBuilder().
|
||||
WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob,
|
||||
projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))).
|
||||
WithInterceptorFuncs(interceptor.Funcs{
|
||||
Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
|
||||
if tc.fail(key, obj) {
|
||||
return apierrors.NewInternalError(errors.New("etcd unavailable"))
|
||||
}
|
||||
return cl.Get(ctx, key, obj, opts...)
|
||||
},
|
||||
})
|
||||
_, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder)
|
||||
rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
|
||||
ctx := context.Background()
|
||||
rj := &backupsv1alpha1.RestoreJob{}
|
||||
if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if _, err := rr.reconcileAltinityRestore(ctx, rj, backup); err == nil {
|
||||
t.Errorf("reconcileAltinityRestore returned no error although the %s could not be read", tc.name)
|
||||
}
|
||||
jobs := &batchv1.JobList{}
|
||||
if err := rr.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil {
|
||||
t.Fatalf("list Jobs: %v", err)
|
||||
}
|
||||
if len(jobs.Items) != 0 {
|
||||
t.Errorf("a restore Job was created although the %s could not be read", tc.name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A backup held for the credentials keeps the archive name it was given from
|
||||
// its persisted StartedAt: the held reconciles write the job's status, and the
|
||||
// Pod created after the hold, and the Backup recorded after it, must still
|
||||
// name the same archive, which is stamped no later than the Backup's takenAt.
|
||||
func TestReconcileAltinity_CredentialsWaitKeepsTheArchiveName(t *testing.T) {
|
||||
startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC))
|
||||
backupJob := &backupsv1alpha1.BackupJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"},
|
||||
Spec: backupsv1alpha1.BackupJobSpec{
|
||||
ApplicationRef: newAltinityClickHouseRef("ch-test"),
|
||||
BackupClassName: "clickhouse-backup",
|
||||
},
|
||||
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt, Phase: backupsv1alpha1.BackupJobPhaseRunning},
|
||||
}
|
||||
secret := projectedCredentials("tenant-test", time.Now())
|
||||
r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"),
|
||||
clientfake.NewClientBuilder().WithObjects(backupJob, newAltinityArchiveStrategy("clickhouse-strategy"), secret))
|
||||
r.CredentialsConfig.TargetSecretName = "cozy-backups-creds"
|
||||
resolved := &ResolvedBackupConfig{
|
||||
StrategyRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new(strategyv1alpha1.GroupVersion.Group),
|
||||
Kind: strategyv1alpha1.AltinityStrategyKind,
|
||||
Name: "clickhouse-strategy",
|
||||
},
|
||||
}
|
||||
ctx := context.Background()
|
||||
const want = "clickhouse-ch-test-20260304T050607Z"
|
||||
reconcile := func(step string) {
|
||||
t.Helper()
|
||||
j := &backupsv1alpha1.BackupJob{}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil {
|
||||
t.Fatalf("%s: get BackupJob: %v", step, err)
|
||||
}
|
||||
if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil {
|
||||
t.Fatalf("%s: reconcileAltinity: %v", step, err)
|
||||
}
|
||||
}
|
||||
|
||||
reconcile("held")
|
||||
job := &batchv1.Job{}
|
||||
if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); !apierrors.IsNotFound(err) {
|
||||
t.Fatalf("the Job exists while the credentials wait should hold it: %v", err)
|
||||
}
|
||||
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(secret), secret); err != nil {
|
||||
t.Fatalf("get projected Secret: %v", err)
|
||||
}
|
||||
secret.Annotations[credentialsUpdatedAtAnnotation] = time.Now().Add(-altinityCredentialsPropagation - time.Second).UTC().Format(time.RFC3339)
|
||||
if err := r.Update(ctx, secret); err != nil {
|
||||
t.Fatalf("age the credentials stamp: %v", err)
|
||||
}
|
||||
reconcile("after the wait")
|
||||
if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); err != nil {
|
||||
t.Fatalf("get backup Job: %v", err)
|
||||
}
|
||||
if got := job.Spec.Template.Spec.Containers[0].Args[1]; got != "--archive="+want {
|
||||
t.Fatalf("backup Pod was rendered with %q after the wait, want --archive=%s", got, want)
|
||||
}
|
||||
|
||||
job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}}
|
||||
if err := r.Status().Update(ctx, job); err != nil {
|
||||
t.Fatalf("mark backup Job complete: %v", err)
|
||||
}
|
||||
reconcile("completed")
|
||||
created := &backupsv1alpha1.Backup{}
|
||||
if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: backupJob.Name}, created); err != nil {
|
||||
t.Fatalf("get Backup: %v", err)
|
||||
}
|
||||
if got := created.Spec.DriverMetadata[altinityArchiveMetaKey]; got != want {
|
||||
t.Errorf("Backup records archive %q, want %q", got, want)
|
||||
}
|
||||
if stamp := created.Spec.TakenAt.UTC().Format(altinityArchiveTimeLayout); stamp < "20260304T050607Z" {
|
||||
t.Errorf("Backup takenAt %s is earlier than its archive's stamp", stamp)
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,14 @@
|
||||
package backupcontroller
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -15,6 +18,8 @@ import (
|
||||
"k8s.io/apimachinery/pkg/types"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
|
||||
"sigs.k8s.io/controller-runtime/pkg/log"
|
||||
"sigs.k8s.io/yaml"
|
||||
)
|
||||
|
||||
// CredentialsProjectionRequeue is the backoff between reconcile attempts
|
||||
@@ -73,9 +78,8 @@ func IsTransient(err error) bool {
|
||||
// BackupCredentialsConfig describes the platform-managed source Secret
|
||||
// produced by the cozy-backups Bucket, the per-tenant projection target
|
||||
// Secret name, and the bucket coordinates (endpoint used to format the
|
||||
// FDB blob_credentials account-host key, region carried verbatim for
|
||||
// drivers like clickhouse-backup that wire it through env, forcePathStyle
|
||||
// projected as a flat key so the chart-emitted ClickHouse sidecar can read
|
||||
// FDB blob_credentials account-host key, region carried verbatim,
|
||||
// forcePathStyle projected so the chart-emitted ClickHouse sidecar can read
|
||||
// a platform value it cannot see directly). Values are wired from
|
||||
// environment variables in main() and the deployment chart populates them
|
||||
// from .Values.backupStorage.
|
||||
@@ -85,11 +89,11 @@ type BackupCredentialsConfig struct {
|
||||
TargetSecretName string
|
||||
Endpoint string
|
||||
Region string
|
||||
// ForcePathStyle is the string "true"/"false" from
|
||||
// backupStorage.forcePathStyle, projected into the credentials Secret
|
||||
// under the forcePathStyle key. The app charts cannot read
|
||||
// backupStorage.* directly, so the ClickHouse sidecar consumes it via
|
||||
// an optional secretKeyRef as S3_FORCE_PATH_STYLE.
|
||||
// ForcePathStyle is backupStorage.forcePathStyle as the chart
|
||||
// normalises it ("true", "false", or "" when unset), projected into the credentials Secret
|
||||
// under the forcePathStyle key and into clickhouse-backup.yml. The app
|
||||
// charts cannot read backupStorage.* directly, so the ClickHouse
|
||||
// sidecar takes it from there.
|
||||
ForcePathStyle string
|
||||
}
|
||||
|
||||
@@ -107,10 +111,11 @@ func (c BackupCredentialsConfig) IsEnabled() bool {
|
||||
// the data so every default Strategy CR can consume the same Secret:
|
||||
//
|
||||
// - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — CNPG, MariaDB, Etcd.
|
||||
// - accessKey / secretKey + bucketName / endpoint / region — ClickHouse
|
||||
// sidecar.
|
||||
// - accessKey / secretKey + bucketName / endpoint / region — Redis
|
||||
// strategy, and ClickHouse sidecars rendered before clickhouse-backup.yml.
|
||||
// - cloud — Velero credentials file ([default] section).
|
||||
// - blob_credentials.json — FoundationDB backup_agent.
|
||||
// - clickhouse-backup.yml — ClickHouse sidecar, mounted as its config file.
|
||||
//
|
||||
// The source Secret is read in two formats:
|
||||
//
|
||||
@@ -156,8 +161,8 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr
|
||||
if creds.endpoint == "" {
|
||||
// Scheme-strip the env-var fallback to match the source-Secret path
|
||||
// (parseSourceSecret strips there). The projected `endpoint` key is
|
||||
// consumed verbatim as S3_ENDPOINT by the ClickHouse sidecar, which
|
||||
// expects a bare host:port — see https://cozystack.io/docs/v1/operations/services/backup-classes/.
|
||||
// a bare host:port and each consumer adds its own scheme — see
|
||||
// https://cozystack.io/docs/v1/operations/services/backup-classes/.
|
||||
creds.endpoint = stripScheme(cfg.Endpoint)
|
||||
}
|
||||
if creds.endpoint == "" {
|
||||
@@ -177,6 +182,20 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr
|
||||
}
|
||||
|
||||
cloud := buildVeleroCredentialsFile(creds.accessKey, creds.secretKey)
|
||||
clickHouseConfig, pathStyleErr, err := buildClickHouseBackupConfig(creds, cfg.ForcePathStyle)
|
||||
if pathStyleErr != nil {
|
||||
// The chart refuses such a value at render time, so only an env set
|
||||
// around it gets here. It concerns the ClickHouse file alone, and
|
||||
// failing the projection would take every other driver's
|
||||
// credentials down with it.
|
||||
log.FromContext(ctx).Error(pathStyleErr, "left force_path_style out of "+clickHouseBackupConfigKey, "namespace", targetNamespace)
|
||||
}
|
||||
if err != nil {
|
||||
return &ProjectionError{
|
||||
Reason: ReasonSourceMalformed,
|
||||
Message: fmt.Sprintf("build %s: %v", clickHouseBackupConfigKey, err),
|
||||
}
|
||||
}
|
||||
blob, err := buildFDBBlobCredentials(creds.endpoint, creds.accessKey, creds.secretKey)
|
||||
if err != nil {
|
||||
return &ProjectionError{
|
||||
@@ -231,28 +250,36 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr
|
||||
if target.Data == nil {
|
||||
target.Data = map[string][]byte{}
|
||||
}
|
||||
before := maps.Clone(target.Data)
|
||||
target.Data["AWS_ACCESS_KEY_ID"] = []byte(creds.accessKey)
|
||||
target.Data["AWS_SECRET_ACCESS_KEY"] = []byte(creds.secretKey)
|
||||
// Mirror under the legacy bucket-controller key names so
|
||||
// chart-emitted sidecars (clickhouse-backup) can consume the
|
||||
// same Secret without a separate field map.
|
||||
// Mirror under the legacy bucket-controller key names so the Redis
|
||||
// strategy and older ClickHouse charts can consume the same Secret
|
||||
// without a separate field map.
|
||||
target.Data["accessKey"] = []byte(creds.accessKey)
|
||||
target.Data["secretKey"] = []byte(creds.secretKey)
|
||||
target.Data["cloud"] = []byte(cloud)
|
||||
target.Data["blob_credentials.json"] = blob
|
||||
target.Data[clickHouseBackupConfigKey] = clickHouseConfig
|
||||
// Always overwrite endpoint/bucket/region (delete if empty in
|
||||
// source) so a re-projection cannot leave stale values lingering
|
||||
// after the bucket-controller rotates the source. Consumers
|
||||
// (ClickHouse sidecar via secretKeyRef) must see exactly what the
|
||||
// source carries, not a half-stale Secret.
|
||||
// after the bucket-controller rotates the source. Consumers must
|
||||
// see exactly what the source carries, not a half-stale Secret.
|
||||
setOrDelete(target.Data, "endpoint", creds.endpoint)
|
||||
setOrDelete(target.Data, "bucketName", creds.bucket)
|
||||
setOrDelete(target.Data, "region", creds.region)
|
||||
// forcePathStyle comes from the platform config (backupStorage.
|
||||
// forcePathStyle), not the source Secret: the ClickHouse sidecar
|
||||
// reads it as S3_FORCE_PATH_STYLE because the app chart cannot see
|
||||
// forcePathStyle), not the source Secret: ClickHouse sidecars
|
||||
// rendered before clickhouse-backup.yml read it as
|
||||
// S3_FORCE_PATH_STYLE because the app chart cannot see
|
||||
// backupStorage.* directly.
|
||||
setOrDelete(target.Data, "forcePathStyle", cfg.ForcePathStyle)
|
||||
if !maps.EqualFunc(before, target.Data, bytes.Equal) {
|
||||
if target.Annotations == nil {
|
||||
target.Annotations = map[string]string{}
|
||||
}
|
||||
target.Annotations[credentialsUpdatedAtAnnotation] = time.Now().UTC().Format(time.RFC3339)
|
||||
}
|
||||
return nil
|
||||
}); err != nil {
|
||||
// If the mutator refused to overwrite an unowned Secret, surface
|
||||
@@ -414,6 +441,48 @@ func buildFDBBlobCredentials(endpoint, accessKey, secretKey string) ([]byte, err
|
||||
return json.Marshal(doc)
|
||||
}
|
||||
|
||||
// credentialsUpdatedAtAnnotation records when the projected content last
|
||||
// changed. A consumer that mounts the Secret as a volume sees a change only
|
||||
// once kubelet rewrites the volume, so it uses this to tell how fresh its
|
||||
// copy can be.
|
||||
const credentialsUpdatedAtAnnotation = "backups.cozystack.io/credentials-updated-at"
|
||||
|
||||
// clickHouseBackupConfigKey holds the s3 section of a clickhouse-backup
|
||||
// config. The ClickHouse chart mounts it as the sidecar's config file
|
||||
// instead of passing the values as environment: clickhouse-backup reloads
|
||||
// the file in each API handler that reaches S3, among them the
|
||||
// create_remote, restore_remote and list calls the strategy makes, and
|
||||
// kubelet rewrites a mounted Secret in place, so credentials projected
|
||||
// after the pod started, or rotated since, reach a running sidecar.
|
||||
const clickHouseBackupConfigKey = "clickhouse-backup.yml"
|
||||
|
||||
// buildClickHouseBackupConfig renders that file. The projected endpoint is
|
||||
// a bare host, and clickhouse-backup's S3 client rejects one without a
|
||||
// scheme. A forcePathStyle that is not a boolean is left out of the file
|
||||
// and reported in pathStyleErr; err is a failure to render the file at all.
|
||||
func buildClickHouseBackupConfig(creds parsedCreds, forcePathStyle string) (config []byte, pathStyleErr, err error) {
|
||||
s3 := map[string]any{
|
||||
"bucket": creds.bucket,
|
||||
"endpoint": "https://" + creds.endpoint,
|
||||
"access_key": creds.accessKey,
|
||||
"secret_key": creds.secretKey,
|
||||
}
|
||||
// An empty region in the file would override clickhouse-backup's own
|
||||
// us-east-1 default instead of leaving it in place.
|
||||
if creds.region != "" {
|
||||
s3["region"] = creds.region
|
||||
}
|
||||
if forcePathStyle != "" {
|
||||
if pathStyle, perr := strconv.ParseBool(forcePathStyle); perr != nil {
|
||||
pathStyleErr = fmt.Errorf("forcePathStyle %q is not a boolean", forcePathStyle)
|
||||
} else {
|
||||
s3["force_path_style"] = pathStyle
|
||||
}
|
||||
}
|
||||
config, err = yaml.Marshal(map[string]any{"s3": s3})
|
||||
return config, pathStyleErr, err
|
||||
}
|
||||
|
||||
// stripScheme returns the endpoint stripped of any leading scheme prefix
|
||||
// (http:// or https://) and preserves the path portion if present. The
|
||||
// resulting form is what `host[:port][/path]` callers (MariaDB strategy,
|
||||
|
||||
@@ -5,7 +5,11 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-logr/logr/funcr"
|
||||
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
@@ -13,6 +17,8 @@ import (
|
||||
"k8s.io/apimachinery/pkg/types"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client/fake"
|
||||
"sigs.k8s.io/controller-runtime/pkg/log"
|
||||
"sigs.k8s.io/yaml"
|
||||
)
|
||||
|
||||
func newFakeClient(objs ...client.Object) client.Client {
|
||||
@@ -664,3 +670,188 @@ func TestProject_HappyPath_HTTPEndpoint(t *testing.T) {
|
||||
t.Errorf("blob account host not stripped: %+v", blob.Accounts)
|
||||
}
|
||||
}
|
||||
|
||||
// The ClickHouse sidecar reads its S3 settings from this key as a mounted
|
||||
// file, which kubelet refreshes in place: clickhouse-backup reloads its
|
||||
// config file in each API handler that reaches S3, while its environment
|
||||
// is fixed at start.
|
||||
func TestProject_WritesClickHouseBackupConfig(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
forcePathStyle string
|
||||
wantPathStyle *bool
|
||||
}{
|
||||
{name: "path style on", forcePathStyle: "true", wantPathStyle: ptrBool(true)},
|
||||
{name: "path style off", forcePathStyle: "false", wantPathStyle: ptrBool(false)},
|
||||
{name: "path style unset", forcePathStyle: ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
c := newFakeClient(flatSourceSecret())
|
||||
ctx := context.Background()
|
||||
cfg := defaultCfg()
|
||||
cfg.ForcePathStyle = tc.forcePathStyle
|
||||
if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
got := &corev1.Secret{}
|
||||
if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil {
|
||||
t.Fatalf("projected Secret not found: %v", err)
|
||||
}
|
||||
var doc struct {
|
||||
S3 map[string]any `json:"s3"`
|
||||
}
|
||||
if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil {
|
||||
t.Fatalf("%s is not YAML: %v\n%s", clickHouseBackupConfigKey, err, got.Data[clickHouseBackupConfigKey])
|
||||
}
|
||||
for key, want := range map[string]string{
|
||||
"bucket": "cozy-backups",
|
||||
"endpoint": "https://seaweedfs-s3.tenant-root.svc:8333",
|
||||
"region": "us-east-1",
|
||||
"access_key": "AK",
|
||||
"secret_key": "SK",
|
||||
} {
|
||||
if doc.S3[key] != want {
|
||||
t.Errorf("s3.%s = %v, want %q", key, doc.S3[key], want)
|
||||
}
|
||||
}
|
||||
pathStyle, set := doc.S3["force_path_style"]
|
||||
switch {
|
||||
case tc.wantPathStyle == nil && set:
|
||||
t.Errorf("s3.force_path_style = %v, want it left to the clickhouse-backup default", pathStyle)
|
||||
case tc.wantPathStyle != nil && pathStyle != *tc.wantPathStyle:
|
||||
t.Errorf("s3.force_path_style = %v, want %v", pathStyle, *tc.wantPathStyle)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// clickhouse-backup defaults the S3 region to us-east-1; an empty value in
|
||||
// its config file would replace that default rather than fall back to it.
|
||||
func TestProject_ClickHouseBackupConfigOmitsAnEmptyRegion(t *testing.T) {
|
||||
c := newFakeClient(flatSourceSecret())
|
||||
ctx := context.Background()
|
||||
cfg := defaultCfg()
|
||||
cfg.Region = ""
|
||||
if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
got := &corev1.Secret{}
|
||||
if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil {
|
||||
t.Fatalf("projected Secret not found: %v", err)
|
||||
}
|
||||
var doc struct {
|
||||
S3 map[string]any `json:"s3"`
|
||||
}
|
||||
if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil {
|
||||
t.Fatalf("%s is not YAML: %v", clickHouseBackupConfigKey, err)
|
||||
}
|
||||
if region, set := doc.S3["region"]; set {
|
||||
t.Errorf("s3.region = %q, want it left to the clickhouse-backup default", region)
|
||||
}
|
||||
}
|
||||
|
||||
// The parse only serves the ClickHouse file, while the same projection
|
||||
// carries every other driver's credentials. An unparseable platform value
|
||||
// must cost ClickHouse its path-style setting and say so, not fail the
|
||||
// projection for all of them.
|
||||
func TestProject_MalformedForcePathStyleSkipsOnlyTheClickHouseSetting(t *testing.T) {
|
||||
c := newFakeClient(flatSourceSecret())
|
||||
var logged []string
|
||||
ctx := log.IntoContext(context.Background(), funcr.New(func(prefix, args string) {
|
||||
logged = append(logged, args)
|
||||
}, funcr.Options{}))
|
||||
cfg := defaultCfg()
|
||||
cfg.ForcePathStyle = "yes"
|
||||
if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil {
|
||||
t.Fatalf("projection failed for every driver over a ClickHouse-only setting: %v", err)
|
||||
}
|
||||
got := &corev1.Secret{}
|
||||
if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil {
|
||||
t.Fatalf("projected Secret not found: %v", err)
|
||||
}
|
||||
if string(got.Data["AWS_ACCESS_KEY_ID"]) != "AK" {
|
||||
t.Errorf("AWS_ACCESS_KEY_ID = %q, want the projected key", got.Data["AWS_ACCESS_KEY_ID"])
|
||||
}
|
||||
if string(got.Data["forcePathStyle"]) != "yes" {
|
||||
t.Errorf("forcePathStyle = %q, want the platform value carried verbatim", got.Data["forcePathStyle"])
|
||||
}
|
||||
var doc struct {
|
||||
S3 map[string]any `json:"s3"`
|
||||
}
|
||||
if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil {
|
||||
t.Fatalf("%s is not YAML: %v", clickHouseBackupConfigKey, err)
|
||||
}
|
||||
if doc.S3["access_key"] != "AK" {
|
||||
t.Errorf("s3.access_key = %v, want the rest of the file written", doc.S3["access_key"])
|
||||
}
|
||||
if pathStyle, set := doc.S3["force_path_style"]; set {
|
||||
t.Errorf("s3.force_path_style = %v, want it left out", pathStyle)
|
||||
}
|
||||
found := false
|
||||
for _, line := range logged {
|
||||
if strings.Contains(line, `"error"`) && strings.Contains(line, `\"yes\"`) && strings.Contains(line, "force_path_style") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("no error naming the rejected value and the skipped setting was logged; got %q", logged)
|
||||
}
|
||||
}
|
||||
|
||||
func ptrBool(b bool) *bool { return &b }
|
||||
|
||||
// Consumers that read the Secret through a mounted volume need to know when
|
||||
// its content last changed; an unchanged re-projection must keep the time.
|
||||
func TestProject_StampsWhenContentChanges(t *testing.T) {
|
||||
src := flatSourceSecret()
|
||||
c := newFakeClient(src)
|
||||
ctx := context.Background()
|
||||
key := types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}
|
||||
stamp := func() string {
|
||||
got := &corev1.Secret{}
|
||||
if err := c.Get(ctx, key, got); err != nil {
|
||||
t.Fatalf("projected Secret not found: %v", err)
|
||||
}
|
||||
return got.Annotations[credentialsUpdatedAtAnnotation]
|
||||
}
|
||||
|
||||
if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil {
|
||||
t.Fatalf("first projection: %v", err)
|
||||
}
|
||||
first := stamp()
|
||||
at, err := time.Parse(time.RFC3339, first)
|
||||
if err != nil || time.Since(at) > time.Minute {
|
||||
t.Fatalf("first projection stamp %q is not a recent RFC3339 time: %v", first, err)
|
||||
}
|
||||
|
||||
old := &corev1.Secret{}
|
||||
if err := c.Get(ctx, key, old); err != nil {
|
||||
t.Fatalf("get projected Secret: %v", err)
|
||||
}
|
||||
old.Annotations[credentialsUpdatedAtAnnotation] = "2000-01-01T00:00:00Z"
|
||||
if err := c.Update(ctx, old); err != nil {
|
||||
t.Fatalf("age the stamp: %v", err)
|
||||
}
|
||||
if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil {
|
||||
t.Fatalf("unchanged re-projection: %v", err)
|
||||
}
|
||||
if got := stamp(); got != "2000-01-01T00:00:00Z" {
|
||||
t.Fatalf("an unchanged re-projection moved the stamp to %q", got)
|
||||
}
|
||||
|
||||
rotated := &corev1.Secret{}
|
||||
if err := c.Get(ctx, types.NamespacedName{Namespace: src.Namespace, Name: src.Name}, rotated); err != nil {
|
||||
t.Fatalf("get source Secret: %v", err)
|
||||
}
|
||||
rotated.Data["secretKey"] = []byte("SK2")
|
||||
if err := c.Update(ctx, rotated); err != nil {
|
||||
t.Fatalf("rotate source Secret: %v", err)
|
||||
}
|
||||
if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil {
|
||||
t.Fatalf("rotated re-projection: %v", err)
|
||||
}
|
||||
if got := stamp(); got == "2000-01-01T00:00:00Z" {
|
||||
t.Fatalf("a rotation did not move the stamp")
|
||||
}
|
||||
}
|
||||
@@ -225,15 +225,20 @@ spec:
|
||||
- name: REMOTE_STORAGE
|
||||
value: s3
|
||||
{{- /*
|
||||
On the useSystemBucket flow every S3 setting but the
|
||||
release's own S3_PATH below, the platform's
|
||||
backupStorage.forcePathStyle included, comes
|
||||
from the clickhouse-backup.yml key the platform projects
|
||||
into cozy-backups-creds, mounted below as the config file.
|
||||
clickhouse-backup reloads that file in each API handler
|
||||
that reaches S3 and kubelet rewrites a mounted Secret in
|
||||
place, so credentials projected after the Pod started, or
|
||||
rotated since, reach the running sidecar; environment
|
||||
would be read only at start and would override the file,
|
||||
so none of those is set here.
|
||||
|
||||
S3_FORCE_PATH_STYLE selects path-style vs vhost-style S3
|
||||
addressing: SeaweedFS (the platform default) needs
|
||||
path-style, while AWS public S3 needs vhost-style in many
|
||||
regions. On the useSystemBucket flow the value is the
|
||||
platform's backupStorage.forcePathStyle, which this app
|
||||
chart cannot read directly, so it is projected into
|
||||
cozy-backups-creds and consumed via an optional
|
||||
secretKeyRef (optional: true so the Pod starts before the
|
||||
first projection). On the legacy paths (own Secret via
|
||||
addressing. On the legacy paths (own Secret via
|
||||
s3CredentialsSecret.name, or a configured endpoint) we
|
||||
cannot peek inside the Secret at render time, so we keep
|
||||
the historical path-style default — AWS users in that mode
|
||||
@@ -241,12 +246,8 @@ spec:
|
||||
clickhouse-backup config.
|
||||
*/}}
|
||||
{{- if .Values.backup.useSystemBucket }}
|
||||
- name: S3_FORCE_PATH_STYLE
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ $sidecarSecretName | quote }}
|
||||
key: forcePathStyle
|
||||
optional: true
|
||||
- name: CLICKHOUSE_BACKUP_CONFIG
|
||||
value: /etc/clickhouse-backup-s3/config.yml
|
||||
{{- else if or (not (eq (default "" .Values.backup.s3CredentialsSecret.name) "")) .Values.backup.endpoint }}
|
||||
- name: S3_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
@@ -282,44 +283,18 @@ spec:
|
||||
*/}}
|
||||
- name: S3_PATH
|
||||
value: {{ $s3Path | quote }}
|
||||
{{- /*
|
||||
optional: true in the useSystemBucket path so the Pod
|
||||
starts even before the platform projector materialises
|
||||
cozy-backups-creds in this namespace (projection runs at
|
||||
first BackupJob time; the sidecar surfaces backup errors
|
||||
only when its HTTP API is exercised). Without this gate
|
||||
every fresh ClickHouse release with useSystemBucket=true
|
||||
would hang in CreateContainerConfigError until a
|
||||
BackupJob fires. Legacy path keeps the strict reference
|
||||
shape — its Secret is chart-emitted and present at
|
||||
install time.
|
||||
*/}}
|
||||
{{- $secretOptional := .Values.backup.useSystemBucket }}
|
||||
{{- if not .Values.backup.useSystemBucket }}
|
||||
- name: S3_BUCKET
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $bucketKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
|
||||
{{- /*
|
||||
clickhouse-backup's aws-sdk-go-v2 client rejects a scheme-less
|
||||
S3_ENDPOINT ("was not a valid URI"). On the useSystemBucket flow
|
||||
the projected cozy-backups-creds.endpoint is a bare host, so
|
||||
prepend https:// (the platform's CNPG endpoint helper does the
|
||||
same). Legacy / BYO-Secret endpoints are tenant-supplied with
|
||||
their own scheme and are consumed verbatim.
|
||||
*/}}
|
||||
{{- if .Values.backup.useSystemBucket }}
|
||||
- name: S3_ENDPOINT_HOST
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $endpointKey | quote }}, optional: true } }
|
||||
- name: S3_ENDPOINT
|
||||
value: "https://$(S3_ENDPOINT_HOST)"
|
||||
{{- else }}
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $bucketKey | quote }} } }
|
||||
- name: S3_ENDPOINT
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $endpointKey | quote }} } }
|
||||
{{- end }}
|
||||
- name: S3_REGION
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $regionKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $regionKey | quote }} } }
|
||||
- name: S3_ACCESS_KEY
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $accessKeyKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $accessKeyKey | quote }} } }
|
||||
- name: S3_SECRET_KEY
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $secretKeyKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } }
|
||||
valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $secretKeyKey | quote }} } }
|
||||
{{- end }}
|
||||
{{- if $endpointCAName }}
|
||||
# Trust the mounted private-CA bundle for the S3 endpoint (see
|
||||
# the `backup.endpointCA` field in values.yaml). SSL_CERT_DIR,
|
||||
@@ -343,6 +318,11 @@ spec:
|
||||
- name: backup-s3-ca
|
||||
mountPath: /etc/ssl/cozy-s3-ca
|
||||
readOnly: true
|
||||
{{- end }}
|
||||
{{- if .Values.backup.useSystemBucket }}
|
||||
- name: backup-s3-config
|
||||
mountPath: /etc/clickhouse-backup-s3
|
||||
readOnly: true
|
||||
{{- end }}
|
||||
securityContext:
|
||||
# clickhouse-server stores data under /var/lib/clickhouse with
|
||||
@@ -366,13 +346,14 @@ spec:
|
||||
runAsGroup: 0
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
{{- if $endpointCAName }}
|
||||
# Private-CA bundle for the backup sidecar, projected as ca.crt for
|
||||
# SSL_CERT_DIR above. Not optional on purpose: a missing Secret holds
|
||||
# the whole ClickHouse Pod in ContainerCreating (naming the Secret in
|
||||
# `kubectl describe`) rather than mounting an empty trust dir and
|
||||
# failing later inside clickhouse-backup with an opaque x509 error.
|
||||
{{- if or $endpointCAName .Values.backup.useSystemBucket }}
|
||||
volumes:
|
||||
{{- if $endpointCAName }}
|
||||
# Private-CA bundle for the backup sidecar, projected as ca.crt for
|
||||
# SSL_CERT_DIR above. Not optional on purpose: a missing Secret holds
|
||||
# the whole ClickHouse Pod in ContainerCreating (naming the Secret in
|
||||
# `kubectl describe`) rather than mounting an empty trust dir and
|
||||
# failing later inside clickhouse-backup with an opaque x509 error.
|
||||
- name: backup-s3-ca
|
||||
secret:
|
||||
secretName: {{ $endpointCAName | quote }}
|
||||
@@ -380,6 +361,18 @@ spec:
|
||||
- key: {{ $endpointCAKey | quote }}
|
||||
path: ca.crt
|
||||
{{- end }}
|
||||
{{- if .Values.backup.useSystemBucket }}
|
||||
# Optional: the platform projects cozy-backups-creds into this
|
||||
# namespace at the first BackupJob, after the Pod has started.
|
||||
- name: backup-s3-config
|
||||
secret:
|
||||
secretName: {{ $sidecarSecretName | quote }}
|
||||
optional: true
|
||||
items:
|
||||
- key: clickhouse-backup.yml
|
||||
path: config.yml
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
serviceTemplates:
|
||||
- name: svc-template
|
||||
|
||||
@@ -2,10 +2,12 @@ suite: backup sidecar S3 wiring (useSystemBucket vs legacy, path-override guard)
|
||||
|
||||
# Locks in the clickhouse-backup sidecar S3 wiring this PR introduced on the
|
||||
# ClickHouseInstallation PodTemplate:
|
||||
# - useSystemBucket=true points every S3_* env at the platform-projected
|
||||
# cozy-backups-creds Secret with optional:true (the Pod starts before the
|
||||
# first projection) and scopes S3_PATH to <namespace>/<release>; the
|
||||
# chart-managed <release>-backup-s3 / <release>-backup Secrets are skipped;
|
||||
# - useSystemBucket=true mounts the clickhouse-backup.yml key of the
|
||||
# platform-projected cozy-backups-creds Secret as the sidecar's config
|
||||
# file with optional:true (the Pod starts before the first projection),
|
||||
# sets no other S3 setting in the environment, and scopes S3_PATH to
|
||||
# <namespace>/<release>; the chart-managed <release>-backup-s3 /
|
||||
# <release>-backup Secrets are skipped;
|
||||
# - the legacy flow keeps the chart-emitted <release>-backup-s3 Secret and a
|
||||
# strict (non-optional) secretKeyRef;
|
||||
# - s3PathOverride + useSystemBucket fails fast at render time.
|
||||
@@ -25,7 +27,7 @@ set:
|
||||
_cluster: {}
|
||||
|
||||
tests:
|
||||
- it: "useSystemBucket=true: sidecar reads cozy-backups-creds (optional) and scopes S3_PATH"
|
||||
- it: "useSystemBucket=true: sidecar reads its S3 settings from the projected config file and scopes S3_PATH"
|
||||
template: templates/clickhouse.yaml
|
||||
documentSelector:
|
||||
path: kind
|
||||
@@ -45,28 +47,30 @@ tests:
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_PATH")].value
|
||||
value: tenant-test/ch-test
|
||||
# clickhouse-backup reloads this file in each API handler that reaches
|
||||
# S3 and kubelet rewrites the mounted Secret in place, so credentials
|
||||
# projected after the Pod started, or rotated since, reach the running
|
||||
# sidecar.
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.name
|
||||
value: cozy-backups-creds
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="CLICKHOUSE_BACKUP_CONFIG")].value
|
||||
value: /etc/clickhouse-backup-s3/config.yml
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.optional
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].volumeMounts[?(@.name=="backup-s3-config")].mountPath
|
||||
value: /etc/clickhouse-backup-s3
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].volumeMounts[?(@.name=="backup-s3-config")].readOnly
|
||||
value: true
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_SECRET_KEY")].valueFrom.secretKeyRef.name
|
||||
value: cozy-backups-creds
|
||||
# The projected endpoint is a bare host; clickhouse-backup needs a URI, so
|
||||
# the chart prepends https:// via env-var expansion on this flow.
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")].valueFrom.secretKeyRef.name
|
||||
value: cozy-backups-creds
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")].valueFrom.secretKeyRef.key
|
||||
value: endpoint
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT")].value
|
||||
value: https://$(S3_ENDPOINT_HOST)
|
||||
path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-config")].secret
|
||||
value:
|
||||
secretName: cozy-backups-creds
|
||||
# The Pod starts before the first projection.
|
||||
optional: true
|
||||
items:
|
||||
- key: clickhouse-backup.yml
|
||||
path: config.yml
|
||||
|
||||
- it: "useSystemBucket=true: S3_FORCE_PATH_STYLE is sourced from cozy-backups-creds, not hardcoded"
|
||||
- it: "useSystemBucket=true: no S3 setting is fixed in the sidecar environment"
|
||||
template: templates/clickhouse.yaml
|
||||
documentSelector:
|
||||
path: kind
|
||||
@@ -76,18 +80,21 @@ tests:
|
||||
enabled: true
|
||||
useSystemBucket: true
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.name
|
||||
value: cozy-backups-creds
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.key
|
||||
value: forcePathStyle
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.optional
|
||||
value: true
|
||||
# Environment overrides the config file and is read only at start.
|
||||
- notExists:
|
||||
# must no longer be the hardcoded literal value: "true"
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].value
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")]
|
||||
- notExists:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT")]
|
||||
- notExists:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")]
|
||||
- notExists:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_REGION")]
|
||||
- notExists:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ACCESS_KEY")]
|
||||
- notExists:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_SECRET_KEY")]
|
||||
- notExists:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")]
|
||||
|
||||
- it: "useSystemBucket=true: chart-managed backup Secrets are skipped"
|
||||
set:
|
||||
@@ -240,7 +247,10 @@ tests:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="SSL_CERT_DIR")].value
|
||||
value: /etc/ssl/cozy-s3-ca:/etc/ssl/certs
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.name
|
||||
path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-ca")].secret.secretName
|
||||
value: my-s3-ca
|
||||
- equal:
|
||||
path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-config")].secret.secretName
|
||||
value: cozy-backups-creds
|
||||
|
||||
# backup.enabled=false renders no sidecar at all, so the CA must not leak a
|
||||
|
||||
@@ -190,3 +190,28 @@ true
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
backupStorage.forcePathStyle as "true", "false", or "" when unset. Its
|
||||
consumers read it differently: the controller and Velero parse it with
|
||||
strconv.ParseBool, the etcd and MongoDB strategies write it into a boolean
|
||||
field that Helm decodes as YAML 1.1 (so "yes" or "off" is a boolean there),
|
||||
and the Kafka and RabbitMQ scripts compare the string with "false".
|
||||
Normalising it here means a spelling cannot mean path style to some drivers
|
||||
and not to others. The spellings accepted are those of strconv.ParseBool and
|
||||
of YAML 1.1 booleans, a superset of the boolean spellings the etcd and
|
||||
MongoDB strategies accepted; any other value fails the render.
|
||||
*/}}
|
||||
{{- define "backupstrategy-controller.forcePathStyle" -}}
|
||||
{{- $value := .Values.backupStorage.forcePathStyle -}}
|
||||
{{- if not (kindIs "invalid" $value) -}}
|
||||
{{- $raw := toString $value -}}
|
||||
{{- if has $raw (list "1" "t" "T" "true" "True" "TRUE" "y" "Y" "yes" "Yes" "YES" "on" "On" "ON") -}}
|
||||
true
|
||||
{{- else if has $raw (list "0" "f" "F" "false" "False" "FALSE" "n" "N" "no" "No" "NO" "off" "Off" "OFF") -}}
|
||||
false
|
||||
{{- else if ne $raw "" -}}
|
||||
{{- fail (printf "backupStorage.forcePathStyle must be a boolean or unset, got %q" $raw) -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
@@ -47,7 +47,7 @@ spec:
|
||||
- name: BACKUP_STORAGE_REGION
|
||||
value: {{ .Values.backupStorage.region | quote }}
|
||||
- name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: {{ .Values.backupStorage.forcePathStyle | quote }}
|
||||
value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
|
||||
- name: BACKUP_STORAGE_SYSTEM_NAMESPACES
|
||||
value: {{ .Values.backupStorage.systemNamespaces | join "," | quote }}
|
||||
{{- if .Values.backupStorage.reconcileDefaultObjects }}
|
||||
|
||||
@@ -15,7 +15,7 @@ spec:
|
||||
endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }}
|
||||
key: {{ printf "{{ .Application.metadata.namespace }}/{{ .Application.metadata.name }}/" | quote }}
|
||||
region: {{ .Values.backupStorage.region | quote }}
|
||||
forcePathStyle: {{ .Values.backupStorage.forcePathStyle }}
|
||||
forcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . }}
|
||||
# The etcd-operator reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
|
||||
# from this Secret by default (no explicit key fields like the
|
||||
# MariaDB strategy's *SecretKeyRef). The projector writes exactly
|
||||
|
||||
@@ -69,7 +69,7 @@ spec:
|
||||
- name: S3_REGION
|
||||
value: {{ .Values.backupStorage.region | quote }}
|
||||
- name: S3_FORCE_PATH_STYLE
|
||||
value: {{ .Values.backupStorage.forcePathStyle | quote }}
|
||||
value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
|
||||
command: ["/usr/bin/bash", "-c"]
|
||||
# The driver script is kept as a standalone shell file and
|
||||
# embedded verbatim so hack/kafka-backup-script.bats can execute
|
||||
|
||||
@@ -76,6 +76,6 @@ spec:
|
||||
prefix: {{ printf "{{ .Application.metadata.namespace }}/{{ .Application.metadata.name }}" | quote }}
|
||||
region: {{ .Values.backupStorage.region | quote }}
|
||||
credentialsSecret: cozy-backups-creds
|
||||
forcePathStyle: {{ .Values.backupStorage.forcePathStyle }}
|
||||
forcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . }}
|
||||
insecureSkipTLSVerify: {{ .Values.backupStorage.insecureSkipTLSVerify }}
|
||||
{{- end -}}
|
||||
@@ -98,7 +98,7 @@ spec:
|
||||
- name: S3_REGION
|
||||
value: {{ .Values.backupStorage.region | quote }}
|
||||
- name: S3_FORCE_PATH_STYLE
|
||||
value: {{ .Values.backupStorage.forcePathStyle | quote }}
|
||||
value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
|
||||
@@ -31,7 +31,7 @@ spec:
|
||||
key: cloud
|
||||
config:
|
||||
region: {{ .Values.backupStorage.region | quote }}
|
||||
s3ForcePathStyle: {{ .Values.backupStorage.forcePathStyle | quote }}
|
||||
s3ForcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . | quote }}
|
||||
s3Url: {{ include "backupstrategy-controller.endpoint" . | quote }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -54,3 +54,12 @@ tests:
|
||||
name: BACKUP_STORAGE_SECRET_NAMESPACE
|
||||
value: tenant-root
|
||||
any: true
|
||||
|
||||
# A value no consumer took as a boolean is refused once, at render time; see
|
||||
# tests/force_path_style_test.yaml for what each consumer receives.
|
||||
- it: "forcePathStyle that is no boolean spelling fails the render and names the value"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "maybe"
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorPattern: 'backupStorage\.forcePathStyle must be a boolean or unset, got "maybe"'
|
||||
@@ -0,0 +1,501 @@
|
||||
suite: forcePathStyle — one normalised boolean for every consumer
|
||||
|
||||
# backupStorage.forcePathStyle reaches drivers that read it differently: the
|
||||
# controller and Velero parse it with strconv.ParseBool, the etcd and MongoDB
|
||||
# strategies write it into a boolean field Helm decodes as YAML 1.1, and the
|
||||
# Kafka and RabbitMQ scripts compare the string with "false". The chart accepts
|
||||
# the spellings of both strconv.ParseBool and YAML 1.1, hands each consumer
|
||||
# "true" or "false", and refuses anything else, so a spelling cannot mean path
|
||||
# style to one driver and not to another.
|
||||
|
||||
templates:
|
||||
- templates/deployment.yaml
|
||||
- templates/strategy-etcd-default.yaml
|
||||
- templates/strategy-mongodb-default.yaml
|
||||
- templates/strategy-kafka-default.yaml
|
||||
- templates/strategy-rabbitmq-default.yaml
|
||||
- templates/velero-bsl.yaml
|
||||
|
||||
release:
|
||||
name: t
|
||||
namespace: cozy-backup-controller
|
||||
|
||||
set:
|
||||
backupStorage.bucketNameOverride: test-bucket
|
||||
|
||||
tests:
|
||||
- it: "forcePathStyle '1' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "1"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 't' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "t"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'T' reaches every consumer as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "T"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: true
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
- equal:
|
||||
path: spec.template.s3.forcePathStyle
|
||||
value: true
|
||||
template: templates/strategy-mongodb-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/strategy-kafka-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/strategy-rabbitmq-default.yaml
|
||||
- equal:
|
||||
path: spec.config.s3ForcePathStyle
|
||||
value: "true"
|
||||
template: templates/velero-bsl.yaml
|
||||
|
||||
- it: "forcePathStyle 'true' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "true"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'True' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "True"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'TRUE' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "TRUE"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'y' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "y"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'Y' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "Y"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'yes' reaches every consumer as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "yes"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: true
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
- equal:
|
||||
path: spec.template.s3.forcePathStyle
|
||||
value: true
|
||||
template: templates/strategy-mongodb-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/strategy-kafka-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/strategy-rabbitmq-default.yaml
|
||||
- equal:
|
||||
path: spec.config.s3ForcePathStyle
|
||||
value: "true"
|
||||
template: templates/velero-bsl.yaml
|
||||
|
||||
- it: "forcePathStyle 'Yes' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "Yes"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'YES' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "YES"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'on' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "on"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'On' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "On"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'ON' reaches the controller as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "ON"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle '0' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "0"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'f' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "f"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'F' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "F"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'false' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "false"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'False' reaches every consumer as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "False"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: false
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
- equal:
|
||||
path: spec.template.s3.forcePathStyle
|
||||
value: false
|
||||
template: templates/strategy-mongodb-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/strategy-kafka-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/strategy-rabbitmq-default.yaml
|
||||
- equal:
|
||||
path: spec.config.s3ForcePathStyle
|
||||
value: "false"
|
||||
template: templates/velero-bsl.yaml
|
||||
|
||||
- it: "forcePathStyle 'FALSE' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "FALSE"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'n' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "n"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'N' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "N"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'no' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "no"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'No' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "No"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'NO' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "NO"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'off' reaches every consumer as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "off"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: false
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
- equal:
|
||||
path: spec.template.s3.forcePathStyle
|
||||
value: false
|
||||
template: templates/strategy-mongodb-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/strategy-kafka-default.yaml
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/strategy-rabbitmq-default.yaml
|
||||
- equal:
|
||||
path: spec.config.s3ForcePathStyle
|
||||
value: "false"
|
||||
template: templates/velero-bsl.yaml
|
||||
|
||||
- it: "forcePathStyle 'Off' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "Off"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle 'OFF' reaches the controller as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: "OFF"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/deployment.yaml
|
||||
|
||||
- it: "forcePathStyle as a YAML boolean (the default) reaches every consumer as true"
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: "true"
|
||||
template: templates/deployment.yaml
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: true
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
|
||||
- it: "forcePathStyle as a YAML boolean false reaches the scripts as false"
|
||||
set:
|
||||
backupStorage.forcePathStyle: false
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: S3_FORCE_PATH_STYLE
|
||||
value: "false"
|
||||
template: templates/strategy-kafka-default.yaml
|
||||
- equal:
|
||||
path: spec.template.s3.forcePathStyle
|
||||
value: false
|
||||
template: templates/strategy-mongodb-default.yaml
|
||||
|
||||
- it: "forcePathStyle as the number 1 reaches the boolean fields as true"
|
||||
set:
|
||||
backupStorage.forcePathStyle: 1
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: true
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
|
||||
# Unset leaves each driver its own default, as before.
|
||||
- it: "forcePathStyle null stays unset for every consumer"
|
||||
set:
|
||||
backupStorage.forcePathStyle: null
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: ""
|
||||
template: templates/deployment.yaml
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: null
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
- equal:
|
||||
path: spec.template.s3.forcePathStyle
|
||||
value: null
|
||||
template: templates/strategy-mongodb-default.yaml
|
||||
|
||||
- it: "forcePathStyle empty stays unset for every consumer"
|
||||
set:
|
||||
backupStorage.forcePathStyle: ""
|
||||
asserts:
|
||||
- contains:
|
||||
path: spec.template.spec.containers[0].env
|
||||
content:
|
||||
name: BACKUP_STORAGE_FORCE_PATH_STYLE
|
||||
value: ""
|
||||
template: templates/deployment.yaml
|
||||
- equal:
|
||||
path: spec.template.destination.s3.forcePathStyle
|
||||
value: null
|
||||
template: templates/strategy-etcd-default.yaml
|
||||
@@ -105,10 +105,6 @@ backupStorage:
|
||||
endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333
|
||||
# region is re-projected into cozy-backups-creds on the next reconcile
|
||||
# (1-minute tick for system namespaces, on-demand for tenant namespaces).
|
||||
# However, charts that consume the region via secretKeyRef in env vars —
|
||||
# the chart-emitted ClickHouse sidecar on `useSystemBucket: true` is the
|
||||
# only such consumer today — only pick up the new value on Pod restart.
|
||||
# Plan a rollout of those Pods when you change this value.
|
||||
region: us-east-1
|
||||
forcePathStyle: true
|
||||
# systemSecretName — the human-friendly Secret produced by the Bucket
|
||||
|
||||
Reference in new issue
Block a user