test(clickhouse): back up and restore a running release through the system bucket

A ClickHouse release on the system bucket starts before the platform
projects cozy-backups-creds into its namespace, and no suite exercised
that order: the round-trips bring their own Bucket and endpoint, so the
sidecar never had to pick up credentials that arrived after it started.

The new Test starts the release without the projected Secret and runs a
backup and an in-place restore through cozy-default against the bucket
as advertised. In CI that host neither resolves nor has a certificate,
so the step makes it reachable from tenant-root for its own duration
and proves the served certificate before any job starts, which keeps a
DNS or TLS problem from reading as a backup failure.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
This commit is contained in:
Alexey Artamonov committed 2026-11-04 19:31:29 +03:00
1 parent f5f50d2a77
commit e752952047
4 files changed
+431

No files matched your search

@@ -377,3 +377,179 @@ spec:
set -eu
cd ../../..
NAMESPACE="$NAMESPACE" examples/backups/clickhouse/cleanup.sh
---
# Backup and in-place restore through the shipped cozy-default BackupClass and
# the platform system bucket, for a ClickHouse whose Pod started before the
# platform projected cozy-backups-creds into its namespace. The sidecar has to
# pick those credentials up while running; until it could, the first BackupJob
# of such a release failed with "no EC2 IMDS role found" and kept failing until
# the Pod was restarted.
#
# Unlike clickhouse-2-backup-roundtrip this targets the system bucket as
# advertised, https://s3.example.org, so system-bucket-roundtrip.sh makes that
# name reachable from tenant-root for the duration of its own step and undoes
# it on exit: see its header for the two shared objects it touches and the
# cert-manager behaviour that relies on. Runs in tenant-root, where the system
# bucket's SeaweedFS and its Ingress controller sit.
#
# The precondition step deletes tenant-root's projected cozy-backups-creds,
# which the mongodb and rabbitmq round-trips leave there after their
# cozy-default BackupJobs. Nothing reads it between suites, and the projector
# writes it again on the next BackupJob, which is the moment under test.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: clickhouse-3-system-bucket-roundtrip
spec:
namespace: tenant-root
catch:
- events: {}
- describe:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
name: clickhouse-sysbucket
- describe:
apiVersion: cert-manager.io/v1
kind: Certificate
name: e2e-system-bucket-s3
- describe:
apiVersion: cert-manager.io/v1
kind: Certificate
name: seaweedfs-s3-ingress-tls
- describe:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: ingress-seaweedfs-s3
- describe:
apiVersion: backups.cozystack.io/v1alpha1
kind: BackupJob
name: sysbucket-backup
- describe:
apiVersion: backups.cozystack.io/v1alpha1
kind: RestoreJob
name: sysbucket-restore
- podLogs:
selector: app.kubernetes.io/instance=clickhouse-sysbucket
container: clickhouse-backup
tail: 200
- podLogs:
selector: app=clickhouse-sysbucket-keeper
tail: 100
steps:
- name: preconditions
try:
- script:
timeout: 2m
content: |
set -eu
if kubectl -n "$NAMESPACE" get secret seaweedfs-s3-ingress-tls -o name >/dev/null 2>&1; then
echo "Secret $NAMESPACE/seaweedfs-s3-ingress-tls already exists; this Test would have to replace it" >&2
exit 1
fi
if kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF 'BEGIN e2e-clickhouse-system-bucket'; then
echo "kube-system/coredns still carries the e2e-clickhouse-system-bucket block from an earlier run" >&2
exit 1
fi
# Fixed names in a persistent namespace: a leftover Succeeded job
# would satisfy the waits in the run step without a new attempt.
kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete secret cozy-backups-creds --ignore-not-found --wait=true --timeout=30s
check:
($error == null): true
- name: create-certificate-and-probe
try:
- apply:
file: system-bucket-tls.yaml
- assert:
timeout: 2m
resource:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: e2e-system-bucket-s3
status:
(conditions[?type == 'Ready']):
- status: "True"
- assert:
timeout: 3m
resource:
apiVersion: v1
kind: Pod
metadata:
name: e2e-system-bucket-probe
status:
(conditions[?type == 'Ready']):
- status: "True"
- name: create-clickhouse
try:
- apply:
file: clickhouse-system-bucket.yaml
# 11m: above the 600s install timeout cozystack-api gives this release;
# see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
- assert:
timeout: 11m
resource:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: clickhouse-sysbucket
status:
(conditions[?type == 'Ready']):
- status: "True"
- assert:
timeout: 5m
resource:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: chi-clickhouse-sysbucket-clickhouse-0-0
status:
readyReplicas: 1
# The Pod is running with no projected credentials: the state the first
# BackupJob of a fresh release meets.
- error:
timeout: 30s
resource:
apiVersion: v1
kind: Secret
metadata:
name: cozy-backups-creds
- name: run
try:
# Inner budgets of system-bucket-roundtrip.sh:
# 240s probe: CoreDNS picking up the rewrite and the Ingress serving
# the Test certificate
# 480s BackupJob to Succeeded, including the driver's 2m hold
# 480s RestoreJob to Succeeded, same hold
# 90s the EXIT trap: Corefile replace plus a 60s Secret delete
# 1290s (21m30s)
# 25m leaves 3m30s for the SQL and kubectl calls between those waits, so
# an inner wait is what fires and names what stalled.
- script:
timeout: 25m
content: |
set -eu
cd ../../..
NAMESPACE="$NAMESPACE" hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh
check:
($error == null): true
finally:
# Chainsaw did not apply these: the jobs come from the script and the
# Backup from the controller, and deleting a BackupJob reclaims no Backup.
# The Certificate's Secret outlives the Certificate Chainsaw deletes.
- description: system-bucket round-trip cleanup
script:
timeout: 3m
content: |
set -eu
backup=$(kubectl -n "$NAMESPACE" get backupjob sysbucket-backup --ignore-not-found -o jsonpath='{.status.backupRef.name}')
kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s
kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s
if [ -n "$backup" ]; then
kubectl -n "$NAMESPACE" delete backup "$backup" --ignore-not-found --wait=true --timeout=60s
fi
kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s
@@ -0,0 +1,28 @@
# ClickHouse on the platform system bucket for the
# clickhouse-3-system-bucket-roundtrip Test. No S3 coordinates are given: the
# sidecar reads them from the clickhouse-backup.yml key the platform projects
# into cozy-backups-creds at the first BackupJob, after this Pod has started.
# backup.endpointCA trusts the SeaweedFS CA that signs the certificate the Test
# serves for s3.example.org.
apiVersion: apps.cozystack.io/v1alpha1
kind: ClickHouse
metadata:
name: sysbucket
spec:
size: 5Gi
logStorageSize: 1Gi
shards: 1
replicas: 1
resourcesPreset: nano
resources: {}
backup:
enabled: true
useSystemBucket: true
endpointCA:
name: e2e-system-bucket-s3-tls
key: ca.crt
clickhouseKeeper:
enabled: true
replicas: 1
size: 1Gi
resourcesPreset: micro
+185
View File
@@ -0,0 +1,185 @@
#!/bin/bash
# Backup and in-place restore of the `sysbucket` ClickHouse through the
# cozy-default BackupClass, against the platform system bucket. Run from the
# repository root by the clickhouse-3-system-bucket-roundtrip Test, which has
# already started the release with no cozy-backups-creds in its namespace.
#
# The system bucket is addressed as https://s3.example.org, the S3 Ingress host
# the COSI driver advertises. In CI that name does not resolve and its ACME
# certificate is never issued, so for the duration of this script, and only of
# this script, two shared objects are changed and the EXIT trap below puts both
# back:
#
# - the S3 Ingress TLS Secret seaweedfs-s3-ingress-tls is created from the
# Test's own certificate (SeaweedFS CA, s3.example.org). It relies on
# cert-manager leaving a Secret's data alone while the Certificate that
# names it has no completed issuance, verified with cert-manager v1.20.2:
# a fresh ACME order would replace this certificate, and ingress-nginx
# would then serve one the sidecar does not trust, which fails the probe
# below rather than the backup.
# - the host CoreDNS Corefile gets one marked `rewrite name exact` line for
# s3.example.org. The host CoreDNS is Talos-managed and imports no snippet
# files, so the block goes into the live Corefile the way ouroboros writes
# its own; ouroboros is off in CI and no other suite uses this name.
set -euo pipefail
source examples/backups/clickhouse/00-helpers.sh
: "${NAMESPACE:?}"
CLICKHOUSE_NAME=sysbucket
BACKUPJOB_NAME=sysbucket-backup
RESTOREJOB_NAME=sysbucket-restore
S3_HOST=s3.example.org
INGRESS_SECRET=seaweedfs-s3-ingress-tls
CERT_SECRET=e2e-system-bucket-s3-tls
PROBE_POD=e2e-system-bucket-probe
OWNER_LABEL=cozystack.io/e2e-owner=clickhouse-system-bucket
DNS_BEGIN=' # === BEGIN e2e-clickhouse-system-bucket ==='
DNS_END=' # === END e2e-clickhouse-system-bucket ==='
DNS_TARGET="root-ingress-controller.${NAMESPACE}.svc.cozy.local"
# Kubelet refreshes a ConfigMap volume within about a minute and CoreDNS's
# reload plugin polls every 30s; the rest is the 30s answer cache.
PROBE_TIMEOUT=240
# Each covers the driver's 2m hold after the credentials change plus the
# strategy Job itself.
JOB_TIMEOUT=480
workdir=$(mktemp -d)
corefile_has_block() {
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF -- "$DNS_BEGIN"
}
remove_dns_block() {
kubectl -n kube-system get configmap coredns -o json >"$workdir/coredns.json"
local begins
begins=$(jq -r '.data.Corefile' "$workdir/coredns.json" | grep -cF -- "$DNS_BEGIN" || true)
if [ "$begins" = 0 ]; then
return 0
fi
if [ "$begins" != 1 ] || ! jq -r '.data.Corefile' "$workdir/coredns.json" | grep -qF -- "$DNS_END"; then
log_error "kube-system/coredns holds ${begins} '${DNS_BEGIN# *}' markers or no END marker; reconcile the Corefile by hand"
return 1
fi
jq --arg b "$DNS_BEGIN" --arg e "$DNS_END" \
'.data.Corefile |= (split("\n") | (index($b)) as $i | (index($e)) as $j | .[:$i] + .[$j+1:] | join("\n"))' \
"$workdir/coredns.json" >"$workdir/coredns-clean.json"
kubectl replace -f "$workdir/coredns-clean.json" >/dev/null
log_success "Removed the ${S3_HOST} rewrite from kube-system/coredns"
}
restore_shared_state() {
local rc=$?
set +e
remove_dns_block || rc=1
if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -l "$OWNER_LABEL" -o name 2>/dev/null | grep -q .; then
kubectl -n "$NAMESPACE" delete secret "$INGRESS_SECRET" --wait=true --timeout=60s || rc=1
fi
rm -rf "$workdir"
exit "$rc"
}
if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -o name >/dev/null 2>&1; then
log_error "Secret ${NAMESPACE}/${INGRESS_SECRET} already exists; this Test serves its own certificate under that name and will not overwrite one it did not create"
exit 1
fi
if corefile_has_block; then
log_error "kube-system/coredns already carries the ${S3_HOST} rewrite block, left by an earlier run; remove it by hand"
exit 1
fi
trap restore_shared_state EXIT
log_step "Serving the Test certificate for ${S3_HOST} through the platform S3 Ingress"
kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o json \
| jq --arg name "$INGRESS_SECRET" '{apiVersion, kind, type,
metadata: {name: $name, labels: {"cozystack.io/e2e-owner": "clickhouse-system-bucket"}},
data: {"tls.crt": .data["tls.crt"], "tls.key": .data["tls.key"], "ca.crt": .data["ca.crt"]}}' \
| kubectl -n "$NAMESPACE" create -f -
log_step "Rewriting ${S3_HOST} to ${DNS_TARGET} in the host CoreDNS"
kubectl -n kube-system get configmap coredns -o json >"$workdir/coredns.json"
jq -e '.data.Corefile | startswith(".:53 {\n")' "$workdir/coredns.json" >/dev/null \
|| { log_error "kube-system/coredns Corefile does not open with '.:53 {'; refusing to edit it"; exit 1; }
# The resourceVersion read above travels with the replace, so a concurrent
# writer makes it fail instead of being overwritten.
jq --arg block "${DNS_BEGIN}"$'\n'" rewrite name exact ${S3_HOST} ${DNS_TARGET}"$'\n'"${DNS_END}" \
'.data.Corefile |= sub("^\\.:53 \\{\n"; ".:53 {\n" + $block + "\n")' \
"$workdir/coredns.json" >"$workdir/coredns-new.json"
kubectl replace -f "$workdir/coredns-new.json" >/dev/null
log_step "Checking that ${S3_HOST} serves the Test certificate from inside ${NAMESPACE}"
kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.ca\.crt}' | base64 -d >"$workdir/ca.crt"
want=$(kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -fingerprint -sha256)
elapsed=0
while true; do
# --cacert makes curl verify both the chain and the host name; %{certs}
# prints the chain it was served, leaf first.
if served=$(kubectl -n "$NAMESPACE" exec -i "$PROBE_POD" -- sh -c \
"cat >/tmp/ca.crt && curl -sS -o /dev/null --max-time 10 --cacert /tmp/ca.crt -w '%{certs}' https://${S3_HOST}/" \
<"$workdir/ca.crt" 2>"$workdir/probe.err"); then
got=$(printf '%s\n' "$served" | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p;/END CERTIFICATE/q' \
| openssl x509 -noout -fingerprint -sha256)
if [ "$got" = "$want" ]; then
log_success "${S3_HOST} resolves in ${NAMESPACE} and serves the Test certificate"
break
fi
echo "served ${got}, want ${want}" >"$workdir/probe.err"
fi
if [ "$elapsed" -ge "$PROBE_TIMEOUT" ]; then
log_error "${S3_HOST} did not serve the Test certificate within ${PROBE_TIMEOUT}s: $(cat "$workdir/probe.err")"
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' >&2 || true
kubectl -n "$NAMESPACE" describe ingress ingress-seaweedfs-s3 >&2 || true
exit 1
fi
sleep 10
elapsed=$((elapsed + 10))
done
log_step "Writing the sentinel row"
clickhouse_query "$CLICKHOUSE_NAME" "CREATE TABLE IF NOT EXISTS default.sentinel (id UInt32, name String) ENGINE = MergeTree ORDER BY id"
clickhouse_query "$CLICKHOUSE_NAME" "INSERT INTO default.sentinel VALUES (1, 'before-backup')"
log_step "Backing up through cozy-default"
kubectl apply -f - <<EOF
apiVersion: backups.cozystack.io/v1alpha1
kind: BackupJob
metadata:
name: ${BACKUPJOB_NAME}
namespace: ${NAMESPACE}
spec:
applicationRef:
apiGroup: apps.cozystack.io
kind: ClickHouse
name: ${CLICKHOUSE_NAME}
backupClassName: cozy-default
EOF
wait_for_field backupjob "$BACKUPJOB_NAME" '{.status.phase}' Succeeded "$NAMESPACE" "$JOB_TIMEOUT" Failed
backup_ref=$(kubectl -n "$NAMESPACE" get backupjob "$BACKUPJOB_NAME" -o jsonpath='{.status.backupRef.name}')
[ -n "$backup_ref" ] || { log_error "BackupJob ${BACKUPJOB_NAME} succeeded with an empty backupRef"; exit 1; }
# The sidecar started before this Secret existed; a backup that succeeded
# against it is the behaviour under test, and the key it read is this one.
endpoint=$(kubectl -n "$NAMESPACE" get secret cozy-backups-creds -o jsonpath='{.data.clickhouse-backup\.yml}' | base64 -d \
| sed -n 's/^ *endpoint: *//p')
[ "$endpoint" = "https://${S3_HOST}" ] \
|| { log_error "cozy-backups-creds clickhouse-backup.yml endpoint is '${endpoint}', expected https://${S3_HOST}"; exit 1; }
log_step "Restoring in place through cozy-default"
clickhouse_query "$CLICKHOUSE_NAME" "DROP TABLE IF EXISTS default.sentinel SYNC"
kubectl apply -f - <<EOF
apiVersion: backups.cozystack.io/v1alpha1
kind: RestoreJob
metadata:
name: ${RESTOREJOB_NAME}
namespace: ${NAMESPACE}
spec:
backupRef:
name: ${backup_ref}
EOF
wait_for_field restorejob "$RESTOREJOB_NAME" '{.status.phase}' Succeeded "$NAMESPACE" "$JOB_TIMEOUT" Failed
count=$(clickhouse_query "$CLICKHOUSE_NAME" "SELECT count() FROM default.sentinel" | tr -d '[:space:]')
[[ "$count" =~ ^[0-9]+$ ]] || { log_error "non-numeric sentinel count: '${count}'"; exit 1; }
(( count >= 1 )) || { log_error "sentinel row count after restore is ${count}; expected >= 1"; exit 1; }
log_success "System-bucket round-trip verified: ${count} sentinel row(s) restored"
@@ -0,0 +1,42 @@
# Certificate and probe Pod for the clickhouse-3-system-bucket-roundtrip Test.
#
# The certificate is signed by the SeaweedFS CA already present in tenant-root,
# so its Secret carries that CA as ca.crt: the ClickHouse release trusts it via
# backup.endpointCA, and the round-trip script copies the same key pair under
# the name the platform's S3 Ingress serves for s3.example.org.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: e2e-system-bucket-s3
spec:
secretName: e2e-system-bucket-s3-tls
dnsNames:
- s3.example.org
issuerRef:
kind: Issuer
name: seaweedfs-ca-issuer
---
# Resolves and fetches s3.example.org from inside tenant-root, where the
# ClickHouse sidecar runs, so the script can prove the DNS rewrite and the
# served certificate before it spends minutes on a BackupJob. The image is the
# one hack/e2e-install-cozystack.bats pre-pulls; it carries curl but no openssl.
apiVersion: v1
kind: Pod
metadata:
name: e2e-system-bucket-probe
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 0
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: probe
image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47
command: [sleep, "3600"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]