Files
cozystack/examples/backups/clickhouse/01-create-strategy.sh
T
Andrey KolkovandClaude Opus 4.8 82aa967674 test(e2e): correct the round-trip budget and its stale comments
Second review pass on this branch.

1. The op timeout equalled the inner sum instead of exceeding it: the comment
   enumerated 3900s of inner waits and then set 65m, and 3900s IS 65m — so there
   was zero headroom for the applies and SQL calls between those waits, and the
   claim of "room" was false. Sized the inner budgets against what CI actually
   takes instead (the bucket suite completes in ~37s and a backup-enabled
   ClickHouse plus Keeper reaches Ready in ~98s, measured on run 30353894332):
   bucket claim/access 300->180 and the three job waits 600->480, summing to
   3300s, under a 60m op timeout. Shrinking the inner budgets rather than
   raising the ceiling also keeps this suite closer to its postgres and mariadb
   siblings on a 180-minute job cap.

2. The strategy Pod's securityContext comment still justified root with "Pod
   runs apk add at startup", which the previous commit made conditional — the
   default path now runs the platform's pre-baked image and installs nothing.
   Scoped the note to the alpine fallback and said explicitly that the field set
   matches cozy-default-altinity, runAsNonRoot deliberately unset in both.

3. clickhouse-backup.yaml still referenced clickhouse-backup-{1-contracts,
   2-roundtrip}: the file moved unchanged, so the Test rename never reached its
   own header and a reader following either name found nothing.

4. wait_sts_ready's timeout diagnostic could not print anything. It selected
   clickhouse.altinity.com/chi=${name#chi-}, which yields
   clickhouse-<app>-clickhouse-0-0 while the label carries the CHI name
   (clickhouse-<app>), and 2>/dev/null then swallowed kubectl's "No resources
   found". Ask for the StatefulSet's own first replica by name instead and let
   both streams through; verified it now surfaces the Pod events that explain
   why the replica never came up.

Also dropped the clickhouse bats case added in the previous commit: it asserted
what the generic examples/backups/<app> -> <app> mapping already guarantees, so
it did not lock the invariant its comment claimed. That invariant is now
recorded where it belongs — in the mapping's own comment in select-e2e.sh.

The dedicated ch-backup-client image this example now inherits is tracked in
#3420; noted there that the command: override is load-bearing in two files.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00

185 lines
9.1 KiB
Bash
Executable File

#!/bin/bash
# Step 01: Create the cluster-scoped Altinity backup strategy. The strategy
# drives Altinity's clickhouse-backup HTTP API exposed by a sidecar inside
# every chi-* Pod (see packages/apps/clickhouse/templates/clickhouse.yaml).
# The strategy Pod itself is a tiny curl/jq client that POSTs to the sidecar
# and polls the action log.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/00-helpers.sh"
print_header "Step 01: Create Altinity backup strategy"
# Reuse the image the platform's own cozy-default-altinity strategy runs, so the
# demo Pod needs no `apk add` at run time. That matters twice over: the strategy
# runs once per BackupJob and RestoreJob, so fetching packages from the Alpine
# CDN each time makes every backup depend on outbound internet — it fails
# outright on an air-gapped cluster, and it turns a CDN hiccup into a failed
# backup — and this same script is the e2e round-trip's harness, where that
# would surface as an unrelated red E2E run.
#
# Read it off the shipped strategy rather than hardcoding a tag, so the demo
# tracks whatever the platform is pinned to. Falls back to alpine + `apk add`
# only when that strategy is absent (a cluster without the backupstrategy
# controller chart), which is also the one case where the demo has to bootstrap
# its own tooling.
CH_BACKUP_CLIENT_IMAGE="${CH_BACKUP_CLIENT_IMAGE:-$(kubectl get altinity.strategy.backups.cozystack.io cozy-default-altinity \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="ch-backup-client")].image}' 2>/dev/null || true)}"
APK_BOOTSTRAP=""
if [[ -z "$CH_BACKUP_CLIENT_IMAGE" ]]; then
CH_BACKUP_CLIENT_IMAGE="alpine:3.19"
APK_BOOTSTRAP="apk add --no-cache curl jq >/dev/null"
log_warning "cozy-default-altinity not found; falling back to ${CH_BACKUP_CLIENT_IMAGE} + apk add (needs outbound internet)."
else
log_info "Reusing the platform strategy's client image: ${CH_BACKUP_CLIENT_IMAGE}"
fi
log_command "kubectl apply -f - (Altinity strategy: $STRATEGY_NAME)"
kubectl apply -f - <<EOF
apiVersion: strategy.backups.cozystack.io/v1alpha1
kind: Altinity
metadata:
name: ${STRATEGY_NAME}
spec:
template:
spec:
restartPolicy: Never
containers:
- name: ch-backup-client
image: ${CH_BACKUP_CLIENT_IMAGE}
imagePullPolicy: IfNotPresent
# API_USERNAME / API_PASSWORD pulled from the chart-emitted
# backup-api-auth Secret in the application's namespace. The
# sidecar rejects unauthenticated requests; without these env
# vars every curl below would 401. Secret name follows the
# actual Helm release shape (clickhouse-rd prefixes the user-
# facing app name with "clickhouse-").
env:
- name: API_USERNAME
valueFrom:
secretKeyRef:
name: clickhouse-{{ .Release.Name }}-backup-api-auth
key: username
- name: API_PASSWORD
valueFrom:
secretKeyRef:
name: clickhouse-{{ .Release.Name }}-backup-api-auth
key: password
command: ["/bin/sh","-c"]
args:
- |
set -euo pipefail
${APK_BOOTSTRAP}
host="chi-clickhouse-{{ .Release.Name }}-clickhouse-0-0.{{ .Release.Namespace }}.svc"
api="http://\${host}:7171"
# \${API_USERNAME} / \${API_PASSWORD} come from the
# backup-api-auth Secret mounted via env (see above).
auth="-u \${API_USERNAME}:\${API_PASSWORD}"
# Sidecar may still be coming up when the strategy Pod fires.
# Block on /backup/status until it answers (max ~5 minutes).
for _ in \$(seq 1 60); do
if curl -fsS \${auth} --max-time 2 "\${api}/backup/status" >/dev/null 2>&1; then break; fi
sleep 5
done
# Backup names embed the source release prefix. The sidecar's
# S3_PATH already scopes list/remote to this release's prefix
# inside the bucket; the prefix on the name is defence in
# depth - a misconfigured tenant sharing S3_PATH across
# releases (or a to-copy restore where s3PathOverride pins
# the destination at the source's prefix) still picks the
# right archive.
#
# Backup mode: .Release.Name is the source app, names use
# that prefix. Restore mode: .Release.Name is the *target*
# app (in-place=source, to-copy=destination), but we filter
# by the *source* via .Backup.ApplicationRef.Name so to-copy
# also lands on the correct archive.
if [ "{{ .Mode }}" = "backup" ]; then
prefix="clickhouse-{{ .Release.Name }}-"
target="\${prefix}\$(date -u +%Y%m%dT%H%M%SZ)"
echo "create_remote \${target}"
curl -fsS \${auth} -X POST "\${api}/backup/create_remote?name=\${target}" >/dev/null
else
prefix="clickhouse-{{ .Backup.ApplicationRef.Name }}-"
# Pick the latest archive whose name starts with the
# source-release prefix. Falls through to "no remote
# backup found" if S3_PATH is misconfigured AND none of
# the listed names match.
target=\$(curl -fsS \${auth} --max-time 60 "\${api}/backup/list/remote" \\
| jq -rs --arg prefix "\${prefix}" \\
'[.[] | select((.name // "") | startswith(\$prefix))] | sort_by(.created // "") | last.name // ""')
if [ -z "\${target}" ] || [ "\${target}" = "null" ]; then
echo "no remote backup found" >&2
exit 1
fi
echo "restore_remote \${target}"
# Caller is responsible for dropping the target table(s)
# before invoking this strategy in restore mode (see
# 06-restore-in-place.sh).
# The clickhouse-backup HTTP API surfaces restore_remote as
# POST /backup/restore_remote/{name} (path-encoded). The
# query-string form returns 404. Omitting ?schema and ?data
# restores both schema and data (the default); pass them
# only to restrict the operation. This script does not pass
# ?rm: callers are expected to drop conflicting tables
# themselves (see 06-restore-in-place.sh, which DROPs the
# sentinel before submitting the RestoreJob).
curl -fsS \${auth} -X POST "\${api}/backup/restore_remote/\${target}" >/dev/null
fi
# Poll the action log. clickhouse-backup emits NDJSON; we match
# on the backup name suffix because the embedded flags vary
# between operations (e.g. "restore_remote --schema --data X"
# vs. "create_remote X"). The strategy Pod runs one operation
# at a time, so the most recent matching action is ours.
while true; do
row=\$(curl -fsS \${auth} --max-time 30 "\${api}/backup/actions" \\
| jq -rs --arg suffix "\${target}" \\
'[.[] | select((.command // "") | endswith(\$suffix))] | sort_by(.start) | last')
status=\$(echo "\${row}" | jq -r '.status // ""')
case "\${status}" in
success) echo "operation succeeded" ; exit 0 ;;
error)
err=\$(echo "\${row}" | jq -r '.error // "unknown"')
echo "operation failed: \${err}" >&2
exit 1
;;
esac
sleep 5
done
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 200m
memory: 128Mi
securityContext:
# Same field set as the platform's cozy-default-altinity strategy
# (packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml):
# no privilege escalation, all caps dropped, default seccomp
# profile, and runAsNonRoot deliberately left unset rather than
# false — the Pod only runs curl and jq, and neither the platform
# strategy nor this one pins a UID.
#
# On the default path there is no `apk add` to need root: the image
# above is the platform's, with curl+jq already baked in. Only the
# alpine:3.19 fallback (taken when cozy-default-altinity is absent)
# installs packages at startup, and that is the one case where a
# cluster enforcing PSA "restricted" has to supply its own image
# with the tools pre-baked.
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
EOF
log_success "Altinity strategy '${STRATEGY_NAME}' created."
echo -e "\n${GREEN}${BOLD}Next:${NC} ./02-create-backupclass.sh"