mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-28 15:47:33 +09:00
Myasnikov Daniil
0ba5337526
Deriving the safety-snapshot target from the projected cozy-backups-creds Secret is right, but detecting the admin-managed external-S3 case (provisionBucket: false) by "the projected endpoint is absent" is not: that state does not exist. The projector substitutes BACKUP_STORAGE_ENDPOINT and fails ReasonSourceMalformed when both sources are empty, so a projected Secret always carries a non-empty, scheme-stripped endpoint. So for provisionBucket: false against a plaintext external S3 the projected endpoint is a bare host, the script saw it non-empty, forced https:// and never consulted the Strategy CR — a regression against main, which read the CR verbatim and passed http:// through. Being a pre-upgrade hook with no ETCD_ADOPT_SKIP_BACKUP plumbed through migration-hook.yaml, that is a total upgrade block with no operator escape. Classify on data instead: match the projected bucketName against the .status.bucketName of any BucketClaim. The COSI driver assigns that name and the projector republishes exactly it (packages/system/bucket user-credentials.yaml reads BucketInfo spec.bucketName), so a match means the creds describe a COSI bucket => the host is the always-TLS S3 ingress => force https, as the 1.6 chart helper does. No match means external S3. Keying on data rather than on the claim's name and namespace matters twice. backupStorage.namespace/.bucketName are supported Package-CR overrides that this hook cannot see, so a name lookup would miss a renamed bucket and fall back to the v1.5.x plaintext CR — the original P0. And a BucketClaim wedged Terminating (its cosi bucketclaim-protection finalizer outlives an uninstalled COSI controller) keeps answering a name lookup while its status names some other bucket. Split the two paths by which source is authoritative. On COSI the projected coordinates win: the bucket is the COSI-assigned name the CR can only reproduce through a live lookup that may never have run. On external S3 the Strategy CR wins WHOLESALE, not just for the endpoint — the CR renders backupStorage.* and points at this same cozy-backups-creds, so CR coordinates plus these credentials is exactly what the cluster's own BackupJobs already use, and the bucket an operator will look in. Keeping the Secret's bucket while taking the CR's endpoint assembled a pairing nothing else produces: the projector copies bucketName straight from the admin-managed source Secret and never from backupStorage.bucketName, so the two can disagree outright and the snapshot could land in a bucket the platform never writes. Relying on the CR is safe on this path alone, because bucketName short-circuits to values before the BucketClaim lookup that races on the COSI path, so the CR is guaranteed present. Match with a single jq `any`, never `jq -r | grep -q`: grep exits on the first match and SIGPIPEs jq, and pipefail reports jq's 141 rather than grep's 0, so a MATCH returns as a failure and the cluster is misclassified external. It is a race on the list outgrowing one stdio write (~8KB) with only a last-line match safe, so it strikes exactly the big multi-tenant clusters that can least afford it, and backoffLimit 3 can decide differently on each retry. Measured with the match first, 20 runs each: 300 claims went 0/20 correct (rc=141 every time) and 2000 likewise, while `any` is 20/20 at 3, 300 and 2000. Refuse rather than guess when the answer is not knowable. Three states qualify: the BucketClaim API is unreadable; the list is not interpretable (no producer emits that, but without a shape gate it reads as "no claim" = external = the plaintext P0, silently); or the bucket is claimed ONLY by a Terminating claim. That last one is ambiguous — an admin who moved COSI -> external S3 reusing the bucket name, versus a still-COSI cluster whose claim is mid-recreate — and both readings fail as the same confusing handshake error when wrong, so a guess buys no reliability while reading it as "external" specifically resurrects the plaintext-:8333 P0. A live claim still wins over an unrelated Terminating one, so healthy clusters are unaffected. Each refusal names the condition and the operator action; the migration is idempotent, so re-running after resolving it is the remedy. Behaviour change worth noting: a cluster on external S3 that still has COSI installed now refuses on a sustained BucketClaim-API failure where main would have proceeded off the Strategy CR. That is the intended trade — main proceeded only because it never had to tell the two apart — but it does mean an apiserver outage can now block this hook where it previously did not. ETCD_ADOPT_SKIP_BACKUP is still not plumbed through migration-hook.yaml, so the messages deliberately point at resolving the condition rather than at an escape hatch the operator cannot reach. Force the scheme only onto a non-empty host: an absent endpoint must degrade to the CR as main did, not to a literal "https://" that would pass the final non-empty guard and reach etcd-migrate as a destination. The old test modelled external S3 by dropping the projected endpoint entirely — a state the projector cannot produce — so it green-lit the regression. It now models what the projector really writes, and the suite pins the stale-claim, overridden-coordinates, empty-endpoint, 300-claim-multi-tenant, Terminating-match, hybrid-coordinate and unreadable/uninterpretable API cases. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
249 lines
14 KiB
Bash
Executable File
249 lines
14 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Fake kubectl for hack/migration-50-etcd-adopt.bats. It serves only the calls
|
|
# migration 50 makes; behaviour is driven by FAKE_* env and every invocation is
|
|
# appended to $FAKE_CMDLOG so the test can assert on order and content.
|
|
#
|
|
# FAKE_CMDLOG file to append a one-line record of each call to
|
|
# FAKE_STAGE_DIR if set, staged Secret YAML is written to <dir>/<ns>.json
|
|
# FAKE_LEGACY_CRD "1" => the etcd.aenix.io CRD exists (default 1)
|
|
# FAKE_CLUSTERS newline list of "<ns> <name>" legacy clusters
|
|
# FAKE_STRATEGY "1" => the cozy-default-etcd Strategy CR resolves (default 1)
|
|
# FAKE_STRATEGY_ENDPOINT the CR's s3.endpoint (default: the v1.5.x plaintext
|
|
# seaweedfs default)
|
|
# FAKE_STRATEGY_BUCKET the CR's s3.bucket (default cozy-backups-7f3a)
|
|
# FAKE_STRATEGY_REGION the CR's s3.region (default us-east-1)
|
|
# FAKE_STRATEGY_FPS the CR's s3.forcePathStyle (default true)
|
|
# FAKE_CREDS "1" => cozy-backups-creds (with AWS_* keys) exists (default 1)
|
|
# FAKE_CREDS_ENDPOINT the projected `endpoint` value; always a BARE host.
|
|
# Set to the EMPTY string to drop the key entirely
|
|
# (default s3.example.com)
|
|
# FAKE_CREDS_BUCKET the projected `bucketName` value (default
|
|
# cozy-backups-7f3a). Copied from the admin's source
|
|
# Secret on the external path, so it can disagree
|
|
# with the CR's bucket outright.
|
|
# FAKE_CREDS_REGION the projected `region` value (default us-east-1)
|
|
# FAKE_CREDS_FPS the projected `forcePathStyle` value (default true)
|
|
# FAKE_CLAIM_CRD "1" => the bucketclaims.objectstorage.k8s.io CRD
|
|
# exists, i.e. COSI is installed (default 1)
|
|
# "0" => genuinely absent: `--ignore-not-found -o name`
|
|
# prints nothing at rc=0
|
|
# "error" => the read FAILS (rc!=0) -- a transient API
|
|
# error, which is NOT the same answer as absent
|
|
# FAKE_CLAIM_LIST_ERR "1" => the cluster-wide claim list read fails (default 0)
|
|
# FAKE_CLAIM_LIST_SHAPE emit an unexpected list document instead of a List:
|
|
# "items-not-array" | "no-items" | "not-json"
|
|
# (default: a well-formed List). No real producer
|
|
# emits these; they pin that an uninterpretable list
|
|
# is never silently read as "no claim" = external.
|
|
# FAKE_CLAIMS newline list of BucketClaims, one per line, as
|
|
# "<ns> <name> <status.bucketName|-> <deletionTimestamp|->"
|
|
# (default: one healthy claim provisioned into the same
|
|
# bucket the projected Secret advertises => COSI path)
|
|
# FAKE_CLAIM_FILLER N synthetic extra claims appended AFTER FAKE_CLAIMS, with
|
|
# realistically long bucket-<uuid> status names (default 0).
|
|
# Models a big multi-tenant cluster: the FAKE_CLAIMS entries
|
|
# stay FIRST, i.e. the matching claim is NOT last, which is
|
|
# the position a first-match-exit consumer gets wrong.
|
|
# FAKE_CERTS "1" => server/peer cert-manager Certificates exist and
|
|
# initially LACK the native wildcard SAN (default 0,
|
|
# i.e. no Certificate -> ensure_wildcard_sans skips)
|
|
set -u
|
|
|
|
log() { [ -n "${FAKE_CMDLOG:-}" ] && printf '%s\n' "$*" >> "$FAKE_CMDLOG"; }
|
|
log "KUBECTL $*"
|
|
|
|
args="$*"
|
|
state_dir="${FAKE_CMDLOG%/*}"
|
|
# arg_after KEY -- echoes the token following KEY in the argument vector.
|
|
arg_after() { k="$1"; shift; p=""; for a in "$@"; do [ "$p" = "$k" ] && { printf '%s' "$a"; return; }; p="$a"; done; }
|
|
|
|
b64() { printf '%s' "$1" | base64 | tr -d '\n'; }
|
|
|
|
# Strategy CR JSON as the version being upgraded FROM rendered it. The default
|
|
# endpoint is the static v1.5.x one: plaintext against SeaweedFS's :8333 TLS
|
|
# listener, i.e. unusable. The script must NOT take its endpoint from here on the
|
|
# COSI path. FAKE_STRATEGY=0 models the other v1.5.x shape, where the CR was
|
|
# never rendered at all (its template is guarded by a bucketName lookup that
|
|
# races the BucketClaim on first install) — a shape reachable ONLY under
|
|
# provisionBucket: true, since the lookup is skipped entirely for external S3.
|
|
strategy_endpoint="${FAKE_STRATEGY_ENDPOINT:-http://seaweedfs-s3.tenant-root.svc.cozy.local:8333}"
|
|
strategy_json='{"spec":{"template":{"destination":{"s3":{"endpoint":"'"$strategy_endpoint"'","bucket":"'"${FAKE_STRATEGY_BUCKET:-cozy-backups-7f3a}"'","region":"'"${FAKE_STRATEGY_REGION:-us-east-1}"'","forcePathStyle":'"${FAKE_STRATEGY_FPS:-true}"'}}}}}'
|
|
# cozy-backups-creds as eagerly projected into cozy-velero (carries owner/anno
|
|
# metadata the stage copy must strip, plus the AWS_* keys). It also carries a
|
|
# tenantresource label here so the test can prove the stage copy strips it.
|
|
#
|
|
# The projector ALWAYS writes a bare-host `endpoint`: it scheme-strips it and
|
|
# refuses to project (ReasonSourceMalformed) when it would be empty, so there is
|
|
# no reachable state in which it is missing — that is why its presence cannot
|
|
# discriminate COSI from external S3, and why the BucketClaim data does. The
|
|
# empty FAKE_CREDS_ENDPOINT (drop the key) models NO reachable projector state;
|
|
# it exists only to pin the defensive CR fallback.
|
|
# bucketName/region/forcePathStyle ARE droppable: the projector's setOrDelete
|
|
# deletes a key whose value is empty. FAKE_CREDS_COORDS=0 models that.
|
|
creds_endpoint_key=''
|
|
[ -z "${FAKE_CREDS_ENDPOINT-s3.example.com}" ] \
|
|
|| creds_endpoint_key='"endpoint":"'"$(b64 "${FAKE_CREDS_ENDPOINT-s3.example.com}")"'",'
|
|
creds_coords='"bucketName":"'"$(b64 "${FAKE_CREDS_BUCKET:-cozy-backups-7f3a}")"'","region":"'"$(b64 "${FAKE_CREDS_REGION:-us-east-1}")"'","forcePathStyle":"'"$(b64 "${FAKE_CREDS_FPS:-true}")"'",'
|
|
[ "${FAKE_CREDS_COORDS:-1}" = "1" ] || creds_coords=''
|
|
creds_json='{"apiVersion":"v1","kind":"Secret","metadata":{"name":"cozy-backups-creds","namespace":"cozy-velero","resourceVersion":"123","uid":"abc","ownerReferences":[{"name":"projector"}],"labels":{"internal.cozystack.io/tenantresource":"true"},"annotations":{"reconcile.io/x":"y"}},"data":{'"$creds_endpoint_key$creds_coords"'"AWS_ACCESS_KEY_ID":"QUtJQUVYQU1QTEU=","AWS_SECRET_ACCESS_KEY":"c2VjcmV0a2V5"}}'
|
|
|
|
# BucketClaims as data. The COSI driver assigns .status.bucketName and the
|
|
# projector republishes exactly that as the `bucketName` key, so the DEFAULT
|
|
# claim is provisioned into the same bucket the creds advertise => COSI path.
|
|
# A claim whose status names a DIFFERENT bucket (a stale/Terminating leftover)
|
|
# must not classify the cluster as COSI, which is why the probe matches on
|
|
# status rather than on the claim's existence.
|
|
FAKE_CLAIMS="${FAKE_CLAIMS-tenant-root bucket-cozy-backups cozy-backups-7f3a -}"
|
|
|
|
# claims_all emits FAKE_CLAIMS followed by FAKE_CLAIM_FILLER synthetic claims.
|
|
# Filler status names are full-length bucket-<uuid> strings because it is the
|
|
# BYTE SIZE of the rendered list, not the claim count, that pushes a streaming
|
|
# consumer past a pipe write boundary.
|
|
claims_all() {
|
|
printf '%s\n' "$FAKE_CLAIMS"
|
|
local i=1 n="${FAKE_CLAIM_FILLER:-0}"
|
|
while [ "$i" -le "$n" ]; do
|
|
printf 'tenant-f%04d bucket-f%04d bucket-%08x-956e-4630-91f4-e1d265de5f51 -\n' "$i" "$i" "$i"
|
|
i=$((i + 1))
|
|
done
|
|
}
|
|
|
|
# claims_json renders claims_all as a real `kubectl get -A -o json` List.
|
|
claims_json() {
|
|
local first=1 out='{"apiVersion":"v1","kind":"List","items":[' ns name bn dts item
|
|
while read -r ns name bn dts; do
|
|
[ -n "${ns:-}" ] || continue
|
|
item='{"apiVersion":"objectstorage.k8s.io/v1alpha1","kind":"BucketClaim","metadata":{"namespace":"'"$ns"'","name":"'"$name"'","finalizers":["cosi.objectstorage.k8s.io/bucketclaim-protection"]'
|
|
[ "${dts:--}" = "-" ] || item="$item"',"deletionTimestamp":"'"$dts"'"'
|
|
item="$item"'}'
|
|
if [ "${bn:--}" = "-" ]; then item="$item"',"status":{}}'; else item="$item"',"status":{"bucketName":"'"$bn"'"}}'; fi
|
|
[ "$first" = 1 ] && { out="$out$item"; first=0; } || out="$out,$item"
|
|
done < <(claims_all)
|
|
printf '%s]}\n' "$out"
|
|
}
|
|
|
|
# claim_exists <ns> <name> -- 0 iff FAKE_CLAIMS lists that claim, regardless of
|
|
# its status. Models the name-keyed single GET, which succeeds for a Terminating
|
|
# claim too (the bucketclaim-protection finalizer keeps the object around).
|
|
claim_exists() {
|
|
local want_ns="$1" want_name="$2" ns name bn dts
|
|
while read -r ns name bn dts; do
|
|
[ -n "${ns:-}" ] || continue
|
|
[ "$ns" = "$want_ns" ] && [ "$name" = "$want_name" ] && return 0
|
|
done <<EOF
|
|
$FAKE_CLAIMS
|
|
EOF
|
|
return 1
|
|
}
|
|
|
|
emit_clusters_name() { printf '%s\n' "${FAKE_CLUSTERS:-}" | while read -r ns name; do [ -n "$ns" ] && printf 'etcdcluster.etcd.aenix.io/%s\n' "$name"; done; }
|
|
emit_clusters_jsonpath() { printf '%s\n' "${FAKE_CLUSTERS:-}" | while read -r ns name; do [ -n "$ns" ] && printf '%s %s\n' "$ns" "$name"; done; }
|
|
|
|
case "$args" in
|
|
# version stamp (kubectl apply --filename -) and creds copy
|
|
# (kubectl -n ns apply --server-side --force-conflicts -f -)
|
|
*"apply --filename -"*) cat >/dev/null 2>&1 || true; log "STAMP"; exit 0 ;;
|
|
*"apply"*" -f -"*)
|
|
ns=""; prev=""; for a in "$@"; do [ "$prev" = "-n" ] && ns="$a"; prev="$a"; done
|
|
if [ -n "${FAKE_STAGE_DIR:-}" ]; then cat > "$FAKE_STAGE_DIR/$ns.json" 2>/dev/null || true; else cat >/dev/null 2>&1 || true; fi
|
|
log "STAGE $ns"; exit 0 ;;
|
|
|
|
*"get crd etcdclusters.etcd.aenix.io"*)
|
|
[ "${FAKE_LEGACY_CRD:-1}" = "1" ] && exit 0 || exit 1 ;;
|
|
*"get crd etcdclusters.etcd-operator.cozystack.io"*)
|
|
[ "${FAKE_V2_CRD:-0}" = "1" ] && exit 0 || exit 1 ;;
|
|
*"get crd etcdbackup"*) exit 1 ;; # no orphaned schedule/backup CRDs
|
|
*"apply"*"etcd-operator-crds"*) log "APPLY-CRDS"; exit 0 ;; # baked v1alpha2 CRDs
|
|
*" wait "*"established"*) exit 0 ;; # crd established wait
|
|
|
|
*"etcds.strategy.backups.cozystack.io"*)
|
|
[ "${FAKE_STRATEGY:-1}" = "1" ] || exit 1
|
|
printf '%s\n' "$strategy_json"; exit 0 ;;
|
|
|
|
# COSI presence. With --ignore-not-found real kubectl distinguishes a genuinely
|
|
# absent CRD (rc=0, no output) from a failed read (rc!=0) — the script must
|
|
# treat only the former as "external".
|
|
*"get crd bucketclaims.objectstorage.k8s.io"*)
|
|
log "CLAIM-CRD-PROBE"
|
|
case "${FAKE_CLAIM_CRD:-1}" in
|
|
1) printf 'customresourcedefinition.apiextensions.k8s.io/bucketclaims.objectstorage.k8s.io\n'; exit 0 ;;
|
|
0) exit 0 ;; # --ignore-not-found: absent is a rc=0 empty answer
|
|
*) exit 1 ;; # transient read error
|
|
esac ;;
|
|
|
|
# Cluster-wide claim list: the data the probe matches the projected bucketName
|
|
# against.
|
|
*"bucketclaims.objectstorage.k8s.io"*"-A"*)
|
|
log "BUCKETCLAIM-LIST"
|
|
[ "${FAKE_CLAIM_LIST_ERR:-0}" = "1" ] && exit 1
|
|
[ "${FAKE_CLAIM_CRD:-1}" = "1" ] || exit 1
|
|
case "${FAKE_CLAIM_LIST_SHAPE:-}" in
|
|
items-not-array) printf '{"items":"oops"}\n'; exit 0 ;;
|
|
no-items) printf '{"apiVersion":"v1","kind":"List"}\n'; exit 0 ;;
|
|
not-json) printf 'definitely not json\n'; exit 0 ;;
|
|
esac
|
|
claims_json; exit 0 ;;
|
|
|
|
# Name-keyed single GET. Nothing in the migration issues this any more; kept so
|
|
# the pre-fix script can still be driven against this fixture to prove the new
|
|
# tests bite.
|
|
*"bucketclaims.objectstorage.k8s.io"*)
|
|
log "BUCKETCLAIM-GET"
|
|
[ "${FAKE_CLAIM_CRD:-1}" = "1" ] || exit 1
|
|
claim_exists "$(arg_after -n "$@")" "$(arg_after bucketclaims.objectstorage.k8s.io "$@")" || exit 1
|
|
printf 'bucketclaim.objectstorage.k8s.io/x\n'; exit 0 ;;
|
|
|
|
*"get secret cozy-backups-creds"*) # both-keys validation probe AND stage source
|
|
[ "${FAKE_CREDS:-1}" = "1" ] || exit 1
|
|
printf '%s\n' "$creds_json"; exit 0 ;;
|
|
|
|
*"etcdclusters.etcd.aenix.io"*"-o name"*) emit_clusters_name; exit 0 ;;
|
|
*"etcdclusters.etcd.aenix.io"*"jsonpath"*) emit_clusters_jsonpath; exit 0 ;;
|
|
|
|
# --- ensure_wildcard_sans path (only when FAKE_CERTS=1) -------------------
|
|
# cert-manager re-issue is modelled statefully: a Certificate initially lacks
|
|
# the native wildcard; after the script patches it, the Secret's alt-names
|
|
# annotation reports the wildcard (a per-secret flag file marks the re-issue).
|
|
*"patch certificate.cert-manager.io"*)
|
|
cert="$(arg_after certificate.cert-manager.io "$@")"
|
|
: > "$state_dir/reissued-${cert}-tls"
|
|
log "PATCH-CERT $cert"; exit 0 ;;
|
|
# jsonpath '{.spec.dnsNames[*]}' probe used by _san_present: emit a
|
|
# SPACE-separated SAN list (matches real kubectl jsonpath[*] output), NOT the
|
|
# -o json blob below. FAKE_CERT_SUPERSTRING=1 injects a longer SAN that
|
|
# CONTAINS the native wildcard as a substring (*.etcd.<ns>.svc inside
|
|
# *.etcd.<ns>.svc.cluster.local) but is not an exact match — exercises the
|
|
# exact-match guard (a substring match would wrongly skip the re-issue patch).
|
|
*"get certificate.cert-manager.io"*"dnsNames"*)
|
|
ns="$(arg_after -n "$@")"
|
|
if [ "${FAKE_CERT_SUPERSTRING:-0}" = "1" ]; then
|
|
printf 'etcd etcd.%s.svc *.etcd.%s.svc.cluster.local\n' "$ns" "$ns"
|
|
else
|
|
printf 'etcd etcd.%s.svc\n' "$ns"
|
|
fi
|
|
exit 0 ;;
|
|
*"get certificate.cert-manager.io"*"-o json"*)
|
|
ns="$(arg_after -n "$@")"
|
|
printf '{"spec":{"dnsNames":["etcd","etcd.%s.svc"]}}\n' "$ns"; exit 0 ;;
|
|
*"get certificate.cert-manager.io"*)
|
|
[ "${FAKE_CERTS:-0}" = "1" ] && exit 0 || exit 1 ;;
|
|
*"get secret"*"alt-names"*)
|
|
ns="$(arg_after -n "$@")"; secret="$(arg_after secret "$@")"
|
|
if [ -f "$state_dir/reissued-$secret" ]; then
|
|
printf 'etcd,etcd.%s.svc,*.etcd.%s.svc\n' "$ns" "$ns"
|
|
else
|
|
printf 'etcd,etcd.%s.svc\n' "$ns"
|
|
fi
|
|
exit 0 ;;
|
|
|
|
*"scale deploy"*)
|
|
case "$args" in
|
|
*"--replicas=0"*) log "SCALE 0" ;;
|
|
*"--replicas=1"*) log "SCALE 1" ;;
|
|
esac
|
|
exit 0 ;;
|
|
*"rollout status"*) exit 0 ;;
|
|
|
|
*) exit 0 ;;
|
|
esac
|