Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.
docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The backup round-trips drive examples/backups, whose bucket waits gave
up at 300s and whose postgres waits at 360s, under the 600s install
timeout cozystack-api gives each of those releases. The kafka example's
bucket wait passed no budget and took the helper's 300s default. A
failed install there reported only that the wait ran out.
Raise each to 660s, and grow the Chainsaw op around every walkthrough
by what its waits gained, so a failure at any later wait keeps the
headroom it had and is still reported by the script rather than cut
off by the op. The kafka and kafka-metadata Kafka waits stay: their
7h install timeout cannot be cleared inside the E2E job cap.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every backup walkthrough carried its own wait_for_field, and most their
own wait_hr_ready and wait_deleted: twelve, eight and two copies, which
had already drifted into five, three and two bodies. A fix made in one
copy did not reach the others, and a diagnostics change carried by hand
between two of them once went into the wrong function and was caught
only by reading the diff.
The helpers now live in examples/backups/_lib/wait-helpers.sh, sourced
by each walkthrough's 00-helpers.sh. Where copies differed, the shared
body keeps the diagnostics any of them had: wait_for_field leaves
kubectl's stderr attached, and wait_hr_ready prints the conditions and
history dump on Stalled or timeout. Call sites and budgets are unchanged.
A new unit test fails when any walkthrough defines one of these helpers
again, which replaces the byte-identity pin between copies.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Those walkthroughs gave their database HelmReleases 300s, under the
600s install timeout the generated releases carry, so a failed install
surfaced as a bare timeout rather than its own reason. None of these
charts sets a startup probe, so 660s clears both floors, as it does
for mariadb.
Each Chainsaw op that runs one of the scripts grows by exactly what
the waits it reaches gained, so it keeps the margin it had. The
clickhouse op still sits above the sum of every inner ceiling. The
rabbitmq one covers a failed install at the source wait, while the
ceilings up to its target wait add up to more than the op, as they
did before. The mongodb round-trip runs with SKIP_RESTORE, so only its
source wait counts there.
postgres keeps its 360s: whether a suite pays the extra minutes on
every red run is a per-suite call, and this one is not made here.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The walkthrough gave each MariaDB HelmRelease 300s, less than the
release can legitimately take, so a slow first boot failed the harness
and a failed install surfaced as a bare timeout with no reason.
A wait here has two floors and must beat the larger. The chart grants a
310s startup budget, and the install can span it: helm-controller
defaults to the poller strategy, and kstatus reads the operator CR's
Ready=False as InProgress. The generated release also carries a 600s
install timeout, and a wait equal to it expires first, because its
clock starts at the apply and the install's only once the chart is
fetched. The install timeout is the binding floor, hence 660s.
On timeout and on Stalled, wait_hr_ready now dumps every condition and
the release history: a scheduled retry can overwrite the Ready message
before anyone reads it. It also stops folding different answers into one
"(none recorded)" line. An absent release, a release deleted during the
wait, a lookup kubectl could not answer, an empty history and a failed
history read each say so, with kubectl's own stderr kept, because that
message is usually the cause. A release seen earlier still gets its dump
when only the last lookup failed.
Every backup example whose wait_hr_ready carries this body takes the
same change, so the copies stay byte-identical.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on #3385:
- Move the endpointCA rationale out of the Helm template (lllamnyp): the
SSL_CERT_DIR and CA-volume blocks were multi-paragraph essays. Trim each to
a few lines stating only the non-obvious decision (SSL_CERT_DIR vs
AWS_CA_BUNDLE; the deliberate non-optional mount) and point at the
values.yaml endpointCA field for the full description.
- wait_deleted (00-helpers.sh): poll with 'get --ignore-not-found' and treat
only an empty result as deletion, so an RBAC/API-server/transport failure is
no longer read as success and cleanup can't settle on a false positive.
- Backup demo CA Secret: scope the default name to CLICKHOUSE_NAME, stamp an
ownership label on the copy, refuse to overwrite a Secret the demo doesn't
own, and delete on cleanup only when that label is present — so a foreign
Secret sharing the name is never clobbered or removed.
- README: document CH_BACKUP_CA_SECRET and the release-scoped, demo-owned CA
Secret; spell out the admin-copy path (export CH_BACKUP_CA_SECRET before
steps 04 and 07).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings).
1. The backup-s3-ca volume carries no `optional: true` while the adjacent
useSystemBucket credential refs deliberately do, and the asymmetry was
undocumented — an operator naming a Secret that does not exist gets the whole
ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the
fail-closed behaviour and wrote down why the two cases differ: credentials are
projected by the platform after install, so a strict ref would wedge every
fresh release until the first BackupJob, whereas this CA is named by the
tenant, must already exist, and is trust material — optional would mount an
empty dir, the Pod would look healthy, and the first backup would fail deep
inside clickhouse-backup with an x509 error naming neither the Secret nor the
chart. The comment also states the blast radius plainly.
2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read
was reported as "CA not found" followed by advice (set S3_CA_SECRET) that
cannot fix a permissions problem. It now separates present / absent /
not-allowed-to-look and says so, and the README no longer labels step 03
simply "tenant": copying the CA reads a Secret in tenant-root, which is a
platform capability. Added a section spelling out the three ways to satisfy
that (admin credentials, a one-off admin copy, or skipping the CA entirely on
a publicly-trusted endpoint).
3. Caveat, and a usability regression this branch introduced: on a cluster with
a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default
the user never chose. It now distinguishes an explicitly named Secret (still
an error when missing — a typo should not be guessed past) from the unset
default, where it warns and continues with backup.endpointCA unset. Verified
all four paths against a fake kubectl: Forbidden, absent+default,
absent+explicit, and present.
4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy
`schedule` CronJob writes to S3 through restic and does not consume it, so a
private-CA endpoint combined with `schedule` still fails TLS. Documented on
the field itself, so it lands in values.schema.json, the README and the
cozyrds openAPISchema.
The PR body's claim about select-e2e.sh/select-install.sh mappings was stale
after the restructure that removed them; corrected there rather than in code.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Recommendations from the third review pass (which returned LGTM; none of these
blocked, all three are cheap and make the failure paths honest).
1. The round-trip's catch named nothing. An unnamed `describe HelmRelease` in
tenant-root dumps every release on the root tenant of a live cluster and
buries the two that matter, so name them (mariadb-2-backup-roundtrip names
its release for the same reason). Raised podLogs tail 100 -> 200, since the
sidecar logs every action and an S3 error sits well back from the tail by the
time a step fails, and added the Keeper Pods: chkeeper.yaml labels them
`app: <release>-keeper`, so the existing app.kubernetes.io/instance selectors
missed them entirely — and a source ClickHouse that never reaches a ready
replica is usually Keeper failing to form a quorum, which explains itself
only there.
2. SSL_CERT_DIR is now `/etc/ssl/cozy-s3-ca:/etc/ssl/certs`. The single-directory
form works — Go reads the default bundle FILE regardless, which is what the
earlier measurement showed — but that left the public-CA half of the contract
resting on a file this image happens to ship. Naming the system directory
costs nothing and survives a base-image bump that moves the bundle; verified
against public AWS S3 that both forms still verify (403 auth, no x509). The
mount path stays the bare directory, and the unit test now pins both halves
separately — it caught the over-broad edit that had briefly made mountPath a
colon list.
3. cleanup.sh returned while the uninstalls it triggered were still running, so
the next suite (etcd) could start with helm-controller still draining this
one, and the pre-clean could hand step 04 an application whose predecessor
was still going away. It now waits for both ClickHouse releases and the
Bucket to actually disappear, and — since the script deliberately runs
without `set -e` so one absent resource cannot abandon the rest — records
those failures and exits non-zero, rather than reporting a teardown that did
not settle as success. wait_deleted polls `get` instead of using
`kubectl wait --for=delete`, which errors on an already-absent resource: the
normal case here, and indistinguishable from a real failure.
Sized the two cleanup step budgets to 12m for the same reason the run step is
60m: cleanup.sh's waits now sum to 540s, which the previous 8m did not clear.
The Keeper-sized-from-the-server-resources bug the review found outside this
diff is filed as #3487.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Second review pass on this branch.
1. The op timeout equalled the inner sum instead of exceeding it: the comment
enumerated 3900s of inner waits and then set 65m, and 3900s IS 65m — so there
was zero headroom for the applies and SQL calls between those waits, and the
claim of "room" was false. Sized the inner budgets against what CI actually
takes instead (the bucket suite completes in ~37s and a backup-enabled
ClickHouse plus Keeper reaches Ready in ~98s, measured on run 30353894332):
bucket claim/access 300->180 and the three job waits 600->480, summing to
3300s, under a 60m op timeout. Shrinking the inner budgets rather than
raising the ceiling also keeps this suite closer to its postgres and mariadb
siblings on a 180-minute job cap.
2. The strategy Pod's securityContext comment still justified root with "Pod
runs apk add at startup", which the previous commit made conditional — the
default path now runs the platform's pre-baked image and installs nothing.
Scoped the note to the alpine fallback and said explicitly that the field set
matches cozy-default-altinity, runAsNonRoot deliberately unset in both.
3. clickhouse-backup.yaml still referenced clickhouse-backup-{1-contracts,
2-roundtrip}: the file moved unchanged, so the Test rename never reached its
own header and a reader following either name found nothing.
4. wait_sts_ready's timeout diagnostic could not print anything. It selected
clickhouse.altinity.com/chi=${name#chi-}, which yields
clickhouse-<app>-clickhouse-0-0 while the label carries the CHI name
(clickhouse-<app>), and 2>/dev/null then swallowed kubectl's "No resources
found". Ask for the StatefulSet's own first replica by name instead and let
both streams through; verified it now surfaces the Pod events that explain
why the replica never came up.
Also dropped the clickhouse bats case added in the previous commit: it asserted
what the generic examples/backups/<app> -> <app> mapping already guarantees, so
it did not lock the invariant its comment claimed. That invariant is now
recorded where it belongs — in the mapping's own comment in select-e2e.sh.
The dedicated ch-backup-client image this example now inherits is tracked in
#3420; noted there that the command: override is load-bearing in two files.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review findings on the previous two commits.
1. TIA no longer selected the suite that runs the harness. select-e2e.sh maps
examples/backups/<app>/** to suite <app>, so with the round-trip living in a
separate clickhouse-backup/ dir an edit to the harness selected `clickhouse`
— which did not execute those scripts — while the suite that does was left
out. Fixed the way postgres/mariadb/etcd already do it: both Tests move into
hack/e2e-chainsaw/clickhouse/ as clickhouse-1-backup-contracts and
clickhouse-2-backup-roundtrip. That also deletes every special case the
earlier commits added (src_to_suites, suite_to_source, and the bats case that
guarded them — both scripts are now byte-identical to main), and the suite
list in hack/e2e-chainsaw/README.md stays correct because no dir is added.
The bats case is replaced by one asserting the invariant that actually
matters: an examples/backups/clickhouse edit selects the suite that runs it.
2. The docs claimed a fail-fast the harness did not have, and the budget was
sized as if it did. wait_for_field polled to the full timeout with no
terminal-value escape, and the HelmRelease/StatefulSet waits were bare
`kubectl wait` with no existence backstop and no Stalled fail-fast. Ported
wait_for_field's fail_value parameter and wait_hr_ready from the postgres
helpers, added wait_sts_ready, and routed all eight waits through them.
Verified against a fake kubectl: a Failed BackupJob now bails in 3s instead
of 600s, a Stalled HelmRelease immediately. The inner budgets sum to 3900s,
which the old 30m op timeout did not cover — a hang would have been SIGKILLed
with no diagnostics, the failure mode .chainsaw.yaml calls strictly worse — so
the timeout is now 65m with the arithmetic written out.
3. The strategy Pod ran `apk add --no-cache curl jq` on every backup and
restore. Harmless while nothing in CI ran it; a per-PR dependency on the
Alpine CDN once it does, and broken outright air-gapped — which is exactly
why the shipped strategy bakes curl+jq into its image. Step 01 now reads that
image off cozy-default-altinity and drops the apk add, falling back to
alpine only when the platform strategy is absent. Also bounded the two
polling curls with --max-time; a hung connection previously stalled the loop
until the outer timeout.
4. 91-scenario-user-backup.md still documented the old tenant-test default.
5. .chainsaw.yaml claimed all app tests run in tenant-test, which the postgres
and mariadb round-trips already contradicted before this branch.
The round-trip header now also states its scope plainly: it exercises the
example's strategy, not the shipped cozy-default-altinity.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The round-trip Test was skipping: it exited 0 behind CLICKHOUSE_E2E_S3_ROUNDTRIP,
so "PASS (1.01s)" meant the gate fired, not that backup and restore worked. Only
the contracts Test ran, which proves the chart still exposes the sidecar and its
auth Secret — not that a byte ever reaches S3 and comes back.
The gate existed because the sidecar could not verify the in-cluster SeaweedFS
certificate. backup.endpointCA closes that, so adopt the shape mariadb and
postgres already use to run their round-trips un-gated:
- examples: resolve the S3 endpoint from S3_ENDPOINT when set (BucketInfo
advertises the external ingress URL, an unroutable placeholder in CI), copy the
seaweedfs CA (auto-discovered from its cert-manager Certificate when the
fullname default is absent) into a per-release Secret, and reference it from
backup.endpointCA on both the source and the to-copy target. NAMESPACE now
defaults to tenant-root, matching the other backup demos.
- suite: drop the gate, run in tenant-root — an isolated tenant's Cilium egress
allowlist blocks its Pods from reaching tenant-root's seaweedfs — and pre-clean
with cleanup.sh, because tenant-root is shared and a leftover Succeeded
BackupJob would satisfy the harness's wait without a backup running.
cleanup.sh also drops the copied CA Secret; the platform's own is never touched.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Myasnikov Daniil