Commit Graph
10 Commits
Author SHA1 Message Date
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Aleksei Sviridkin 04d469fc85 fix(tests): let the backup walkthrough waits outlive their installs
The backup round-trips drive examples/backups, whose bucket waits gave
up at 300s and whose postgres waits at 360s, under the 600s install
timeout cozystack-api gives each of those releases. The kafka example's
bucket wait passed no budget and took the helper's 300s default. A
failed install there reported only that the wait ran out.

Raise each to 660s, and grow the Chainsaw op around every walkthrough
by what its waits gained, so a failure at any later wait keeps the
headroom it had and is still reported by the script rather than cut
off by the op. The kafka and kafka-metadata Kafka waits stay: their
7h install timeout cannot be cleared inside the E2E job cap.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:22 +03:00
Aleksei Sviridkin ead7048b35 refactor(backups): define the walkthrough wait helpers once
Every backup walkthrough carried its own wait_for_field, and most their
own wait_hr_ready and wait_deleted: twelve, eight and two copies, which
had already drifted into five, three and two bodies. A fix made in one
copy did not reach the others, and a diagnostics change carried by hand
between two of them once went into the wrong function and was caught
only by reading the diff.

The helpers now live in examples/backups/_lib/wait-helpers.sh, sourced
by each walkthrough's 00-helpers.sh. Where copies differed, the shared
body keeps the diagnostics any of them had: wait_for_field leaves
kubectl's stderr attached, and wait_hr_ready prints the conditions and
history dump on Stalled or timeout. Call sites and budgets are unchanged.

A new unit test fails when any walkthrough defines one of these helpers
again, which replaces the byte-identity pin between copies.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 02:21:01 +03:00
Alexey Artamonov f02ca8cbc2 test(backups): log in as an application user after each restore
The backup examples and e2e suites no longer set passwords. They read
the generated ones from <release>-credentials with jq, since a username
may contain a dot that jsonpath treats as a path step, and log in as an
application user after each restore, so a restore that leaves the
credentials unusable fails the run. The MariaDB run also checks that an
account created out of band on the source is absent from the copy. The
unreachable-target restore step asserts a terminal Failed attributed to
the recovery window, since the reason itself is best effort.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-24 18:11:07 +03:00
Aleksei Sviridkin 289acb2ed6 fix(tests): let the mariadb backup example cover a first boot
The walkthrough gave each MariaDB HelmRelease 300s, less than the
release can legitimately take, so a slow first boot failed the harness
and a failed install surfaced as a bare timeout with no reason.

A wait here has two floors and must beat the larger. The chart grants a
310s startup budget, and the install can span it: helm-controller
defaults to the poller strategy, and kstatus reads the operator CR's
Ready=False as InProgress. The generated release also carries a 600s
install timeout, and a wait equal to it expires first, because its
clock starts at the apply and the install's only once the chart is
fetched. The install timeout is the binding floor, hence 660s.

On timeout and on Stalled, wait_hr_ready now dumps every condition and
the release history: a scheduled retry can overwrite the Ready message
before anyone reads it. It also stops folding different answers into one
"(none recorded)" line. An absent release, a release deleted during the
wait, a lookup kubectl could not answer, an empty history and a failed
history read each say so, with kubectl's own stderr kept, because that
message is usually the cause. A release seen earlier still gets its dump
when only the last lookup failed.

Every backup example whose wait_hr_ready carries this body takes the
same change, so the copies stay byte-identical.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-23 11:56:10 +03:00
Myasnikov Daniil 186547d835 test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.

The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.

Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.

Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.

The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.

Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.

One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.

Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-05 13:19:17 +05:00
Andrey KolkovandClaude Opus 4.8 2829f7a300 docs(mariadb): address review notes on backup example
- 00-bucket.yaml: correct the Secret-name comment to include the
  "bucket-" release prefix (bucket-<bucket>-<user>), matching the
  README and 00-helpers.sh.
- README: note that S3_CA_NAMESPACE defaults to tenant-root
  independently of NAMESPACE, so a non-root tenant run must override
  it (and S3_CA_SECRET) too.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 10:44:42 +04:00
Andrey KolkovandClaude Opus 4.8 d1f2822018 test(mariadb): add backup/restore round-trip e2e (chainsaw) via example harness
Reworks the never-merged backup-mariadb-e2e branch onto the modern
Chainsaw approach (docs/agents/e2e-testing.md). The bats port that
branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the
whole hack/e2e-apps/*.bats app suite has been replaced by
hack/e2e-chainsaw/, and the mariadb strategy controller + example
manifests it depended on have since landed on main independently. Only
the e2e coverage was still missing.

Convert examples/backups/mariadb/ from static numbered YAML into the
same executable harness the other backup demos use (00-helpers.sh,
run-all.sh, cleanup.sh), keeping the numbered manifests as the
human-readable flow that run-all.sh drives — so the documented flow and
the automated test can never drift. run-all.sh provisions a Bucket,
derives the two per-app Secrets the strategy references from BucketInfo
+ the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires
an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and
asserts the sentinel round-tripped through S3 into the copy while the
source is left untouched.

Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to
hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring
postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant
cannot reach tenant-root's seaweedfs across the Cilium egress policy),
skipDelete since the harness (not Chainsaw) owns the resources, fail-fast
on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics.
Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs
the Cozystack jobs own only by label.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 10:44:42 +04:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 13:49:59 +04:00
Andrey Kolkov 62605a198b feat(platform): implement backup strategy for mariadb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-12 13:56:29 +04:00