mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 10:37:21 +09:00
Myasnikov Daniil
186547d835
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos containers instead of QEMU guests, so a tenant worker sits at L2 rather than L3. Measured in CI on commits where both substrates ran the same tree: whole-job wall clock down 24-49 minutes, essentially all of it in Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to 466-809s and stop depending on the runner's vgif flag. The node-join soft-red gate goes with the substrate, which is what it was for. It existed because nested virtualisation on the shared runners degraded far enough that a worker registering in minutes elsewhere could miss any deadline this test could afford; #3513 established that the discriminator was the host's kvm_amd vgif flag, and removing a nesting level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from all four workflows, and a missed deadline is an ordinary red again. The ::warning annotation survives, because the reason a run went red should be legible without opening the log, and so does the 124-versus-everything -else test, because "the workers were slow" and "the wait never ran" send a reader to different places. Done by swapping the substrate inside the existing e2e jobs rather than by promoting a separate container job. Those jobs carry more than a substrate -- the GitHub App token, the report-overrun guard, the images list, the SSH breakpoint -- and promoting a job that never had them would have dropped four features without saying so. Three properties of container mode drive the design, and each fails quietly rather than loudly. machine.kernel.modules is a silent no-op -- kernel_module_spec.go returns early on ModeContainer with no error and no event -- so the host loads openvswitch and zfs before the nodes start and the workflow step asserts both rather than trusting them; a missing module surfaces an hour later looking like a CNI or a storage regression. Docker gives a privileged container its own tmpfs /dev seeded once at creation, so /dev/kvm works but ZFS zvols created later never appear and linstor-csi fails ControllerPublishVolume; the compose file binds the host devtmpfs, which is also why talosctl cluster create docker can never carry LINSTOR. And there is no ARP VIP, so 192.168.123.11 stands in for the QEMU lane's .10. The QEMU machinery stays in the tree and keeps running: nightly.yaml and e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated StorageClass, its Immediate binding mode, the tenant StorageClass propagation that only applies to remotely-accessible classes, and the Cozystack Talos node image with its extensions all keep nightly and release-candidate coverage. None of them is exercised per pull request any more. That is the trade. Nothing on the PR path builds or downloads a nocloud disk now, so build-talos drops the talos-nocloud step and the talos-image artifact. build-talos itself stays: finalize consumes its digest fragment. One property this does not fix, recorded beside the runner class because it will be read off a red run eventually: a container node reports the HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters that with kubelet systemReserved. That corrects the scheduler and not what a pod reads from /proc, so a workload sizing itself from /proc/cpuinfo or /proc/meminfo is configured for the whole runner. Also carried here because the same files carry them: the ghcr.io pull-through mirror removal (see #4007 -- whichever lands first makes the other a no-op), the authoritative HelmRelease readiness gate, the backup-credential preflight, the prepull guard against an image-less container fragment, and folding the two standalone OIDC suites into the latest-version tenant cluster. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
ClickHouse backup/restore example
Heads up — most clusters do not need this walk-through. Cozystack ships a platform-managed
cozy-defaultBackupClasstogether with the system bucketcozy-backups. To use the default flow, setbackup.enabled: trueandbackup.useSystemBucket: trueon the ClickHouse release; tenants do not provision a Bucket or supply S3 credentials. See Backup Classes for the supported BackupJob / Plan flow againstcozy-default. The walk-through below covers the legacy path that wires a per-app Bucket, custom strategy CR, and bespoke BackupClass — useful when an admin needs a tuned non-default policy.
This directory shows how to back up and restore a Cozystack-managed
ClickHouse application using the cluster's Altinity backup strategy
driver. The chart materialises an Altinity clickhouse-backup
sidecar inside every chi-* Pod (when backup.enabled=true) that exposes
the tool's HTTP API on port 7171 and shares /var/lib/clickhouse with
clickhouse-server. The strategy Pod itself is a tiny curl + jq client
that POSTs to the sidecar and polls the action log.
Step order
| File | Role | Triggered by |
|---|---|---|
00-helpers.sh |
Shared bash helpers and env defaults; sourced by every step. | n/a |
01-create-strategy.sh |
Creates the cluster-scoped Altinity strategy that wraps clickhouse-backup. |
admin |
02-create-backupclass.sh |
Maps apps.cozystack.io/ClickHouse to that strategy. |
admin |
03-create-bucket.sh |
Provisions a Bucket, caches its S3 coordinates into .bucket-info.env (chmod 600; raw access keys), and copies the S3 endpoint CA into a per-release Secret so the sidecar can verify a self-signed endpoint. cleanup.sh removes both. |
tenant, plus admin for the CA copy — see below |
04-create-clickhouse.sh |
Provisions a ClickHouse instance with backup.enabled=true (chart emits the backup-s3 Secret + sidecar) and writes a sentinel row. |
tenant |
05-create-backupjob.sh |
Submits a BackupJob and waits for Succeeded. |
tenant |
06-restore-in-place.sh |
Drops the sentinel and restores into the same instance via RestoreJob. |
tenant |
07-restore-to-copy.sh |
Provisions a second ClickHouse and restores into it via RestoreJob.spec.targetApplicationRef. |
tenant |
cleanup.sh |
Removes everything created by the demo. | admin or tenant |
run-all.sh |
Convenience runner that executes 01..07 in order. | demo |
90-scenario-admin-prepare.md |
Narrative for the admin preparation steps. | docs |
91-scenario-user-backup.md |
Narrative for the user backup flow. | docs |
92-scenario-user-restore.md |
Narrative for the user restore flow (in-place and to-copy). | docs |
Environment variables (overrides)
All variables come from 00-helpers.sh:
| Variable | Default | Description |
|---|---|---|
NAMESPACE |
tenant-root |
Tenant namespace for the demo. tenant-root because the S3 client is the clickhouse-backup sidecar inside the tenant and cozystack's shared SeaweedFS lives there; an isolated tenant's Cilium egress allowlist blocks its Pods from reaching tenant-root's SeaweedFS. |
CLICKHOUSE_NAME |
clickhouse-test |
Source ClickHouse application name. |
CLICKHOUSE_RESTORE_NAME |
clickhouse-restore |
Target ClickHouse for the to-copy restore. |
BUCKET_NAME |
clickhouse-backups |
Cozystack Bucket to provision. |
BACKUPCLASS_NAME |
clickhouse-backup |
BackupClass name (cluster-scoped). |
STRATEGY_NAME |
altinity |
Strategy name (cluster-scoped). |
BACKUPJOB_NAME |
clickhouse-backup-job |
BackupJob name. |
RESTOREJOB_INPLACE_NAME |
clickhouse-restore-inplace |
In-place RestoreJob name. |
RESTOREJOB_TOCOPY_NAME |
clickhouse-restore-to-copy |
To-copy RestoreJob name. |
S3_ENDPOINT |
(from BucketInfo) | Overrides the S3 endpoint the sidecar uses. BucketInfo advertises the external ingress URL, which in-cluster Pods cannot always reach or TLS-validate; set https://seaweedfs-s3.tenant-root:8333 to use the in-cluster endpoint. |
S3_CA_SECRET |
seaweedfs-ca-cert |
Secret holding the S3 endpoint's CA. Step 03 copies its CA into a per-release Secret wired to backup.endpointCA, so the sidecar can verify a self-signed endpoint. Auto-discovered from the seaweedfs cert-manager Certificate if this name is absent. Set to "" on a publicly-trusted endpoint to skip the copy and leave backup.endpointCA unset. |
S3_CA_NAMESPACE |
tenant-root |
Namespace of S3_CA_SECRET. Independent of NAMESPACE: the shared SeaweedFS and its CA live in tenant-root even when the demo runs elsewhere. |
S3_CA_KEY |
ca.crt |
Key inside S3_CA_SECRET holding the PEM bundle. |
CH_CA_SECRET_NAME |
<CLICKHOUSE_NAME>-backup-s3-ca |
Per-release Secret the CA is copied into; referenced by backup.endpointCA. Scoped to the instance name so two instances in one namespace don't collide, and stamped with a cozystack.io/owned-by-demo label so step 03 refuses to overwrite a Secret it doesn't own and cleanup.sh deletes only one this demo created. |
CH_BACKUP_CA_SECRET |
(set by step 03) | Secret name steps 04 and 07 wire into backup.endpointCA. Step 03 sets it to CH_CA_SECRET_NAME after copying the CA and leaves it empty when S3_CA_SECRET="". On the admin-copy path, export it yourself to the pre-copied Secret before step 04, since step 03 does no copy and leaves it empty. |
Prerequisites
- Cozystack cluster with the backup-controller and backupstrategy-controller installed.
kubectl,jq, and (for04+) the ClickHouse operator deployed by the chart.
Who can run step 03
Step 03 is otherwise a tenant action, but copying the S3 endpoint CA reads a Secret in S3_CA_NAMESPACE (tenant-root by default), which is a platform/admin capability rather than a tenant one. Three ways out, in order of preference:
- run step 03 with credentials that can read that namespace (the admin path this demo assumes);
- have an admin copy the CA into
<NAMESPACE>/<CH_CA_SECRET_NAME>once, then run the demo withS3_CA_SECRET=""andexport CH_BACKUP_CA_SECRET="<CH_CA_SECRET_NAME>"before steps 04 and 07, so both ClickHouse flows wirebackup.endpointCAto that pre-copied Secret; - skip it entirely with
S3_CA_SECRET=""when the S3 endpoint's certificate is signed by a publicly-trusted CA — nothing needs a CA bundle then.
The script distinguishes the three outcomes so a permissions problem does not present as a missing Secret: it stops with an explicit permissions message on Forbidden, and on a genuinely absent default (no seaweedfs-ca-cert, no seaweedfs Certificate to discover, and S3_CA_SECRET not set explicitly) it warns and continues with backup.endpointCA unset instead of failing — which is the right outcome on a cluster whose endpoint is publicly trusted. Naming a Secret that does not exist is still an error.