mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
Myasnikov DaniilandClaude Fable 5
9297e51170
Reconcile the Chainsaw e2e migration with ~500 commits of main drift: - keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's _lib/ rename with the Chainsaw finally/snapshot deltas re-applied - resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted; their post-branch assertion drift is re-ported into the Chainsaw suites in follow-up commits - drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup, serviceexposure) from _lib/ relocation; ported as Chainsaw suites in follow-up commits - docs/agents/e2e-testing.md: merge main's plan-job TIA wording with the Chainsaw suite phrasing Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
126 lines
6.9 KiB
Markdown
126 lines
6.9 KiB
Markdown
# Etcd backup/restore example
|
|
|
|
> **Heads up — most clusters do not need this walk-through.** Cozystack
|
|
> ships a platform-managed `cozy-default` `BackupClass` together with
|
|
> the system bucket `cozy-backups`. Tenants reference `cozy-default`
|
|
> directly from BackupJob / Plan / RestoreJob without provisioning a
|
|
> Bucket or supplying S3 credentials. See
|
|
> [Backup Classes](../../../docs/operations/backup-classes.md).
|
|
> The walk-through below covers the **legacy** path that wires a
|
|
> per-app Bucket and bespoke BackupClass — useful for tuned non-default
|
|
> policies.
|
|
|
|
This directory shows how to back up and restore a Cozystack-managed `Etcd`
|
|
application using the cluster's `Etcd` backup strategy driver. The driver
|
|
delegates to the [etcd-operator][etcd-operator]: each Cozystack `BackupJob`
|
|
materialises an `etcd-operator.cozystack.io/v1alpha2 EtcdSnapshot` CR (a one-shot Job that
|
|
streams `etcdctl snapshot save` output to S3), and each `RestoreJob`
|
|
performs an in-place restore by suspending the Etcd HelmRelease,
|
|
deleting the chart-rendered `EtcdCluster`, and recreating it with
|
|
`spec.bootstrap.restore.source.s3` populated from the artefact.
|
|
|
|
## Supported destinations
|
|
|
|
This driver supports **S3 destinations only**. The strategy CR's
|
|
`spec.template.destination` schema rejects `pvc` at admission time. The
|
|
upstream etcd-operator's PVC backup-write path
|
|
(`<mount>/<subPath>/<backupName>.db`, plus
|
|
`BACKUP_INCLUDE_REVISION` / `BACKUP_TIMESTAMP` suffixes the agent adds at
|
|
write time) disagrees with its PVC restore-read path (opens
|
|
`<mount>/<subPath>` as a file, or `<mount>/snapshot.db` when subPath is
|
|
empty), so a PVC backup taken through this strategy would be
|
|
unrestoreable. The S3 path uses the operator's `BackupSnapshot.URI`
|
|
status field to recover the full final S3 object key including all
|
|
agent-injected suffixes — see
|
|
`s3KeyFromArtifactURI` in the driver.
|
|
|
|
PVC support will land once upstream gains symmetric semantics
|
|
(equivalent of `BackupSnapshot.URI` for PVC) or makes the restore-side
|
|
read the same path the backup-write side produced.
|
|
|
|
## Restore semantics
|
|
|
|
This driver supports **in-place restore only**. To-copy restore is
|
|
rejected by the driver with a terminal `phase=Failed`. The PRIMARY
|
|
reason is API-level: `RestoreJob.spec.targetApplicationRef` is a
|
|
`TypedLocalObjectReference`, which has no namespace field — the
|
|
restore target is always the SAME namespace as the source `Backup`,
|
|
so there is no API surface today for a "fresh cluster in a different
|
|
namespace" flow. A SECONDARY chart-level constraint compounds this:
|
|
`packages/extra/etcd/templates/check-release-name.yaml` pins the
|
|
Helm release name to `etcd`, so two `apps.cozystack.io/Etcd`
|
|
applications cannot coexist in the same namespace regardless of
|
|
whether `targetApplicationRef` were extended to cross namespaces.
|
|
|
|
To get "a fresh cluster with the snapshot's data" today, deploy a
|
|
new `Etcd` app in a different namespace and hand-author an
|
|
`EtcdCluster` with `spec.bootstrap.restore.source.s3` pointing at
|
|
the Backup's S3 coordinates from `Backup.spec.driverMetadata`.
|
|
Enabling a managed to-copy flow in this driver would require both
|
|
extending `RestoreJob.spec.targetApplicationRef` to a namespaced
|
|
reference upstream AND lifting the chart's release-name pin — the
|
|
chart change alone is not sufficient.
|
|
|
|
## Step order
|
|
|
|
| File | Role | Triggered by |
|
|
|---|---|---|
|
|
| `00-helpers.sh` | Shared bash helpers and env defaults; sourced by every `01..05` step and by `cleanup.sh`. | n/a |
|
|
| `01-create-strategy.sh` | Creates the cluster-scoped `Etcd` strategy. | admin |
|
|
| `02-create-bucket.sh` | Provisions a `Bucket`, mints an `<app>-etcd-backup-creds` Secret in the source namespace, and creates the `BackupClass` bound to the Etcd strategy with resolved S3 coordinates. Caches raw creds in `.bucket-info.env` (chmod 600); `cleanup.sh` removes it. | tenant |
|
|
| `03-create-etcd-src.sh` | Provisions the source `Etcd` application, waits for `EtcdCluster.status.conditions[Available]=True`, and writes a sentinel key under `etcdctl`. | tenant |
|
|
| `04-create-backupjob.sh` | Submits a `BackupJob` and waits for `phase=Succeeded`. | tenant |
|
|
| `05-restore-in-place.sh` | Mutates the sentinel, submits a `RestoreJob` (no `targetApplicationRef`), and verifies the sentinel is round-tripped back to its pre-mutation value. | tenant |
|
|
| `cleanup.sh` | Best-effort teardown. | admin or tenant |
|
|
| `run-all.sh` | Convenience runner: 01..05 in order. | demo |
|
|
|
|
## Environment variables (overrides)
|
|
|
|
All variables come from `00-helpers.sh`:
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `NAMESPACE` | `tenant-root` | Tenant namespace for the demo. |
|
|
| `ETCD_NAME` | `etcd` | Source Etcd application name. Must be `etcd` because the chart pins the release name. |
|
|
| `BUCKET_NAME` | `etcd-backups` | Cozystack `Bucket` to provision. |
|
|
| `BACKUPCLASS_NAME` | `etcd-default` | BackupClass name (cluster-scoped). |
|
|
| `STRATEGY_NAME` | `etcd-strategy-default` | Strategy name (cluster-scoped). |
|
|
| `BACKUPJOB_NAME` | `etcd-backup-job` | BackupJob name. |
|
|
| `RESTOREJOB_INPLACE_NAME` | `etcd-restore-inplace` | In-place RestoreJob name. |
|
|
|
|
## Prerequisites
|
|
|
|
- Cozystack cluster with `backup-controller` and `backupstrategy-controller`
|
|
running, with the Etcd dispatch wired (see
|
|
`internal/backupcontroller/etcdstrategy_controller.go`).
|
|
- `kubectl`, `jq`.
|
|
- The `etcd-operator.cozystack.io` CRDs installed by
|
|
`packages/system/etcd-operator-crds`. The Cozystack flow specifically depends
|
|
on `EtcdCluster` (the driver re-creates it with
|
|
`spec.bootstrap.restore.source.s3` during in-place restore) and `EtcdSnapshot`
|
|
(one materialised per `BackupJob`). There is no separate `EtcdRestore` CRD —
|
|
restore goes through `EtcdCluster.spec.bootstrap`.
|
|
- The Cozystack `etcd-rd` ApplicationDefinition installed so
|
|
`apps.cozystack.io/Etcd` renders a HelmRelease.
|
|
|
|
## Notes
|
|
|
|
- The chart's `backup.*` values block (`backup.enabled`, `backup.schedule`,
|
|
`backup.destinationPath`, ...) is marked **DEPRECATED**. It continues to
|
|
render unchanged for existing tenants; new tenants should use this
|
|
`BackupClass` flow instead. See `packages/extra/etcd/README.md`.
|
|
- `02-create-bucket.sh` reads the bucket's `region` from the
|
|
`BucketInfo` Secret. SeaweedFS (the default object-storage backend
|
|
for Cozystack `Bucket` resources) does not populate
|
|
`.spec.secretS3.region`, so the script falls back to `us-east-1`. The
|
|
fallback only matters for S3 SDKs that require a non-empty region;
|
|
override it by setting `ETCD_REGION` in your shell before sourcing
|
|
`00-helpers.sh` if you point the demo at a non-SeaweedFS endpoint
|
|
whose region differs.
|
|
- Step 05 only verifies the happy path. Edge cases (stuck HR resume, failed bootstrap, mid-restore controller crash) belong in the etcd Chainsaw suite at `hack/e2e-chainsaw/etcd/`.
|
|
- The destructive in-place flow deletes the `EtcdCluster` (and the
|
|
operator drops the member PVCs alongside it). All client traffic to
|
|
etcd is unavailable for the duration of the restore. Plan a window.
|
|
|
|
[etcd-operator]: https://github.com/cozystack/etcd-operator
|