Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.
docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on #3385:
- Move the endpointCA rationale out of the Helm template (lllamnyp): the
SSL_CERT_DIR and CA-volume blocks were multi-paragraph essays. Trim each to
a few lines stating only the non-obvious decision (SSL_CERT_DIR vs
AWS_CA_BUNDLE; the deliberate non-optional mount) and point at the
values.yaml endpointCA field for the full description.
- wait_deleted (00-helpers.sh): poll with 'get --ignore-not-found' and treat
only an empty result as deletion, so an RBAC/API-server/transport failure is
no longer read as success and cleanup can't settle on a false positive.
- Backup demo CA Secret: scope the default name to CLICKHOUSE_NAME, stamp an
ownership label on the copy, refuse to overwrite a Secret the demo doesn't
own, and delete on cleanup only when that label is present — so a foreign
Secret sharing the name is never clobbered or removed.
- README: document CH_BACKUP_CA_SECRET and the release-scoped, demo-owned CA
Secret; spell out the admin-copy path (export CH_BACKUP_CA_SECRET before
steps 04 and 07).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings).
1. The backup-s3-ca volume carries no `optional: true` while the adjacent
useSystemBucket credential refs deliberately do, and the asymmetry was
undocumented — an operator naming a Secret that does not exist gets the whole
ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the
fail-closed behaviour and wrote down why the two cases differ: credentials are
projected by the platform after install, so a strict ref would wedge every
fresh release until the first BackupJob, whereas this CA is named by the
tenant, must already exist, and is trust material — optional would mount an
empty dir, the Pod would look healthy, and the first backup would fail deep
inside clickhouse-backup with an x509 error naming neither the Secret nor the
chart. The comment also states the blast radius plainly.
2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read
was reported as "CA not found" followed by advice (set S3_CA_SECRET) that
cannot fix a permissions problem. It now separates present / absent /
not-allowed-to-look and says so, and the README no longer labels step 03
simply "tenant": copying the CA reads a Secret in tenant-root, which is a
platform capability. Added a section spelling out the three ways to satisfy
that (admin credentials, a one-off admin copy, or skipping the CA entirely on
a publicly-trusted endpoint).
3. Caveat, and a usability regression this branch introduced: on a cluster with
a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default
the user never chose. It now distinguishes an explicitly named Secret (still
an error when missing — a typo should not be guessed past) from the unset
default, where it warns and continues with backup.endpointCA unset. Verified
all four paths against a fake kubectl: Forbidden, absent+default,
absent+explicit, and present.
4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy
`schedule` CronJob writes to S3 through restic and does not consume it, so a
private-CA endpoint combined with `schedule` still fails TLS. Documented on
the field itself, so it lands in values.schema.json, the README and the
cozyrds openAPISchema.
The PR body's claim about select-e2e.sh/select-install.sh mappings was stale
after the restructure that removed them; corrected there rather than in code.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The round-trip Test was skipping: it exited 0 behind CLICKHOUSE_E2E_S3_ROUNDTRIP,
so "PASS (1.01s)" meant the gate fired, not that backup and restore worked. Only
the contracts Test ran, which proves the chart still exposes the sidecar and its
auth Secret — not that a byte ever reaches S3 and comes back.
The gate existed because the sidecar could not verify the in-cluster SeaweedFS
certificate. backup.endpointCA closes that, so adopt the shape mariadb and
postgres already use to run their round-trips un-gated:
- examples: resolve the S3 endpoint from S3_ENDPOINT when set (BucketInfo
advertises the external ingress URL, an unroutable placeholder in CI), copy the
seaweedfs CA (auto-discovered from its cert-manager Certificate when the
fullname default is absent) into a per-release Secret, and reference it from
backup.endpointCA on both the source and the to-copy target. NAMESPACE now
defaults to tenant-root, matching the other backup demos.
- suite: drop the gate, run in tenant-root — an isolated tenant's Cilium egress
allowlist blocks its Pods from reaching tenant-root's seaweedfs — and pre-clean
with cleanup.sh, because tenant-root is shared and a leftover Succeeded
BackupJob would satisfy the harness's wait without a backup running.
cleanup.sh also drops the copied CA Secret; the platform's own is never touched.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>