Files
cozystack/hack/e2e-chainsaw
Aleksei Sviridkin f9ea0da3de feat(backups): Kafka topic-metadata backup via a dedicated strategy driver (#4000)
## What this PR does

Adds a dedicated `strategy.backups.cozystack.io/Kafka` Go driver that
backs up the **topic metadata** of a Cozystack-managed `Kafka`
application and wires it into the platform `cozy-default` flow — the
counterpart to the app-specific drivers (CNPG, MariaDB, Altinity,
MongoDB). It is a typed variant of the generic Job strategy: the
controller renders a one-shot batch Job that talks to the Kafka Admin
API, and adds on top an applicationRef Kind gate, a `Ready` precondition
on the Strimzi `Kafka` cluster, and a per-run object key via
`.BackupName` so distinct runs never overwrite each other.

What it captures: every non-internal topic's partition count,
replication factor and non-default configs (via `kafka-topics
--describe`), stored as one small object per run in the system bucket;
restore recreates them with `kafka-topics --create` + `kafka-configs
--alter`. Deliberately **out of scope**: message payloads (a
volume-snapshot strategy's job), consumer-group offsets (a log position,
only meaningful with a byte-identical data restore), ACLs, quotas, and
`KafkaUser`s. No PITR. Restored topics are created directly through the
Admin API, so on the target they are unmanaged (no `KafkaTopic` CR) —
the accepted tradeoff of an Admin-API-only driver.

Both restore flows are implemented: **in-place** recreates dropped
topics into the source, and **to-copy** applies the source's topics onto
a freshly-bootstrapped empty `Kafka` (the "bootstrap a cluster, then
apply the metadata state" flow).

Contents:
- Driver: `api/backups/strategy/v1alpha1/kafka_types.go`,
`internal/backupcontroller/kafkastrategy_controller.go` (+ unit tests
for its pure helpers), `internal/backupcontroller/kafkatypes/` (Strimzi
`Kafka` readiness gate), dispatch cases in both controllers +
`supportedBackupStrategyKinds()`, scheme registration, generated CRD +
deepcopy.
- Chart: `strategy-kafka-default.yaml` (gated on the bucket name), a
`cozy-default` binding, read RBAC on `kafka.strimzi.io/kafkas`, and the
Strimzi Kafka client image value.
- Docs: a Kafka section in `docs/operations/backup-classes.md`.
- Example + e2e: `examples/backups/kafka-metadata/` and
`hack/e2e-chainsaw/kafka-metadata/` (`kafka-3-metadata-roundtrip`), an
honest round-trip that proves a per-run `retention.ms` sentinel survives
both restores.

The original `packages/apps/kafka` chart is left untouched (it has no
legacy backup mechanism).

Validated: `go build ./...`, `go vet`, `go test
./internal/backupcontroller/...` (pure helpers plus a reconcile
Kind-gate test and the RestoreJob cleanup regression test), `helm
unittest` on the package (which now renders
`strategy-kafka-default.yaml` and asserts its
kind/name/`artifactURITemplate`), the `select-install`/`select-e2e` bats
suites, `helm template` of the strategy CR, `bash -n` of the rendered
script, and `make generate` (clean). The end-to-end round-trip
(`kafka-3-metadata-roundtrip`) is enabled and mapped like the sibling
backup suites, so CI builds this branch's controller and runs it live
against a fresh sandbox; it has not been merged-and-run before, so that
first live run must go green before merge.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. This PR adds a new `strategy.backups.cozystack.io/Kafka` CRD, a Go
driver, chart wiring, docs and examples. It does not add or rename a
`packages/apps`/`packages/extra` package, does not change any
`ApplicationDefinition`, `values.schema.json`, version enum,
`release.prefix`, output Secret/Service, the
`Package`/`Plan`/`RestoreJob` CRDs the provider types by hand, any
`hack/*.mk` anchor or make-target behaviour, node prerequisites,
labels/annotations, or telemetry metrics. A brand-new
`strategy.backups.cozystack.io` CRD travels to the Terraform provider as
opaque JSON (it types only `Plan`/`RestoreJob`/`Package`), so it keeps
working untouched. None of the downstream triggers match.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(backups): add a Kafka topic-metadata backup strategy bound by cozy-default — backs up topic definitions and configs (partitions, replication factor, non-default configs) via the Kafka Admin API, with in-place and to-copy restore; message data, consumer offsets, ACLs, quotas and KafkaUsers are out of scope
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added Kafka topic-metadata backup and restore support, excluding
message data.
- Supports ad-hoc and scheduled backups, in-place restoration, and
restoration to a separate Kafka cluster.
- Recreates missing topics and restores compatible configurations while
validating partition and replication-factor constraints.
- Waits for Kafka readiness and cleans up backup artifacts after
completion.
- Backup and restore jobs now enforce configurable deadlines to prevent
stalled operations.

- **Documentation**
- Added guidance covering Kafka metadata backups, restoration
limitations, and S3 configuration.

- **Tests**
- Added end-to-end coverage for backup, restoration, validation, and
cleanup scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-14 19:21:43 +04:00
..

Chainsaw E2E tests

The Cozystack E2E app suite, written for Kyverno Chainsaw — a declarative, Kubernetes-native E2E test framework (CNCF, part of the Kyverno project; successor to KUTTL).

This directory replaces the per-app BATS suite that used to live in hack/e2e-apps/*.bats. The CI e2e job runs chainsaw test hack/e2e-chainsaw/ after make install-cozystack; cluster bootstrap (hack/e2e-prepare-cluster.bats, hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS.

Layout

hack/e2e-chainsaw/
├── .chainsaw.yaml          # global Configuration (namespace, timeouts, JUnit report)
├── _lib/                   # shell reused by the script-heavy suites
│   ├── run-kubernetes.sh   #   tenant Kubernetes bringup + LB/NFS/ouroboros checks
│   └── remediation-guard.sh
├── <app>/
│   ├── chainsaw-test.yaml  # one or more Test definitions
│   └── <app>.yaml          # the manifests the test applies
└── ...

Suites: postgres, bucket, mariadb, mongodb, redis, valkey, qdrant, clickhouse, kafka, rabbitmq, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, opensearch, kubernetes-latest, kubernetes-previous, securitygroup.

What Chainsaw buys over the BATS suite

  • Readable tests — a list of resources and expected states, not shell control flow. The timeout N sh -ec "until kubectl get ..." + kubectl wait pair that appeared ~100 times collapses into a single assert that polls existence and state together.
  • Distinguishable failure modes — "object never created", "created but never ready", and "wrong status" all surface as a structured diff instead of an opaque shell timeout.
  • Automatic diagnostics — catch blocks dump events, describe, and pod logs on any failure (the BATS suite only did this in harbor.bats, by hand).
  • Automatic cleanup — resources Chainsaw applied are deleted (and waited on) in reverse order; finally covers operator-created leftovers.
  • CI-ready output — a JUnit chainsaw-report.xml is emitted for the test summary.

Test anatomy

Each suite is a directory with a chainsaw-test.yaml (one or more Test docs) plus the manifests it applies. A step has try (operations: apply, assert, error, script, delete, …), optional catch (diagnostics on failure), and optional finally (teardown regardless of outcome).

Declarative suites assert on status.conditions and concrete fields. Inherently imperative suites keep their logic in script steps:

  • openbao — bao operator init/unseal via kubectl exec.
  • kuberture — jq on Service annotations + external-dns split-horizon log assertions.
  • gateway — admission-rejection cases that need kubectl --as impersonation or error-message greps; the tenant apex is read from the namespace namespace.cozystack.io/host label at runtime so they pass on any host.
  • kubernetes-latest/previous — wrap _lib/run-kubernetes.sh (Kamaji bringup, port-forward, LB/NFS/ouroboros). The script cds to the repo root (the test dir is hack/e2e-chainsaw/<suite>, so repo root is ../../..).

Conventions / gotchas (Chainsaw v0.2.15)

  • Condition asserts use the filter-as-list form, not the documented [0] index:
    status:
      (conditions[?type == 'Ready']):
      - status: "True"
    
    The (conditions[?type == 'Ready'])[0]: {status: "True"} idiom fails with "field not found in the input object".
  • A reading of a failure diff: the "actual" side is a projection onto the expected fields — status: {} means the JMESPath expression failed, not that status is empty. Read the error line above the diff first.
  • error with a bare resource ref (name only) asserts the resource does not exist (used for "PVC reclaimed", "no backup schedule", "child has no Gateway HR").
  • script runs in the test directory with $NAMESPACE set to the config namespace (tenant-test).

Running locally

Requires a cluster with Cozystack installed and a tenant-test namespace (the environment hack/e2e-install-cozystack.bats produces). bucket additionally needs mc and nc on the host; kuberture/openbao/harbor need jq.

# install chainsaw
go install github.com/kyverno/chainsaw@latest   # or: brew install kyverno/chainsaw/chainsaw

# run all suites (config picked up from .chainsaw.yaml)
cd hack/e2e-chainsaw
chainsaw test

# run a single suite
chainsaw test postgres/

kubernetes-latest/kubernetes-previous each bring up a full tenant Kubernetes cluster (~25 min, real worker nodes) and vminstance boots a nested VM — expect a long run for the full set.

CI integration

The e2e job in .github/workflows/pull-requests.yaml runs make -C packages/core/testing test-chainsaw (which execs chainsaw test hack/e2e-chainsaw/ inside the e2e sandbox) and uploads chainsaw-report.xml. The sandbox image (packages/core/testing/images/e2e-sandbox/Dockerfile) ships the chainsaw binary alongside kubectl/helm/mc/jq/nc.