Commit Graph
26 Commits
Author SHA1 Message Date
Aleksei Sviridkin a7ac37be65 test(e2e): let HR-Ready waits outlast the install timeout of the release they wait on (#4471)
## What this PR does

Stacked on #4468, which holds the raised backup waits to a start margin;
review that one first. This PR's own diff is the six commits on top of
it.

Most HR-Ready waits in the E2E suites gave up before the release they
were waiting on could fail its install. cozystack-api gives every
release it generates a 600s install timeout, and the Chainsaw asserts in
front of those releases sat at 5 or 6 minutes. When an install genuinely
failed, the suite reported a bare assert timeout against a release that
was still installing, and the reason the install failed, which the
release records when its own timeout fires, was never reached. Red runs
failed just as often as they should; they only said less.

I raised every HR-Ready wait above the install timeout of its own
release, by the 60s start margin #4468 introduces:

- Chainsaw asserts on releases cozystack-api generates go to 11m, across
bucket, clickhouse, etcd, external-dns, foundationdb, harbor, mongodb,
openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and
vminstance.
- Asserts on nested `<name>-system` releases take their floor from the
template that renders them: 11m for external-dns, openbao and qdrant
(10m `spec.timeout`), 16m for `harbor-test-system` (15m). `harbor-test`
sat exactly on its 600s floor, which expires first because the assert's
clock starts before the install's; it goes to 11m too.
- The kuberture Package release takes its 10m from cozystack-operator
and goes to 11m.
- In the backup walkthroughs, the bucket waits go from 300s to 660s in
clickhouse, kafka, mariadb, mongodb, postgres, rabbitmq and redis, and
the two postgres waits go from 360s to 660s. The kafka bucket wait
passed no budget and silently took the helper's 300s default, which is
why earlier scans for literal budgets missed it. Each Chainsaw op that
runs a walkthrough grows by exactly what its waits gained (clickhouse
78m, kafka 46m, mariadb 48m, mongodb 42m, postgres 66m, rabbitmq 48m,
redis 36m), so a failure at any later wait keeps the headroom it had and
is still reported by the script instead of being cut off by the op.

Two waits stay as they are, on purpose:

- kafka: `kafka-rd` sets a 7h install timeout, which no budget inside
the 215-minute E2E job can clear. The kafka assert and the Kafka waits
in the kafka and kafka-metadata walkthroughs keep their numbers, and the
assert now says why in-line.
- `vm-instance-test`: `vm-instance-rd` disables the install wait, so its
release goes Ready as soon as helm has applied and there is no floor to
clear. Earlier lists of this problem included it by mistake.

The cost falls on red runs only: a suite whose release never goes Ready
now spends up to six more minutes before it reports, and in exchange the
report carries the release's own reason. Green runs are unchanged.
Suites run one after another in a single job capped at 215 minutes, so a
systemic failure that reddens many suites now reaches that cap sooner,
and a job cancelled at the cap loses its log and `cozyreport.tgz`. I
took that trade because a failed install that says why is worth more
than a red run that ends a few minutes earlier.

`docs/agents/e2e-testing.md` stopped being true in several places once
the waits were raised: it described most of the tree as under the floor,
mariadb as the only suite raised, the walkthrough bucket waits as under
it, and harbor's asserts as sitting on and under theirs. I rewrote those
sentences, and the one that pointed at "every wait above the band" of
5-6m, kept the start-margin wording from #4468, and added that a
walkthrough wait passing no budget takes the helper's default. The
comments in `hack/mariadb-first-boot-waits.bats` that quoted the old op
and bucket figures are restated; its existing checks and constants are
unchanged, because every op grew by exactly its bucket delta and each
remainder still sits at exactly its constant. I also added postgres to
the table that file holds the walkthroughs to, so its raised waits and
its op cannot drift back apart unnoticed, and a test that holds each
table suite's bucket wait to the bucket release's floor, read from the
api-server default with `bucket-rd` checked for an override; the op
check alone could not catch a bucket wait dropping back, since a lower
wait only widens the slack. redis stays out: its walkthrough waits on no
redis HelmRelease, only on the bucket and on the sentinel master, so the
table's floor check has nothing to read there.

How I checked it, since E2E cannot run locally:

- I re-ran the enumeration from the sizing section of
`docs/agents/e2e-testing.md` over every
`hack/e2e-chainsaw/*/chainsaw-test.yaml` and every `wait_hr_ready` call
under `examples/backups/`, literal or default, and held each against its
own release's floor. Every `Ready=True` assert and every `wait_hr_ready`
call a suite drives now clears its floor, except the kafka ones named
above. The opt-in etcd round-trip uses bare `kubectl wait hr` calls at
300s instead of that helper; they are listed in #4470. The `Ready=False`
assert on the rejected redis upgrade is not a readiness wait and is left
out.
- `make bats-unit-mariadb-first-boot-waits` passes, 31 of 31. For the
new postgres row, putting the op back to 50m fails only the postgres
slack test, and a postgres wait at 659s fails only the postgres floor
test. Setting any one table suite's bucket wait to 659s fails only the
bucket test, naming only that suite, and an install-timeout override on
`bucket-rd` fails it too. I ran the mutations under plain `bats` so a
fail-fast runner could not hide a second failure.
- `hack/select-e2e.sh` over this diff selects bucket, clickhouse, etcd,
external-dns, foundationdb, harbor, kafka, kuberture, mariadb, mongodb,
openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and
vminstance.

Out of scope, and tracked in #4470: the tenant addon waits in
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` (an edit there escalates the
whole E2E run), the redis and kafka-metadata `wait_hr_ready` helpers
that print no reason on timeout, and the opt-in etcd round-trip's bare
`kubectl wait` calls.

Known and left for later, none of them changed here:

- Only the mariadb Chainsaw asserts and the walkthroughs in the guard's
table are held by a test. The other raised asserts, and the redis and
kafka bucket waits, rest on literals with in-line reasons; #4470 covers
extending the guard.
- The Kafka waits in `examples/backups/kafka/04-create-kafka.sh`,
`07-restore-to-copy.sh` and `kafka-metadata/run-all.sh` stay at 300s
against the 7h floor without an in-line reason at the call site. The
doc's rule asks that of Chainsaw asserts only, and the kafka Chainsaw
assert carries it.

The postgres waits are the ones #3748 is about. Their budget now clears
the install-timeout floor and a test holds it there; what a CNPG first
boot costs on this platform is still unmeasured, so that issue stays
open.

Closes #3771

Closes #3753

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff. It touches E2E suites, backup walkthrough scripts, a unit-lane
bats file and an agent doc, none of which any downstream repository
restates.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tests): HelmRelease readiness waits in the E2E suites and backup walkthroughs now outlast the install timeout of the release they wait on, so a failed install reports its own reason instead of a bare timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Extended readiness and backup round-trip test timeouts across multiple
services to accommodate slower startup and provisioning.
* Added checks to verify that bucket readiness waits meet configured
installation-timeout requirements.

* **Documentation**
* Updated testing guidance to clarify how readiness waits relate to
release-specific timeouts and overall test budgets.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 12:15:58 +03:00
Aleksei Sviridkin 04d469fc85 fix(tests): let the backup walkthrough waits outlive their installs
The backup round-trips drive examples/backups, whose bucket waits gave
up at 300s and whose postgres waits at 360s, under the 600s install
timeout cozystack-api gives each of those releases. The kafka example's
bucket wait passed no budget and took the helper's 300s default. A
failed install there reported only that the wait ran out.

Raise each to 660s, and grow the Chainsaw op around every walkthrough
by what its waits gained, so a failure at any later wait keeps the
headroom it had and is still reported by the script rather than cut
off by the op. The kafka and kafka-metadata Kafka waits stay: their
7h install timeout cannot be cleared inside the E2E job cap.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:22 +03:00
Aleksei Sviridkin 49cefe0dff fix(tests): size HR-Ready asserts on nested and Package releases
A nested <name>-system release takes its install timeout from the
template that renders it, not from cozystack-api, and a Package release
takes it from cozystack-operator. The external-dns, openbao and qdrant
templates set 10m and harbor's sets 15m; the operator gives 10m. Their
asserts sat at 5m or 10m, so a failed install reported as a bare
timeout. Raise each above its own floor: 11m, and 16m for
harbor-test-system. harbor-test was on its 600s floor exactly, which
expires first because its clock starts before the install's; it goes
to 11m with the generated parents here.

kafka keeps its 6m: kafka-rd sets a 7h install timeout that no budget
inside the E2E job cap can clear, and the assert now says so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:21 +03:00
Aleksei Sviridkin 3303cc59c7 fix(e2e): delete the kafka suite's broker and controller PVCs
The kafka chart sets deleteClaim: false on both node pools, so deleting
an app leaves its volumes behind on purpose. The suite deleted its apps
and never the volumes. The 35 GiB a run leaves in tenant-test, 20 GiB
of it on one node, kept that LINSTOR node short of space for the
kubernetes suites that run after it. The backup round-trip leaves its
two clusters' volumes in tenant-root the same way.

Each test now deletes the PVCs by strimzi.io/cluster once its apps are
gone, and waits for them to go. The kafka test does it from a step
cleanup, so a failed run reclaims them too.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 15:27:18 +03:00
Aleksei Sviridkin 7002397e40 test(kafka): assert that uninstall leaves no node pool or broker behind
The Kafka suites passed while the node pools of the apps they had
deleted, and the brokers those pools own, kept running into the suites
that came after. Each suite now checks, after its apps are gone, that
no KafkaNodePool and no pod labelled strimzi.io/cluster=<cluster>
remains in the namespace.

The kafka suite deletes its app explicitly so it can look afterwards.
The metadata roundtrip's cleanup does not wait for the uninstalls, so
its checks carry a budget sized from the pre-delete Job's deadline.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 07:35:32 +03:00
Andrey Kolkov 7ce85a2f7f test(kafka): pin the strategy script's guards with a hermetic bats suite
The in-Pod script had grown eight fail-closed guards across as many
rounds, and the only thing exercising any of them was a 40-minute
cluster test that drives the happy path plus one refusal. The partition
listing guard in particular - the whole of the previous round's second
fix - could be deleted with nothing in the tree going red. Add
hack/kafka-topic-backup-strategy.bats on the shape of
hack/kafka-backup-script.bats: render the strategy through a stub
kubectl exactly as step 01 applies it, lift the container body out of
it, and execute it against stubbed kafka-*.sh, a stubbed curl and a
directory standing in for S3, with fault injection for the broker
behaviours the guards exist for. Deleting that listing guard now fails
one case; the script gains BIN/SCRATCH/CA_FILE overrides so it can run
outside the Pod, each defaulting to the Pod's value.

The same rounds' smaller findings: the manifest's topic name is held to
Kafka's charset like the topics parameter, since it reaches unquoted
expansions with globbing on; the shape check after --create compares
the replication factor as well as the partition count, and refuses a
replication factor below the topic's effective min.insync.replicas
before replaying, because kafka-console-producer only logs a rejected
acks=all write and the run would otherwise end at the count check with
the cause unnamed; the manifest's replication factor must agree across
a topic's lines; topic_records carries its own || return 1 on each
command, as a function called as the operand of || runs with errexit
off; the CA lookup in step 03 distinguishes a denied read from an
absent Secret, and the README's persona column says the copy needs the
admin; the e2e comment no longer claims the 2-label endpoint fails the
serving cert's SANs, which include '*.tenant-root'.

Rebased onto main, which moved the chart to KRaft: the demo's Kafka CRs
drop the zookeeper block the chart no longer has.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
2026-09-23 18:00:15 +04:00
Andrey Kolkov 5bd029c46a fix(kafka): restore with the captured replication factor and a complete listing
Record each topic's ReplicationFactor in the backup manifest, off the
same --describe line the partition count already comes from, and
recreate the topic with it on restore. The value used to be a
BackupClass parameter defaulting to 1, and the Cozystack Kafka chart
sets the broker-level min.insync.replicas to 2 as soon as kafka.replicas
reaches 2: kafka-console-producer writes with acks=all, the broker
rejects every write into a topic with fewer replicas than that, and the
producer logs the rejection and exits 0. A restore onto any multi-broker
cluster therefore landed zero records and failed the post-replay count
with a message naming the wrong cause; in place, the documented delete
had already taken the live topic with it. replicationFactor stays as an
override for restoring into a smaller cluster, and a value the target
cannot host now fails at --create in Kafka's own words.

Hold the pre-replay emptiness check to the topic's partition count, as
the backup side already does. kafka-get-offsets skips a partition it
could not read and still exits 0, so a leader election in flight dropped
the populated partitions from the listing, the surviving empty ones
summed to zero, and the replay appended a full copy into a live topic -
the exact outcome the check was moved ahead of the replay to prevent.
Both restore-side counts now go through one helper that refuses a short
listing. The data-file-versus-manifest check runs both ways before
anything is created, and the replay walks the manifest topic by topic,
announcing each one, so a retried Pod's log says which topics landed.

The demo now runs two brokers with two-replica topics, which is the
shape the chart puts min.insync.replicas=2 on; the restore steps assert
the restored topic's replication factor, and step 06 submits a second
RestoreJob against the populated topic and requires it to be refused
with the topic untouched.

Smaller findings from the same round: the topics parameter is held to
Kafka's name charset (a literal "*" no longer path-expands) and refused
on a repeated name, which produced a Succeeded backup no restore could
balance; the strategy verifies S3 TLS against the seaweedfs CA projected
into the <app>-backup-s3 Secret instead of curl -k, following the
rabbitmq example, so the e2e endpoint moves to the .svc name the serving
cert covers; the image lookup keeps digest-pinned references; the
default namespace is tenant-root, the only one the demo ever ran in;
step 05 caches the resolved Backup name for the restore steps; the decoy
count subtracts begin offsets like the topic count; the bats test name
carries its count again; and the scenario walk-throughs the sibling
examples ship are added.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov d405a36e3c fix(kafka): validate the backup's partition set against the topic
The drain guard only covers a short read within a partition it was told
about. kafka-get-offsets lists just the partitions whose lookup
succeeded - GetOffsetShell logs a per-partition KafkaException to stderr,
skips it and still exits 0 - so a partition lost to a leader election got
no manifest line, no data file and no drain check, and the tarball was
uploaded with the Backup reporting Succeeded. The restore-side count
check could not catch it either, since it sums the same short manifest,
and a missing highest-numbered partition made restore recreate the topic
with fewer partitions, re-placing every keyed record.

Read the topic's own PartitionCount before the drain and fail closed when
the offset listing is short. A short "begins" needs no separate check:
its per-partition default of 0 can only overstate n, which the existing
drain check then catches.

Also run the shared BucketAccess preflight in the e2e - accessGranted
does not prove the S3 server reloaded the new identity, and a fresh grant
answering the SigV4 PUT with 403 reds the run as a backup defect - and
state the .chainsaw.yaml namespace override as the rule it is rather than
as a claim about which Tests match a name pattern.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 9673af6aca fix(kafka): guard short backup drains and settle example teardown
- Short-drain guard: kafka-console-consumer exits 0 after a timed-out
  short read, so a truncated drain was tarred and reported Succeeded.
  Compare the drained line count against end-begin per partition and
  fail the backup on any shortfall, making the README's "fails closed"
  promise real for compacted/transactional topics.
- Settling teardown: cleanup.sh returned before the Kafka apps and
  Bucket had uninstalled and could not report failure, letting the
  neighbouring chainsaw/kafka Test contend on PVC reclaim. Add a
  wait_deleted helper and a teardown_rc gate that waits for the heavy
  objects and exits non-zero when the teardown does not settle.
- Fully qualify the ambiguous delete kinds (backup/bucket/backupclass/
  restorejob/backupjob) so a bare shortcut cannot target a foreign group
  and silently leave the Cozystack artifact behind.
- Default KAFKA_IMAGE from the running operator (newest STRIMZI_KAFKA_
  IMAGES entry), falling back to the literal only when the operator
  cannot be read; drop the duplicate resolution note.
- Make the strategy and throwaway CLI Pods restricted-clean
  (runAsNonRoot + JSON-patch securityContext on the CLI Pods).
- gitignore .source-dump.txt; replace the deprecated "small" preset with
  c1.small; capture the to-copy cluster in the e2e catch; document the
  all-topics risk; de-enumerate the decaying suite-list comments.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov d38b538d98 test(kafka): fail fast on terminal backup phases and capture strategy Job logs
Two CI-quality fixes, both diagnostics/wall-clock only (neither can turn a
run falsely green):

- The chainsaw comment claimed run-all.sh fails fast on terminal
  BackupJob/RestoreJob phases and Stalled HelmReleases, but the kafka helpers
  never implemented it (carried over from mariadb). A BackupJob that settled
  phase=Failed was polled the full 600s and then mislabelled 'Timeout',
  burning ~10 of the 40 chainsaw minutes per already-red run. Port the mariadb
  variants: wait_for_field gains a terminal fail_value arg (call sites pass
  Failed), and a new wait_hr_ready fails fast on Stalled=True with an existence
  backstop. HR waits in steps 03/04/07 now use it; the Strimzi Kafka CR waits
  keep their genuine Ready condition. run-all.sh runs under set -e, so a
  non-zero step aborts the flow immediately, matching the comment.

- The catch block only captured the Strimzi broker Pods
  (app.kubernetes.io/instance=kafka-kafka-test); the generic Job strategy runs
  the drain/replay in controller-created Pods labelled
  job.strategy.backups.cozystack.io/mode=backup|restore (never that instance
  label) and not TTL-reaped, so the actual S3/SigV4 error left no log in the
  report. Add a second podLogs selector keyed on that label.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 00943392cc fix(kafka): resolve a single Kafka image for backup e2e harness
STRIMZI_KAFKA_IMAGES is a newline-separated version=image map with one
entry per supported Kafka version. The old 'sed | tr -d [:space:]' collapsed
every line into one string, producing an invalid multi-image reference
(quay.io/...3.8.0quay.io/...3.8.1...) that failed 'kubectl run' with
'invalid reference format' and broke kafka-2-backup-roundtrip.

Split on whitespace, keep image lines, strip the version prefix and take the
last (newest) image so a single valid reference is exported.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Andrey Kolkov 7842b88676 feat(kafka): add topic-data backup/restore example and e2e roundtrip
Add examples/backups/kafka, a worked backup/restore of a Cozystack-managed
Kafka application's topic data via the generic Job backup strategy - the
same pattern as examples/backups/nats, with no purpose-built backup image:
a stock Strimzi Kafka image (kafka-*.sh CLI + bash/curl/tar) runs one
shell script that branches on .Mode.

Consistency is a frozen-end-offset cut: kafka-get-offsets --time -1 is read
once per partition at backup start and the consumer drains only up to that
offset, so anything produced during the run is excluded and every partition
is captured as of the same instant. Restore recreates each topic with its
original partition count and replays every partition file; keyed records
reproduce into their original partition via the default partitioner.

This is deliberately a DATA-only backup: topic configs, ACLs, SCRAM users
and consumer-group offsets are not captured, and re-produced records get
fresh offsets. The README spells out the scope and points at a
volume-snapshot strategy for faithful, offset-preserving backups.

Add kafka-2-backup-roundtrip to hack/e2e-chainsaw/kafka, driving the
example run-all.sh as the harness (Bucket -> source Kafka + seeded topic
-> BackupJob -> in-place restore -> to-copy restore + verify) so the test
and the documented flow cannot drift, mirroring mariadb-2-backup-roundtrip.
The image is resolved from the running operator so the backup Pod reuses
the cached Kafka image.

Verified end to end on a live cluster: freeze cut orders:0 [0,8),
orders:1 [0,11), orders:2 [0,11) = 30 records; in-place and to-copy
restores both return 30 with the original 8/11/11 partition distribution.

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
2026-09-23 17:44:17 +04:00
Alexey Artamonov 0b9332877e test(kafka): drop the false-green ZK->KRaft migration chainsaw suite
The kafka-zk-to-kraft-migration suite did not exercise the pre-upgrade hook,
it reimplemented a simplified flow beside it: a controller pool named
"controller" instead of the hashed c-<hash>, no strimzi.io/next-node-ids pin,
and a single broker so the controller-vs-broker node-id collision could never
occur. That is exactly the defect the hook exists to prevent, so the suite
stayed green on a revision whose in-place migration lost every message —
reproduced by hand 2/2 on a live cluster. A suite that cannot fail on this
code is worse than no suite: it reads as migration coverage and invites trust
it has not earned, while costing ~380s on every Kafka PR and asserting only
that a PVC object exists, never reading a record back. The migration hook's
bats suite and the live upgrade runs are the real coverage; the Strimzi bump
would delete this suite regardless.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-22 16:05:44 +03:00
Alexey Artamonov 8ef230f0bf Merge remote-tracking branch 'origin/main' into kafka-multi-instance
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>

# Conflicts:
#	docs/storage-immutability.md
#	packages/apps/kafka/tests/topics_test.yaml
#	packages/system/kafka-rd/cozyrds/kafka.yaml
2026-09-21 12:05:33 +03:00
Yan Bondarenko 8ec76a3614 fix(kafka): make topics[].config optional
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-18 17:30:06 +04:00
Alexey Artamonov 353f37e4fe test(kafka): widen the fresh-create e2e cleanup budget for KRaft
The fresh-create Kafka suite relied on the shared 5m cleanup default, but a
KRaft cluster tears down broker and controller node pools, the entity
operator, and their PVCs — more work than a single ZooKeeper node. Under a
contended CI run (concurrent Postgres/OpenSearch/Redis suites) that teardown
overran 5m and failed the suite after every assertion had already passed, so
cleanup latency, not the test, was deciding the verdict. Give this suite its
own 8m cleanup timeout; the migration suite runs in its own namespace and is
unaffected.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 053ef16976 feat(kafka): allow multiple KRaft Kafka clusters per namespace
Fresh KRaft clusters no longer need to be alone in a namespace. Their
node pools get short, release-hashed names — "b-<hash>" for brokers,
"c-<hash>" for controllers — instead of the fixed "kafka"/"controller".
Each release hashes differently, so the names are unique within the
namespace (KafkaNodePool object names are namespace-unique) and any
number of Kafka instances coexist. The hash keeps the pool name a fixed
length: Strimzi derives every pod/PVC name as <cluster>-<pool>-<id> and
the cluster name is already the release name, so embedding the release
name in the pool name too would double it into every pod hostname and
blow the 63-character DNS label limit for long release names. A
render-time guard fails with a clear message when even the hashed
controller node name would still exceed 63 characters.

The fixed "kafka" name is still required to adopt an existing ZooKeeper
cluster's <cluster>-kafka-N brokers and data in place during migration,
so kafka.brokerPoolName resolves to "kafka" for a migrating/migrated
cluster and "b-<hash>" for a fresh one. A second ZooKeeper->KRaft
migration into a namespace that already holds a "kafka" pool owned by
another cluster fails closed, both at render time and in the pre-upgrade
hook (whose owner probe uses --ignore-not-found + an explicit exit-code
check, so a transient kubectl error aborts rather than silently waiving
the collision guard). This replaces the previous blanket "one Kafka per
namespace" guard, which over-restricted the fresh case.

The pool names are resolved from a live Helm lookup; that is safe because
Helm propagates any non-NotFound read error and fails the whole render
(pkg/engine/lookup_func.go), so a transient apiserver/RBAC miss retries
rather than silently flipping a live cluster's pool name — an empty
lookup means genuine absence or a never-applied client-side render.

The migration hook and workload monitors follow the same naming. Tests:
helm-unittest fixtures (via kubernetesProvider) cover every resolution
branch and the fail-closed guards; hack/kafka-migration-hook.bats drives
the migration Job's real shell script against a stub kubectl across the
fresh / read-error / already-KRaft / owner-collision / happy-path /
stuck / retry cases. Verified on a live cluster: in-place ZooKeeper->KRaft
migration preserved all data and adopted the original broker PVCs, and
two fresh clusters coexist with distinct pools, pods and PVCs.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 1a68157564 fix(kafka): harden migration hook and isolate the migration e2e
Addresses the second review round on the ZK->KRaft migration.

Migration hook:
- Fully qualify the Strimzi resources as kafkas.kafka.strimzi.io /
  kafkanodepools.kafka.strimzi.io. The bare plural "kafkas" also exists in this
  chart's apps.cozystack.io group, so a bare name resolves by per-cluster
  discovery priority and can hit the tenant kind — which the hook Role does not
  grant, failing the pre-upgrade hook deterministically.
- Classify NotFound structurally via `kubectl get --ignore-not-found` instead of
  substring-matching stderr, so an unrelated "... not found" error can no longer
  be mistaken for a fresh install and permanently skip a live ZooKeeper cluster.
  This also drops the jq dependency.
- Treat a KRaft-native / already-migrated CR (annotated strimzi.io/kraft=enabled)
  as done, never driving it backwards into kraft=migration.
- Keep only `before-hook-creation` on the Job's hook-delete-policy so a failed
  attempt's pod logs survive for a post-mortem until the next run.
- Route the kubectl image through cozy-lib.image (cluster registry mirror) and
  pin it by digest (clastix/kubectl:v1.32@sha256:...), matching the other
  cozystack hooks, instead of an unpinned docker.io tag.

Controller quorum:
- Fix the controller node-pool replica count at creation instead of deriving it
  from kafka.replicas on every reconcile. Strimzi 0.45 cannot scale a controller
  pool, so a day-2 edit of kafka.replicas (e.g. 1 -> 3) must not flip the quorum;
  the helper now keeps the existing "controller" pool's replica count when it is
  present.

Docs:
- Document that migration to KRaft is one-way: neither a version-field downgrade
  nor a pre-KRaft chart rollback is supported once migrated (Strimzi has no
  KRaft->ZooKeeper rollback after completion).

E2E:
- Run the ZK->KRaft chainsaw migration in its own namespace instead of the
  shared tenant-test. It raw-applies Strimzi resources plus a namespace-wide
  LimitRange and leaves deleteClaim:false PVCs, which polluted the sibling kafka
  suite. Raise the seed readiness timeout for CI headroom. Validated against a
  live cluster: migration adopts the broker PVC in place and reaches KRaft under
  a 128Mi LimitRange, in isolation.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov f9676961f3 fix(kafka): set resources on the migration node pools
The ZK->KRaft pre-upgrade hook created the adopting broker ("kafka") and
"controller" KafkaNodePools without a resources block. Those hook-created pools
govern the Kafka pods for the entire migration window — Helm applies the
resourced steady-state kafkanodepools.yaml only after the pre-upgrade hook
finishes. In a tenant namespace the LimitRange defaults every container to
128Mi, so the migration's controller (and rolled broker) JVMs OOM, never reach
KRaftPostMigration, and the hook times out — wedging the migration and thrashing
it on every reconcile.

Give both hook pools the same resources the steady-state pools get
(cozy-lib.resources.defaultingSanitize from kafka.resourcesPreset/resources).
This also avoids a redundant second broker roll (resource-less -> resourced)
after the hook completes.

Tests: a helm-unittest assertion that both migration pools render a resources
block; the Chainsaw ZK->KRaft suite now seeds under a 128Mi LimitRange (and
gives the seed cluster explicit resources), reproducing the tenant-quota
condition, and asserts the migration still reaches KRaft with the broker PVC
adopted. Validated green against a live cluster under the LimitRange.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 649f7e0b63 fix(kafka): enforce one Kafka per namespace; simplify node pool names
Follow-up to the KRaft-migration review. The broker KafkaNodePool must be named
exactly "kafka" so Strimzi adopts an existing ZooKeeper cluster's
<cluster>-kafka-N brokers and data in place during migration, but KafkaNodePool
object names are unique within a namespace — so two Kafka clusters cannot both
own a "kafka" pool there. This is an upstream constraint and Strimzi's own
guidance is one Kafka cluster per namespace
(strimzi/strimzi-kafka-operator discussions/11120).

Adopt that model instead of the earlier fresh-vs-migrated conditional naming:

- New kafka.assertSingleInstance helper fails the render (with a clear message
  and the upstream reference) if the namespace already holds another Kafka
  cluster. Reconciles/upgrades of the same release are allowed; lookup returns
  nothing during `helm template`, so unit tests never trip it.
- Node pools are now named "kafka" (broker) and "controller" unconditionally,
  which also adopts existing brokers on migration and yields clean
  <cluster>-kafka-N / <cluster>-controller-N names. The fresh-vs-migrated
  brokerPoolName helper is removed.
- WorkloadMonitor pool-name selectors and the migration hook updated to match;
  unit tests updated for the fixed pool names.
- README documents the one-Kafka-per-namespace rule and links the Strimzi
  discussion.

The ZK->KRaft Chainsaw migration suite (seed ZooKeeper cluster, migrate, assert
the original broker PVC survives and no wrongly-named PVC is created, cluster
reaches KRaft) was run green against a live cluster.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 710cdd18bb fix(kafka): address KRaft-migration review — broker pool adoption, hook safety, tests
Addresses the branch review on the ZK->KRaft migration path.

CRITICAL — broker node pool adoption. Strimzi derives broker/PVC names as
<cluster>-<pool>-<id>; a pre-node-pool ZooKeeper cluster has brokers
<cluster>-kafka-N, so the adopting broker pool must be named exactly "kafka"
or the operator provisions fresh empty brokers and orphans the tenant's data.
A new kafka.brokerPoolName helper resolves the broker pool to "kafka" for an
existing (migrating) cluster and keeps the release-scoped "<release>-broker"
for fresh installs, so multiple Kafka instances still coexist in one namespace.
The migration hook creates the pool as "kafka" and fails closed if a "kafka"
pool already belongs to another cluster (node pool names are namespace-unique,
so in-place migration of co-namespaced ZK clusters is unsupported). Wired
through kafkanodepools.yaml, workloadmonitor.yaml and the hook.

MAJOR — migration robustness.
- Raise the HelmRelease timeout: the migration Job can run tens of minutes but
  the release timeout defaults to 10m, so helm-controller aborted and thrashed
  the pre-upgrade hook. Add release.cozystack.io/helm-install-timeout=45m on the
  kafka ApplicationDefinition, above the Job's activeDeadlineSeconds (2400s).
- Idempotent migration: do not re-issue kraft=migration once the cluster is past
  the ZooKeeper phase (an invalid backward transition Strimzi rejects on retry).
- Fail closed on the decision gates: distinguish a genuine NotFound (skip) from
  any other kubectl error (abort), and never treat an empty kafkaMetadataState on
  an existing CR as "migrated".

MINOR — version downgrade documented as unsupported; integer (not lexical)
compare in the version gate; deleteClaim/PVC-retention and the one-per-namespace
migration limitation documented in the README; docs/storage-immutability.md
updated for KRaft.

Tests — helm-unittest coverage for the broker pool name, the hook guards and the
idempotent/fail-closed logic; a bats check for the helm-install-timeout budget
ordering; and a Chainsaw ZK->KRaft migration suite that seeds a ZooKeeper
cluster, drives the migration, and asserts the original broker PVC survives (not
recreated) and the cluster reaches KRaft. The migration sequence and the
Chainsaw assertions were validated against a live cluster.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Matthieu 42db2ca141 feat(kafka): migrate from ZooKeeper to KRaft mode with automatic migration
Replace ZooKeeper-based Kafka with KRaft (Kafka Raft) metadata mode.
Strimzi 0.45 is the last line that still supports ZooKeeper, so this
prepares the transition before ZooKeeper support is removed upstream.

Chart changes:
- The Kafka CR always renders strimzi.io/kraft=enabled with node pools
  and no spec.zookeeper block. Storage, replicas and resources move to
  separate broker + controller KafkaNodePools (a homogeneous separate-pool
  layout works for both fresh installs and migrations; Strimzi 0.45 does
  not support migration into a combined pool).
- Kafka version is now user-configurable via the standard files/versions.yaml
  pattern (v3.9 default, v3.8 available); appVersion bumped to 3.9.0.
- WorkloadMonitors track the broker and controller pools (no zookeeper
  monitor); the dashboard RBAC grants the controller monitor and drops the
  zookeeper one.

Migration of existing ZooKeeper clusters is handled by a Helm pre-upgrade
Job (templates/migration-hook.yaml) gated by a <release>-kafka-deployed-version
ConfigMap, following the established seaweedfs/etcd hook pattern. The Job
creates the broker + controller pools with Helm ownership labels, drives the
Strimzi ZK->KRaft state machine (node-pools=enabled, kraft=migration ->
kraft=enabled once KRaftPostMigration is reached) and is skipped once the
ConfigMap records version >= "1".

Testing:
- Helm unit tests cover the Kafka CR, node pools, migration hook, metrics,
  topics, version map, dashboard RBAC and workload monitors.
- The e2e suite is ported to the current Chainsaw framework
  (hack/e2e-chainsaw/kafka): it asserts the KRaft topology (both node pools,
  kafkaMetadataState=KRaft), the absence of ZooKeeper, and bootstrap
  endpoints.

Rebased onto current main: TLS listener support, entity-operator resource
guards, x-cozystack-options schema annotations and the WorkloadMonitor
replica-path fix are all preserved and reconciled with KRaft.

Signed-off-by: Matthieu <matthieu@hidora.com>
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Yan Bondarenko 9d2fddd4e6 test(kafka): assert release topics exist before the deletion check
The post-delete absence assertions could pass vacuously if the two
release topics were never created. Asserting both topics first makes
the E2E prove that the pre-delete cleanup actually removed them.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-15 13:47:04 +03:00
Yan Bondarenko 44ba13b527 fix(kafka): delete release topics before the topic operator is removed
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-15 13:47:04 +03:00
Myasnikov DaniilandClaude Fable 5 21e21a9234 test(kafka): raise fixture presets off the 128Mi OOM threshold
Main raised kafka/zookeeper resourcesPreset nano -> c1.small
(de9c81ce6, #3052) after the Chainsaw port was cut; the fixture still
pinned both to nano.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-08 18:14:04 +05:00
Myasnikov DaniilandClaude Opus 4.8 e22d84f2c8 test(e2e): migrate app suite from BATS to Chainsaw and wire into CI
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.

- Declarative ports assert on status/conditions and concrete fields; the
  repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
  collapses into a single polling assert with structured failure diffs and
  automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
  init/unseal, kuberture split-horizon probes, vminstance, the gateway
  admission/impersonation cases, and kubernetes-latest/previous, which wrap
  the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
  runtime, so they are host-independent.

CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.

Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-25 13:42:34 +05:00