Commit Graph
7777 Commits
Author SHA1 Message Date
Timur Tukaev 94885c2e93 docs(governance): remove the unused Directors role (#4296)
## What this PR does

Removes the **Directors** role from `GOVERNANCE.md`. The role
(non-technical leaders, elected by the maintainers, one vote each in
decisions) was defined but never used — no Directors were identified or
elected. It was intended for non-technical leadership, but those people
ended up maintaining parts of the project directly, so the role stayed
empty. Removing it so the documented decision-making body matches actual
leadership (maintainers, by consensus with a simple-majority fallback).
It can be reintroduced if a real need arises.

Addresses cozystack/community#79.

### Downstream repositories
- [x] No downstream repository is affected by this change

### Release note
```release-note
docs(governance): remove the unused Directors role from GOVERNANCE.md
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated governance guidance to remove the Directors role and related
leadership descriptions.
  * Clarified that voting is conducted by maintainers.
* Updated consensus language to reflect the revised decision-making
process.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-17 13:07:45 +03:00
coderabbitai[bot] a295e29b8a fix(kafka): clean up pre-delete hook ServiceAccount and RBAC on success 2026-09-17 09:58:12 +00:00
Alexey Artamonov 353f37e4fe test(kafka): widen the fresh-create e2e cleanup budget for KRaft
The fresh-create Kafka suite relied on the shared 5m cleanup default, but a
KRaft cluster tears down broker and controller node pools, the entity
operator, and their PVCs — more work than a single ZooKeeper node. Under a
contended CI run (concurrent Postgres/OpenSearch/Redis suites) that teardown
overran 5m and failed the suite after every assertion had already passed, so
cleanup latency, not the test, was deciding the verdict. Give this suite its
own 8m cleanup timeout; the migration suite runs in its own namespace and is
unaffected.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Andrey Kolkov 6e18ceef53 fix(kafka): fail closed when the migration hook cannot list Kafka CRs
The "free the kafka pool" gate counted other ZooKeeper clusters through a
`kubectl ... | awk` pipeline under `set -eu` without `pipefail`, its
kubectl error dropped by `2>/dev/null`. A failed read then counted zero and
kept the shared pool, stranding any co-namespaced cluster still waiting for
the name. Separate the read status from the count, mirroring the drain read
below, and cover it with a bats case.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-17 11:09:10 +03:00
Alexey Artamonov cd9f4d6a27 feat(kafka): auto-migrate every ZooKeeper Kafka in a namespace to KRaft
Extends the ZooKeeper->KRaft migration hook so that N Kafka clusters
sharing one namespace all reach KRaft automatically, with no manual
steps. This closes the last gap in the migration: previously only one
cluster per namespace could migrate in place (the broker pool must be
named exactly "kafka" to adopt the existing <cluster>-kafka-N brokers,
and that name is namespace-unique), so a second co-namespaced ZooKeeper
cluster hit the render guard and its HelmRelease never rendered again --
stuck on ZooKeeper with no way forward once a future Strimzi drops the
migration.

The clusters now serialize on the "kafka" name automatically. Whichever
upgrade wins the name (claimed atomically via kubectl create) migrates
in place; a cluster that loses fails its hook fast and its next reconcile
retries. Once a migrated cluster sees other ZooKeeper clusters still
waiting, it frees the name: it stands up its own release-hashed
"b-<hash>" broker pool (node IDs pinned clear of the existing broker and
controller ranges), enables Cruise Control, drains every partition off
the "kafka"-pool brokers with a remove-brokers KafkaRebalance, deletes
the drained pool, and removes Cruise Control again. The next waiting
cluster's retry adopts the freed name; the last one to migrate keeps
"kafka". Everything happens inside the one namespace -- the hook's Role
is namespaced and it never reads or touches another.

Robustness details learned from live runs:
- The drain rebalance sets skipHardGoalCheck: true. A mandatory broker
  evacuation must not be blocked by Cruise Control's capacity hard goals;
  under a small resources preset the capacity model can briefly
  over-estimate CPU from noisy samples and reject the proposal with
  CpuCapacityGoal even when real CPU is near idle.
- The drain wait refreshes the proposal (strimzi.io/rebalance=refresh) on
  any NotReady and keeps polling, rather than treating it as terminal: a
  just-enabled Cruise Control is unreachable for a minute or two, and its
  model settles as metric windows age out. This also makes the drain
  resumable -- a re-run refreshes a stale rebalance instead of reading its
  old failure forever.
- Cruise Control is enabled with explicit resources (it is a JVM and a
  tenant LimitRange would OOM it at 128Mi), transiently -- kafka.yaml
  never sets spec.cruiseControl, so Helm's server-side apply does not
  fight the hook patching it on and off.

Every phase is idempotent so a Job killed mid-drain (the drain scales
with data size and can run for hours) resumes on retry; the Job deadline
and the release's helm-install-timeout are raised accordingly.

Verified on a live cluster across five runs of two co-namespaced
ZooKeeper Kafkas migrating to KRaft: both reach KRaft automatically, the
first drains onto its b-<hash> pool and frees the name, the second adopts
"kafka" in place (broker node IDs and PVCs unchanged), and each cluster's
data is byte-for-byte identical before and after. helm-unittest 87/87 and
hack/kafka-migration-hook.bats 14/14 (the real hook script against a stub
kubectl) cover the freeing decision, the atomic name claim, the drain's
refresh/skip-hard-goal handling, and the "delete the pool only after the
rebalance is Ready" guard.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 5cbf16088f fix(api): cap Kafka app name to the KRaft controller pod-hostname budget
A Kafka cluster's KRaft controller pods are named
"<release>-c-<hash>-<id>" (8-hex hash, id at least one digit), and that
derived pod hostname must fit the 63-char DNS-1123 label limit. That is
tighter than the 53-char Helm release-name budget validateNameLength
enforced, so a release name of 51-53 characters was admitted but then
failed the chart render (kafka.assertNameLength) forever, an
un-recoverable upgrade dead-end for an existing cluster whose name is
immutable. Cap the Kafka app name at admission, mirroring the existing
Kubernetes worker-pool cap, so the overflow surfaces on the Kafka CR the
operator is editing instead of at render time on a pod Strimzi can never
create. The chart's render guard remains the second line of defense and
also catches larger clusters whose higher node ids need more digits.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov cb7578bb31 fix(kafka): data-safe, Helm-clean ZooKeeper->KRaft migration
Two live-reproduced blockers in the in-place ZK->KRaft migration are
fixed, plus the concurrency guard behind them.

1. Data loss (node IDs). The migration created the broker and controller
   KafkaNodePools together without pinning node IDs. On Strimzi 0.45 the
   controller pool deterministically took the existing broker IDs
   [0..N-1], so the brokers were recreated empty at [N..] and their data
   PVCs were orphaned (reproduced 2/2 on dev9: 20000 messages -> 0,
   partitions leaderless, migration wedged in KRaftMigration). Naming the
   broker pool "kafka" is necessary but NOT sufficient: Strimzi adopts by
   node id, not pool name. The controller pool now pins
   strimzi.io/next-node-ids above the broker range, so the broker pool
   keeps [0..N-1] and adopts the existing <cluster>-kafka-N brokers and
   their data in place.

2. Release stuck "failed" (Helm lifecycle). The pre-upgrade hook creates
   the pools, and the chart also declared them as regular manifests, so
   Helm's main apply tried to reconcile a resource that exists in the
   cluster but was absent from the previous (ZooKeeper) release revision
   and aborted with "original object KafkaNodePool ... not found",
   leaving the HelmRelease failed even though the cluster migrated fine
   (reproduced 3/3 under server-side apply, the production
   helm-controller path). This is unavoidable while the pools are Helm
   main-manifest resources across the ZK->KRaft boundary, so the pools
   are now managed entirely by the migration Job (a pre-install,
   pre-upgrade hook) on every install and upgrade, and removed from the
   regular manifests. The Job ensures both pools (creating or updating
   them idempotently), then drives the migration state machine.
   templates/kafkanodepools.yaml and the deployed-version gate ConfigMap
   (templates/version.yaml) are removed; the broker pool name is resolved
   per path by kafka.brokerPoolName ("b-<hash>" fresh, "kafka"
   migrating), and the render-time name-length guard moves to the Kafka
   CR template.

3. Concurrency (one migration per namespace). The collision guard read
   the "kafka" pool owner and then applied, a non-atomic read-then-write
   two concurrent per-namespace migrations could both pass. When the pool
   is absent the hook now uses "kubectl create" as the atomic arbiter:
   exactly one concurrent creator wins, the loser re-reads the owner and
   aborts.

Verified live on dev9 under server-side apply: a fresh install comes up
KRaft with pools at [0,1,2]/[3,4,5] and the release is "deployed"; an
in-place migration of a 20000-message ZooKeeper cluster ends "deployed",
adopts the broker PVCs in place (same UIDs), and the KRaft data is
byte-for-byte identical to the pre-migration ZooKeeper data (matching
sha256 over every partition:offset:key:value). helm-unittest 86/86 and
hack/kafka-migration-hook.bats 12/12 (the real hook script against a stub
kubectl) cover the resolution branches, the fail-closed reads, the atomic
collision guard and the finalize guard.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Andrey Kolkov 1a5cb049ae fix(kafka): match KRaft controller dashboard panel to hashed pool name
The controller commit-latency panel selected controller pods with
pod=~"$strimzi_cluster_name-controller-.*", but the controller node pool
is named c-<hash> (kafka.controllerPoolName), so Strimzi names controller
pods <cluster>-c-<hash>-<id>. The literal -controller- never matched, so
the panel was permanently empty. Select the pool by its -c- token instead.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 053ef16976 feat(kafka): allow multiple KRaft Kafka clusters per namespace
Fresh KRaft clusters no longer need to be alone in a namespace. Their
node pools get short, release-hashed names — "b-<hash>" for brokers,
"c-<hash>" for controllers — instead of the fixed "kafka"/"controller".
Each release hashes differently, so the names are unique within the
namespace (KafkaNodePool object names are namespace-unique) and any
number of Kafka instances coexist. The hash keeps the pool name a fixed
length: Strimzi derives every pod/PVC name as <cluster>-<pool>-<id> and
the cluster name is already the release name, so embedding the release
name in the pool name too would double it into every pod hostname and
blow the 63-character DNS label limit for long release names. A
render-time guard fails with a clear message when even the hashed
controller node name would still exceed 63 characters.

The fixed "kafka" name is still required to adopt an existing ZooKeeper
cluster's <cluster>-kafka-N brokers and data in place during migration,
so kafka.brokerPoolName resolves to "kafka" for a migrating/migrated
cluster and "b-<hash>" for a fresh one. A second ZooKeeper->KRaft
migration into a namespace that already holds a "kafka" pool owned by
another cluster fails closed, both at render time and in the pre-upgrade
hook (whose owner probe uses --ignore-not-found + an explicit exit-code
check, so a transient kubectl error aborts rather than silently waiving
the collision guard). This replaces the previous blanket "one Kafka per
namespace" guard, which over-restricted the fresh case.

The pool names are resolved from a live Helm lookup; that is safe because
Helm propagates any non-NotFound read error and fails the whole render
(pkg/engine/lookup_func.go), so a transient apiserver/RBAC miss retries
rather than silently flipping a live cluster's pool name — an empty
lookup means genuine absence or a never-applied client-side render.

The migration hook and workload monitors follow the same naming. Tests:
helm-unittest fixtures (via kubernetesProvider) cover every resolution
branch and the fail-closed guards; hack/kafka-migration-hook.bats drives
the migration Job's real shell script against a stub kubectl across the
fresh / read-error / already-KRaft / owner-collision / happy-path /
stuck / retry cases. Verified on a live cluster: in-place ZooKeeper->KRaft
migration preserved all data and adopted the original broker PVCs, and
two fresh clusters coexist with distinct pools, pods and PVCs.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Andrey KolkovandClaude Opus 4.8 03619f3ad2 test(kafka): drop source-grepping migration-timeout drift-guard
The removed bats file grepped two static literals in two separate
packages (helm-install-timeout in system/kafka-rd, activeDeadlineSeconds
in apps/kafka) and asserted an ordering between them. Substring matching
cannot verify the contract it targets: the real invariant is between the
running migration Job and the release timeout helm-controller actually
enforces on the pre-upgrade hook, not between two YAML literals in the
tree. It also greened only for the minute-format literal and errored on
any legitimate reformat (e.g. "1h" / "2700s").

The timeout > Job-deadline relationship is already documented inline at
both values. Its machine-checkable home is an e2e that upgrades an
existing release through the HelmRelease pre-upgrade hook and confirms a
real migration completes inside the release timeout (the chainsaw suite
drives migration via kubectl, bypassing that path).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-17 11:09:10 +03:00
Alexey Artamonov dae4d95444 fix(kafka): replace dead ZooKeeper panel with KRaft controller metrics
The strimzi-kafka Grafana dashboard still had a 'ZK request latency' panel
(kafka_server_zookeeperclientmetrics_zookeeperrequestlatencyms), which goes
permanently empty after the ZooKeeper->KRaft migration. Replace it with a
KRaft controller commit-latency panel (kafka_server_raftmetrics_commit_latency_avg,
scraped from the controller pods). Metric names verified against a live KRaft
cluster. The PVC-exclusion filter that references zookeeper stays valid and is
left as is.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 1a68157564 fix(kafka): harden migration hook and isolate the migration e2e
Addresses the second review round on the ZK->KRaft migration.

Migration hook:
- Fully qualify the Strimzi resources as kafkas.kafka.strimzi.io /
  kafkanodepools.kafka.strimzi.io. The bare plural "kafkas" also exists in this
  chart's apps.cozystack.io group, so a bare name resolves by per-cluster
  discovery priority and can hit the tenant kind — which the hook Role does not
  grant, failing the pre-upgrade hook deterministically.
- Classify NotFound structurally via `kubectl get --ignore-not-found` instead of
  substring-matching stderr, so an unrelated "... not found" error can no longer
  be mistaken for a fresh install and permanently skip a live ZooKeeper cluster.
  This also drops the jq dependency.
- Treat a KRaft-native / already-migrated CR (annotated strimzi.io/kraft=enabled)
  as done, never driving it backwards into kraft=migration.
- Keep only `before-hook-creation` on the Job's hook-delete-policy so a failed
  attempt's pod logs survive for a post-mortem until the next run.
- Route the kubectl image through cozy-lib.image (cluster registry mirror) and
  pin it by digest (clastix/kubectl:v1.32@sha256:...), matching the other
  cozystack hooks, instead of an unpinned docker.io tag.

Controller quorum:
- Fix the controller node-pool replica count at creation instead of deriving it
  from kafka.replicas on every reconcile. Strimzi 0.45 cannot scale a controller
  pool, so a day-2 edit of kafka.replicas (e.g. 1 -> 3) must not flip the quorum;
  the helper now keeps the existing "controller" pool's replica count when it is
  present.

Docs:
- Document that migration to KRaft is one-way: neither a version-field downgrade
  nor a pre-KRaft chart rollback is supported once migrated (Strimzi has no
  KRaft->ZooKeeper rollback after completion).

E2E:
- Run the ZK->KRaft chainsaw migration in its own namespace instead of the
  shared tenant-test. It raw-applies Strimzi resources plus a namespace-wide
  LimitRange and leaves deleteClaim:false PVCs, which polluted the sibling kafka
  suite. Raise the seed readiness timeout for CI headroom. Validated against a
  live cluster: migration adopts the broker PVC in place and reaches KRaft under
  a 128Mi LimitRange, in isolation.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov f9676961f3 fix(kafka): set resources on the migration node pools
The ZK->KRaft pre-upgrade hook created the adopting broker ("kafka") and
"controller" KafkaNodePools without a resources block. Those hook-created pools
govern the Kafka pods for the entire migration window — Helm applies the
resourced steady-state kafkanodepools.yaml only after the pre-upgrade hook
finishes. In a tenant namespace the LimitRange defaults every container to
128Mi, so the migration's controller (and rolled broker) JVMs OOM, never reach
KRaftPostMigration, and the hook times out — wedging the migration and thrashing
it on every reconcile.

Give both hook pools the same resources the steady-state pools get
(cozy-lib.resources.defaultingSanitize from kafka.resourcesPreset/resources).
This also avoids a redundant second broker roll (resource-less -> resourced)
after the hook completes.

Tests: a helm-unittest assertion that both migration pools render a resources
block; the Chainsaw ZK->KRaft suite now seeds under a 128Mi LimitRange (and
gives the seed cluster explicit resources), reproducing the tenant-quota
condition, and asserts the migration still reaches KRaft with the broker PVC
adopted. Validated green against a live cluster under the LimitRange.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 649f7e0b63 fix(kafka): enforce one Kafka per namespace; simplify node pool names
Follow-up to the KRaft-migration review. The broker KafkaNodePool must be named
exactly "kafka" so Strimzi adopts an existing ZooKeeper cluster's
<cluster>-kafka-N brokers and data in place during migration, but KafkaNodePool
object names are unique within a namespace — so two Kafka clusters cannot both
own a "kafka" pool there. This is an upstream constraint and Strimzi's own
guidance is one Kafka cluster per namespace
(strimzi/strimzi-kafka-operator discussions/11120).

Adopt that model instead of the earlier fresh-vs-migrated conditional naming:

- New kafka.assertSingleInstance helper fails the render (with a clear message
  and the upstream reference) if the namespace already holds another Kafka
  cluster. Reconciles/upgrades of the same release are allowed; lookup returns
  nothing during `helm template`, so unit tests never trip it.
- Node pools are now named "kafka" (broker) and "controller" unconditionally,
  which also adopts existing brokers on migration and yields clean
  <cluster>-kafka-N / <cluster>-controller-N names. The fresh-vs-migrated
  brokerPoolName helper is removed.
- WorkloadMonitor pool-name selectors and the migration hook updated to match;
  unit tests updated for the fixed pool names.
- README documents the one-Kafka-per-namespace rule and links the Strimzi
  discussion.

The ZK->KRaft Chainsaw migration suite (seed ZooKeeper cluster, migrate, assert
the original broker PVC survives and no wrongly-named PVC is created, cluster
reaches KRaft) was run green against a live cluster.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov 710cdd18bb fix(kafka): address KRaft-migration review — broker pool adoption, hook safety, tests
Addresses the branch review on the ZK->KRaft migration path.

CRITICAL — broker node pool adoption. Strimzi derives broker/PVC names as
<cluster>-<pool>-<id>; a pre-node-pool ZooKeeper cluster has brokers
<cluster>-kafka-N, so the adopting broker pool must be named exactly "kafka"
or the operator provisions fresh empty brokers and orphans the tenant's data.
A new kafka.brokerPoolName helper resolves the broker pool to "kafka" for an
existing (migrating) cluster and keeps the release-scoped "<release>-broker"
for fresh installs, so multiple Kafka instances still coexist in one namespace.
The migration hook creates the pool as "kafka" and fails closed if a "kafka"
pool already belongs to another cluster (node pool names are namespace-unique,
so in-place migration of co-namespaced ZK clusters is unsupported). Wired
through kafkanodepools.yaml, workloadmonitor.yaml and the hook.

MAJOR — migration robustness.
- Raise the HelmRelease timeout: the migration Job can run tens of minutes but
  the release timeout defaults to 10m, so helm-controller aborted and thrashed
  the pre-upgrade hook. Add release.cozystack.io/helm-install-timeout=45m on the
  kafka ApplicationDefinition, above the Job's activeDeadlineSeconds (2400s).
- Idempotent migration: do not re-issue kraft=migration once the cluster is past
  the ZooKeeper phase (an invalid backward transition Strimzi rejects on retry).
- Fail closed on the decision gates: distinguish a genuine NotFound (skip) from
  any other kubectl error (abort), and never treat an empty kafkaMetadataState on
  an existing CR as "migrated".

MINOR — version downgrade documented as unsupported; integer (not lexical)
compare in the version gate; deleteClaim/PVC-retention and the one-per-namespace
migration limitation documented in the README; docs/storage-immutability.md
updated for KRaft.

Tests — helm-unittest coverage for the broker pool name, the hook guards and the
idempotent/fail-closed logic; a bats check for the helm-install-timeout budget
ordering; and a Chainsaw ZK->KRaft migration suite that seeds a ZooKeeper
cluster, drives the migration, and asserts the original broker PVC survives (not
recreated) and the cluster reaches KRaft. The migration sequence and the
Chainsaw assertions were validated against a live cluster.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Alexey Artamonov cbf104d6b3 fix(kafka): address branch-review — restore CA cert secrets, guard KRaft finalize
Follow-up to the KRaft migration addressing the branch review.

- kafka-rd cozyrds: restore the two public CA certs in secrets.include
  (clients-ca-cert + cluster-ca-cert). The rebase had collapsed them to
  the bare clients-ca, which is Strimzi's clients-CA *private key* secret
  — a private-key exposure to the tenant and a TLS regression (clients
  need cluster-ca-cert to trust the broker listener). Covered by the
  existing hack/check-kafka-rd-secrets.bats.

- migration-hook: guard the KRaft finalize step. The wait loop only exited
  via break on success and fell through to unconditionally apply
  strimzi.io/kraft=enabled on timeout — unsafe and irreversible if the
  cluster is still in ZooKeeper/KRaftMigration. Add a migration_ready flag
  and exit 1 when the safe state (KRaftPostMigration+) is not reached,
  leaving the deployed-version ConfigMap unstamped so Helm retries. New
  matchRegex assertions in migration-hook_test.yaml cover the guard.

- migration-hook: raise activeDeadlineSeconds 1800 -> 3600 so a
  legitimately-progressing migration is not SIGKILLed before the in-script
  waits and error paths can run.

- Extract the duplicated controller-quorum logic into a shared
  kafka.controllerReplicas helper (used by kafkanodepools, workloadmonitor
  and migration-hook) so the three stay in lockstep.

- Reflow the README migration section to one line per paragraph per the
  repo prose rule.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
Matthieu 42db2ca141 feat(kafka): migrate from ZooKeeper to KRaft mode with automatic migration
Replace ZooKeeper-based Kafka with KRaft (Kafka Raft) metadata mode.
Strimzi 0.45 is the last line that still supports ZooKeeper, so this
prepares the transition before ZooKeeper support is removed upstream.

Chart changes:
- The Kafka CR always renders strimzi.io/kraft=enabled with node pools
  and no spec.zookeeper block. Storage, replicas and resources move to
  separate broker + controller KafkaNodePools (a homogeneous separate-pool
  layout works for both fresh installs and migrations; Strimzi 0.45 does
  not support migration into a combined pool).
- Kafka version is now user-configurable via the standard files/versions.yaml
  pattern (v3.9 default, v3.8 available); appVersion bumped to 3.9.0.
- WorkloadMonitors track the broker and controller pools (no zookeeper
  monitor); the dashboard RBAC grants the controller monitor and drops the
  zookeeper one.

Migration of existing ZooKeeper clusters is handled by a Helm pre-upgrade
Job (templates/migration-hook.yaml) gated by a <release>-kafka-deployed-version
ConfigMap, following the established seaweedfs/etcd hook pattern. The Job
creates the broker + controller pools with Helm ownership labels, drives the
Strimzi ZK->KRaft state machine (node-pools=enabled, kraft=migration ->
kraft=enabled once KRaftPostMigration is reached) and is skipped once the
ConfigMap records version >= "1".

Testing:
- Helm unit tests cover the Kafka CR, node pools, migration hook, metrics,
  topics, version map, dashboard RBAC and workload monitors.
- The e2e suite is ported to the current Chainsaw framework
  (hack/e2e-chainsaw/kafka): it asserts the KRaft topology (both node pools,
  kafkaMetadataState=KRaft), the absence of ZooKeeper, and bootstrap
  endpoints.

Rebased onto current main: TLS listener support, entity-operator resource
guards, x-cozystack-options schema annotations and the WorkloadMonitor
replica-path fix are all preserved and reconciled with KRaft.

Signed-off-by: Matthieu <matthieu@hidora.com>
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-17 11:09:10 +03:00
IvanHunters 928286ba42 fix(test): run the artifact-ignores suite under the runner's own shell
hack/cozytest.sh is `#!/bin/sh` and sources the translated suite, so on a
runner where /bin/sh is dash the here-string is a parse error and the whole
suite dies before its first assertion. macOS hides it: /bin/sh there is bash
in sh mode and accepts `<<<`.

A heredoc keeps the loop body in the current shell, so the accumulated
`missing` still survives to the assertion below it.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-17 09:49:41 +03:00
IvanHunters 617cba1621 fix(installer): keep node_modules out of the packages artifact
`flux push artifact` walks the whole packages tree and reads neither .gitignore nor .sourceignore, so a working copy where somebody ran `pnpm install` under the vendored console ships its dependency tree inside the bundle. Helm then refuses the dashboard chart on every cluster that pulls it, because one dependency file sits above the 5MB per-file limit, and the build itself reports nothing. Passing --ignore-paths replaces the built-in list rather than extending it, so the defaults are restated alongside node_modules/.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-17 09:34:18 +03:00
IvanHunters 3615958240 fix(bucket): stop the ACME cleanup hook from talking to localhost, and gate the class
The hook's four kubectl calls carried --request-timeout=30s. A non-zero
value there makes kubectl skip its in-cluster credentials: client-go
consults them only while the assembled config still equals the default,
and the flag lands in that config, so every call went to kubectl's
built-in default cluster and was refused.

This package failed more quietly than kafka did. Each call is guarded by
`|| echo WARNING` and the script sets no -e, so the hook exited 0 and
printed "ACME cleanup complete."; hook-succeeded then deleted the Job that
held those warnings. The ACME solver objects it exists to reap were left
behind, and the only record went with the Job. Its own suite matched the
rendered command including the flag, so the broken form was pinned.

The gate is the point of this change. Two packages had it, neither was
visible in a rendered manifest, and nothing stops the next hook from
reaching for the same flag, since it reads as ordinary defensive practice.
hack/check-hook-request-timeout.bats fails any non-vendored template whose
kubectl call carries a non-zero --request-timeout, skipping files that name
KUBECONFIG because an explicit kubeconfig is not served by the in-cluster
branch. That per-file judgement is the coarse part and is written down in
the file rather than left to be inferred: apps/kubernetes keeps the flag on
two calls that pass their own kubeconfig, and stays green.

Bounding the work is still available where it belongs: --timeout for a
watch, activeDeadlineSeconds for the Job, which is what this hook already
had at 120s.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-17 09:32:07 +03:00
IvanHunters e63b8595d3 fix(kafka): drop the request timeout that hid the apiserver from the hook
A non-zero --request-timeout makes kubectl skip its in-cluster credentials
altogether. client-go consults them only while the assembled config still
equals the default, and the flag lands in that config, so the branch that
would load the ServiceAccount token never runs. kubectl falls back to its
built-in default cluster, localhost:8080, is refused, and the script reads
the refusal as a failed delete and exits 1. A non-zero pre-delete hook
returns from Uninstall.Run before any resource is removed, so no Kafka
release could finish deleting and helm-controller retried forever.

Measured on the pinned image rather than argued from the docs: with the
token and ca.crt mounted and KUBERNETES_SERVICE_HOST set, the same delete
reaches the cluster without the flag, goes to localhost:8080 with
--request-timeout=20s, goes to localhost:8080 with 1m, and reaches the
cluster again with 0.

The three in-tree hooks that work do not pass it. apps/kubernetes names it
in a comment only, apps/harbor and apps/mariadb never mention it, and
apps/tenant records that its DELETE calls "carry kubectl's default
--request-timeout of 0". The watch stays bounded by --timeout, and the Job
by activeDeadlineSeconds, which is that same shape.

The suite now rejects the flag. It needs the "=" because the explanation
in the script names the flag too, and matching the bare name would fail on
the comment rather than on a use.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-17 09:31:49 +03:00
IvanHunters 64d5689434 fix(kafka): keep the pre-delete hook's credentials as long as its Job
The Job carries no hook-succeeded, so it outlives the hook run and
ttlSecondsAfterFinished reclaims it, which is what leaves a readable
record when the finalizer outlives the wait. Its ServiceAccount, Role and
RoleBinding still carried hook-succeeded and hook-failed, so Helm reaped
them at the end of the run either way and left the Job behind them.

A Job whose ServiceAccount is gone still starts a pod. kubelet mounts the
projected volume without a token, client-go finds no in-cluster config,
kubectl falls back to localhost:8080, and the script takes its catch-all
branch and exits 1. A non-zero pre-delete hook returns from Uninstall.Run
before any resource goes, so the release never finishes deleting and
helm-controller retries it: the e2e run on the merge commit shows six
such pods in five minutes, in two namespaces, every one of them dead the
same way. The hook meant to prevent one stuck Kafka release was instead
stopping all of them.

Give all four objects the same before-hook-creation policy, so the
credentials live exactly as long as the consumer and the next run
replaces them. Both other in-tree hooks that reach the apiserver,
apps/kubernetes and apps/harbor, are symmetric this way already.

The suite now pins the policy on each of the three, because nothing
compared them: the asymmetry rendered cleanly, passed 37 helm-unittest
cases and 5 bats cases, and only a cluster showed it.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-17 09:31:49 +03:00
IvanHunters 6818795588 fix(seaweedfs): raise the bucket ceiling by sizing volumes smaller (#4251)
## What this PR does

Raises how many buckets a SeaweedFS instance can serve, by lowering the
master's `volumeSizeLimitMB` from 1000 to 400 and its two-copy growth
count from 6 to 3. The size limit is exposed on the app chart, where an
operator can reach it. Existing instances stay at 1000 unless they
already name a limit, so for them the upgrade changes only the growth
count. A platform migration does that where migrations run; on the
`default` variant it is a manual step before upgrading. A unit test pins
each constant separately, and an e2e case creates several buckets and
requires each to take a write.

Late in a full e2e run every S3 `PUT` starts timing out and a backup
suite fails its access preflight. It reads like a credentials or COSI
problem and is not one: the instance has no volume slots left, so the
master cannot grow a volume for a newly created bucket. The disk is
still almost entirely free, which is what makes it look like a flake.
The COSI objects say nothing either way, the claim is ready and the
access is granted.

### The ceiling

A volume server started with `-max 0` takes a slot count derived from
disk rather than a fixed one, replication `001` spends a slot on each of
the two servers per logical volume, and a collection claims `COPY_2`
volumes each time it grows. So the ceiling is set by the product of the
size limit and `COPY_2`, and moving either number alone moves it. The
mechanism is upstream in `MaybeAdjustVolumeMax`
(`weed/storage/store.go`) and `findVolumeCount`
(`weed/topology/volume_growth.go`); what matters for choosing the
numbers is only that upshot.

Do not divide the slot figures out of the nominal PVC size, because what
the server counts is the space the filesystem makes available and not
the volume's advertised size. At the upstream 1000 and 6 a 10Gi volume
PVC gives nine slots per server and two collections, which the
platform's own `cozy-backups` bucket and a single tenant bucket already
use. That same PVC was seen to give nineteen slots at 500, which puts
400 at twenty-three to twenty-four by the same division, or eight
collections. The span is the observation's own: the free space behind
nineteen at 500 fits more than one filesystem geometry, and those divide
differently at 400. Eight either way, because a collection that cannot
get a whole batch of `COPY_2` keeps the volumes it did get and writes
into them, so the count rounds up rather than down.

Every figure here is for the shipped `replicationFactor: 2`. The master
picks which of its four growth counts to apply from the replication the
filer asks for, and the filer takes that from `replicationFactor`, so at
1 or 3 copies it reads `COPY_1` or `COPY_3` and the `COPY_2` this PR
sets does not apply. Lowering the size limit still raises the ceiling
there, since it is the divisor whatever the replication is.

I measured this instead of only deriving it, on the real `weed` 4.31
binary: one master, two volume servers with `-max 0`, each on its own 1
GB filesystem image, replication `001`, then one `assign` per fresh
collection until the master refused. Scaled 10:1 against the 10Gi PVC,
so the limits scale with it.

| `volumeSizeLimitMB` | `COPY_2` | slots reported | collections served |
| --- | --- | --- | --- |
| 100 | 6 | 10 | 2 |
| 100 | 3 | 10 | 4 |
| 50 | 3 | 20 | 7 |
| 40 | 3 | 25 | 9 |
| 25 | 6 | 40 | 7 |

The fourth row is the pair this PR ships. The refusal arrives as `No
writable volumes and no free volumes left`, with `volumes == max` on
both servers and the disk nearly empty. The reported `max` matched the
space the filesystem made available over the limit in every run, which
is the rule the PVC figures above are divided with. Those runs sit on a
1 GB image, so their counts are the rule's, not the PVC's: a 10Gi volume
has its own filesystem geometry and divides to its own numbers.

### Why both constants move

The last two rows are the argument: lowering the size limit and lowering
the growth count reach the same ceiling. They do not cost the same. Open
descriptors scale with stored bytes rather than with the PVC, so at 400
a 10 TiB server keeps roughly 52k of them against 21k upstream, where
taking the whole reduction out of the size limit would have put it at
105k, well past a 65536 `nofile` limit where one is set. Two stay open
per volume, a `.dat` and an `.idx`; the `.vif` beside them is written
and closed. That cost lands hardest on large instances, which never had
a ceiling problem in the first place: at any of these limits a multi-TiB
PVC has slots to spare. Lowering the growth count costs no descriptors
at all. It costs write spread, since a bucket then spreads its writes
over three volumes rather than six, and that cost is flat, it does not
grow with the instance.

So the reduction is split across both. Neither number is a new bet: 3 is
what upstream already uses for three-copy replication.

One price is shared by both routes. A collection absorbs
`volumeSizeLimitMB × COPY_2` before it needs another grow, so growth
requests arrive five times as often as they did at 6000 MB. A collection
also starts with half as many writable volumes at `replicationFactor:
2`, and a volume server takes the writes into each volume one at a time,
so a heavily parallel writer spreads over fewer volumes until the
collection grows again. That holds for instances kept at 1000 too.

### On upgrade

On a variant that runs platform migrations, the upgrade does not change
the limit on an existing instance unless its operator already set one.
Platform migration 57 writes `master.volumeSizeLimitMB: 1000` into every
existing SeaweedFS instance that has not chosen a limit, except one
whose `master` is not an object (below), so 400 applies only to
instances created after this release. The `default` variant runs no
platform migrations. A cluster there that installed SeaweedFS by hand
has to apply the same patch before upgrading, and the operations guide
has the command. Both guide steps end with a `complete` line. `jq`
prints it only after reading the whole list, and the pinning loop stops
at the first failed patch before it, so a run without that line is
incomplete, whatever else it printed. A chosen limit is a whole number
from 1 to 30000 and is kept. Anything else counts as not chosen and
becomes 1000: the previous chart ignored the key and ran 1000 in its
place, and the new schema would refuse it. A kept value other than 1000
takes effect with the upgrade itself. A lower one can strand buckets and
a higher one takes the room new buckets grow into, so the guide has a
query that lists those instances before upgrading. An instance kept at
1000 still gains: at `replicationFactor: 2` a grow now claims three
volumes instead of six, so it has room for about twice as many new
buckets.

Three shapes get their own line in the Job log, with what they hold. A
non-object `spec.values` is pinned too. helm-controller v1.5.0 ignores
values it cannot read as a map and renders the chart defaults, so that
instance runs at 1000 today and would drop to 400. The merge patch
replaces the unreadable value, and nothing that applies is lost. A
non-object `master` is left alone. Every v1 schema already refuses it,
so that release does not render today, and the log says to set the limit
in the same edit that repairs it. A `valuesFrom` with entries besides
the `cozystack-values` Secret is named and pinned anyway.
helm-controller merges `spec.values` over those entries, so a limit set
there does not apply, unless it is set through `targetPath`, which can
still win depending on the entries before it. The guide steps print the
same three.

The value goes into the application's HelmRelease, whose `spec.values`
is the application spec, so it covers an instance created through the
API and one the tenant module renders alike, the platform's own in
`tenant-root` included. The tenant module renders that release without
`spec.values`, which a new tenant chart test pins, and a Helm upgrade
leaves alone a key that neither the previous nor the new manifest
carries: I measured that on a kind cluster with the real HelmRelease CRD
and Helm 4.3.0, on client-side, server-side and forced server-side
apply, and read `createPatch` at the v4.1.1 that helm-controller v1.5.0
runs. Where migrations run, the order comes from the platform release
itself. The migration is its pre-upgrade hook, and the
`cozystack-packages` source every application chart is built from is a
manifest of that same release, applied only after the hooks finish, so
no instance meets the new chart unstamped and a failed migration stops
the platform upgrade before that source moves. That holds while the
platform source is pinned by digest, as it is out of the box; a source
that follows a tag, a semver range or a branch lets source-controller
move `cozystack-packages` on its own schedule, and an instance there can
run at 400 until the migration has run. Until `cozystack-packages`
moves, each patched release still runs its old chart, which ignores the
key: no v1 release reads it, and none closes the `master` object in its
values schema. The migration's test runs the script under the migrations
image's shell against a fake `kubectl` that refuses any selector but the
one cozystack-api lists SeaweedFS with, because an empty fleet is a
legitimate answer and a mistyped selector would otherwise pass while
pinning nothing. The same test runs the two guide steps, found by a
marker comment above each, against the same fake and checks that they
select the same releases as the migration.

Lowering an existing instance is left to its operator, because it is not
free. Its volumes between the new limit and the old go oversized on the
next heartbeat and leave the master's writable set, and each affected
collection needs one successful grow before its next write lands.
Nothing is lost and the volumes stay readable. Those grows compete for
one pool, and the budget is per collection rather than per server. An
oversized volume is charged no headroom while one below the limit is
charged all of its unfilled room, so the slots open to a stranded
collection are free space less that room, divided by the limit; a grow
then claims a whole `COPY_2` batch and takes as much of it as is there,
and the volumes it creates are empty and charge their headroom straight
back. Measured on the rig: two oversized collections with two free slots
between them, and the first to write took both, leaving the second
refused afterwards with the disk still reporting a quarter of itself
free. Collections that keep writing are not free either: they were grown
six at a time, and each of their volumes below 400 MB can hold back a
whole 400 MB. Measured at a scaled pair, a server with free space for
one batch per collection refused the only grow it needed, because a
second, nearly empty collection held the room, and the same free space
without that collection let the grow through. The master also grows, on
its own schedule and with no write, any collection with no writable
volume at or below 90 percent of the limit, which at 400 is 360 MB;
measured at the same scaled pair, a collection with every volume in that
band got two new volumes per server within one pass while a roomy one
was left alone. So the precondition on every volume server is a batch
for each collection with no volume at or below that line plus 400 MB for
each volume still below the limit, and no volume size exempts an
instance from it, since a larger volume carries more buckets and the
budget scales with them. The `minFreeSpacePercent` reserve only comes
into it on a server whose writable volumes have all passed the limit,
since nothing there is charged headroom. It is five percent of what the
filesystem reports rather than of the PVC's nominal size, so `All >=
8000 MiB` is where it reaches one 400 MB limit; a 10Gi PVC measures 9979
MiB and clears that, and such a server then refuses a grow only once its
disk is down to the reserve. Everywhere else the headroom charge
decides, and a refusal can come with the disk far from full whatever the
reserve. Deleting a collection gives back a slot for each of its
volumes, however little it stored, since the headroom charged to them
leaves with them; only a server already at its reserve, where every
volume is read-only and charged nothing, gets back no more than the
bytes. Deleted objects return room only through a vacuum. The room left
inside those volumes for further writes is given up and does not return
as capacity, since such a volume is charged no headroom while its own
bytes still come out of the budget. Lowering the growth count takes
nothing away by contrast: no path removes a volume from the writable set
for being surplus to the batch size, so an existing collection keeps
what it has and converges as those volumes fill.
[`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md)
has the check for it, and the way back.

The ceiling cannot come out lower at any supported `replicationFactor`,
which is the part worth stating plainly. The divisor halves, and the
space the slot count is derived from grows at the same time, because a
volume past the new limit stops being charged headroom against it.
`COPY_1` and `COPY_3` are untouched, so at 1 and 3 copies the limit
alone still moves the ceiling upward.

### Reaching the value

Neither constant was reachable by an operator. The data-plane chart is
pulled by `ExternalArtifact`, and the component publishing it declares
no `install` section, so the reconciler drops a `Package`-level values
override for it before any merge happens. Editing the chart was the only
route, on a value whose right setting depends on the PVC size the
operator chose. `master.volumeSizeLimitMB` is on the app chart now, with
a schema entry.

Worth being precise about which file the two numbers end up living in,
because exposing one of them moves it. The app chart's HelmRelease sends
the size limit on every release, so
`packages/extra/seaweedfs/values.yaml` holds the 400 installs run with,
and `packages/system/seaweedfs/values.yaml` keeps upstream's 1000 as the
fallback a render without it gets. That fallback is deliberate: a
platform upgrade can re-render the data-plane release from its artifact
before the parent has sent anything new, and a lower value there would
then reach the master first and drop the limit ahead of an operator's
own setting. Nothing sends the growth count, so the data-plane chart's
value is the live one for that half, and it is the part that does land
in that window, where a grow costs 3000 MB against the 6000 MB it cost
before. The budget the pair produces is written out next to the growth
count, with a pointer from the size limit to it, so the two do not drift
apart.

The field is bounded at both ends, because either end takes an instance
down. Zero is not a smaller limit but an off switch: the volume server
leaves its slot calculation untouched at zero, keeps reporting no
capacity, and nothing is ever grown. Past 30000 the master does not
start at all, exiting on `volumeSizeLimitMB should be smaller than
30000`. The bound lives in the values schema, through the generator's
`@minimum` and `@maximum` annotations, so Helm refuses the value before
any template runs. The apps API reads that schema only for defaults, so
it accepts a value out of range, and the release then fails to render. A
case pins each edge along with the value just outside it. An explicit
null gets past both bounds, because Helm drops a null key before it
validates, and would reach the master as an empty `-volumeSizeLimitMB=`
that the shipped image exits on at start, so the template requires the
value and a case pins that as well.

There is no second bound against the disk, and not for want of trying: a
limit at or above the volume's own size does not break an instance.
`MaybeAdjustVolumeMax` floors the slot count at one, and a 255 MiB disk
against a 1000 MB limit reported `Max 1`, took an assign for a fresh
collection and served back what was written to it. Such a server is
capped at a single slot rather than dead, so a render refusal would stop
a working release on every reconcile. The field description says instead
that a value close to or above `volume.size` leaves each volume server a
single slot.

The growth count is deliberately not exposed beside the limit. A field
wired to `COPY_2` would do nothing at 1 or 3 copies, for the reason
above, while still looking like a setting, and a knob that is silently
inert for two of the three supported values is worse than no knob.
Making it track the replication factor is a feature with its own matrix,
and it would rest on a path where the master's own default replication
is separately broken. The size limit has no such dependency, so it is
the one that ships.

### Picking a value

400 is sized for the shipped `volume.size: 10Gi`, which is where the
slot and collection figures above come from. Going up, it holds until
the two descriptors per volume meet a 65536 `nofile` limit, roughly 12.5
TiB per volume server. Going down, an instance needs 1600 MB of
filesystem per volume server to serve a second collection at all, since
the first claims the opening three slots.

### What the tests cover

The existing bucket case creates one bucket, which sits inside the
ceiling at any sizing, so it could not tell a working ceiling from an
exhausted one. The new e2e case creates three beside the platform's own
backup bucket and requires each to accept an object, which is the only
signal the exhaustion produces. Each `mc` call there carries its own
timeout, since a hung PUT is exactly that signal and the step timeout
alone would not end it. It was observed red at the old sizing in [run
34948279363](https://github.com/cozystack/cozystack/actions/runs/34948279363),
which ran the case against the unchanged constants: the pre-existing
single-bucket case passed in 18s beside it, and the new one failed on
the write to the second bucket after the first succeeded. Three rather
than two so the count clears the ceiling without depending on how many
slots the unnamed default collection holds, which the suite cannot
observe.

The unit tests pin each constant on its own, because losing one is a
silent capacity regression rather than something a render would catch.
The app-chart test asserts an overridden size limit as well as the
default, since asserting the default alone passes just as well against a
template that ignores the value and writes the number as a literal. It
also pins both ends of the accepted range and the first value outside
each, the `-max 0` zero on all four render paths it is written to, and
the 10Gi volume default the slot figures above are measured for, which
nothing else guarded. The data-plane suite pins the five percent
free-space reserve as well, since the 8000 MiB threshold above is
computed from it and nothing else reads it.

The capacity case also reads the master's slot accounting in its
`catch`. A grow failure in the master log does not say whether the
shipped limit is too coarse for the PVC or an earlier collection in the
same run is still holding the budget, and the per-server `Max` against
`Volumes` does.

Closes #4239, closes #4245, closes #4248

### Screenshots

No UI change.

### Downstream repositories

Walked against the diff. The new field lands on
`packages/extra/seaweedfs`, and the provider models bucket, generic,
qdrant, redis and tenant, with no SeaweedFS resource among them, so
there is no schema or expand/flatten pair over there for it to drift
from. No package is added, renamed or removed, so the website app lists
are untouched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(seaweedfs): raise how many buckets a new SeaweedFS instance can serve on the default 10Gi volume PVC from two to eight at `replicationFactor: 2`, by defaulting the master volume size limit to 400 MB instead of 1000 and lowering the two-copy growth count from 6 to 3, and expose the limit as `master.volumeSizeLimitMB` on the app chart. **Action required on the `default` variant only:** it runs no platform migrations, so before upgrading, set `master.volumeSizeLimitMB: 1000` on every SeaweedFS instance that has no valid value and whose `master` is an object, with the step in [`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md). If this is skipped, every such instance drops to 400 on the upgrade, and a bucket whose volumes have all reached 400 MB takes no writes until it can grow. On the variants that run platform migrations, migration 57 does this: it sets `master.volumeSizeLimitMB: 1000` on every instance that has not chosen a value, except one whose `master` is not an object, which no v1 chart renders and which it names in its Job's log instead, so nothing changes for an instance unless it already names a different value, which the previous chart accepted and ignored and this one applies; the same guide lists such instances before the upgrade, and the migration names them in its Job's log. An existing value that is not a whole number from 1 to 30000 is replaced with 1000, which is what the previous chart applied in its place. Instances at 1000 still get roughly twice the room for new buckets from the lower growth count. The upgrade rolls every SeaweedFS master StatefulSet once, since the growth count is part of the master's environment, and an instance that names a value other than 1000 rolls a second time when that value arrives. The pin lives in the instance's spec, so a tool that replaces the whole resource from a manifest without `master.volumeSizeLimitMB` removes it and the instance takes 400; add the field to such manifests before upgrading. Lowering the limit on an instance that already holds data takes every volume at or above the new value out of the master's writable set, and raising it takes the room new buckets grow into, so the same guide describes the free-space check to run first and how to set it back.
```
2026-09-17 09:29:05 +03:00
IvanHunters c983390b39 ci(e2e): surface overlay drift and remove both causes of it (#4261)
## What this PR does

Six commits against one script: one reports a silent degradation, three
remove causes of it, two repair the tests and the reasons recorded
beside them.

**Why it matters.** A file whose config differs from the current-main
artifact beyond its image lines keeps its committed ref. That is the
right safe answer, and it is not a neutral one: the component then runs
its last RELEASE image in a lane whose job is to exercise the tree under
review, and the only trace is one line among hundreds. Seven packages
took that path on the run behind #4257. One of them,
`backupstrategy-controller`, served an image from v1.6.0 (22 July)
against a tree carrying four backup strategy drivers added after it, so
five of that run's seven chainsaw failures came from there, and three
unrelated pull requests each spent a review round establishing that the
red lane was not theirs.

After this PR that list is empty. Against the public artifact at its own
revision, with the tree checked out at that same revision:

```text
before  drift=5  overlaid=29    core/platform, cozystack-api, dashboard,
                                kubeovn-webhook, linstor-gui
after   drift=0  overlaid=34
```

**Naming the drifted packages.** One GitHub annotation after the
existing tally. Which files get overlaid does not change, and a run with
no drift emits nothing.

**Cause one: a suffixed image key.** The classifier accepted a key only
when it was exactly `image`, `repository`, `registry`, `tag` or
`digest`. A chart rendering several images names the extras after what
they are for, so `backupstrategy-controller` carries
`chBackupClientImage`, `rabbitmqBackupClientImage` and
`redisBackupClientImage`. The first two passed only through the
`@sha256:` branch; the third is pinned by tag alone, so it read as a
config change and cost the whole file its overlay. The fix accepts a
prefix before `image`, and two things keep it narrow. A letter is
required before `image`, which a key beginning with it has nothing to
supply, so `imagePullPolicy` and `imagePullSecrets` stay config; the
trailing colon is not what does that, and they match neither with it nor
without. The value must also hold a repository path, which keeps out
operator-settable knobs named after an image: `vddkImage: ""` in
`core/platform` and `migration-controller` is configuration the build
never stamps, and absorbing it would be the opposite of what this script
is for.

**Cause two: a blank line the build removes itself.** `yq -i` re-emits
the document when it stamps a digest, and a blank line separating two
top-level keys does not come back. So a file the PR never touched
differs from the artifact on a line carrying no configuration at all,
and loses its overlay for it:

```text
$ diff repo/system/cozystack-api/values.yaml artifact/system/cozystack-api/values.yaml
4d3
<
6c5
<   image: ...cozystack-api:v1.6.0@sha256:c4d374a9...
---
>   image: ...cozystack-api:main@sha256:2fb87cf8...
```

Running the stamping assignment over the committed file reproduces that
diff exactly, which is the second signal behind the diagnosis rather
than the first one twice. Blank lines are now discounted before the
test; a line with content still counts, so a genuine config difference
keeps the committed ref as before.

This accounted for all five remaining packages, and it retires the
explanation offered in #4265, which ruled `yq` reformatting out. That
was wrong: the round-trip check behind it did not cover the blank line.

`cozystack-api` and `dashboard` are not peripheral here. An e2e result
about either was a result about the last release rather than about
`main`.

**Repairing the tests.** `hack/bats-no-exit-trap.bats` audits every
`.bats` file under `hack` against the debt its own header declares, and
this file was installing three traps beyond it. An EXIT trap inside an
`@test` body replaces the one the bats binary installs, so a failing
test prints no TAP line at all and the suite reads as green. The answer
is to stop installing the handler rather than to raise the number, so
those tests clean up at the end of the body instead.

### Tests

Eight cases in `hack/overlay-main-images_test.bats`, each held by a
mutation of the line it describes:

- drift is summarised with the package names and the count; suppressing
the annotation or dropping the names reddens the suite
- a clean run emits no annotation
- a suffixed image key pinned by tag alone overlays instead of drifting;
reverting the classifier reddens it
- a key merely STARTING with `image` is still config
- an `*Image` key whose value is operator configuration rather than a
path still drifts, so `vddkImage` is not absorbed
- two drifted files in one package count as one package, and the name
carries no `/images` suffix
- a blank line lost to `yq` is not a config change; removing the filter
reddens it
- a real config line alongside a blank line still counts as drift;
widening the filter to drop every line reddens this one and two others

The suite's plan and its `ok` count agree at 21, which is the
reconciliation the trap audit exists to protect. Green under the `bats`
binary and under `hack/cozytest.sh`, which is what `make
bats-unit-tests` runs and the only one of the two whose result decides
the gate.

### What this does not close

`redisBackupClientImage` is still pinned by tag with no digest, and the
tag it names does not exist:

```text
$ curl .../cozystack/redis-backup/tags/list
{"tags":["nightly","0.0.0-nightly.20260911...","0.0.0-nightly.20260912...", ...]}
```

Only nightly tags are published, so `redis-backup:v1.6.0` resolves to
nothing. Pinning it needs a published release tag first, which is #4264
rather than this PR.

Making the overlay line-wise, so image lines transfer even when other
lines genuinely differ, would go further still. It trades away the
guarantee the script is built on, that a file comes wholly from one
source, and that guarantee is deliberate and commented: a maintainer
decision rather than a quiet change from a reviewer.

### Screenshots

Not a UI change.

### Downstream repositories


- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note


```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* GitHub Actions now warns when image overlay processing detects
configuration drift.
  * Warnings list affected package names in sorted, de-duplicated form.
* Image references with alphabetic prefixes are recognized during
overlay processing.
  * Blank-line-only differences no longer trigger drift warnings.

* **Tests**
* Added coverage for image references, similarly named configuration
keys, blank-line differences, and multi-package drift reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-17 09:28:03 +03:00
Timur Tukaev 69c4fadf8a docs(security): document Security Champion rotation and response team (#4223)
## What this PR does

Surfaces the **Security Champion rotation policy** and the **security
response team's composition and diversity** into the public
`SECURITY.md`. Until now these were defined only in the CVE-scanning
pipeline's private governance document, which the General Technical
Review flagged as a gap (the policy existed but was unpublished, and the
rotating role was unassigned).

Specifically:
- Adds a **"Security response team and rotation"** subsection: names the
response-team maintainers, notes the team spans more than one
organization (Ænix, HIDORA) and more than one country, and documents the
rotating **Security Champion** role — its responsibilities, a **6-month
rotation** among maintainers, and a **one-week handoff overlap**.
- Records **@tym83** as the current Security Champion, filling the role
this file previously described as unfilled.
- Updates the Trivy paragraph accordingly (the triage-clock owner is no
longer "currently unfilled").

This is a policy change to a `CODEOWNERS`-protected file and is intended
for maintainer (CODEOWNER) review rather than a wording check.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
docs(security): document the Security Champion rotation policy and security response team in SECURITY.md
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added information about the security response team and its rotating
Security Champion role.
* Clarified ownership of security triage timing and response
coordination.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-17 06:07:59 +03:00
Aleksei Sviridkin a2db2aeb8c fix(seaweedfs): raise the number of buckets a volume server can serve
A volume server started with -max 0 takes a slot count derived from disk
rather than a fixed one, replication 001 spends a slot on each of the
two servers per logical volume, and a collection claims COPY_2 volumes
whenever it grows. So the bucket ceiling is set by the product of the
size limit and COPY_2, and moving either number alone moves it. The
mechanism is upstream in MaybeAdjustVolumeMax (weed/storage/store.go)
and findVolumeCount (weed/topology/volume_growth.go).

Do not divide these out of the nominal PVC size, because the count
follows the space the filesystem makes available and not the volume's
advertised size: the shipped 10Gi volume PVC was seen to give nine slots
per server at the upstream 1000 and 19 at 500, which puts 400 at 23 to
24 by the same division, or eight collections against two. The span is
the observation's own, since the free space behind 19 at 500 fits more
than one filesystem geometry and those divide differently at 400; eight
either way, because a collection that cannot get a whole batch of COPY_2
keeps the volumes it did get and writes into them. The division itself
is measured: on a 1 GB image the master reports max as the filesystem's
available space over the limit in every run, 25 slots and nine
collections at the 40 and 3 that scale to this pair. Two is what the
platform's own cozy-backups bucket and a single tenant bucket already
use. Past that every S3 PUT to a new bucket times out while the disk
still reads as nearly empty, and no COSI object reports anything wrong,
because the claim and the access are both granted.

400 and 3 make the budget 1200 MB per collection. Splitting the
reduction across both numbers rather than taking it out of the size
limit alone keeps the cost off the curve that grows: open descriptors
track stored bytes divided by the limit, so a 10 TiB server keeps
roughly 52k of them at this limit against 21k upstream, where moving the
size limit alone would have put it at 105k. A .dat and an .idx stay open
per volume; the .vif beside them is written and closed. The ceiling
reached is the same either way.

Both figures describe the shipped replicationFactor of 2. The master
picks its growth count from the replication the filer asks for, so at 1
and 3 copies it reads COPY_1 and COPY_3 instead and this COPY_2 does not
apply; lowering the size limit still raises the ceiling there.

The size limit was out of an operator's reach: it lived in the
data-plane chart, which is pulled by ExternalArtifact from a component
that declares no install section, so a Package-level values override for
it is dropped before any merge. The app chart now declares
master.volumeSizeLimitMB and sends it on every release, and 400 is set
there. The data-plane chart keeps upstream's 1000 as its fallback on
purpose. A platform upgrade can re-render that release from its artifact
before the parent has sent anything new, and a lower fallback would then
reach the master first and drop the limit ahead of an operator's own
setting. COPY_2 is sent by nothing and lives in the data-plane chart, so
it does land in that window, where a grow costs 3000 MB against the 6000
MB it cost before.

The app chart also writes maxVolumes into dataDirs on every release, and
because dataDirs is a list Helm replaces the data-plane chart's copy
wholesale, so the zero that makes the slot count auto-derived at all
comes from there. Losing it drops the server to the subchart's fixed
seven without failing any render.

The field is bounded at both ends, because either end takes an instance
down. Zero is not a smaller limit but an off switch: the volume server
leaves its slot calculation untouched at zero, keeps reporting no
capacity, and nothing is ever grown. Past 30000 the master does not
start at all. The bound sits in the values schema rather than in a
template guard, so Helm refuses the value before any template runs. The
apps API does not: it reads that schema only for defaults, so it accepts
a value out of range, and the release then fails to render. An explicit
null gets past both bounds, because Helm drops the key before it
validates, and would render an empty -volumeSizeLimitMB= that the master
exits on at start, measured on the shipped image, so the template
requires the value.

No bound against the disk goes with it, though the field invites one. A
limit at or above the volume's own size does not break an instance:
MaybeAdjustVolumeMax floors the slot count at one, measured on 4.31 with
a 255 MiB disk against a 1000 MB limit, which reported Max 1 and served
a write it then read back. Such a server is capped at a single slot
rather than dead, so refusing it at render would be a capacity policy
that stops a working release on every reconcile, not only on the
upgrade. The field's description carries the relation instead, where the
value is chosen.

The growth count is deliberately not exposed beside the limit. The
master picks which of its four growth counts to apply from the
replication the filer requests, and the filer takes that from
replicationFactor, so a field wired to COPY_2 would do nothing at 1 or 3
copies while still appearing to be a setting. A knob that is silently
inert for two of the three supported values is worse than no knob.

Two further prices. A collection absorbs the budget before it needs
another grow, so growth requests arrive five times as often as at the
upstream pair. A collection also starts with half as many writable
volumes, and a volume server takes the writes into each volume one at a
time (syncWrite holds that volume's dataFileAccessLock), so a heavily
parallel writer spreads over fewer volumes until the collection grows
again; that holds for instances kept at 1000 too. And lowering the limit
on an instance that already holds data carries the other one: its
volumes between the new limit and the old go oversized on the next
heartbeat and leave the master's writable set, and each affected
collection needs one successful grow before its next write lands.
Nothing is lost and the volumes stay readable; the room left inside them
for further writes is given up and does not return as capacity.

Those grows compete for one pool, and the pool is smaller than it looks.
An oversized volume is charged no headroom, while one below the limit is
charged all of its unfilled room; a grow then claims a whole COPY_2
batch and takes as much of it as is there, and the volumes it creates
are empty and charge their headroom straight back. Measured on a local
SeaweedFS 4.31 setup: two oversized collections with two free slots
between them, the first to write took both, and the second stayed
refused with the disk still reporting a quarter of itself free.
Collections that need no grow are not free either, since their volumes
below the new limit were created six at a time and each can hold back a
whole limit. Also measured there, with the limit and batch scaled down:
a server with free space for one batch per collection refused the grow
its one affected collection needed, because a second, nearly empty
collection held the room, and the same free space without it let the
grow through. The minFreeSpacePercent reserve only comes into it on a
server whose writable volumes have all passed the limit, where nothing
is charged headroom: from 8000 MiB of filesystem upward, a 10Gi PVC
measuring 9979, five percent is at least 400 MB, and such a server then
refuses a grow only once its disk is down to the reserve. Everywhere
else the headroom charge decides, and a refusal can arrive with the disk
far from full whatever the reserve. Deleting a collection gives the room
back a slot per volume, however little it stored, since the headroom
charged to those volumes leaves with them.

So, on a variant that runs platform migrations, the upgrade does not
change the limit on an existing instance unless its operator already set
one, which the previous chart accepted and ignored. Platform migration
57 writes master.volumeSizeLimitMB: 1000 into every existing SeaweedFS
instance that has not chosen a limit, and 400 applies only to instances
created from here on. It leaves alone only an instance whose master is
not an object, which no v1 chart renders, and names that one in its log.
The default variant runs no platform migrations, so a cluster there that
installed SeaweedFS by hand gets no pin, and the document below gives
the same patch to apply before upgrading. A chosen limit is a whole
number the new chart accepts; anything else counts as not chosen and is
replaced with 1000, which is what the previous chart ran in its place,
since it ignored the key and the new chart would refuse to render it. A
valid value already set is kept, and since the previous chart ignored
it, the upgrade is when it starts to apply; the document below lists the
ones that change the limit ahead of it. Raised values are listed as
well: existing buckets keep writing, but every volume below the new
limit is charged up to it, and on a local SeaweedFS 4.31 setup seven
nearly empty volumes left no free slot once the limit went up tenfold,
so a new bucket was refused. The value goes into the application's
HelmRelease, whose spec.values is the application spec, for an instance
created through the API and for one the tenant module renders alike. The
tenant module never renders spec.values on that release, and a Helm
upgrade leaves a key alone that neither manifest carries: measured with
Helm 4.3.0 on client-side, server-side and forced server-side apply, and
read in createPatch at the v4.1.1 that helm-controller v1.5.0 runs.
Where the migration runs, it is a pre-upgrade hook of the platform
release, and the cozystack-packages source every application chart comes
from is a manifest of that same release, applied only after its hooks;
so no instance meets the new chart unstamped, and a failed migration
stops the platform upgrade before that source moves. That holds while
the platform source is pinned by digest, as it is out of the box; one
that follows a tag, a semver range or a branch lets source-controller
move cozystack-packages on its own schedule. Until cozystack-packages
moves, each patched release keeps its old chart, which ignores the key,
since no v1 release reads it or closes the master object in its schema.
An instance kept at 1000 still gains, because at a replication factor of
2 a grow now claims three volumes rather than six, so it has room for
roughly twice as many new buckets.

Lowering an existing instance is left to its operator, and
docs/operations/seaweedfs-lowering-the-volume-size-limit.md is the check
for it. It lists each server's volume files and adds one limit for every
volume below the limit and a batch for every collection with no volume
at or below 90 percent of it, since the master grows such a collection
on its own schedule without a write; measured the same way, a collection
with every volume in that band and no writes got two new volumes per
server within one pass. Where each disk type is served by as many volume
servers as the replication factor, as the shipped two at a factor of 2
are, each collection's volumes sit on every server of that type, and
that bounds the claim from above without assuming how the collections
grew. The master places a new volume on any server of the requested disk
type with room rather than where its collection already lives, whichever
group or pool that server belongs to, and it reads an empty disk type as
hdd; under MultiZone it also grows each zone on its own. So the document
keeps zoned instances, and any disk type served by more servers than
copies, at 1000.

BREAKING CHANGE: on the default variant, which runs no platform
migrations, set master.volumeSizeLimitMB to 1000 before upgrading on
every SeaweedFS instance that has no valid value and whose master is an
object, with the step in
docs/operations/seaweedfs-lowering-the-volume-size-limit.md. On the
variants that run platform migrations, migration 57 does this. If the
step is skipped, every such instance drops to 400 on the upgrade: each
of its volumes at or above 400 MB leaves the master's writable set, and
a bucket whose volumes have all crossed that line takes no writes until
it can grow.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-17 00:54:48 +03:00
Aleksei Sviridkin c038cf25d3 test(seaweedfs): cover the bucket count a volume server can serve
A volume server started with -max 0 derives the slots it has left from
the free space on its disk, less the room its unfilled volumes still
claim, divided by the master's volume size limit, and each collection
claims a batch of those slots on every volume server the replication
spans. The number of buckets an instance can hold therefore has a
ceiling well below what the free bytes suggest, and crossing it leaves
every COSI object reporting ready while the S3 PUT times out.

The existing case creates one bucket, which sits inside that ceiling at
any sizing, so nothing here told a working ceiling from an exhausted
one. This case creates three beside the platform's own backup bucket and
requires each to accept an object, which is the only signal the
exhaustion produces.

Three rather than two because the count has to clear the ceiling whether
or not cozy-backups and the unnamed default collection have grown yet,
which this suite cannot observe.

BucketAccess reconciles separately from the claim it belongs to, and the
write step reads the Secret that access produces, so readiness of the
release and the claim is not enough to enter it. Without the access
assertions the step can fail on a missing Secret, which looks the same
in the report as the capacity failure it is here to measure.

The catch reads the master's slot accounting as well as its log. A grow
failure in the log does not say whether the shipped limit is too coarse
for the PVC or an earlier collection in the same run still holds the
budget, and the per-server Max against Volumes does. The read runs
inside the master, whose image ships curl, so it needs nothing on the
host and adds no second fixed port to the reason this suite cannot run
in parallel. That reason now covers two cases rather than one, which the
shared config says.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-16 23:16:14 +03:00
Andrei Kvapil 4953b512a5 chore(platform): freeze HTTPCache on the last artifact that ships it
The earlier shape of this migration orphaned every instance into plain
objects, which left its owner without a way to manage or remove it. A
cluster that still runs HTTPCache instead keeps the application, served
from a copy of the packages OCIRepository pinned to the digest of the
last release that contains it. The copy carries no owner references and
no Helm metadata, and the PackageSource is repointed at it, annotated
keep and disowned, so neither Helm nor the operator prunes it. Helm
reads that annotation off the live object during the upgrade prune,
which is what makes disowning in a pre-upgrade hook work at all.

Names are deliberately unchanged: the ApplicationDefinition inside that
artifact hardcodes its chartRef, so renaming the PackageSource would
leave every release pointing at an artifact nobody produces.

A cluster with no instance keeps nothing. The naas bundle installs the
rd component regardless of use, so freezing unconditionally would strand
a pinned source on clusters that never touched the application.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-16 11:00:54 +02:00
Timur Tukaev 2192547f1e docs(governance): remove the unused Directors role
GOVERNANCE.md defined a Directors role (non-technical leaders, elected by the maintainers, one vote in decisions), but no Directors were ever identified or elected. The role was intended for non-technical leadership, but those people ended up maintaining parts of the project directly, so it stayed unused. Remove it so the documented decision-making body matches actual leadership (maintainers, by consensus with a simple-majority fallback); it can be reintroduced if a real need arises. Addresses cozystack/community#79.

Signed-off-by: Timur Tukaev <6355522@gmail.com>
2026-09-16 08:45:41 +05:00
Andrei Kvapil c0f9f08de7 chore(platform): detach HTTPCache instances before the package is pruned
Pruning the http-cache PackageSource takes its chart artifacts with it,
so without a migration every http-cache HelmRelease loses its chart and
stays Ready=False for good, with HelmReleaseNotReady firing on every
cluster that runs the naas bundle. Deleting the leftover Package by hand
is worse: it takes the ApplicationDefinition with it while the pods of a
surviving instance are still running.

Migration 57 orphans each tenant release instead, the way migration 55
does, so the workloads stay in the tenant namespace as plain objects,
and then removes the platform side. The Package goes before
hr/http-cache-rd because the operator re-creates a release it owns for
as long as that Package exists.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-16 00:34:51 +02:00
Alexey Artamonov 150e18b3d3 docs(e2e): correct the Chainsaw Test-ordering claim in etcd and clickhouse suites
The etcd suite justified per-scenario teardown with a claim that Chainsaw
runs Tests in name order, not file order. That was wrong, but restating it
as document order is wrong too: at the pinned v0.2.15 Chainsaw guarantees
no order between Tests at all.

The runner registers every Test through testing.T.Run and the body calls
t.Parallel() whenever spec.concurrent is nil or true — the case in both
suites, since neither sets it. Go pauses those subtests and releases them
together once the parent returns, and execution.parallel maps only to
-test.parallel, a concurrency cap rather than an order. So parallel: 1
serialises the Tests without ordering them: not file, not document, not
name. The E2E run on this branch's own head shows it — clickhouse-2 ran
before clickhouse-1 with both etcd Tests interleaved between the clickhouse
pair.

The teardown decision the false claim justified still holds, for a reason
that does not touch order: each etcd scenario is self-contained under
skipDelete: true with a step-level finally, and v0.2.15 has no test-level
finally, so a scenario is reclaimed only if it reclaims itself — true of
whichever Test runs last regardless of order. Drop the ordering clause from
the etcd teardown rationale and the ordering sentence from the clickhouse
header rather than restating either. Comment-only; no behaviour changes.

Verified against Chainsaw v0.2.15 source: pkg/runner/runner.go (t.Run plus
t.Parallel gated on spec.concurrent) and pkg/runner/flags/flags.go
(execution.parallel maps to test.parallel).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-15 23:35:32 +03:00
Alexey Artamonov 42a27567ce fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.

Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.

Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.

Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.

Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.

Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.

Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.

Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.

A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.

Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-15 23:34:52 +03:00
IvanHunters b99db147a7 test(ci): isolate the commit-trailer fixture from global tag signing (#4271)
## What this PR does

`make_repo` in `hack/check-commit-trailers.bats` silences
`commit.gpgsign` in the fixture repository but leaves `tag.gpgsign`
alone. Anyone who signs tags has that setting globally, it reaches the
fixture, `git tag start` becomes an annotated tag, and git refuses it
for want of a message. The helper opens all 27 tests, so the whole suite
dies before a single test reaches its body, and `make bats-unit-tests`
takes `make unit-tests` down with it. Details and the reproduction are
in #4270.

The first commit is that one line, next to the one that is already
there. With it the suite went 27/27 on a machine where the global
setting is set, and was unchanged on one where it is not. I ran it both
ways.

CI never saw this. The `Commit trailers` job runs the checker against
explicit SHAs and does not touch the bats file, and a fresh runner has
no global git configuration to leak in. The failure only lands on the
local `make unit-tests` that `docs/agents/contributing.md` asks for
before pushing, which is the worst place for it: an unexplained red in a
file the contributor did not touch.

That is also how the guard went missing in the first place, and it would
go missing again the same way, so the second commit adds `make_repo
holds against a hostile global configuration`. None of the three `git
config` lines in `make_repo` is exercised on a runner: delete any of
them and CI stays green. The test supplies the hostile configuration
itself rather than waiting for a contributor to have one, so each guard
is held by the suite. Pulling them one at a time gives three distinct
reds: `tag.gpgsign` gives `fatal: no tag message?`, `commit.gpgsign`
gives `gpg failed to sign the data`, and `core.hooksPath` lets a global
`commit-msg` hook rewrite the message and fails the assertion on it. The
suite is 28/28 both with the global settings and without, so a runner
sees no change.

`hack/release-changelog-behaviour.bats` has gaps of the same kind in its
own fixture, which sets `commit.gpgsign` and nothing else. The missing
`tag.gpgsign` is inert there because it creates no tag. The missing
`core.hooksPath` is not: a global `pre-commit` hook that exits non-zero
kills every `git commit` it makes. I left both alone rather than widen
this diff, and noted them in the issue.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. The only `hack/`-related
trigger is `cozystack/ccp`, and it fires on moving or renaming something
under `hack/` or changing what a make target does. This adds a line
inside an existing file; the filename and the make target behave as
before, except that the target stops failing.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Repository fixtures now remain reliable despite conflicting global Git
settings, including commit and tag signing, failing signing tools, and
commit-message hooks.
- Fixture setup preserves lightweight tags and the original seed commit
message, ensuring consistent repository initialization across
environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-15 15:09:10 +03:00
IvanHunters a19fd0274f fix(kafka): delete release topics before the topic operator is removed (#3938)
## What this PR does

Fixes Kafka application deletion when a release-owned KafkaTopic still
has Strimzi's `strimzi.io/topic-operator` finalizer. Without ordering,
Helm removes the topic operator with the release, the KafkaTopic cannot
finish deleting, and reinstalling the same release name times out on the
leftover object (#3793).

A namespace-scoped pre-delete hook deletes only KafkaTopics carrying the
current release label while the operator is still running. It never
removes finalizers directly. The hook uses a digest-pinned kubectl
image, restricted RBAC, a 270-second job deadline, a 180-second deletion
timeout, and a 20-second request timeout. The E2E absence checks allow
five minutes, matching the hook's Helm budget.

A wait that times out with the finalizer still held exits 0 on purpose,
so the uninstall can finish rather than wedging the release under
helm-controller's retry loop. That is a deliberate trade and the hook
says so on stderr; the Job survives an hour under
`ttlSecondsAfterFinished` so the record outlives the run.

Validation:

- `helm unittest packages/apps/kafka`: 37/37 passed, covering the hook
resources, RBAC wiring, the digest pin and the
`_cluster.images-registry` override, the release-scoped selector, and
the Job deadline and delete-policy settings.
- `hack/cozytest.sh hack/kafka-pre-delete-hook.bats`: 5/5 passed. The
suite renders the hook and runs the script against a fake kubectl,
because every property worth pinning there is an exit code and matching
manifest text cannot establish one. Run under the runner `make
bats-unit-tests` uses rather than the `bats` binary, which do not accept
the same test file.
- Both suites were mutation-checked: putting the timeout branch back to
`exit 1`, flipping the catch-all to `exit 0`, dropping
`--ignore-not-found`, inverting the guard, widening the selector to a
sibling release, and setting `--wait=false` each turn one of them red.
- The raw template parses with `yq`; all leading Helm-expression scalars
are quoted.
- Chainsaw asserts both release topics are absent after deletion, but
the full E2E suite was not run locally because it requires a disposable
cluster.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the chart hook and E2E test. No
values/schema/API/default or downstream hardcoded contract changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(kafka): delete release-owned KafkaTopics before removing the Strimzi topic operator
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Kafka deletion now automatically removes associated Kafka topics
before the Kafka resource is removed.
- Cleanup waits for topics to disappear before checking ZooKeeper
storage reclamation.
- Deletion remains resilient when topic resources or their definitions
are absent, or when finalizers delay removal.
- Cleanup failures are reported appropriately without blocking deletion
when a timeout occurs.

- **Tests**
- Added coverage for cleanup behavior, execution settings, permissions,
and failure scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-15 15:00:37 +03:00
IvanHunters 379c763fc3 test(ci): drop an inert key from the fixture's hostile configuration
tag.forceSignAnnotated governs annotated tags and `git tag <name>` creates
none, so with tag.gpgsign unset the tag still resolves to a commit object. The
key cannot make the new test fail and guards nothing the fixture could lose;
carrying it implied a fourth guard that does not exist.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:52:49 +03:00
IvanHunters ef3bfd71d3 docs(ci): record what actually narrows the overlay key branch
Two comments credited the trailing colon with keeping imagePullPolicy and
imagePullSecrets out of the key branch. It does not: the branch requires a
letter before image, which a key beginning with image has nothing to supply, so
neither matches with the colon or without it, even when the value carries a
slash. The colon earns its place elsewhere, as the point the value scan starts
from, and removing it reddens the suffixed-key test rather than the pull-policy
one.

A reader acts on the recorded reason, so a wrong one costs more than none.

Refer to that test by name rather than by ordinal: the number comes from the
runner's output order and moves when anything is inserted above it.

While here, give the pull-policy test a failure message about pull policies:
it read 'a PR-edited package was overlaid', which belongs to a different case.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:50:19 +03:00
IvanHunters 609a014cc4 fix(ci): make the clean-run overlay test able to fail, and narrow the key branch
`! grep -q …` is a pipeline preceded by `!`, which POSIX exempts from set -e,
and hack/cozytest.sh appends `return 0` to every converted test function, so a
failed negation was discarded twice over. That line was the only assertion in
'no annotation when nothing drifted': mutating the drift>0 guard to `true` left
the suite green under both runners. Replace all three negations with an
explicit if/exit; the mutation now reddens test 19.

The prefixed *Image branch matched any key ending in image, including
operator-settable ones the build never stamps: vddkImage (core/platform,
migration-controller) defaults to "" and a change there was absorbed from the
artifact in silence, the opposite of what this script is for. Require the value
to hold a repository path, which a ref always does and "" does not.

The annotation counted ref-bearing files while naming deduplicated dirnames, so
one package with a drifted values.yaml and a drifted images/*.tag read as two,
and named <pkg>/images, which is not a package. Count the named list instead.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:49:52 +03:00
IvanHunters e9f666bc25 fix(ci): stop a blank line lost to yq from costing a package its overlay
The overlay keeps a package on its committed release ref when the artifact copy
differs from the repo on any line that is not image-reference-bearing. A blank
line is such a line, and the build produces one on its own: yq -i re-emits the
document when it stamps a digest, and a blank line separating two top-level keys
does not come back.

So a file the PR never touched differs from the artifact on a line carrying no
configuration, and loses its overlay for it. That was the whole of the remaining
drift list. Against the public nightly artifact at its own revision, with the
tree checked out at that same revision:

  before  drift=5  overlaid=29   core/platform, cozystack-api, dashboard,
                                 kubeovn-webhook, linstor-gui
  after   drift=0  overlaid=34

Each of the five differed from the artifact by one blank line and its own ref,
nothing else, which is reproducible locally by running the stamping assignment
over the committed file. cozystack-api and dashboard are not peripheral: an e2e
result about either was a result about the last release rather than about main.

Discount blank lines before the test. A line with content still counts, so a
real config difference keeps the committed ref as before.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:49:52 +03:00
IvanHunters 80a41e4e6f test(ci): stop the overlay tests installing EXIT traps, and pin the key branch
hack/bats-no-exit-trap.bats audits every .bats file under hack against the debt
its own header declares, and this file was installing three traps beyond it.

An EXIT trap inside an @test body replaces the one the bats binary installs for
its own bookkeeping, so a test that then fails prints no TAP line at all and the
suite reads as green. That is the reason the count is exact rather than a
ceiling, and the reason the answer here is to stop installing the handler rather
than to raise the number. Clean up at the end of the body instead: both runners
set -e, so on failure the cleanup is unreachable and the scratch directory
survives for inspection.

The key branch had nothing holding its shape. A test now pins each half: that a
suffixed key such as clientImage is read as a ref, and that a key merely
beginning with image is not.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:49:51 +03:00
IvanHunters 8ac99b67cc fix(ci): read a suffixed image key as a ref, not a config change
The classifier accepted a key only when it was exactly image, repository,
registry, tag or digest. A chart that renders several images names the extras
after what they are for, so backupstrategy-controller carries
chBackupClientImage, rabbitmqBackupClientImage and redisBackupClientImage. The
first two reach the classifier through the @sha256: branch; the third is pinned
by tag alone, so it read as a config change and cost the WHOLE file its overlay.

That package therefore kept its committed release ref in every lane that did not
rebuild it, and served an image from v1.6.0 (22 July) against a tree carrying
four backup strategy drivers added after it. Five of the seven chainsaw failures
behind #4257 trace to that, across three unrelated pull requests.

Accept a prefix before image in the key branch.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:49:46 +03:00
IvanHunters 618ab58482 ci(e2e): name the packages that kept release refs instead of current main
A file whose config differs from the artifact beyond its image lines keeps the
committed ref. That is the safe answer, and it is not a neutral one: the
component then runs its last release image in a lane whose job is to exercise
the tree under review, and the only trace is one line among hundreds of overlay
lines.

Six packages took that path on a recent run, backupstrategy-controller among
them, which is why four backup round-trip suites refused Kinds the tree
supports and three unrelated pull requests each spent a round establishing the
red lane was not theirs (#4257).

Summarise the drifted packages in one GitHub annotation. Nothing about which
files are overlaid changes; a run with no drift emits nothing.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:49:38 +03:00
IvanHunters 80ab355d66 test(ci): hold the fixture's config guards from CI
make_repo silences commit.gpgsign, tag.gpgsign and core.hooksPath because a
contributor's global configuration reaches a fresh repository. A runner carries
no such configuration, so none of the three is exercised there: the suite stays
green with any of them deleted, and the failure lands only on the machine of
whoever signs their commits or tags. That is how tag.gpgsign came to be missing
without anything going red.

The new test supplies the hostile configuration itself, so removing a guard now
fails in CI rather than on a contributor's first local run.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 14:37:29 +03:00
IvanHunters 5a777e9cfc test(ci): isolate the commit-trailer fixture from global tag signing
make_repo silences commit.gpgsign in the fixture repository but not
tag.gpgsign, so a contributor who signs tags has that setting reach the
fixture. git tag start then becomes an annotated tag, git refuses it for
want of a message, and since the helper opens all 27 tests the suite
fails before a single one reaches its body, taking make unit-tests with
it.

CI never saw it: the Commit trailers job runs the checker against
explicit SHAs and does not touch this file, and a fresh runner has no
global configuration to leak. The failure only ever lands on the local
run that contributing.md asks for before pushing.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 13:59:45 +03:00
IvanHunters 637098f222 test(kafka): anchor the release selector the hook suites assert on
Both suites matched "strimzi.io/cluster=kafka-test" as a bare substring,
and that is a prefix of the sibling release kafka-test2 and of
kafka-test-topics. Rendering the hook with either selector left both
suites green, so neither pinned the scoping they exist to pin. Anchor on
the leading "-l " and on the trailing delimiter; both variants now red in
both suites, and the six earlier hook mutations still red exactly where
they did.

Drop the notMatchRegex on a leading slash. It cannot fail on its own: the
anchored image pattern above rejects a leading slash first, and an empty
_cluster.images-registry renders no slash to begin with. Its rationale
moves into that pattern's comment.

Record why activeDeadlineSeconds and the 180s wait have to move together.
A deadline that fires mid-wait kills the Job, and with backoffLimit: 0 the
hook then returns non-zero, which is the wedged uninstall the deliberate
exit 0 on the timeout branch exists to avoid.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 13:48:08 +03:00
IvanHunters 415aa82409 test(kafka): drive the hook suite through the runner CI actually uses
The suite used bats' run helper. make bats-unit-tests does not invoke the bats
binary: it runs hack/cozytest.sh over every hack/*.bats, and that runner
implements @test and little else, so the file died at 'run: command not found'
and took make unit-tests with it. Under bats it was 5/5, which is where the 5/5
in the previous commit came from. Save $? by hand instead, the shape
app-cleanup-hooks_test.bats already uses; verified under both runners.

The fake kubectl no longer defaults DELETE_RC to 0. An unset value made every
failure-path assertion vacuous; now it exits 64 and says so. Cover both
RESTMapper wordings for an absent CRD rather than just the first.

Widen the image pattern to v[0-9]+\.[0-9]+(\.[0-9]+)? like harbor and mariadb:
pinning v1\.32 reintroduced the breakage the digest was made pattern-based to
avoid, one Renovate bump later. Assert the _cluster.images-registry override
and the no-leading-slash default, neither of which was covered: the hook is the
one image in this chart whose pull failure surfaces at uninstall.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 13:47:04 +03:00
IvanHunters 3326e0fb9e test(kafka): run the pre-delete hook against a fake kubectl
The hook's exit code is the whole contract: a non-zero one returns from
Uninstall.Run before any release resource goes, keeping the topic operator
alive; a zero one lets Helm remove it. delete_test.yaml matches manifest text,
so it stayed green when the timeout branch was put back to exit 1, when the
CRD-absent branch was, and when the selector was widened past this release.

Add a bats suite in the shape hack/app-cleanup-hooks_test.bats already uses for
qdrant and harbor: render the script, run it against a fake kubectl, pin the
exit code of each branch. Each of the three mutations above now turns it red.

Drop hook-succeeded from the delete policy and reclaim the Job with
ttlSecondsAfterFinished instead. The script exits 0 on the degraded path too,
where topics still hold the finalizer, so deleting on success removed the only
record of that seconds after it was written. Assert both in delete_test.yaml.

Pin the hook image by shape rather than by literal digest: Renovate rewrites
digests under packages/*/templates/hooks/ and nothing rewrites tests/, so the
literal turns the next bump red for no defect.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-15 13:47:04 +03:00
Yan Bondarenko 210b7fe30c fix(kafka): stop the pre-delete hook from wedging the uninstall
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-15 13:47:04 +03:00
Yan Bondarenko 9d2fddd4e6 test(kafka): assert release topics exist before the deletion check
The post-delete absence assertions could pass vacuously if the two
release topics were never created. Asserting both topics first makes
the E2E prove that the pre-delete cleanup actually removed them.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-15 13:47:04 +03:00
Yan Bondarenko 44ba13b527 fix(kafka): delete release topics before the topic operator is removed
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-15 13:47:04 +03:00
Aleksei Sviridkin 070dcb1e30 fix(e2e): wait for the redis replica link instead of sampling it once (#4252)
## What this PR does

The `verify-tenant-ca-projection` step in the redis suite read `INFO
replication` from both pods once and failed the run when a replica
reported `role:slave` without `master_link_status:up`. That value is
still in motion at that point, so one sampling decided the result.

Redis reports the link up only once a replica reaches
`REPL_STATE_CONNECTED`. The operator's readiness script admits a replica
on two other predicates: `master_host` being off `127.0.0.1`, and
`master_sync_in_progress` not being `1`. `master_host` is set the moment
`REPLICAOF` is issued, ahead of the link coming up, and
`master_sync_in_progress` is `1` only during the transfer phase, so
every handshake state between the two is ready and unlinked. A dropped
link re-enters the same window. `readyReplicas: 2` therefore establishes
nothing about the link, and the comment on the step calling the check
deterministic was wrong.

The step now polls the topology to a bounded deadline and fails only if
it does not settle, reporting each pod's observed state and the elapsed
time. Nothing models the link declaratively: the RedisFailover CRD
defines no status at all. So the poll carries the
`TODO(e2e-replace-fixed-timeouts):` marker the e2e conventions require
for a wait with no object behind it.

What the step does now is wait for convergence where it used to sample
once. No claim about flake rate follows from that. A green run says
nothing about a failure that shows up in a minority of runs.

Shell embedded in a chainsaw step has no unit-test surface in this
repository, so this change is exercised by the suite and nothing else.

Two pre-existing diagnostics defects in the same file turned up while
reading it, tracked in #4246 and #4250. Neither is touched here. The
first one matters a little more after this change: the poll runs its
read path about a dozen times per step instead of twice.

Closes #4238

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map against the diff rather than the title. The
change edits one file under `hack/e2e-chainsaw/`, moves and renames
nothing, and alters no make target, so none of the listed triggers fire.
The node-prerequisite trigger that reaches `ansible-cozystack` and
`talm` is `hack/e2e-prepare-cluster.bats`, which this does not touch.

### Release note

```release-note
fix(e2e): the redis TLS suite waits for the replica link to come up instead of reading it once, so a link that is still establishing no longer fails the run
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Tests**
- Improved Redis TLS topology validation by allowing replication status
time to stabilize before verification.
- Extended the test operation timeout to reduce failures caused by slow
environment startup or synchronization.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-15 12:09:21 +04:00