mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 04:33:36 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
7777
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
94885c2e93 |
docs(governance): remove the unused Directors role (#4296)
## What this PR does Removes the **Directors** role from `GOVERNANCE.md`. The role (non-technical leaders, elected by the maintainers, one vote each in decisions) was defined but never used — no Directors were identified or elected. It was intended for non-technical leadership, but those people ended up maintaining parts of the project directly, so the role stayed empty. Removing it so the documented decision-making body matches actual leadership (maintainers, by consensus with a simple-majority fallback). It can be reintroduced if a real need arises. Addresses cozystack/community#79. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note docs(governance): remove the unused Directors role from GOVERNANCE.md ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated governance guidance to remove the Directors role and related leadership descriptions. * Clarified that voting is conducted by maintainers. * Updated consensus language to reflect the revised decision-making process. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a295e29b8a | fix(kafka): clean up pre-delete hook ServiceAccount and RBAC on success | ||
|
|
353f37e4fe |
test(kafka): widen the fresh-create e2e cleanup budget for KRaft
The fresh-create Kafka suite relied on the shared 5m cleanup default, but a KRaft cluster tears down broker and controller node pools, the entity operator, and their PVCs — more work than a single ZooKeeper node. Under a contended CI run (concurrent Postgres/OpenSearch/Redis suites) that teardown overran 5m and failed the suite after every assertion had already passed, so cleanup latency, not the test, was deciding the verdict. Give this suite its own 8m cleanup timeout; the migration suite runs in its own namespace and is unaffected. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6e18ceef53 |
fix(kafka): fail closed when the migration hook cannot list Kafka CRs
The "free the kafka pool" gate counted other ZooKeeper clusters through a `kubectl ... | awk` pipeline under `set -eu` without `pipefail`, its kubectl error dropped by `2>/dev/null`. A failed read then counted zero and kept the shared pool, stranding any co-namespaced cluster still waiting for the name. Separate the read status from the count, mirroring the drain read below, and cover it with a bats case. Assisted-By: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
cd9f4d6a27 |
feat(kafka): auto-migrate every ZooKeeper Kafka in a namespace to KRaft
Extends the ZooKeeper->KRaft migration hook so that N Kafka clusters sharing one namespace all reach KRaft automatically, with no manual steps. This closes the last gap in the migration: previously only one cluster per namespace could migrate in place (the broker pool must be named exactly "kafka" to adopt the existing <cluster>-kafka-N brokers, and that name is namespace-unique), so a second co-namespaced ZooKeeper cluster hit the render guard and its HelmRelease never rendered again -- stuck on ZooKeeper with no way forward once a future Strimzi drops the migration. The clusters now serialize on the "kafka" name automatically. Whichever upgrade wins the name (claimed atomically via kubectl create) migrates in place; a cluster that loses fails its hook fast and its next reconcile retries. Once a migrated cluster sees other ZooKeeper clusters still waiting, it frees the name: it stands up its own release-hashed "b-<hash>" broker pool (node IDs pinned clear of the existing broker and controller ranges), enables Cruise Control, drains every partition off the "kafka"-pool brokers with a remove-brokers KafkaRebalance, deletes the drained pool, and removes Cruise Control again. The next waiting cluster's retry adopts the freed name; the last one to migrate keeps "kafka". Everything happens inside the one namespace -- the hook's Role is namespaced and it never reads or touches another. Robustness details learned from live runs: - The drain rebalance sets skipHardGoalCheck: true. A mandatory broker evacuation must not be blocked by Cruise Control's capacity hard goals; under a small resources preset the capacity model can briefly over-estimate CPU from noisy samples and reject the proposal with CpuCapacityGoal even when real CPU is near idle. - The drain wait refreshes the proposal (strimzi.io/rebalance=refresh) on any NotReady and keeps polling, rather than treating it as terminal: a just-enabled Cruise Control is unreachable for a minute or two, and its model settles as metric windows age out. This also makes the drain resumable -- a re-run refreshes a stale rebalance instead of reading its old failure forever. - Cruise Control is enabled with explicit resources (it is a JVM and a tenant LimitRange would OOM it at 128Mi), transiently -- kafka.yaml never sets spec.cruiseControl, so Helm's server-side apply does not fight the hook patching it on and off. Every phase is idempotent so a Job killed mid-drain (the drain scales with data size and can run for hours) resumes on retry; the Job deadline and the release's helm-install-timeout are raised accordingly. Verified on a live cluster across five runs of two co-namespaced ZooKeeper Kafkas migrating to KRaft: both reach KRaft automatically, the first drains onto its b-<hash> pool and frees the name, the second adopts "kafka" in place (broker node IDs and PVCs unchanged), and each cluster's data is byte-for-byte identical before and after. helm-unittest 87/87 and hack/kafka-migration-hook.bats 14/14 (the real hook script against a stub kubectl) cover the freeing decision, the atomic name claim, the drain's refresh/skip-hard-goal handling, and the "delete the pool only after the rebalance is Ready" guard. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5cbf16088f |
fix(api): cap Kafka app name to the KRaft controller pod-hostname budget
A Kafka cluster's KRaft controller pods are named "<release>-c-<hash>-<id>" (8-hex hash, id at least one digit), and that derived pod hostname must fit the 63-char DNS-1123 label limit. That is tighter than the 53-char Helm release-name budget validateNameLength enforced, so a release name of 51-53 characters was admitted but then failed the chart render (kafka.assertNameLength) forever, an un-recoverable upgrade dead-end for an existing cluster whose name is immutable. Cap the Kafka app name at admission, mirroring the existing Kubernetes worker-pool cap, so the overflow surfaces on the Kafka CR the operator is editing instead of at render time on a pod Strimzi can never create. The chart's render guard remains the second line of defense and also catches larger clusters whose higher node ids need more digits. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
cb7578bb31 |
fix(kafka): data-safe, Helm-clean ZooKeeper->KRaft migration
Two live-reproduced blockers in the in-place ZK->KRaft migration are
fixed, plus the concurrency guard behind them.
1. Data loss (node IDs). The migration created the broker and controller
KafkaNodePools together without pinning node IDs. On Strimzi 0.45 the
controller pool deterministically took the existing broker IDs
[0..N-1], so the brokers were recreated empty at [N..] and their data
PVCs were orphaned (reproduced 2/2 on dev9: 20000 messages -> 0,
partitions leaderless, migration wedged in KRaftMigration). Naming the
broker pool "kafka" is necessary but NOT sufficient: Strimzi adopts by
node id, not pool name. The controller pool now pins
strimzi.io/next-node-ids above the broker range, so the broker pool
keeps [0..N-1] and adopts the existing <cluster>-kafka-N brokers and
their data in place.
2. Release stuck "failed" (Helm lifecycle). The pre-upgrade hook creates
the pools, and the chart also declared them as regular manifests, so
Helm's main apply tried to reconcile a resource that exists in the
cluster but was absent from the previous (ZooKeeper) release revision
and aborted with "original object KafkaNodePool ... not found",
leaving the HelmRelease failed even though the cluster migrated fine
(reproduced 3/3 under server-side apply, the production
helm-controller path). This is unavoidable while the pools are Helm
main-manifest resources across the ZK->KRaft boundary, so the pools
are now managed entirely by the migration Job (a pre-install,
pre-upgrade hook) on every install and upgrade, and removed from the
regular manifests. The Job ensures both pools (creating or updating
them idempotently), then drives the migration state machine.
templates/kafkanodepools.yaml and the deployed-version gate ConfigMap
(templates/version.yaml) are removed; the broker pool name is resolved
per path by kafka.brokerPoolName ("b-<hash>" fresh, "kafka"
migrating), and the render-time name-length guard moves to the Kafka
CR template.
3. Concurrency (one migration per namespace). The collision guard read
the "kafka" pool owner and then applied, a non-atomic read-then-write
two concurrent per-namespace migrations could both pass. When the pool
is absent the hook now uses "kubectl create" as the atomic arbiter:
exactly one concurrent creator wins, the loser re-reads the owner and
aborts.
Verified live on dev9 under server-side apply: a fresh install comes up
KRaft with pools at [0,1,2]/[3,4,5] and the release is "deployed"; an
in-place migration of a 20000-message ZooKeeper cluster ends "deployed",
adopts the broker PVCs in place (same UIDs), and the KRaft data is
byte-for-byte identical to the pre-migration ZooKeeper data (matching
sha256 over every partition:offset:key:value). helm-unittest 86/86 and
hack/kafka-migration-hook.bats 12/12 (the real hook script against a stub
kubectl) cover the resolution branches, the fail-closed reads, the atomic
collision guard and the finalize guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
1a5cb049ae |
fix(kafka): match KRaft controller dashboard panel to hashed pool name
The controller commit-latency panel selected controller pods with pod=~"$strimzi_cluster_name-controller-.*", but the controller node pool is named c-<hash> (kafka.controllerPoolName), so Strimzi names controller pods <cluster>-c-<hash>-<id>. The literal -controller- never matched, so the panel was permanently empty. Select the pool by its -c- token instead. Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
053ef16976 |
feat(kafka): allow multiple KRaft Kafka clusters per namespace
Fresh KRaft clusters no longer need to be alone in a namespace. Their node pools get short, release-hashed names — "b-<hash>" for brokers, "c-<hash>" for controllers — instead of the fixed "kafka"/"controller". Each release hashes differently, so the names are unique within the namespace (KafkaNodePool object names are namespace-unique) and any number of Kafka instances coexist. The hash keeps the pool name a fixed length: Strimzi derives every pod/PVC name as <cluster>-<pool>-<id> and the cluster name is already the release name, so embedding the release name in the pool name too would double it into every pod hostname and blow the 63-character DNS label limit for long release names. A render-time guard fails with a clear message when even the hashed controller node name would still exceed 63 characters. The fixed "kafka" name is still required to adopt an existing ZooKeeper cluster's <cluster>-kafka-N brokers and data in place during migration, so kafka.brokerPoolName resolves to "kafka" for a migrating/migrated cluster and "b-<hash>" for a fresh one. A second ZooKeeper->KRaft migration into a namespace that already holds a "kafka" pool owned by another cluster fails closed, both at render time and in the pre-upgrade hook (whose owner probe uses --ignore-not-found + an explicit exit-code check, so a transient kubectl error aborts rather than silently waiving the collision guard). This replaces the previous blanket "one Kafka per namespace" guard, which over-restricted the fresh case. The pool names are resolved from a live Helm lookup; that is safe because Helm propagates any non-NotFound read error and fails the whole render (pkg/engine/lookup_func.go), so a transient apiserver/RBAC miss retries rather than silently flipping a live cluster's pool name — an empty lookup means genuine absence or a never-applied client-side render. The migration hook and workload monitors follow the same naming. Tests: helm-unittest fixtures (via kubernetesProvider) cover every resolution branch and the fail-closed guards; hack/kafka-migration-hook.bats drives the migration Job's real shell script against a stub kubectl across the fresh / read-error / already-KRaft / owner-collision / happy-path / stuck / retry cases. Verified on a live cluster: in-place ZooKeeper->KRaft migration preserved all data and adopted the original broker PVCs, and two fresh clusters coexist with distinct pools, pods and PVCs. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
03619f3ad2 |
test(kafka): drop source-grepping migration-timeout drift-guard
The removed bats file grepped two static literals in two separate packages (helm-install-timeout in system/kafka-rd, activeDeadlineSeconds in apps/kafka) and asserted an ordering between them. Substring matching cannot verify the contract it targets: the real invariant is between the running migration Job and the release timeout helm-controller actually enforces on the pre-upgrade hook, not between two YAML literals in the tree. It also greened only for the minute-format literal and errored on any legitimate reformat (e.g. "1h" / "2700s"). The timeout > Job-deadline relationship is already documented inline at both values. Its machine-checkable home is an e2e that upgrades an existing release through the HelmRelease pre-upgrade hook and confirms a real migration completes inside the release timeout (the chainsaw suite drives migration via kubectl, bypassing that path). Signed-off-by: Andrey Kolkov <androndo@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
dae4d95444 |
fix(kafka): replace dead ZooKeeper panel with KRaft controller metrics
The strimzi-kafka Grafana dashboard still had a 'ZK request latency' panel (kafka_server_zookeeperclientmetrics_zookeeperrequestlatencyms), which goes permanently empty after the ZooKeeper->KRaft migration. Replace it with a KRaft controller commit-latency panel (kafka_server_raftmetrics_commit_latency_avg, scraped from the controller pods). Metric names verified against a live KRaft cluster. The PVC-exclusion filter that references zookeeper stays valid and is left as is. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1a68157564 |
fix(kafka): harden migration hook and isolate the migration e2e
Addresses the second review round on the ZK->KRaft migration. Migration hook: - Fully qualify the Strimzi resources as kafkas.kafka.strimzi.io / kafkanodepools.kafka.strimzi.io. The bare plural "kafkas" also exists in this chart's apps.cozystack.io group, so a bare name resolves by per-cluster discovery priority and can hit the tenant kind — which the hook Role does not grant, failing the pre-upgrade hook deterministically. - Classify NotFound structurally via `kubectl get --ignore-not-found` instead of substring-matching stderr, so an unrelated "... not found" error can no longer be mistaken for a fresh install and permanently skip a live ZooKeeper cluster. This also drops the jq dependency. - Treat a KRaft-native / already-migrated CR (annotated strimzi.io/kraft=enabled) as done, never driving it backwards into kraft=migration. - Keep only `before-hook-creation` on the Job's hook-delete-policy so a failed attempt's pod logs survive for a post-mortem until the next run. - Route the kubectl image through cozy-lib.image (cluster registry mirror) and pin it by digest (clastix/kubectl:v1.32@sha256:...), matching the other cozystack hooks, instead of an unpinned docker.io tag. Controller quorum: - Fix the controller node-pool replica count at creation instead of deriving it from kafka.replicas on every reconcile. Strimzi 0.45 cannot scale a controller pool, so a day-2 edit of kafka.replicas (e.g. 1 -> 3) must not flip the quorum; the helper now keeps the existing "controller" pool's replica count when it is present. Docs: - Document that migration to KRaft is one-way: neither a version-field downgrade nor a pre-KRaft chart rollback is supported once migrated (Strimzi has no KRaft->ZooKeeper rollback after completion). E2E: - Run the ZK->KRaft chainsaw migration in its own namespace instead of the shared tenant-test. It raw-applies Strimzi resources plus a namespace-wide LimitRange and leaves deleteClaim:false PVCs, which polluted the sibling kafka suite. Raise the seed readiness timeout for CI headroom. Validated against a live cluster: migration adopts the broker PVC in place and reaches KRaft under a 128Mi LimitRange, in isolation. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f9676961f3 |
fix(kafka): set resources on the migration node pools
The ZK->KRaft pre-upgrade hook created the adopting broker ("kafka") and
"controller" KafkaNodePools without a resources block. Those hook-created pools
govern the Kafka pods for the entire migration window — Helm applies the
resourced steady-state kafkanodepools.yaml only after the pre-upgrade hook
finishes. In a tenant namespace the LimitRange defaults every container to
128Mi, so the migration's controller (and rolled broker) JVMs OOM, never reach
KRaftPostMigration, and the hook times out — wedging the migration and thrashing
it on every reconcile.
Give both hook pools the same resources the steady-state pools get
(cozy-lib.resources.defaultingSanitize from kafka.resourcesPreset/resources).
This also avoids a redundant second broker roll (resource-less -> resourced)
after the hook completes.
Tests: a helm-unittest assertion that both migration pools render a resources
block; the Chainsaw ZK->KRaft suite now seeds under a 128Mi LimitRange (and
gives the seed cluster explicit resources), reproducing the tenant-quota
condition, and asserts the migration still reaches KRaft with the broker PVC
adopted. Validated green against a live cluster under the LimitRange.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
649f7e0b63 |
fix(kafka): enforce one Kafka per namespace; simplify node pool names
Follow-up to the KRaft-migration review. The broker KafkaNodePool must be named exactly "kafka" so Strimzi adopts an existing ZooKeeper cluster's <cluster>-kafka-N brokers and data in place during migration, but KafkaNodePool object names are unique within a namespace — so two Kafka clusters cannot both own a "kafka" pool there. This is an upstream constraint and Strimzi's own guidance is one Kafka cluster per namespace (strimzi/strimzi-kafka-operator discussions/11120). Adopt that model instead of the earlier fresh-vs-migrated conditional naming: - New kafka.assertSingleInstance helper fails the render (with a clear message and the upstream reference) if the namespace already holds another Kafka cluster. Reconciles/upgrades of the same release are allowed; lookup returns nothing during `helm template`, so unit tests never trip it. - Node pools are now named "kafka" (broker) and "controller" unconditionally, which also adopts existing brokers on migration and yields clean <cluster>-kafka-N / <cluster>-controller-N names. The fresh-vs-migrated brokerPoolName helper is removed. - WorkloadMonitor pool-name selectors and the migration hook updated to match; unit tests updated for the fixed pool names. - README documents the one-Kafka-per-namespace rule and links the Strimzi discussion. The ZK->KRaft Chainsaw migration suite (seed ZooKeeper cluster, migrate, assert the original broker PVC survives and no wrongly-named PVC is created, cluster reaches KRaft) was run green against a live cluster. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
710cdd18bb |
fix(kafka): address KRaft-migration review — broker pool adoption, hook safety, tests
Addresses the branch review on the ZK->KRaft migration path. CRITICAL — broker node pool adoption. Strimzi derives broker/PVC names as <cluster>-<pool>-<id>; a pre-node-pool ZooKeeper cluster has brokers <cluster>-kafka-N, so the adopting broker pool must be named exactly "kafka" or the operator provisions fresh empty brokers and orphans the tenant's data. A new kafka.brokerPoolName helper resolves the broker pool to "kafka" for an existing (migrating) cluster and keeps the release-scoped "<release>-broker" for fresh installs, so multiple Kafka instances still coexist in one namespace. The migration hook creates the pool as "kafka" and fails closed if a "kafka" pool already belongs to another cluster (node pool names are namespace-unique, so in-place migration of co-namespaced ZK clusters is unsupported). Wired through kafkanodepools.yaml, workloadmonitor.yaml and the hook. MAJOR — migration robustness. - Raise the HelmRelease timeout: the migration Job can run tens of minutes but the release timeout defaults to 10m, so helm-controller aborted and thrashed the pre-upgrade hook. Add release.cozystack.io/helm-install-timeout=45m on the kafka ApplicationDefinition, above the Job's activeDeadlineSeconds (2400s). - Idempotent migration: do not re-issue kraft=migration once the cluster is past the ZooKeeper phase (an invalid backward transition Strimzi rejects on retry). - Fail closed on the decision gates: distinguish a genuine NotFound (skip) from any other kubectl error (abort), and never treat an empty kafkaMetadataState on an existing CR as "migrated". MINOR — version downgrade documented as unsupported; integer (not lexical) compare in the version gate; deleteClaim/PVC-retention and the one-per-namespace migration limitation documented in the README; docs/storage-immutability.md updated for KRaft. Tests — helm-unittest coverage for the broker pool name, the hook guards and the idempotent/fail-closed logic; a bats check for the helm-install-timeout budget ordering; and a Chainsaw ZK->KRaft migration suite that seeds a ZooKeeper cluster, drives the migration, and asserts the original broker PVC survives (not recreated) and the cluster reaches KRaft. The migration sequence and the Chainsaw assertions were validated against a live cluster. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
cbf104d6b3 |
fix(kafka): address branch-review — restore CA cert secrets, guard KRaft finalize
Follow-up to the KRaft migration addressing the branch review. - kafka-rd cozyrds: restore the two public CA certs in secrets.include (clients-ca-cert + cluster-ca-cert). The rebase had collapsed them to the bare clients-ca, which is Strimzi's clients-CA *private key* secret — a private-key exposure to the tenant and a TLS regression (clients need cluster-ca-cert to trust the broker listener). Covered by the existing hack/check-kafka-rd-secrets.bats. - migration-hook: guard the KRaft finalize step. The wait loop only exited via break on success and fell through to unconditionally apply strimzi.io/kraft=enabled on timeout — unsafe and irreversible if the cluster is still in ZooKeeper/KRaftMigration. Add a migration_ready flag and exit 1 when the safe state (KRaftPostMigration+) is not reached, leaving the deployed-version ConfigMap unstamped so Helm retries. New matchRegex assertions in migration-hook_test.yaml cover the guard. - migration-hook: raise activeDeadlineSeconds 1800 -> 3600 so a legitimately-progressing migration is not SIGKILLed before the in-script waits and error paths can run. - Extract the duplicated controller-quorum logic into a shared kafka.controllerReplicas helper (used by kafkanodepools, workloadmonitor and migration-hook) so the three stay in lockstep. - Reflow the README migration section to one line per paragraph per the repo prose rule. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
42db2ca141 |
feat(kafka): migrate from ZooKeeper to KRaft mode with automatic migration
Replace ZooKeeper-based Kafka with KRaft (Kafka Raft) metadata mode. Strimzi 0.45 is the last line that still supports ZooKeeper, so this prepares the transition before ZooKeeper support is removed upstream. Chart changes: - The Kafka CR always renders strimzi.io/kraft=enabled with node pools and no spec.zookeeper block. Storage, replicas and resources move to separate broker + controller KafkaNodePools (a homogeneous separate-pool layout works for both fresh installs and migrations; Strimzi 0.45 does not support migration into a combined pool). - Kafka version is now user-configurable via the standard files/versions.yaml pattern (v3.9 default, v3.8 available); appVersion bumped to 3.9.0. - WorkloadMonitors track the broker and controller pools (no zookeeper monitor); the dashboard RBAC grants the controller monitor and drops the zookeeper one. Migration of existing ZooKeeper clusters is handled by a Helm pre-upgrade Job (templates/migration-hook.yaml) gated by a <release>-kafka-deployed-version ConfigMap, following the established seaweedfs/etcd hook pattern. The Job creates the broker + controller pools with Helm ownership labels, drives the Strimzi ZK->KRaft state machine (node-pools=enabled, kraft=migration -> kraft=enabled once KRaftPostMigration is reached) and is skipped once the ConfigMap records version >= "1". Testing: - Helm unit tests cover the Kafka CR, node pools, migration hook, metrics, topics, version map, dashboard RBAC and workload monitors. - The e2e suite is ported to the current Chainsaw framework (hack/e2e-chainsaw/kafka): it asserts the KRaft topology (both node pools, kafkaMetadataState=KRaft), the absence of ZooKeeper, and bootstrap endpoints. Rebased onto current main: TLS listener support, entity-operator resource guards, x-cozystack-options schema annotations and the WorkloadMonitor replica-path fix are all preserved and reconciled with KRaft. Signed-off-by: Matthieu <matthieu@hidora.com> Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
928286ba42 |
fix(test): run the artifact-ignores suite under the runner's own shell
hack/cozytest.sh is `#!/bin/sh` and sources the translated suite, so on a runner where /bin/sh is dash the here-string is a parse error and the whole suite dies before its first assertion. macOS hides it: /bin/sh there is bash in sh mode and accepts `<<<`. A heredoc keeps the loop body in the current shell, so the accumulated `missing` still survives to the assertion below it. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
617cba1621 |
fix(installer): keep node_modules out of the packages artifact
`flux push artifact` walks the whole packages tree and reads neither .gitignore nor .sourceignore, so a working copy where somebody ran `pnpm install` under the vendored console ships its dependency tree inside the bundle. Helm then refuses the dashboard chart on every cluster that pulls it, because one dependency file sits above the 5MB per-file limit, and the build itself reports nothing. Passing --ignore-paths replaces the built-in list rather than extending it, so the defaults are restated alongside node_modules/. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
3615958240 |
fix(bucket): stop the ACME cleanup hook from talking to localhost, and gate the class
The hook's four kubectl calls carried --request-timeout=30s. A non-zero value there makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so every call went to kubectl's built-in default cluster and was refused. This package failed more quietly than kafka did. Each call is guarded by `|| echo WARNING` and the script sets no -e, so the hook exited 0 and printed "ACME cleanup complete."; hook-succeeded then deleted the Job that held those warnings. The ACME solver objects it exists to reap were left behind, and the only record went with the Job. Its own suite matched the rendered command including the flag, so the broken form was pinned. The gate is the point of this change. Two packages had it, neither was visible in a rendered manifest, and nothing stops the next hook from reaching for the same flag, since it reads as ordinary defensive practice. hack/check-hook-request-timeout.bats fails any non-vendored template whose kubectl call carries a non-zero --request-timeout, skipping files that name KUBECONFIG because an explicit kubeconfig is not served by the in-cluster branch. That per-file judgement is the coarse part and is written down in the file rather than left to be inferred: apps/kubernetes keeps the flag on two calls that pass their own kubeconfig, and stays green. Bounding the work is still available where it belongs: --timeout for a watch, activeDeadlineSeconds for the Job, which is what this hook already had at 120s. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
e63b8595d3 |
fix(kafka): drop the request timeout that hid the apiserver from the hook
A non-zero --request-timeout makes kubectl skip its in-cluster credentials altogether. client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that would load the ServiceAccount token never runs. kubectl falls back to its built-in default cluster, localhost:8080, is refused, and the script reads the refusal as a failed delete and exits 1. A non-zero pre-delete hook returns from Uninstall.Run before any resource is removed, so no Kafka release could finish deleting and helm-controller retried forever. Measured on the pinned image rather than argued from the docs: with the token and ca.crt mounted and KUBERNETES_SERVICE_HOST set, the same delete reaches the cluster without the flag, goes to localhost:8080 with --request-timeout=20s, goes to localhost:8080 with 1m, and reaches the cluster again with 0. The three in-tree hooks that work do not pass it. apps/kubernetes names it in a comment only, apps/harbor and apps/mariadb never mention it, and apps/tenant records that its DELETE calls "carry kubectl's default --request-timeout of 0". The watch stays bounded by --timeout, and the Job by activeDeadlineSeconds, which is that same shape. The suite now rejects the flag. It needs the "=" because the explanation in the script names the flag too, and matching the bare name would fail on the comment rather than on a use. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
64d5689434 |
fix(kafka): keep the pre-delete hook's credentials as long as its Job
The Job carries no hook-succeeded, so it outlives the hook run and ttlSecondsAfterFinished reclaims it, which is what leaves a readable record when the finalizer outlives the wait. Its ServiceAccount, Role and RoleBinding still carried hook-succeeded and hook-failed, so Helm reaped them at the end of the run either way and left the Job behind them. A Job whose ServiceAccount is gone still starts a pod. kubelet mounts the projected volume without a token, client-go finds no in-cluster config, kubectl falls back to localhost:8080, and the script takes its catch-all branch and exits 1. A non-zero pre-delete hook returns from Uninstall.Run before any resource goes, so the release never finishes deleting and helm-controller retries it: the e2e run on the merge commit shows six such pods in five minutes, in two namespaces, every one of them dead the same way. The hook meant to prevent one stuck Kafka release was instead stopping all of them. Give all four objects the same before-hook-creation policy, so the credentials live exactly as long as the consumer and the next run replaces them. Both other in-tree hooks that reach the apiserver, apps/kubernetes and apps/harbor, are symmetric this way already. The suite now pins the policy on each of the three, because nothing compared them: the asymmetry rendered cleanly, passed 37 helm-unittest cases and 5 bats cases, and only a cluster showed it. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
6818795588 |
fix(seaweedfs): raise the bucket ceiling by sizing volumes smaller (#4251)
## What this PR does Raises how many buckets a SeaweedFS instance can serve, by lowering the master's `volumeSizeLimitMB` from 1000 to 400 and its two-copy growth count from 6 to 3. The size limit is exposed on the app chart, where an operator can reach it. Existing instances stay at 1000 unless they already name a limit, so for them the upgrade changes only the growth count. A platform migration does that where migrations run; on the `default` variant it is a manual step before upgrading. A unit test pins each constant separately, and an e2e case creates several buckets and requires each to take a write. Late in a full e2e run every S3 `PUT` starts timing out and a backup suite fails its access preflight. It reads like a credentials or COSI problem and is not one: the instance has no volume slots left, so the master cannot grow a volume for a newly created bucket. The disk is still almost entirely free, which is what makes it look like a flake. The COSI objects say nothing either way, the claim is ready and the access is granted. ### The ceiling A volume server started with `-max 0` takes a slot count derived from disk rather than a fixed one, replication `001` spends a slot on each of the two servers per logical volume, and a collection claims `COPY_2` volumes each time it grows. So the ceiling is set by the product of the size limit and `COPY_2`, and moving either number alone moves it. The mechanism is upstream in `MaybeAdjustVolumeMax` (`weed/storage/store.go`) and `findVolumeCount` (`weed/topology/volume_growth.go`); what matters for choosing the numbers is only that upshot. Do not divide the slot figures out of the nominal PVC size, because what the server counts is the space the filesystem makes available and not the volume's advertised size. At the upstream 1000 and 6 a 10Gi volume PVC gives nine slots per server and two collections, which the platform's own `cozy-backups` bucket and a single tenant bucket already use. That same PVC was seen to give nineteen slots at 500, which puts 400 at twenty-three to twenty-four by the same division, or eight collections. The span is the observation's own: the free space behind nineteen at 500 fits more than one filesystem geometry, and those divide differently at 400. Eight either way, because a collection that cannot get a whole batch of `COPY_2` keeps the volumes it did get and writes into them, so the count rounds up rather than down. Every figure here is for the shipped `replicationFactor: 2`. The master picks which of its four growth counts to apply from the replication the filer asks for, and the filer takes that from `replicationFactor`, so at 1 or 3 copies it reads `COPY_1` or `COPY_3` and the `COPY_2` this PR sets does not apply. Lowering the size limit still raises the ceiling there, since it is the divisor whatever the replication is. I measured this instead of only deriving it, on the real `weed` 4.31 binary: one master, two volume servers with `-max 0`, each on its own 1 GB filesystem image, replication `001`, then one `assign` per fresh collection until the master refused. Scaled 10:1 against the 10Gi PVC, so the limits scale with it. | `volumeSizeLimitMB` | `COPY_2` | slots reported | collections served | | --- | --- | --- | --- | | 100 | 6 | 10 | 2 | | 100 | 3 | 10 | 4 | | 50 | 3 | 20 | 7 | | 40 | 3 | 25 | 9 | | 25 | 6 | 40 | 7 | The fourth row is the pair this PR ships. The refusal arrives as `No writable volumes and no free volumes left`, with `volumes == max` on both servers and the disk nearly empty. The reported `max` matched the space the filesystem made available over the limit in every run, which is the rule the PVC figures above are divided with. Those runs sit on a 1 GB image, so their counts are the rule's, not the PVC's: a 10Gi volume has its own filesystem geometry and divides to its own numbers. ### Why both constants move The last two rows are the argument: lowering the size limit and lowering the growth count reach the same ceiling. They do not cost the same. Open descriptors scale with stored bytes rather than with the PVC, so at 400 a 10 TiB server keeps roughly 52k of them against 21k upstream, where taking the whole reduction out of the size limit would have put it at 105k, well past a 65536 `nofile` limit where one is set. Two stay open per volume, a `.dat` and an `.idx`; the `.vif` beside them is written and closed. That cost lands hardest on large instances, which never had a ceiling problem in the first place: at any of these limits a multi-TiB PVC has slots to spare. Lowering the growth count costs no descriptors at all. It costs write spread, since a bucket then spreads its writes over three volumes rather than six, and that cost is flat, it does not grow with the instance. So the reduction is split across both. Neither number is a new bet: 3 is what upstream already uses for three-copy replication. One price is shared by both routes. A collection absorbs `volumeSizeLimitMB × COPY_2` before it needs another grow, so growth requests arrive five times as often as they did at 6000 MB. A collection also starts with half as many writable volumes at `replicationFactor: 2`, and a volume server takes the writes into each volume one at a time, so a heavily parallel writer spreads over fewer volumes until the collection grows again. That holds for instances kept at 1000 too. ### On upgrade On a variant that runs platform migrations, the upgrade does not change the limit on an existing instance unless its operator already set one. Platform migration 57 writes `master.volumeSizeLimitMB: 1000` into every existing SeaweedFS instance that has not chosen a limit, except one whose `master` is not an object (below), so 400 applies only to instances created after this release. The `default` variant runs no platform migrations. A cluster there that installed SeaweedFS by hand has to apply the same patch before upgrading, and the operations guide has the command. Both guide steps end with a `complete` line. `jq` prints it only after reading the whole list, and the pinning loop stops at the first failed patch before it, so a run without that line is incomplete, whatever else it printed. A chosen limit is a whole number from 1 to 30000 and is kept. Anything else counts as not chosen and becomes 1000: the previous chart ignored the key and ran 1000 in its place, and the new schema would refuse it. A kept value other than 1000 takes effect with the upgrade itself. A lower one can strand buckets and a higher one takes the room new buckets grow into, so the guide has a query that lists those instances before upgrading. An instance kept at 1000 still gains: at `replicationFactor: 2` a grow now claims three volumes instead of six, so it has room for about twice as many new buckets. Three shapes get their own line in the Job log, with what they hold. A non-object `spec.values` is pinned too. helm-controller v1.5.0 ignores values it cannot read as a map and renders the chart defaults, so that instance runs at 1000 today and would drop to 400. The merge patch replaces the unreadable value, and nothing that applies is lost. A non-object `master` is left alone. Every v1 schema already refuses it, so that release does not render today, and the log says to set the limit in the same edit that repairs it. A `valuesFrom` with entries besides the `cozystack-values` Secret is named and pinned anyway. helm-controller merges `spec.values` over those entries, so a limit set there does not apply, unless it is set through `targetPath`, which can still win depending on the entries before it. The guide steps print the same three. The value goes into the application's HelmRelease, whose `spec.values` is the application spec, so it covers an instance created through the API and one the tenant module renders alike, the platform's own in `tenant-root` included. The tenant module renders that release without `spec.values`, which a new tenant chart test pins, and a Helm upgrade leaves alone a key that neither the previous nor the new manifest carries: I measured that on a kind cluster with the real HelmRelease CRD and Helm 4.3.0, on client-side, server-side and forced server-side apply, and read `createPatch` at the v4.1.1 that helm-controller v1.5.0 runs. Where migrations run, the order comes from the platform release itself. The migration is its pre-upgrade hook, and the `cozystack-packages` source every application chart is built from is a manifest of that same release, applied only after the hooks finish, so no instance meets the new chart unstamped and a failed migration stops the platform upgrade before that source moves. That holds while the platform source is pinned by digest, as it is out of the box; a source that follows a tag, a semver range or a branch lets source-controller move `cozystack-packages` on its own schedule, and an instance there can run at 400 until the migration has run. Until `cozystack-packages` moves, each patched release still runs its old chart, which ignores the key: no v1 release reads it, and none closes the `master` object in its values schema. The migration's test runs the script under the migrations image's shell against a fake `kubectl` that refuses any selector but the one cozystack-api lists SeaweedFS with, because an empty fleet is a legitimate answer and a mistyped selector would otherwise pass while pinning nothing. The same test runs the two guide steps, found by a marker comment above each, against the same fake and checks that they select the same releases as the migration. Lowering an existing instance is left to its operator, because it is not free. Its volumes between the new limit and the old go oversized on the next heartbeat and leave the master's writable set, and each affected collection needs one successful grow before its next write lands. Nothing is lost and the volumes stay readable. Those grows compete for one pool, and the budget is per collection rather than per server. An oversized volume is charged no headroom while one below the limit is charged all of its unfilled room, so the slots open to a stranded collection are free space less that room, divided by the limit; a grow then claims a whole `COPY_2` batch and takes as much of it as is there, and the volumes it creates are empty and charge their headroom straight back. Measured on the rig: two oversized collections with two free slots between them, and the first to write took both, leaving the second refused afterwards with the disk still reporting a quarter of itself free. Collections that keep writing are not free either: they were grown six at a time, and each of their volumes below 400 MB can hold back a whole 400 MB. Measured at a scaled pair, a server with free space for one batch per collection refused the only grow it needed, because a second, nearly empty collection held the room, and the same free space without that collection let the grow through. The master also grows, on its own schedule and with no write, any collection with no writable volume at or below 90 percent of the limit, which at 400 is 360 MB; measured at the same scaled pair, a collection with every volume in that band got two new volumes per server within one pass while a roomy one was left alone. So the precondition on every volume server is a batch for each collection with no volume at or below that line plus 400 MB for each volume still below the limit, and no volume size exempts an instance from it, since a larger volume carries more buckets and the budget scales with them. The `minFreeSpacePercent` reserve only comes into it on a server whose writable volumes have all passed the limit, since nothing there is charged headroom. It is five percent of what the filesystem reports rather than of the PVC's nominal size, so `All >= 8000 MiB` is where it reaches one 400 MB limit; a 10Gi PVC measures 9979 MiB and clears that, and such a server then refuses a grow only once its disk is down to the reserve. Everywhere else the headroom charge decides, and a refusal can come with the disk far from full whatever the reserve. Deleting a collection gives back a slot for each of its volumes, however little it stored, since the headroom charged to them leaves with them; only a server already at its reserve, where every volume is read-only and charged nothing, gets back no more than the bytes. Deleted objects return room only through a vacuum. The room left inside those volumes for further writes is given up and does not return as capacity, since such a volume is charged no headroom while its own bytes still come out of the budget. Lowering the growth count takes nothing away by contrast: no path removes a volume from the writable set for being surplus to the batch size, so an existing collection keeps what it has and converges as those volumes fill. [`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md) has the check for it, and the way back. The ceiling cannot come out lower at any supported `replicationFactor`, which is the part worth stating plainly. The divisor halves, and the space the slot count is derived from grows at the same time, because a volume past the new limit stops being charged headroom against it. `COPY_1` and `COPY_3` are untouched, so at 1 and 3 copies the limit alone still moves the ceiling upward. ### Reaching the value Neither constant was reachable by an operator. The data-plane chart is pulled by `ExternalArtifact`, and the component publishing it declares no `install` section, so the reconciler drops a `Package`-level values override for it before any merge happens. Editing the chart was the only route, on a value whose right setting depends on the PVC size the operator chose. `master.volumeSizeLimitMB` is on the app chart now, with a schema entry. Worth being precise about which file the two numbers end up living in, because exposing one of them moves it. The app chart's HelmRelease sends the size limit on every release, so `packages/extra/seaweedfs/values.yaml` holds the 400 installs run with, and `packages/system/seaweedfs/values.yaml` keeps upstream's 1000 as the fallback a render without it gets. That fallback is deliberate: a platform upgrade can re-render the data-plane release from its artifact before the parent has sent anything new, and a lower value there would then reach the master first and drop the limit ahead of an operator's own setting. Nothing sends the growth count, so the data-plane chart's value is the live one for that half, and it is the part that does land in that window, where a grow costs 3000 MB against the 6000 MB it cost before. The budget the pair produces is written out next to the growth count, with a pointer from the size limit to it, so the two do not drift apart. The field is bounded at both ends, because either end takes an instance down. Zero is not a smaller limit but an off switch: the volume server leaves its slot calculation untouched at zero, keeps reporting no capacity, and nothing is ever grown. Past 30000 the master does not start at all, exiting on `volumeSizeLimitMB should be smaller than 30000`. The bound lives in the values schema, through the generator's `@minimum` and `@maximum` annotations, so Helm refuses the value before any template runs. The apps API reads that schema only for defaults, so it accepts a value out of range, and the release then fails to render. A case pins each edge along with the value just outside it. An explicit null gets past both bounds, because Helm drops a null key before it validates, and would reach the master as an empty `-volumeSizeLimitMB=` that the shipped image exits on at start, so the template requires the value and a case pins that as well. There is no second bound against the disk, and not for want of trying: a limit at or above the volume's own size does not break an instance. `MaybeAdjustVolumeMax` floors the slot count at one, and a 255 MiB disk against a 1000 MB limit reported `Max 1`, took an assign for a fresh collection and served back what was written to it. Such a server is capped at a single slot rather than dead, so a render refusal would stop a working release on every reconcile. The field description says instead that a value close to or above `volume.size` leaves each volume server a single slot. The growth count is deliberately not exposed beside the limit. A field wired to `COPY_2` would do nothing at 1 or 3 copies, for the reason above, while still looking like a setting, and a knob that is silently inert for two of the three supported values is worse than no knob. Making it track the replication factor is a feature with its own matrix, and it would rest on a path where the master's own default replication is separately broken. The size limit has no such dependency, so it is the one that ships. ### Picking a value 400 is sized for the shipped `volume.size: 10Gi`, which is where the slot and collection figures above come from. Going up, it holds until the two descriptors per volume meet a 65536 `nofile` limit, roughly 12.5 TiB per volume server. Going down, an instance needs 1600 MB of filesystem per volume server to serve a second collection at all, since the first claims the opening three slots. ### What the tests cover The existing bucket case creates one bucket, which sits inside the ceiling at any sizing, so it could not tell a working ceiling from an exhausted one. The new e2e case creates three beside the platform's own backup bucket and requires each to accept an object, which is the only signal the exhaustion produces. Each `mc` call there carries its own timeout, since a hung PUT is exactly that signal and the step timeout alone would not end it. It was observed red at the old sizing in [run 34948279363](https://github.com/cozystack/cozystack/actions/runs/34948279363), which ran the case against the unchanged constants: the pre-existing single-bucket case passed in 18s beside it, and the new one failed on the write to the second bucket after the first succeeded. Three rather than two so the count clears the ceiling without depending on how many slots the unnamed default collection holds, which the suite cannot observe. The unit tests pin each constant on its own, because losing one is a silent capacity regression rather than something a render would catch. The app-chart test asserts an overridden size limit as well as the default, since asserting the default alone passes just as well against a template that ignores the value and writes the number as a literal. It also pins both ends of the accepted range and the first value outside each, the `-max 0` zero on all four render paths it is written to, and the 10Gi volume default the slot figures above are measured for, which nothing else guarded. The data-plane suite pins the five percent free-space reserve as well, since the 8000 MiB threshold above is computed from it and nothing else reads it. The capacity case also reads the master's slot accounting in its `catch`. A grow failure in the master log does not say whether the shipped limit is too coarse for the PVC or an earlier collection in the same run is still holding the budget, and the per-server `Max` against `Volumes` does. Closes #4239, closes #4245, closes #4248 ### Screenshots No UI change. ### Downstream repositories Walked against the diff. The new field lands on `packages/extra/seaweedfs`, and the provider models bucket, generic, qdrant, redis and tenant, with no SeaweedFS resource among them, so there is no schema or expand/flatten pair over there for it to drift from. No package is added, renamed or removed, so the website app lists are untouched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(seaweedfs): raise how many buckets a new SeaweedFS instance can serve on the default 10Gi volume PVC from two to eight at `replicationFactor: 2`, by defaulting the master volume size limit to 400 MB instead of 1000 and lowering the two-copy growth count from 6 to 3, and expose the limit as `master.volumeSizeLimitMB` on the app chart. **Action required on the `default` variant only:** it runs no platform migrations, so before upgrading, set `master.volumeSizeLimitMB: 1000` on every SeaweedFS instance that has no valid value and whose `master` is an object, with the step in [`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md). If this is skipped, every such instance drops to 400 on the upgrade, and a bucket whose volumes have all reached 400 MB takes no writes until it can grow. On the variants that run platform migrations, migration 57 does this: it sets `master.volumeSizeLimitMB: 1000` on every instance that has not chosen a value, except one whose `master` is not an object, which no v1 chart renders and which it names in its Job's log instead, so nothing changes for an instance unless it already names a different value, which the previous chart accepted and ignored and this one applies; the same guide lists such instances before the upgrade, and the migration names them in its Job's log. An existing value that is not a whole number from 1 to 30000 is replaced with 1000, which is what the previous chart applied in its place. Instances at 1000 still get roughly twice the room for new buckets from the lower growth count. The upgrade rolls every SeaweedFS master StatefulSet once, since the growth count is part of the master's environment, and an instance that names a value other than 1000 rolls a second time when that value arrives. The pin lives in the instance's spec, so a tool that replaces the whole resource from a manifest without `master.volumeSizeLimitMB` removes it and the instance takes 400; add the field to such manifests before upgrading. Lowering the limit on an instance that already holds data takes every volume at or above the new value out of the master's writable set, and raising it takes the room new buckets grow into, so the same guide describes the free-space check to run first and how to set it back. ``` |
||
|
|
c983390b39 |
ci(e2e): surface overlay drift and remove both causes of it (#4261)
## What this PR does Six commits against one script: one reports a silent degradation, three remove causes of it, two repair the tests and the reasons recorded beside them. **Why it matters.** A file whose config differs from the current-main artifact beyond its image lines keeps its committed ref. That is the right safe answer, and it is not a neutral one: the component then runs its last RELEASE image in a lane whose job is to exercise the tree under review, and the only trace is one line among hundreds. Seven packages took that path on the run behind #4257. One of them, `backupstrategy-controller`, served an image from v1.6.0 (22 July) against a tree carrying four backup strategy drivers added after it, so five of that run's seven chainsaw failures came from there, and three unrelated pull requests each spent a review round establishing that the red lane was not theirs. After this PR that list is empty. Against the public artifact at its own revision, with the tree checked out at that same revision: ```text before drift=5 overlaid=29 core/platform, cozystack-api, dashboard, kubeovn-webhook, linstor-gui after drift=0 overlaid=34 ``` **Naming the drifted packages.** One GitHub annotation after the existing tally. Which files get overlaid does not change, and a run with no drift emits nothing. **Cause one: a suffixed image key.** The classifier accepted a key only when it was exactly `image`, `repository`, `registry`, `tag` or `digest`. A chart rendering several images names the extras after what they are for, so `backupstrategy-controller` carries `chBackupClientImage`, `rabbitmqBackupClientImage` and `redisBackupClientImage`. The first two passed only through the `@sha256:` branch; the third is pinned by tag alone, so it read as a config change and cost the whole file its overlay. The fix accepts a prefix before `image`, and two things keep it narrow. A letter is required before `image`, which a key beginning with it has nothing to supply, so `imagePullPolicy` and `imagePullSecrets` stay config; the trailing colon is not what does that, and they match neither with it nor without. The value must also hold a repository path, which keeps out operator-settable knobs named after an image: `vddkImage: ""` in `core/platform` and `migration-controller` is configuration the build never stamps, and absorbing it would be the opposite of what this script is for. **Cause two: a blank line the build removes itself.** `yq -i` re-emits the document when it stamps a digest, and a blank line separating two top-level keys does not come back. So a file the PR never touched differs from the artifact on a line carrying no configuration at all, and loses its overlay for it: ```text $ diff repo/system/cozystack-api/values.yaml artifact/system/cozystack-api/values.yaml 4d3 < 6c5 < image: ...cozystack-api:v1.6.0@sha256:c4d374a9... --- > image: ...cozystack-api:main@sha256:2fb87cf8... ``` Running the stamping assignment over the committed file reproduces that diff exactly, which is the second signal behind the diagnosis rather than the first one twice. Blank lines are now discounted before the test; a line with content still counts, so a genuine config difference keeps the committed ref as before. This accounted for all five remaining packages, and it retires the explanation offered in #4265, which ruled `yq` reformatting out. That was wrong: the round-trip check behind it did not cover the blank line. `cozystack-api` and `dashboard` are not peripheral here. An e2e result about either was a result about the last release rather than about `main`. **Repairing the tests.** `hack/bats-no-exit-trap.bats` audits every `.bats` file under `hack` against the debt its own header declares, and this file was installing three traps beyond it. An EXIT trap inside an `@test` body replaces the one the bats binary installs, so a failing test prints no TAP line at all and the suite reads as green. The answer is to stop installing the handler rather than to raise the number, so those tests clean up at the end of the body instead. ### Tests Eight cases in `hack/overlay-main-images_test.bats`, each held by a mutation of the line it describes: - drift is summarised with the package names and the count; suppressing the annotation or dropping the names reddens the suite - a clean run emits no annotation - a suffixed image key pinned by tag alone overlays instead of drifting; reverting the classifier reddens it - a key merely STARTING with `image` is still config - an `*Image` key whose value is operator configuration rather than a path still drifts, so `vddkImage` is not absorbed - two drifted files in one package count as one package, and the name carries no `/images` suffix - a blank line lost to `yq` is not a config change; removing the filter reddens it - a real config line alongside a blank line still counts as drift; widening the filter to drop every line reddens this one and two others The suite's plan and its `ok` count agree at 21, which is the reconciliation the trap audit exists to protect. Green under the `bats` binary and under `hack/cozytest.sh`, which is what `make bats-unit-tests` runs and the only one of the two whose result decides the gate. ### What this does not close `redisBackupClientImage` is still pinned by tag with no digest, and the tag it names does not exist: ```text $ curl .../cozystack/redis-backup/tags/list {"tags":["nightly","0.0.0-nightly.20260911...","0.0.0-nightly.20260912...", ...]} ``` Only nightly tags are published, so `redis-backup:v1.6.0` resolves to nothing. Pinning it needs a published release tag first, which is #4264 rather than this PR. Making the overlay line-wise, so image lines transfer even when other lines genuinely differ, would go further still. It trades away the guarantee the script is built on, that a file comes wholly from one source, and that guarantee is deliberate and commented: a maintainer decision rather than a quiet change from a reviewer. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * GitHub Actions now warns when image overlay processing detects configuration drift. * Warnings list affected package names in sorted, de-duplicated form. * Image references with alphabetic prefixes are recognized during overlay processing. * Blank-line-only differences no longer trigger drift warnings. * **Tests** * Added coverage for image references, similarly named configuration keys, blank-line differences, and multi-package drift reporting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
69c4fadf8a |
docs(security): document Security Champion rotation and response team (#4223)
## What this PR does Surfaces the **Security Champion rotation policy** and the **security response team's composition and diversity** into the public `SECURITY.md`. Until now these were defined only in the CVE-scanning pipeline's private governance document, which the General Technical Review flagged as a gap (the policy existed but was unpublished, and the rotating role was unassigned). Specifically: - Adds a **"Security response team and rotation"** subsection: names the response-team maintainers, notes the team spans more than one organization (Ænix, HIDORA) and more than one country, and documents the rotating **Security Champion** role — its responsibilities, a **6-month rotation** among maintainers, and a **one-week handoff overlap**. - Records **@tym83** as the current Security Champion, filling the role this file previously described as unfilled. - Updates the Trivy paragraph accordingly (the triage-clock owner is no longer "currently unfilled"). This is a policy change to a `CODEOWNERS`-protected file and is intended for maintainer (CODEOWNER) review rather than a wording check. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note docs(security): document the Security Champion rotation policy and security response team in SECURITY.md ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added information about the security response team and its rotating Security Champion role. * Clarified ownership of security triage timing and response coordination. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a2db2aeb8c |
fix(seaweedfs): raise the number of buckets a volume server can serve
A volume server started with -max 0 takes a slot count derived from disk rather than a fixed one, replication 001 spends a slot on each of the two servers per logical volume, and a collection claims COPY_2 volumes whenever it grows. So the bucket ceiling is set by the product of the size limit and COPY_2, and moving either number alone moves it. The mechanism is upstream in MaybeAdjustVolumeMax (weed/storage/store.go) and findVolumeCount (weed/topology/volume_growth.go). Do not divide these out of the nominal PVC size, because the count follows the space the filesystem makes available and not the volume's advertised size: the shipped 10Gi volume PVC was seen to give nine slots per server at the upstream 1000 and 19 at 500, which puts 400 at 23 to 24 by the same division, or eight collections against two. The span is the observation's own, since the free space behind 19 at 500 fits more than one filesystem geometry and those divide differently at 400; eight either way, because a collection that cannot get a whole batch of COPY_2 keeps the volumes it did get and writes into them. The division itself is measured: on a 1 GB image the master reports max as the filesystem's available space over the limit in every run, 25 slots and nine collections at the 40 and 3 that scale to this pair. Two is what the platform's own cozy-backups bucket and a single tenant bucket already use. Past that every S3 PUT to a new bucket times out while the disk still reads as nearly empty, and no COSI object reports anything wrong, because the claim and the access are both granted. 400 and 3 make the budget 1200 MB per collection. Splitting the reduction across both numbers rather than taking it out of the size limit alone keeps the cost off the curve that grows: open descriptors track stored bytes divided by the limit, so a 10 TiB server keeps roughly 52k of them at this limit against 21k upstream, where moving the size limit alone would have put it at 105k. A .dat and an .idx stay open per volume; the .vif beside them is written and closed. The ceiling reached is the same either way. Both figures describe the shipped replicationFactor of 2. The master picks its growth count from the replication the filer asks for, so at 1 and 3 copies it reads COPY_1 and COPY_3 instead and this COPY_2 does not apply; lowering the size limit still raises the ceiling there. The size limit was out of an operator's reach: it lived in the data-plane chart, which is pulled by ExternalArtifact from a component that declares no install section, so a Package-level values override for it is dropped before any merge. The app chart now declares master.volumeSizeLimitMB and sends it on every release, and 400 is set there. The data-plane chart keeps upstream's 1000 as its fallback on purpose. A platform upgrade can re-render that release from its artifact before the parent has sent anything new, and a lower fallback would then reach the master first and drop the limit ahead of an operator's own setting. COPY_2 is sent by nothing and lives in the data-plane chart, so it does land in that window, where a grow costs 3000 MB against the 6000 MB it cost before. The app chart also writes maxVolumes into dataDirs on every release, and because dataDirs is a list Helm replaces the data-plane chart's copy wholesale, so the zero that makes the slot count auto-derived at all comes from there. Losing it drops the server to the subchart's fixed seven without failing any render. The field is bounded at both ends, because either end takes an instance down. Zero is not a smaller limit but an off switch: the volume server leaves its slot calculation untouched at zero, keeps reporting no capacity, and nothing is ever grown. Past 30000 the master does not start at all. The bound sits in the values schema rather than in a template guard, so Helm refuses the value before any template runs. The apps API does not: it reads that schema only for defaults, so it accepts a value out of range, and the release then fails to render. An explicit null gets past both bounds, because Helm drops the key before it validates, and would render an empty -volumeSizeLimitMB= that the master exits on at start, measured on the shipped image, so the template requires the value. No bound against the disk goes with it, though the field invites one. A limit at or above the volume's own size does not break an instance: MaybeAdjustVolumeMax floors the slot count at one, measured on 4.31 with a 255 MiB disk against a 1000 MB limit, which reported Max 1 and served a write it then read back. Such a server is capped at a single slot rather than dead, so refusing it at render would be a capacity policy that stops a working release on every reconcile, not only on the upgrade. The field's description carries the relation instead, where the value is chosen. The growth count is deliberately not exposed beside the limit. The master picks which of its four growth counts to apply from the replication the filer requests, and the filer takes that from replicationFactor, so a field wired to COPY_2 would do nothing at 1 or 3 copies while still appearing to be a setting. A knob that is silently inert for two of the three supported values is worse than no knob. Two further prices. A collection absorbs the budget before it needs another grow, so growth requests arrive five times as often as at the upstream pair. A collection also starts with half as many writable volumes, and a volume server takes the writes into each volume one at a time (syncWrite holds that volume's dataFileAccessLock), so a heavily parallel writer spreads over fewer volumes until the collection grows again; that holds for instances kept at 1000 too. And lowering the limit on an instance that already holds data carries the other one: its volumes between the new limit and the old go oversized on the next heartbeat and leave the master's writable set, and each affected collection needs one successful grow before its next write lands. Nothing is lost and the volumes stay readable; the room left inside them for further writes is given up and does not return as capacity. Those grows compete for one pool, and the pool is smaller than it looks. An oversized volume is charged no headroom, while one below the limit is charged all of its unfilled room; a grow then claims a whole COPY_2 batch and takes as much of it as is there, and the volumes it creates are empty and charge their headroom straight back. Measured on a local SeaweedFS 4.31 setup: two oversized collections with two free slots between them, the first to write took both, and the second stayed refused with the disk still reporting a quarter of itself free. Collections that need no grow are not free either, since their volumes below the new limit were created six at a time and each can hold back a whole limit. Also measured there, with the limit and batch scaled down: a server with free space for one batch per collection refused the grow its one affected collection needed, because a second, nearly empty collection held the room, and the same free space without it let the grow through. The minFreeSpacePercent reserve only comes into it on a server whose writable volumes have all passed the limit, where nothing is charged headroom: from 8000 MiB of filesystem upward, a 10Gi PVC measuring 9979, five percent is at least 400 MB, and such a server then refuses a grow only once its disk is down to the reserve. Everywhere else the headroom charge decides, and a refusal can arrive with the disk far from full whatever the reserve. Deleting a collection gives the room back a slot per volume, however little it stored, since the headroom charged to those volumes leaves with them. So, on a variant that runs platform migrations, the upgrade does not change the limit on an existing instance unless its operator already set one, which the previous chart accepted and ignored. Platform migration 57 writes master.volumeSizeLimitMB: 1000 into every existing SeaweedFS instance that has not chosen a limit, and 400 applies only to instances created from here on. It leaves alone only an instance whose master is not an object, which no v1 chart renders, and names that one in its log. The default variant runs no platform migrations, so a cluster there that installed SeaweedFS by hand gets no pin, and the document below gives the same patch to apply before upgrading. A chosen limit is a whole number the new chart accepts; anything else counts as not chosen and is replaced with 1000, which is what the previous chart ran in its place, since it ignored the key and the new chart would refuse to render it. A valid value already set is kept, and since the previous chart ignored it, the upgrade is when it starts to apply; the document below lists the ones that change the limit ahead of it. Raised values are listed as well: existing buckets keep writing, but every volume below the new limit is charged up to it, and on a local SeaweedFS 4.31 setup seven nearly empty volumes left no free slot once the limit went up tenfold, so a new bucket was refused. The value goes into the application's HelmRelease, whose spec.values is the application spec, for an instance created through the API and for one the tenant module renders alike. The tenant module never renders spec.values on that release, and a Helm upgrade leaves a key alone that neither manifest carries: measured with Helm 4.3.0 on client-side, server-side and forced server-side apply, and read in createPatch at the v4.1.1 that helm-controller v1.5.0 runs. Where the migration runs, it is a pre-upgrade hook of the platform release, and the cozystack-packages source every application chart comes from is a manifest of that same release, applied only after its hooks; so no instance meets the new chart unstamped, and a failed migration stops the platform upgrade before that source moves. That holds while the platform source is pinned by digest, as it is out of the box; one that follows a tag, a semver range or a branch lets source-controller move cozystack-packages on its own schedule. Until cozystack-packages moves, each patched release keeps its old chart, which ignores the key, since no v1 release reads it or closes the master object in its schema. An instance kept at 1000 still gains, because at a replication factor of 2 a grow now claims three volumes rather than six, so it has room for roughly twice as many new buckets. Lowering an existing instance is left to its operator, and docs/operations/seaweedfs-lowering-the-volume-size-limit.md is the check for it. It lists each server's volume files and adds one limit for every volume below the limit and a batch for every collection with no volume at or below 90 percent of it, since the master grows such a collection on its own schedule without a write; measured the same way, a collection with every volume in that band and no writes got two new volumes per server within one pass. Where each disk type is served by as many volume servers as the replication factor, as the shipped two at a factor of 2 are, each collection's volumes sit on every server of that type, and that bounds the claim from above without assuming how the collections grew. The master places a new volume on any server of the requested disk type with room rather than where its collection already lives, whichever group or pool that server belongs to, and it reads an empty disk type as hdd; under MultiZone it also grows each zone on its own. So the document keeps zoned instances, and any disk type served by more servers than copies, at 1000. BREAKING CHANGE: on the default variant, which runs no platform migrations, set master.volumeSizeLimitMB to 1000 before upgrading on every SeaweedFS instance that has no valid value and whose master is an object, with the step in docs/operations/seaweedfs-lowering-the-volume-size-limit.md. On the variants that run platform migrations, migration 57 does this. If the step is skipped, every such instance drops to 400 on the upgrade: each of its volumes at or above 400 MB leaves the master's writable set, and a bucket whose volumes have all crossed that line takes no writes until it can grow. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
c038cf25d3 |
test(seaweedfs): cover the bucket count a volume server can serve
A volume server started with -max 0 derives the slots it has left from the free space on its disk, less the room its unfilled volumes still claim, divided by the master's volume size limit, and each collection claims a batch of those slots on every volume server the replication spans. The number of buckets an instance can hold therefore has a ceiling well below what the free bytes suggest, and crossing it leaves every COSI object reporting ready while the S3 PUT times out. The existing case creates one bucket, which sits inside that ceiling at any sizing, so nothing here told a working ceiling from an exhausted one. This case creates three beside the platform's own backup bucket and requires each to accept an object, which is the only signal the exhaustion produces. Three rather than two because the count has to clear the ceiling whether or not cozy-backups and the unnamed default collection have grown yet, which this suite cannot observe. BucketAccess reconciles separately from the claim it belongs to, and the write step reads the Secret that access produces, so readiness of the release and the claim is not enough to enter it. Without the access assertions the step can fail on a missing Secret, which looks the same in the report as the capacity failure it is here to measure. The catch reads the master's slot accounting as well as its log. A grow failure in the log does not say whether the shipped limit is too coarse for the PVC or an earlier collection in the same run still holds the budget, and the per-server Max against Volumes does. The read runs inside the master, whose image ships curl, so it needs nothing on the host and adds no second fixed port to the reason this suite cannot run in parallel. That reason now covers two cases rather than one, which the shared config says. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
4953b512a5 |
chore(platform): freeze HTTPCache on the last artifact that ships it
The earlier shape of this migration orphaned every instance into plain objects, which left its owner without a way to manage or remove it. A cluster that still runs HTTPCache instead keeps the application, served from a copy of the packages OCIRepository pinned to the digest of the last release that contains it. The copy carries no owner references and no Helm metadata, and the PackageSource is repointed at it, annotated keep and disowned, so neither Helm nor the operator prunes it. Helm reads that annotation off the live object during the upgrade prune, which is what makes disowning in a pre-upgrade hook work at all. Names are deliberately unchanged: the ApplicationDefinition inside that artifact hardcodes its chartRef, so renaming the PackageSource would leave every release pointing at an artifact nobody produces. A cluster with no instance keeps nothing. The naas bundle installs the rd component regardless of use, so freezing unconditionally would strand a pinned source on clusters that never touched the application. Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
2192547f1e |
docs(governance): remove the unused Directors role
GOVERNANCE.md defined a Directors role (non-technical leaders, elected by the maintainers, one vote in decisions), but no Directors were ever identified or elected. The role was intended for non-technical leadership, but those people ended up maintaining parts of the project directly, so it stayed unused. Remove it so the documented decision-making body matches actual leadership (maintainers, by consensus with a simple-majority fallback); it can be reintroduced if a real need arises. Addresses cozystack/community#79. Signed-off-by: Timur Tukaev <6355522@gmail.com> |
||
|
|
c0f9f08de7 |
chore(platform): detach HTTPCache instances before the package is pruned
Pruning the http-cache PackageSource takes its chart artifacts with it, so without a migration every http-cache HelmRelease loses its chart and stays Ready=False for good, with HelmReleaseNotReady firing on every cluster that runs the naas bundle. Deleting the leftover Package by hand is worse: it takes the ApplicationDefinition with it while the pods of a surviving instance are still running. Migration 57 orphans each tenant release instead, the way migration 55 does, so the workloads stay in the tenant namespace as plain objects, and then removes the platform side. The Package goes before hr/http-cache-rd because the operator re-creates a release it owns for as long as that Package exists. Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
150e18b3d3 |
docs(e2e): correct the Chainsaw Test-ordering claim in etcd and clickhouse suites
The etcd suite justified per-scenario teardown with a claim that Chainsaw runs Tests in name order, not file order. That was wrong, but restating it as document order is wrong too: at the pinned v0.2.15 Chainsaw guarantees no order between Tests at all. The runner registers every Test through testing.T.Run and the body calls t.Parallel() whenever spec.concurrent is nil or true — the case in both suites, since neither sets it. Go pauses those subtests and releases them together once the parent returns, and execution.parallel maps only to -test.parallel, a concurrency cap rather than an order. So parallel: 1 serialises the Tests without ordering them: not file, not document, not name. The E2E run on this branch's own head shows it — clickhouse-2 ran before clickhouse-1 with both etcd Tests interleaved between the clickhouse pair. The teardown decision the false claim justified still holds, for a reason that does not touch order: each etcd scenario is self-contained under skipDelete: true with a step-level finally, and v0.2.15 has no test-level finally, so a scenario is reclaimed only if it reclaims itself — true of whichever Test runs last regardless of order. Drop the ordering clause from the etcd teardown rationale and the ordering sentence from the clickhouse header rather than restating either. Comment-only; no behaviour changes. Verified against Chainsaw v0.2.15 source: pkg/runner/runner.go (t.Run plus t.Parallel gated on spec.concurrent) and pkg/runner/flags/flags.go (execution.parallel maps to test.parallel). Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
42a27567ce |
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
flux-plunger recovers a "has no deployed releases" HelmRelease by briefly suspending it, deleting the stale storage Secret and unsuspending. helm-controller v1.5.x skips the uninstall and drops its finalizer when it handles a delete on a suspended HelmRelease, and it wakes on the same delete event flux-plunger does, so a delete inside that window orphaned the release's DataVolume, PVCs, RBAC and storage Secret. Lifting the suspension on the delete event cannot win that race reliably. Hold the release with a flux-plunger finalizer while it is suspended. The finalizer is applied together with the suspension in one server-side apply. On a delete flux-plunger lifts its suspension and keeps the finalizer, helm-controller reconciles the unsuspended generation and runs the uninstall, and flux-plunger removes the finalizer once helm-controller's own is gone and that generation is observed. This holds whichever controller handles the delete first. A completed recovery lifts the suspension and the finalizer together, and a terminating release never begins recovery. Decide ownership of a suspension by which field manager owns spec.suspend in metadata.managedFields rather than by an annotation. The apiserver transfers the field away the moment an operator, cozyhr resume or a backup controller writes a different value to it, so a later suspension is never misread as flux-plunger's, and one set through the shared flux-client-side-apply manager (platform migrations, the tenant delete hook, a manual kubectl apply) is never claimed. A release deleted under another manager's suspension is left to that manager's decision. Acquire applies without forcing first and, on a conflict, re-fetches and forces only if spec.suspend is still false: a backup controller owns the field at false through a dynamic-client Update, which a non-forced apply can never take, while true is an active suspension to leave alone. Every apply is preconditioned on the resourceVersion and uid of the read that decided on it. Without the resourceVersion, the unforced apply of true over a foreign true is accepted as shared ownership, a window the optimistic-locked merge patch this replaces had closed, and lifting the suspension and the finalizer from a release read as live would delete it without an uninstall if helm-controller had meanwhile handled a delete. Without the uid, an apply to a release that has gone would create an empty one. The apply carries only spec.suspend and the finalizer, so it cannot leak the required spec.interval. Read the release, every re-check and the storage Secret list from the apiserver rather than the informer cache. A lagging cache would let the processed-version guard delete the revision below the one just reclaimed, and the ClusterRole grants secrets get and list but not the watch an informer needs. Emit a Warning when flux-plunger holds a suspension it failed to clear, and Normal events when it leaves an external suspension alone or defers a recovery. An external suspension on a release flux-plunger has recovered before is a Warning instead: it is most likely one an earlier flux-plunger left under the shared manager, which nothing lifts unless an operator does. A release deleted under another manager's suspension is a Normal event when that was a migration's choice, and a Warning when the release carries flux-plunger's processed-version annotation, since then the skipped uninstall is an accident. Keep a single writer with leader election. A second process able to act under the same field manager could clear a suspension the first is still recovering, and no rollout strategy prevents one: the next pod starts as soon as the old one is terminating, and an eviction or a node drain replaces a pod the same way. The old pod keeps its Lease, unreleased, through its stop and drain, and the Lease outlasts both, so the next pod acts only after the old one is done; the price is that a new pod waits for the Lease to run out. The strategy sets maxSurge 0 with maxUnavailable 1 and stays RollingUpdate because existing installs carry the apiserver-defaulted rollingUpdate block and helm-controller applies server-side, where type Recreate over it is rejected on upgrade. Give the finalizer a way out when flux-plunger itself goes. Once the manager has stopped, within a bounded time, and with it every reconcile, flux-plunger drains its finalizers under a deadline that, with the stop, fits inside the pod's termination grace period. A stop that overran its bound is drained too and still reported, and only a pod that led drains. The drain writes its events straight to the apiserver, since the manager's recorder stops with the manager. A recovery in progress is aborted, the suspension and the finalizer lifted together for the next flux-plunger to start again, and a release being deleted has the suspension lifted and keeps the finalizer until helm-controller has run the uninstall. A release helm-controller does not handle in time keeps it. The waiting lives in the controller, where the envtest suites hold it against a real apiserver's generations and finalizers, rather than in a shell script against a model of them. A pre-delete hook on the chart is the backstop for what the drain could not finish, so uninstalling the component never leaves a release that tenant teardown waits on forever. It stops flux-plunger, waits out its grace period, then lifts any remaining suspension and finalizer without waiting again, naming each release it lets go of before helm-controller uninstalled it. It lifts a suspension only where flux-plunger alone holds it, with a JSON patch preconditioned on the resourceVersion and uid it read: kubectl apply --server-side on a release last written by a client-side apply moves that apply's fields to the applying manager and strips every one the body leaves out. The Job's deadline sits under the 300s helm-controller gives a pre-delete hook by default. A Deployment that cannot be stopped or a listing that never succeeds ends the hook before any finalizer is touched, and a finalizer it cannot remove ends it after the sweep; either way it exits non-zero: a failed pre-delete hook leaves the release untouched and helm-controller retries the uninstall, hook included, from a state it can still converge from, where exiting 0 would remove the component and strand the finalizer. A flux-plunger pod still present after its grace period, typically one stuck Terminating on a NotReady node, is only reported, since failing on it would fail every retry until the node returns; because it may still be setting finalizers, they are listed again after the sweep and any found fails the hook. The Job runs once, since helm-controller's retry is the one that starts afresh, and is kept after success so that its report stays readable; helm.sh/hook-output-log-policy would make helm-controller call a nil output writer (fluxcd/helm-controller#1562). A delete still held after ten minutes raises a Warning, and the held state is re-checked on a timer instead of only on helm-controller's next status write. The chart README documents the finalizer and how to clear it. Run the envtest suites on every pull request through a new make target that fetches a kube-apiserver and etcd with a pinned setup-envtest and takes the HelmRelease CRD from the in-tree Flux manifests: the unit tests drive a fake client, and everything the finalizer rests on (apply preconditions, the finalizer set-merge, real managedFields) only holds against a real apiserver. CI caches those binaries, and the target uses them without asking the release index, so a pull request does not depend on their sources being reachable; the control plane gets a longer start timeout because it starts beside the other test goals. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
b99db147a7 |
test(ci): isolate the commit-trailer fixture from global tag signing (#4271)
## What this PR does `make_repo` in `hack/check-commit-trailers.bats` silences `commit.gpgsign` in the fixture repository but leaves `tag.gpgsign` alone. Anyone who signs tags has that setting globally, it reaches the fixture, `git tag start` becomes an annotated tag, and git refuses it for want of a message. The helper opens all 27 tests, so the whole suite dies before a single test reaches its body, and `make bats-unit-tests` takes `make unit-tests` down with it. Details and the reproduction are in #4270. The first commit is that one line, next to the one that is already there. With it the suite went 27/27 on a machine where the global setting is set, and was unchanged on one where it is not. I ran it both ways. CI never saw this. The `Commit trailers` job runs the checker against explicit SHAs and does not touch the bats file, and a fresh runner has no global git configuration to leak in. The failure only lands on the local `make unit-tests` that `docs/agents/contributing.md` asks for before pushing, which is the worst place for it: an unexplained red in a file the contributor did not touch. That is also how the guard went missing in the first place, and it would go missing again the same way, so the second commit adds `make_repo holds against a hostile global configuration`. None of the three `git config` lines in `make_repo` is exercised on a runner: delete any of them and CI stays green. The test supplies the hostile configuration itself rather than waiting for a contributor to have one, so each guard is held by the suite. Pulling them one at a time gives three distinct reds: `tag.gpgsign` gives `fatal: no tag message?`, `commit.gpgsign` gives `gpg failed to sign the data`, and `core.hooksPath` lets a global `commit-msg` hook rewrite the message and fails the assertion on it. The suite is 28/28 both with the global settings and without, so a runner sees no change. `hack/release-changelog-behaviour.bats` has gaps of the same kind in its own fixture, which sets `commit.gpgsign` and nothing else. The missing `tag.gpgsign` is inert there because it creates no tag. The missing `core.hooksPath` is not: a global `pre-commit` hook that exits non-zero kills every `git commit` it makes. I left both alone rather than widen this diff, and noted them in the issue. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. The only `hack/`-related trigger is `cozystack/ccp`, and it fires on moving or renaming something under `hack/` or changing what a make target does. This adds a line inside an existing file; the filename and the make target behave as before, except that the target stops failing. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Repository fixtures now remain reliable despite conflicting global Git settings, including commit and tag signing, failing signing tools, and commit-message hooks. - Fixture setup preserves lightweight tags and the original seed commit message, ensuring consistent repository initialization across environments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a19fd0274f |
fix(kafka): delete release topics before the topic operator is removed (#3938)
## What this PR does Fixes Kafka application deletion when a release-owned KafkaTopic still has Strimzi's `strimzi.io/topic-operator` finalizer. Without ordering, Helm removes the topic operator with the release, the KafkaTopic cannot finish deleting, and reinstalling the same release name times out on the leftover object (#3793). A namespace-scoped pre-delete hook deletes only KafkaTopics carrying the current release label while the operator is still running. It never removes finalizers directly. The hook uses a digest-pinned kubectl image, restricted RBAC, a 270-second job deadline, a 180-second deletion timeout, and a 20-second request timeout. The E2E absence checks allow five minutes, matching the hook's Helm budget. A wait that times out with the finalizer still held exits 0 on purpose, so the uninstall can finish rather than wedging the release under helm-controller's retry loop. That is a deliberate trade and the hook says so on stderr; the Job survives an hour under `ttlSecondsAfterFinished` so the record outlives the run. Validation: - `helm unittest packages/apps/kafka`: 37/37 passed, covering the hook resources, RBAC wiring, the digest pin and the `_cluster.images-registry` override, the release-scoped selector, and the Job deadline and delete-policy settings. - `hack/cozytest.sh hack/kafka-pre-delete-hook.bats`: 5/5 passed. The suite renders the hook and runs the script against a fake kubectl, because every property worth pinning there is an exit code and matching manifest text cannot establish one. Run under the runner `make bats-unit-tests` uses rather than the `bats` binary, which do not accept the same test file. - Both suites were mutation-checked: putting the timeout branch back to `exit 1`, flipping the catch-all to `exit 0`, dropping `--ignore-not-found`, inverting the guard, widening the selector to a sibling release, and setting `--wait=false` each turn one of them red. - The raw template parses with `yq`; all leading Helm-expression scalars are quoted. - Chainsaw asserts both release topics are absent after deletion, but the full E2E suite was not run locally because it requires a disposable cluster. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the chart hook and E2E test. No values/schema/API/default or downstream hardcoded contract changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kafka): delete release-owned KafkaTopics before removing the Strimzi topic operator ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Kafka deletion now automatically removes associated Kafka topics before the Kafka resource is removed. - Cleanup waits for topics to disappear before checking ZooKeeper storage reclamation. - Deletion remains resilient when topic resources or their definitions are absent, or when finalizers delay removal. - Cleanup failures are reported appropriately without blocking deletion when a timeout occurs. - **Tests** - Added coverage for cleanup behavior, execution settings, permissions, and failure scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
379c763fc3 |
test(ci): drop an inert key from the fixture's hostile configuration
tag.forceSignAnnotated governs annotated tags and `git tag <name>` creates none, so with tag.gpgsign unset the tag still resolves to a commit object. The key cannot make the new test fail and guards nothing the fixture could lose; carrying it implied a fourth guard that does not exist. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
ef3bfd71d3 |
docs(ci): record what actually narrows the overlay key branch
Two comments credited the trailing colon with keeping imagePullPolicy and imagePullSecrets out of the key branch. It does not: the branch requires a letter before image, which a key beginning with image has nothing to supply, so neither matches with the colon or without it, even when the value carries a slash. The colon earns its place elsewhere, as the point the value scan starts from, and removing it reddens the suffixed-key test rather than the pull-policy one. A reader acts on the recorded reason, so a wrong one costs more than none. Refer to that test by name rather than by ordinal: the number comes from the runner's output order and moves when anything is inserted above it. While here, give the pull-policy test a failure message about pull policies: it read 'a PR-edited package was overlaid', which belongs to a different case. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
609a014cc4 |
fix(ci): make the clean-run overlay test able to fail, and narrow the key branch
`! grep -q …` is a pipeline preceded by `!`, which POSIX exempts from set -e, and hack/cozytest.sh appends `return 0` to every converted test function, so a failed negation was discarded twice over. That line was the only assertion in 'no annotation when nothing drifted': mutating the drift>0 guard to `true` left the suite green under both runners. Replace all three negations with an explicit if/exit; the mutation now reddens test 19. The prefixed *Image branch matched any key ending in image, including operator-settable ones the build never stamps: vddkImage (core/platform, migration-controller) defaults to "" and a change there was absorbed from the artifact in silence, the opposite of what this script is for. Require the value to hold a repository path, which a ref always does and "" does not. The annotation counted ref-bearing files while naming deduplicated dirnames, so one package with a drifted values.yaml and a drifted images/*.tag read as two, and named <pkg>/images, which is not a package. Count the named list instead. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
e9f666bc25 |
fix(ci): stop a blank line lost to yq from costing a package its overlay
The overlay keeps a package on its committed release ref when the artifact copy
differs from the repo on any line that is not image-reference-bearing. A blank
line is such a line, and the build produces one on its own: yq -i re-emits the
document when it stamps a digest, and a blank line separating two top-level keys
does not come back.
So a file the PR never touched differs from the artifact on a line carrying no
configuration, and loses its overlay for it. That was the whole of the remaining
drift list. Against the public nightly artifact at its own revision, with the
tree checked out at that same revision:
before drift=5 overlaid=29 core/platform, cozystack-api, dashboard,
kubeovn-webhook, linstor-gui
after drift=0 overlaid=34
Each of the five differed from the artifact by one blank line and its own ref,
nothing else, which is reproducible locally by running the stamping assignment
over the committed file. cozystack-api and dashboard are not peripheral: an e2e
result about either was a result about the last release rather than about main.
Discount blank lines before the test. A line with content still counts, so a
real config difference keeps the committed ref as before.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
|
80a41e4e6f |
test(ci): stop the overlay tests installing EXIT traps, and pin the key branch
hack/bats-no-exit-trap.bats audits every .bats file under hack against the debt its own header declares, and this file was installing three traps beyond it. An EXIT trap inside an @test body replaces the one the bats binary installs for its own bookkeeping, so a test that then fails prints no TAP line at all and the suite reads as green. That is the reason the count is exact rather than a ceiling, and the reason the answer here is to stop installing the handler rather than to raise the number. Clean up at the end of the body instead: both runners set -e, so on failure the cleanup is unreachable and the scratch directory survives for inspection. The key branch had nothing holding its shape. A test now pins each half: that a suffixed key such as clientImage is read as a ref, and that a key merely beginning with image is not. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
8ac99b67cc |
fix(ci): read a suffixed image key as a ref, not a config change
The classifier accepted a key only when it was exactly image, repository, registry, tag or digest. A chart that renders several images names the extras after what they are for, so backupstrategy-controller carries chBackupClientImage, rabbitmqBackupClientImage and redisBackupClientImage. The first two reach the classifier through the @sha256: branch; the third is pinned by tag alone, so it read as a config change and cost the WHOLE file its overlay. That package therefore kept its committed release ref in every lane that did not rebuild it, and served an image from v1.6.0 (22 July) against a tree carrying four backup strategy drivers added after it. Five of the seven chainsaw failures behind #4257 trace to that, across three unrelated pull requests. Accept a prefix before image in the key branch. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
618ab58482 |
ci(e2e): name the packages that kept release refs instead of current main
A file whose config differs from the artifact beyond its image lines keeps the committed ref. That is the safe answer, and it is not a neutral one: the component then runs its last release image in a lane whose job is to exercise the tree under review, and the only trace is one line among hundreds of overlay lines. Six packages took that path on a recent run, backupstrategy-controller among them, which is why four backup round-trip suites refused Kinds the tree supports and three unrelated pull requests each spent a round establishing the red lane was not theirs (#4257). Summarise the drifted packages in one GitHub annotation. Nothing about which files are overlaid changes; a run with no drift emits nothing. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
80ab355d66 |
test(ci): hold the fixture's config guards from CI
make_repo silences commit.gpgsign, tag.gpgsign and core.hooksPath because a contributor's global configuration reaches a fresh repository. A runner carries no such configuration, so none of the three is exercised there: the suite stays green with any of them deleted, and the failure lands only on the machine of whoever signs their commits or tags. That is how tag.gpgsign came to be missing without anything going red. The new test supplies the hostile configuration itself, so removing a guard now fails in CI rather than on a contributor's first local run. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
5a777e9cfc |
test(ci): isolate the commit-trailer fixture from global tag signing
make_repo silences commit.gpgsign in the fixture repository but not tag.gpgsign, so a contributor who signs tags has that setting reach the fixture. git tag start then becomes an annotated tag, git refuses it for want of a message, and since the helper opens all 27 tests the suite fails before a single one reaches its body, taking make unit-tests with it. CI never saw it: the Commit trailers job runs the checker against explicit SHAs and does not touch this file, and a fresh runner has no global configuration to leak. The failure only ever lands on the local run that contributing.md asks for before pushing. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
637098f222 |
test(kafka): anchor the release selector the hook suites assert on
Both suites matched "strimzi.io/cluster=kafka-test" as a bare substring, and that is a prefix of the sibling release kafka-test2 and of kafka-test-topics. Rendering the hook with either selector left both suites green, so neither pinned the scoping they exist to pin. Anchor on the leading "-l " and on the trailing delimiter; both variants now red in both suites, and the six earlier hook mutations still red exactly where they did. Drop the notMatchRegex on a leading slash. It cannot fail on its own: the anchored image pattern above rejects a leading slash first, and an empty _cluster.images-registry renders no slash to begin with. Its rationale moves into that pattern's comment. Record why activeDeadlineSeconds and the 180s wait have to move together. A deadline that fires mid-wait kills the Job, and with backoffLimit: 0 the hook then returns non-zero, which is the wedged uninstall the deliberate exit 0 on the timeout branch exists to avoid. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
415aa82409 |
test(kafka): drive the hook suite through the runner CI actually uses
The suite used bats' run helper. make bats-unit-tests does not invoke the bats binary: it runs hack/cozytest.sh over every hack/*.bats, and that runner implements @test and little else, so the file died at 'run: command not found' and took make unit-tests with it. Under bats it was 5/5, which is where the 5/5 in the previous commit came from. Save $? by hand instead, the shape app-cleanup-hooks_test.bats already uses; verified under both runners. The fake kubectl no longer defaults DELETE_RC to 0. An unset value made every failure-path assertion vacuous; now it exits 64 and says so. Cover both RESTMapper wordings for an absent CRD rather than just the first. Widen the image pattern to v[0-9]+\.[0-9]+(\.[0-9]+)? like harbor and mariadb: pinning v1\.32 reintroduced the breakage the digest was made pattern-based to avoid, one Renovate bump later. Assert the _cluster.images-registry override and the no-leading-slash default, neither of which was covered: the hook is the one image in this chart whose pull failure surfaces at uninstall. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
3326e0fb9e |
test(kafka): run the pre-delete hook against a fake kubectl
The hook's exit code is the whole contract: a non-zero one returns from Uninstall.Run before any release resource goes, keeping the topic operator alive; a zero one lets Helm remove it. delete_test.yaml matches manifest text, so it stayed green when the timeout branch was put back to exit 1, when the CRD-absent branch was, and when the selector was widened past this release. Add a bats suite in the shape hack/app-cleanup-hooks_test.bats already uses for qdrant and harbor: render the script, run it against a fake kubectl, pin the exit code of each branch. Each of the three mutations above now turns it red. Drop hook-succeeded from the delete policy and reclaim the Job with ttlSecondsAfterFinished instead. The script exits 0 on the degraded path too, where topics still hold the finalizer, so deleting on success removed the only record of that seconds after it was written. Assert both in delete_test.yaml. Pin the hook image by shape rather than by literal digest: Renovate rewrites digests under packages/*/templates/hooks/ and nothing rewrites tests/, so the literal turns the next bump red for no defect. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
210b7fe30c |
fix(kafka): stop the pre-delete hook from wedging the uninstall
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
9d2fddd4e6 |
test(kafka): assert release topics exist before the deletion check
The post-delete absence assertions could pass vacuously if the two release topics were never created. Asserting both topics first makes the E2E prove that the pre-delete cleanup actually removed them. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
44ba13b527 |
fix(kafka): delete release topics before the topic operator is removed
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
070dcb1e30 |
fix(e2e): wait for the redis replica link instead of sampling it once (#4252)
## What this PR does The `verify-tenant-ca-projection` step in the redis suite read `INFO replication` from both pods once and failed the run when a replica reported `role:slave` without `master_link_status:up`. That value is still in motion at that point, so one sampling decided the result. Redis reports the link up only once a replica reaches `REPL_STATE_CONNECTED`. The operator's readiness script admits a replica on two other predicates: `master_host` being off `127.0.0.1`, and `master_sync_in_progress` not being `1`. `master_host` is set the moment `REPLICAOF` is issued, ahead of the link coming up, and `master_sync_in_progress` is `1` only during the transfer phase, so every handshake state between the two is ready and unlinked. A dropped link re-enters the same window. `readyReplicas: 2` therefore establishes nothing about the link, and the comment on the step calling the check deterministic was wrong. The step now polls the topology to a bounded deadline and fails only if it does not settle, reporting each pod's observed state and the elapsed time. Nothing models the link declaratively: the RedisFailover CRD defines no status at all. So the poll carries the `TODO(e2e-replace-fixed-timeouts):` marker the e2e conventions require for a wait with no object behind it. What the step does now is wait for convergence where it used to sample once. No claim about flake rate follows from that. A green run says nothing about a failure that shows up in a minority of runs. Shell embedded in a chainsaw step has no unit-test surface in this repository, so this change is exercised by the suite and nothing else. Two pre-existing diagnostics defects in the same file turned up while reading it, tracked in #4246 and #4250. Neither is touched here. The first one matters a little more after this change: the poll runs its read path about a dozen times per step instead of twice. Closes #4238 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I walked the trigger map against the diff rather than the title. The change edits one file under `hack/e2e-chainsaw/`, moves and renames nothing, and alters no make target, so none of the listed triggers fire. The node-prerequisite trigger that reaches `ansible-cozystack` and `talm` is `hack/e2e-prepare-cluster.bats`, which this does not touch. ### Release note ```release-note fix(e2e): the redis TLS suite waits for the replica link to come up instead of reading it once, so a link that is still establishing no longer fails the run ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Tests** - Improved Redis TLS topology validation by allowing replication status time to stabilize before verification. - Extended the test operation timeout to reduce failures caused by slow environment startup or synchronization. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |