mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
ace69cb0750d2fa94f45b415e1ea10cbf14715bf
7101
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
ace69cb075 |
docs(controller): say what a nil DataVolume reader means
The field comment still said DataVolumes do not count while the reader is nil, and that the reader appears once the kind is served. Neither holds: the reader is installed only after the informer has synced, and until then a monitor keeps its stored verdict, which still decides Operational. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a6a32d79cd |
fix(controller): recompute stored DataVolume verdicts once the reader is in
A reconcile that runs before the DataVolume watch starts finds no reader, keeps the monitor's stored verdict and is not requeued. The watch's initial replay then delivers only the DataVolumes that still exist, so a monitor whose DataVolume was deleted while the controller was down kept reporting it not ready until the periodic resync, hours later. Installing the reader now queues every monitor that holds a DataVolume verdict, through the controller's own queue, so each is recomputed against the synced cache. Monitors without a verdict are left alone: the replay already reaches every DataVolume they could select. If the monitors cannot be listed, the next discovery poll retries. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9f72f0bda9 |
feat(api): refuse a WorkloadMonitor with an empty selector
An empty selector matches every Pod, PVC, Service and DataVolume in the namespace, so one such monitor goes non-operational on any import there. No chart renders one, but the schema marked the field required without requiring any label, and the API accepted it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
694f4f2ff9 |
fix(controller): report a disk's PVC Workload by its DataVolume
CDI copies the DataVolume labels onto the PVC it creates, so the vm-disk monitor also publishes a Workload for that PVC, and its readiness was bind state alone. Mid-import the PVC is already Bound: the Workload read operational while the monitor, and the Application's WorkloadsReady, said the disk was not populated. A PVC whose controller is a DataVolume the monitor finds not ready now reads not operational. While the DataVolumes cannot be read, its Workload keeps the verdict it has, as the monitor does. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
366b5e80f4 |
perf(controller): skip the monitor status write when nothing changed
While the DataVolume watch syncs, every WorkloadMonitor of every kind is requeued every few seconds, and an informer that cannot sync keeps that window open. Each pass wrote status unconditionally, so a cluster with many monitors sent the API server a steady stream of writes that changed nothing. Comparing against the stored status first leaves the requeue as cache reads only. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
dd72583dfd |
fix(controller): record an event when DataVolumes cannot be read
A failed DataVolume List keeps the monitor's last verdict and retries, and the only trace of it was a controller log line. A monitor with no verdict yet then reads operational while its disks are unreadable, with nothing a tenant can see saying why. A Warning event on the monitor makes the failure visible where the bucket metrics failure already is. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
beeb7994cc |
fix(controller): start the DataVolume watch source only once
A controller-runtime Kind source adds an event handler to the informer when it starts and never removes it; cancelling it on a sync timeout only ends its context. Building a new source on every failed sync attempt therefore left one more handler on the DataVolume informer per retry, each enqueueing the same events, until the controller restarted. The source is now started once and later attempts only wait for the informer to sync. Because the started source keeps delivering events between attempts, the syncing flag stays set until the reader is installed, so a reconcile queued in that gap is requeued as well. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a9b276f99b |
fix(controller): requeue monitors reconciled while the DataVolume watch syncs
Starting the DataVolume source replays every DataVolume to its handler at once, and the handler queues the matching monitors, while the reader is installed only after the source has synced. A monitor reconciled in that window saw no reader, kept its last verdict and returned with nothing left to queue it again, so a disk that failed or finished while the controller was down could keep a wrong WorkloadsReady until the next unrelated event. Such a reconcile still publishes the replica counts and now asks to be requeued after a few seconds, until the reader is in place or the sync attempt gives up. It is a requeue rather than an error because the window is expected: an error would count as a reconcile failure and log a line for every monitor in the cluster while it lasts. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f447ab8ebd |
docs(vm-disk): explain which condition reports the disk populated
Ready follows the Helm release; whether CDI finished is in WorkloadsReady. Say so and where an operator sees it, say that a disk created before this change keeps a release that may wait on the DataVolume until its next update, say that an upload disk reads not ready until the upload completes, and say that a disk waiting for its first consumer counts as ready, so it is not read as broken. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
6170c1615f |
feat(dashboard): turn the status badge amber while a disk is not populated
The console badge followed Ready alone, and Ready follows the Helm release. A vm-disk install does not wait on its DataVolume, so a disk CDI never populated was painted green. The badge now shows the WorkloadsReady condition while Ready is True and WorkloadsReady is not True with reason DataVolumeNotReady. Other causes of WorkloadsReady=False keep the badge as it was: a stopped VM has no pod and reports not operational on purpose, and turning it amber for every kind needs its own decision. A failing Ready still wins, because it is the more specific cause. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
99f3d3b2b7 |
feat(controller): tag a DataVolume cause with a condition reason
WorkloadsReady=False covers different situations: a stopped VM has no pod and its monitor reports it not operational on purpose, while a disk whose DataVolume never populates is broken. A consumer that wants to react to one and not the other had only the message text to go by. The WorkloadMonitor status gains a Reason next to Message, set to DataVolumeNotReady by the DataVolume check, and the Application API carries it into the WorkloadsReady condition in place of the generic reason whenever a monitor that is not operational names one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b867ea1eda |
feat(api): show WorkloadsReady in the Application table
Ready follows the Helm release, so for a kind whose install does not wait on its workloads, a vm-disk among them, READY in kubectl get says only that the manifests were applied. Whether the workloads came up lives in WorkloadsReady, which the default table did not show, so a disk CDI never populated looked healthy on the surface an operator reads first. The table now carries a WORKLOADS column beside READY. It reads <none> for an Application with no WorkloadMonitor, which has no WorkloadsReady condition at all. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
81ffcda86f |
fix(controller): keep publishing monitor status when DataVolumes are unread
The DataVolume List sits on the reconcile path of every WorkloadMonitor, and returning its error before the status write froze the replica counts and Operational of a monitor that selects no DataVolume at all whenever CDI's API hiccupped. The status is now written either way, with the last DataVolume verdict kept whenever the DataVolumes could not be read, and the error still returned so the request is retried. The same applies while no reader is installed yet, so a stuck disk does not read as ready for the window between a controller restart and the DataVolume informer syncing. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
48ece79ffc |
feat(controller): name the DataVolume that keeps a disk not ready
For a vm-disk, WorkloadsReady is the one condition that reports whether CDI populated the disk, and its generic "One or more workloads are not operational" gives an operator nothing to act on: a disk has no workloads, and the stuck DataVolume and its phase are exactly what they need to know. The WorkloadMonitor status gains a message that the controller fills with each selected DataVolume that is not ready and its phase, and the Application API uses the messages of the monitors that are not operational in WorkloadsReady instead of the generic text whenever one of them names a cause. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d1aea3f40e |
test(e2e): assert vm-disk readiness through WorkloadsReady
The VMDisk scenarios waited on the HelmRelease, which does not cover the import behind it. They now wait for the DataVolume to be Ready and then for Ready and WorkloadsReady on the VMDisk itself. The DataVolume wait keeps the combined budget the two old waits had. That alone would pass with the monitor ignoring DataVolumes, since an empty monitor reports operational. A new scenario imports from an unresolvable URL and waits for Ready True with WorkloadsReady False, which only a monitor that reads the DataVolume phase can produce. A DataVolume that has not started is not ready either, so the scenario first waits for the importer to fail, which CDI reports as Running False with reason Error, and only then asserts the VMDisk. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f4b1f08adb |
fix(vm-disk): keep the Helm install from waiting on the DataVolume
Helm's default poller falls back to a resource's Ready condition for kinds it has no rule for, so a vm-disk install could wait on the DataVolume's Ready and hold a helm-controller worker for the whole install timeout while an import runs or never completes. Whether CDI populated the disk is reported by the WorkloadMonitor, so the release now treats the DataVolume as current and Ready of a vm-disk no longer depends on the import. Release settings reach a HelmRelease when its application is created or updated; the reconciler that syncs definitions onto existing releases copies the chart reference, valuesFrom and labels only, so an existing disk picks this up on its next update. The vm-disk-rd package had no unit test target; it gets one, and a test that pins the entry. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bf7e8f5bbb |
fix(controller): count DataVolume readiness in WorkloadMonitor
A vm-disk application reported its workloads operational whatever state its DataVolume was in, so an import or clone that never finished left the disk looking healthy while it was unusable. Its monitor has minReplicas 0 and only pods fed Operational. The monitor now also lists the DataVolumes its selector matches and reports not operational while any of them is in flight, Failed, Unknown or without a phase. UploadReady counts as in flight: CDI holds it from the moment the upload server is ready until the transfer succeeds, so it covers an empty disk and a half-uploaded one alike. Waiting for the first consumer counts as ready, since that is where such a disk rests, and so does a phase this controller does not know, so a CDI release that renames one cannot turn working disks red. This is deliberately not a Helm wait: a readiness gate holds a helm-controller worker for its whole window, and a disk that never populates would keep taking it back. The signal lands in the application's WorkloadsReady condition instead, and Ready keeps following the HelmRelease. CDI ships with the IaaS bundle only and can be installed while this controller is running. A watch on a kind whose CRD is absent keeps the controller from starting, so the DataVolume watch is added from a runnable that polls discovery until the kind is served. It starts on the first reconcile, the earliest point at which a new watch is started at once, rather than on the next tick, so a disk created just after a restart is not read as populated for a poll interval. The reader is installed only once that watch has synced within a bound: a reconcile context has no deadline, and a List on an informer that never syncs would park every WorkloadMonitor reconcile for good. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2df29639df |
fix(clickhouse, workloadmonitor): add the cluster domain to keeper hosts and stop a PVC nil dereference (#4608)
## What this PR does Two small bug fixes, one per commit. Both were first proposed by @officialasishkumar in #2334 and #2336. Those PRs went stale, and part of each was fixed on main in the meantime, so this carries only what is still broken. ClickHouse Keeper hosts rendered without a cluster domain, as `chk-<name>-keeper-cluster1-0-0.<namespace>.svc.` with nothing after `svc.`. The zookeeper block in `clickhouse.yaml` redeclared `$clusterDomain` from `.Values.clusterDomain`, which the chart never defines, and that shadowed the value read from `_cluster.cluster-domain` at the top of the template. Keeper is enabled by default, so every ClickHouse release got these hosts. The keeper name test set `clusterDomain` itself and hid the bug. It now sets only `_cluster.cluster-domain`, like the platform does, and fails without the fix. The WorkloadMonitor controller panicked on a PVC with no `storageClassName`, which is any PVC that takes the cluster default class. The check used `||` where it needed `&&`. controller-runtime recovers the panic, but the reconcile of that monitor fails on every retry and never reaches its status update. A PVC with an empty `storageClassName` also got a resource key with an empty class name instead of `default`. A new table test covers unset, empty and explicit classes. The duplicate `template` key in the ClickHouse backup CronJob from #2334 and the `Operational` status fix from #2336 are already on main, so they are not in this PR. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(clickhouse, workloadmonitor): ClickHouse Keeper node hostnames now include the cluster domain, and the WorkloadMonitor controller no longer panics on a PVC without a storage class. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Workload resource labels now use the default storage class when a PVC’s storage class is unset or empty, while continuing to reflect explicitly configured storage classes. * ClickHouse ZooKeeper hostnames now consistently use the configured cluster domain, or `cozy.local` when no domain is set. This helps hostnames align with the cluster’s domain configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a7ac37be65 |
test(e2e): let HR-Ready waits outlast the install timeout of the release they wait on (#4471)
## What this PR does Stacked on #4468, which holds the raised backup waits to a start margin; review that one first. This PR's own diff is the six commits on top of it. Most HR-Ready waits in the E2E suites gave up before the release they were waiting on could fail its install. cozystack-api gives every release it generates a 600s install timeout, and the Chainsaw asserts in front of those releases sat at 5 or 6 minutes. When an install genuinely failed, the suite reported a bare assert timeout against a release that was still installing, and the reason the install failed, which the release records when its own timeout fires, was never reached. Red runs failed just as often as they should; they only said less. I raised every HR-Ready wait above the install timeout of its own release, by the 60s start margin #4468 introduces: - Chainsaw asserts on releases cozystack-api generates go to 11m, across bucket, clickhouse, etcd, external-dns, foundationdb, harbor, mongodb, openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and vminstance. - Asserts on nested `<name>-system` releases take their floor from the template that renders them: 11m for external-dns, openbao and qdrant (10m `spec.timeout`), 16m for `harbor-test-system` (15m). `harbor-test` sat exactly on its 600s floor, which expires first because the assert's clock starts before the install's; it goes to 11m too. - The kuberture Package release takes its 10m from cozystack-operator and goes to 11m. - In the backup walkthroughs, the bucket waits go from 300s to 660s in clickhouse, kafka, mariadb, mongodb, postgres, rabbitmq and redis, and the two postgres waits go from 360s to 660s. The kafka bucket wait passed no budget and silently took the helper's 300s default, which is why earlier scans for literal budgets missed it. Each Chainsaw op that runs a walkthrough grows by exactly what its waits gained (clickhouse 78m, kafka 46m, mariadb 48m, mongodb 42m, postgres 66m, rabbitmq 48m, redis 36m), so a failure at any later wait keeps the headroom it had and is still reported by the script instead of being cut off by the op. Two waits stay as they are, on purpose: - kafka: `kafka-rd` sets a 7h install timeout, which no budget inside the 215-minute E2E job can clear. The kafka assert and the Kafka waits in the kafka and kafka-metadata walkthroughs keep their numbers, and the assert now says why in-line. - `vm-instance-test`: `vm-instance-rd` disables the install wait, so its release goes Ready as soon as helm has applied and there is no floor to clear. Earlier lists of this problem included it by mistake. The cost falls on red runs only: a suite whose release never goes Ready now spends up to six more minutes before it reports, and in exchange the report carries the release's own reason. Green runs are unchanged. Suites run one after another in a single job capped at 215 minutes, so a systemic failure that reddens many suites now reaches that cap sooner, and a job cancelled at the cap loses its log and `cozyreport.tgz`. I took that trade because a failed install that says why is worth more than a red run that ends a few minutes earlier. `docs/agents/e2e-testing.md` stopped being true in several places once the waits were raised: it described most of the tree as under the floor, mariadb as the only suite raised, the walkthrough bucket waits as under it, and harbor's asserts as sitting on and under theirs. I rewrote those sentences, and the one that pointed at "every wait above the band" of 5-6m, kept the start-margin wording from #4468, and added that a walkthrough wait passing no budget takes the helper's default. The comments in `hack/mariadb-first-boot-waits.bats` that quoted the old op and bucket figures are restated; its existing checks and constants are unchanged, because every op grew by exactly its bucket delta and each remainder still sits at exactly its constant. I also added postgres to the table that file holds the walkthroughs to, so its raised waits and its op cannot drift back apart unnoticed, and a test that holds each table suite's bucket wait to the bucket release's floor, read from the api-server default with `bucket-rd` checked for an override; the op check alone could not catch a bucket wait dropping back, since a lower wait only widens the slack. redis stays out: its walkthrough waits on no redis HelmRelease, only on the bucket and on the sentinel master, so the table's floor check has nothing to read there. How I checked it, since E2E cannot run locally: - I re-ran the enumeration from the sizing section of `docs/agents/e2e-testing.md` over every `hack/e2e-chainsaw/*/chainsaw-test.yaml` and every `wait_hr_ready` call under `examples/backups/`, literal or default, and held each against its own release's floor. Every `Ready=True` assert and every `wait_hr_ready` call a suite drives now clears its floor, except the kafka ones named above. The opt-in etcd round-trip uses bare `kubectl wait hr` calls at 300s instead of that helper; they are listed in #4470. The `Ready=False` assert on the rejected redis upgrade is not a readiness wait and is left out. - `make bats-unit-mariadb-first-boot-waits` passes, 31 of 31. For the new postgres row, putting the op back to 50m fails only the postgres slack test, and a postgres wait at 659s fails only the postgres floor test. Setting any one table suite's bucket wait to 659s fails only the bucket test, naming only that suite, and an install-timeout override on `bucket-rd` fails it too. I ran the mutations under plain `bats` so a fail-fast runner could not hide a second failure. - `hack/select-e2e.sh` over this diff selects bucket, clickhouse, etcd, external-dns, foundationdb, harbor, kafka, kuberture, mariadb, mongodb, openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and vminstance. Out of scope, and tracked in #4470: the tenant addon waits in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` (an edit there escalates the whole E2E run), the redis and kafka-metadata `wait_hr_ready` helpers that print no reason on timeout, and the opt-in etcd round-trip's bare `kubectl wait` calls. Known and left for later, none of them changed here: - Only the mariadb Chainsaw asserts and the walkthroughs in the guard's table are held by a test. The other raised asserts, and the redis and kafka bucket waits, rest on literals with in-line reasons; #4470 covers extending the guard. - The Kafka waits in `examples/backups/kafka/04-create-kafka.sh`, `07-restore-to-copy.sh` and `kafka-metadata/run-all.sh` stay at 300s against the 7h floor without an in-line reason at the call site. The doc's rule asks that of Chainsaw asserts only, and the kafka Chainsaw assert carries it. The postgres waits are the ones #3748 is about. Their budget now clears the install-timeout floor and a test holds it there; what a CNPG first boot costs on this platform is still unmeasured, so that issue stays open. Closes #3771 Closes #3753 ### Screenshots Not a UI change. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against the diff. It touches E2E suites, backup walkthrough scripts, a unit-lane bats file and an agent doc, none of which any downstream repository restates. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(tests): HelmRelease readiness waits in the E2E suites and backup walkthroughs now outlast the install timeout of the release they wait on, so a failed install reports its own reason instead of a bare timeout. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Extended readiness and backup round-trip test timeouts across multiple services to accommodate slower startup and provisioning. * Added checks to verify that bucket readiness waits meet configured installation-timeout requirements. * **Documentation** * Updated testing guidance to clarify how readiness waits relate to release-specific timeouts and overall test budgets. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
360f03bb8b |
feat(monitoring,tenant): shared-central tracing topology (phase 4) (#3775)
## What this PR does Adds the opt-in shared-central traces topology from the distributed tracing design (cozystack/community `design-proposals/distributed-tracing`, rollout item 4): a tenant can send its spans to one VictoriaTraces store in tenant-root instead of running its own, with both write and read isolation between the tenants that share it. **Switching it on.** A tenant opts in with the new Tenant field `tracingCentral`, which needs the tenant's own Monitoring. The tenant chart publishes it to the tenant's monitoring charts as `_namespace.tracingCentral` in the `cozystack-values` Secret, which the apps API does not let an application write, and renders the two network paths the topology needs under the same condition. tenant-root decides whether it hosts the store, with the new Monitoring field `tracingCentralHost` next to a cluster-mode `tracingStorages` entry named `generic`. **Isolation.** VictoriaTraces takes the tenant account from the `AccountID`/`ProjectID` headers and does no authorization of its own, so other tenants reach the shared store only through a vmauth in tenant-root. Each central tenant gets a VMUser that pins an account derived from its namespace name and UID, for both its collector's writes and its Grafana reads, and vmauth overrides any account a client sends. The collector and Grafana log in with credentials the chart generates and keeps. The tenant egress rules allow only pods carrying the collector's or Grafana's label to reach the vmauth proxy port, and a pod that carries such a label without the tenant's credentials gets only 401 from it; the vmauth internal routes, whose `/metrics` lists every tenant's user, are served on a port tenants cannot reach. The tenant-root namespace itself is inside the trust boundary, as it already is for the metrics of tenants that rely on a parent's monitoring: its workloads reach the store directly, and one that learns a tenant's account ID can read and write that account, and whoever tenant-root admits to its Grafana reads the store through a datasource that bypasses vmauth (with the System oidc mode, tenant-root's own groups, which already hold the same access level in every tenant under it); the `tracingCentralHost` description says so. **Sharing the store.** Each tenant's VMUser is capped at four requests in flight per vmauth replica, reads and writes together, and each collector replica exports with two senders, so more collector replicas do not raise a tenant's throughput; above the cap vmauth queues for up to 10s and answers 429, which the exporter retries. The store has one disk and one retention for all of its tenants and nothing reserves space per tenant; `tracingCentralHost` states that, since hosting is tenant-root's decision. The vmauth runs two replicas spread across nodes under a PodDisruptionBudget, so draining one node does not stop every central tenant at once. Every traces store is also disk-bounded now, as the design requires: without `retentionDiskUsageBytes` it drops its oldest day once its filesystem is 80% full. That alone does not keep one tenant from filling the shared disk, since VictoriaTraces always keeps the last two days. **States that would drop spans silently are refused, one race aside.** A live render fails, with a message naming the prerequisite, when a tenant turns central on while tenant-root hosts nothing, and when tenant-root stops hosting while central tenants remain; tenant-root's render also refuses `tracingCentralHost` without a store to host. The second check does not wait on a VMUser that is already being deleted, since its tenant is leaving. Each check reads the other side's objects when it renders, so a tenant turning central on while tenant-root stops hosting can pass both and export to a vmauth that is gone; the `tracingCentralHost` description names this race and its recovery. At runtime, tenant-root's Monitoring release is not Ready while the vmauth has failed. These checks run on a render, so removing the tenant-root Monitoring altogether is not refused, and from then on every Monitoring render of a central tenant fails until it turns central off; the `tracingCentralHost` description says to turn central off on the tenants first. Turning central on keeps the tenant's own local stores and their datasources, so traces stored before the switch stay readable until the tenant removes them; turning it off again leaves the spans written meanwhile in the shared store, unreadable from the tenant until it turns central on again, and spans sent while the collector switches back may be dropped; a deleted tenant's spans likewise stay until retention drops them. Right after a tenant turns central on, or after its credentials Secret is re-created, spans its collector sends before the vmauth has loaded the new login get a 401, which the exporter does not retry, so they are dropped; the tracing guide in #3772 covers it. **Where this departs from the design.** The design has the per-tenant collector inject the account headers and adds exactly one egress rule. Here the account is pinned by the tenant's VMUser at the vmauth and the collector sends none, because the collector runs in the tenant namespace and whatever runs with its pod label could otherwise choose an account; vmauth is also what the design already requires for reads, so both directions go through it. That needs a second egress rule, for the tenant's Grafana to reach the vmauth, beside the collector's. The per-tenant write rate the design gives the collector is not in this PR: no OpenTelemetry distribution has a rate-limiting processor, so #4648 builds the collector with one and holds each central tenant to a rate tenant-root publishes. Until it lands, a tenant writing faster than the disk cap allows over two days can fill the disk the other tenants share. **Consent is per host, not per tenant.** `tracingCentralHost` lets every tenant that sets `tracingCentral` in, nested tenants included, and each gets its own VMUser and its own request cap; there is no allow-list. A tenant-root that needs to limit who shares its store keeps `tracingCentralHost` off. **Upgrade.** Tenants that do not opt in render the same objects as before, and their `cozystack-values` Secret is unchanged. The monitoring charts render the same objects too, with one exception: a VTSingle or VTCluster without `retentionDiskUsageBytes` gains `retention.maxDiskUsagePercent: "80"` in its `extraArgs`, so its spec changes and its storage pods restart once, and a store already more than 80% full drops its oldest days when it starts. The inner monitoring HelmRelease's values gain the new `tracingCentralHost: false` default, as with any new field, and its `healthCheckExprs` gain a `VMAuth` entry, which matches nothing outside a hosting tenant-root. Both change the HelmRelease spec, so every tenant's monitoring release upgrades once, rendering the same objects. A tenant-root that has a cluster-mode `generic` store gets no vmauth until it sets `tracingCentralHost`. **Testing.** helm-unittest covers the three charts the change touches (tenant, monitoring, extra monitoring) and every guard was checked by mutation; a bats test covers what helm-unittest cannot, that the generated password and the collector's rollout checksum agree on a first install. On a test cluster: a spoofed `AccountID` from a tenant is overridden by vmauth, one tenant cannot read another's spans, the vmauth starts under the real tenant network policies (and stays in init without its apiserver label), the guards refuse as described, and the rendered collector config passes `otelcol validate` in the pinned image. There is no end-to-end test yet; one that covers cross-tenant isolation is planned as a follow-up. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: covered by cozystack/cozystack#3772, the tracing guide - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#64 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The Tenant gains the user-facing field `tracingCentral` (bool, default false), which the trigger map sends to terraform-provider-cozystack: its hand-written Tenant schema and model need the field. The follow-up is cozystack/terraform-provider-cozystack#64. The Monitoring gains `tracingCentralHost` (bool, default false); the provider's `cozystack_tenant_module` is a marker without values, so it needs no change. The website reference pages for both are generated from the READMEs on release. The tracing user guide in #3772 covers the per-tenant backend only. A central-mode section in the tracing guide of #3772 (hosting and opting in, the shared disk and retention, turning central off on the tenants before tenant-root stops hosting, and the recovery from the opt-in/stop-hosting race) also takes the lifecycle details the `tracingCentral` description no longer carries. ### Release note ```release-note feat(monitoring,tenant): add an opt-in shared-central traces store. A tenant sets tracingCentral to send its spans to one VictoriaTraces store in tenant-root instead of its own, and tenant-root opts in to hosting it with tracingCentralHost on its Monitoring. Tenants sharing the store are isolated on write and read through a vmauth in tenant-root, and each is capped in the requests it can hold in flight. ``` |
||
|
Aleksei Sviridkin
|
2d0c359f4c |
test(e2e): stop bringing up the monitoring instance nothing looks at (#4023)
## What this PR does This PR turns off monitoring instance of root tenant in e2e install and removes waits on it. Install patched root tenant with `monitoring: true` and then waited up to 15m each on vmalert and vmalertmanager, `vlclusters/generic`, shortterm and longterm vmclusters, `grafana-db` CNPG cluster and grafana deployment, but nothing in the run looks at any of it. No chainsaw suite touches tenant-root monitoring objects. Only one that mentions monitoring is `foundationdb`, it asserts `WorkloadMonitor` whose CRD comes from `cozystack-controller` and exists without monitoring stack. Monitoring OIDC wiring is covered by `packages/system/monitoring/tests/oidc_test.yaml` since #4019 removed orphaned BATS suites. Kubernetes suites run with `monitoringAgents.enabled: false` and tracing is off by default (`tracingStorages` is empty), so they don't depend on this instance either. Those waits were not free. Monitoring stack competes with rest of install for node capacity, and removed block own comment said slow victorialogs start can fail install on PR that never touched monitoring - that's why it was moved to 15m budget. ### What stays Monitoring package stays installed, because `tenant-application` has `dependsOn cozystack.monitoring-application` and without the package platform release is never resolved. Only tenant-root instance is off. Container lane `logsStorages` override for victorialogs also stays. It does nothing while monitoring is off, but keeps a run that turns monitoring back on from creating immutable replicated PVC on lane that has only `local`. Its comment, paragraph in `docs/agents/e2e-testing.md` and comment on Install step env in `pull-requests.yaml` now say that instead of calling root monitoring install-time workload. ### What is lost `curl` through `root-ingress-controller` with `Host: grafana.example.org`, it was the only check that root ingress actually routes to backend. Grafana was the target just because it was installed. Moving it to seaweedfs host needs expected response checked on live stand first, so it's tracked in #4615. ### Savings Not measured. The happy path never spent the 15m budgets, so what this returns is real bring-up time of the stack plus tail risk, and only CI runs can show how much it is. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` |
||
|
Aleksei Sviridkin
|
42faf54932 |
test(e2e): drop readiness asserts that HelmRelease Ready already covers (#4652)
## What this PR does Removes e2e asserts that wait for something an earlier assert in the same test already proves. First step of [#2653](https://github.com/cozystack/cozystack/issues/2653), not the whole issue. qdrant and external-dns render the workload in a nested `-system` HelmRelease, and both suites already assert that HelmRelease Ready. The `readyReplicas: 1` on the StatefulSet and the Deployments goes, the instance-label pin stays because the catch collectors select on it. I checked on a dev cluster that a parent HelmRelease waits for its nested one (a Qdrant with a broken `storageClass` kept the parent in `Progressing` until its install timeout, then it failed naming the nested HelmRelease), details in [this comment](https://github.com/cozystack/cozystack/issues/2653#issuecomment-5929644802). vminstance: both PVC `Bound` asserts follow a DataVolume `Ready` assert, and a DataVolume is not Ready before its PVC is Bound. Left alone on purpose: the qdrant PVC assert (the reclaim-on-delete step only proves anything if the PVC was seen first, same in kafka), openbao sts and PVC (they run after unseal), operator-created workloads, and the Service and Endpoints shape checks. Chainsaw isn't installed on my machine, so the suites are checked by CI only. ### Screenshots No UI changes. ### Downstream repositories The diff only edits `hack/e2e-chainsaw/*/chainsaw-test.yaml`. The trigger map's `hack/` entry is for cozystack/ccp, whose skills anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`, none of which change here. No box ticked. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): the qdrant, external-dns and vminstance suites no longer repeat readiness checks that the HelmRelease Ready assert already covers. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated end-to-end checks to use Helm release readiness for Deployment and StatefulSet scenarios, rather than separately asserting ready replica counts. * Removed the requirement for PersistentVolumeClaims to reach `Bound` in the VMDisk and VMInstance scenarios; DataVolume readiness checks remain. * Retained instance-label checks and clarified that PVC reclaim validation requires the claim to be observed first. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
89c7ea0b46 |
fix(e2e): hold replicated-lane LINSTOR cleanup to a pre-suite baseline and delete kafka suite PVCs (#4651)
## What this PR does This fixes two cleanup failures in the kubernetes suites. The nightly runs failed `kubernetes-previous` and `kubernetes-latest` in the replicated storage lane. PR runs fail `kubernetes-latest` with `LINSTOR pools did not return to their pre-suite baseline within 300s`. The replicated lane asked for too much free space at cleanup. It wanted 90 GiB free on every LINSTOR node, and volumes from earlier suites kept srv2 at 83 GiB, so both suites failed without leaking anything (nightly run 36659663201). A replicated suite still needs those 90 GiB for its workers and CDI scratch volumes, so the floor is now checked before the suite starts. If a node is short there, the suite fails with the per-node table and an error that blames the sandbox, not the suite. Cleanup in both lanes now compares against a baseline recorded before the suite creates anything, and that baseline is no longer free bytes. On a node whose pools are all thick ZFS, `linstor sp l` does not ask the satellite for free space. The controller fetches it live only for nodes with thin pools and otherwise returns the value cached at the satellite's last report. In LINSTOR v1.33.3 that is `FreeCapacityFetcherProto.assembleRequests` filtering on `hasThinPools`, with `CtrlStorPoolListApiCallHandler` falling back to `getFreeCapacityLastUpdated`. The last report can be taken while `zfs destroy` is still freeing blocks in the background. A cleanup that had removed everything then kept polling a frozen figure 0.4 to 0.9 GiB short and failed after 300s. A byte tolerance does not fix that, because the residue is already close to the smallest volume a suite is known to have leaked (1 GiB). The baseline is now the set of ZFS datasets under each node's LINSTOR pool. It is read with `zfs list` inside that node's `linstor-satellite` container. The zpool name comes from the pool's `StorDriver/StorPoolName` property, because it is `data` on QEMU nodes and `data-srvN` in container mode. Cleanup waits until no dataset outside the baseline remains. A destroyed dataset leaves the namespace when `zfs destroy` returns, whether or not its blocks are still being freed, so background freeing no longer fails the check. A real leak, like the 21 GiB scratch volume left on srv3 in an earlier run, is reported by node and dataset name instead of a KiB difference. A node whose pool cannot be read fails the check. A run that fails before it records a baseline no longer compares against the previous suite's file. Every read the new probe makes runs under `timeout -k 5 30`, so a hung controller or satellite fails one poll and the wait still ends with its failure message. The kafka suite deleted its apps but never their PVCs. The chart keeps them on purpose with `deleteClaim: false`. One run left 35 GiB in `tenant-test`, 20 GiB of it on srv2, and the backup round trip left its two clusters' volumes in `tenant-root` the same way. Each test now deletes its clusters' PVCs by `strimzi.io/cluster` after the apps are gone and waits for them. The `kafka` test does it from a step cleanup, so it also runs on a failed run. The tests cover both fixes. `hack/linstor-storage-mode_test.bats` has cases for a clean cleanup, a volume still listed on the first poll and gone on the next while the cached free capacity stays short, a leaked volume, a node that cannot be read, a hung satellite read, a missing or stale baseline, and the replicated-lane floor. Its fake runs the real controller-side `jq` filter against a LINSTOR-shaped listing. `hack/kafka-suite-pvc-cleanup_test.bats` checks that each cluster's PVC delete is in the suite and runs after the app cleanup. One stub line goes into `hack/run-kubernetes-talos-diagnostics_test.bats`, because that test calls `cozy_cleanup` with no baseline and only checks the order of the deletes. The new guards were mutation-tested. Some limits stay as they are. The local lane waits for dataset names to disappear, not for the freed space, and has no free-space floor, so a background destroy can still be freeing blocks when the next suite starts. The 90 GiB precondition in the replicated lane still reads the cached free capacity, and its comment says so. The two `kubectl exec` reads in `cozy_wait_linstor_pool_free` were there before this PR and still have no time limit of their own. Two related problems stay out of this PR. The `kafka-metadata` suite leaks PVCs the same way, tracked in #4641. The stale comment in `examples/backups/kafka/cleanup.sh` and the missing PVC pre-clean in the round trip are in #4649. Closes #4617, closes #4616, closes #4653 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): the kubernetes suites' LINSTOR cleanup compares each node's ZFS datasets against a pre-suite baseline instead of cached free capacity, the replicated lane checks its 90 GiB floor before the suite starts, and the kafka suite deletes the PVCs its clusters keep. ``` |
||
|
Aleksei Sviridkin
|
200a7fff66 |
test(e2e): read the kafka round-trip cleanup order without yq if/else
The check on kafka-2-backup-roundtrip's finally block classified each op with if/then/elif/else, which yq only gained in v4.54.1. The CI runner's yq predates that and rejects the whole expression with a lexer error, so the test failed there while passing locally. Have yq emit the plain facts per op and let awk name it. The round-trip document is cut out before that query, because in a multi-document file yq also evaluates an array literal against the documents a select drops and prints a line for each. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
cb7a579065 |
fix(e2e): compare LINSTOR pool datasets, not free bytes, at cleanup
For a node with only thick ZFS pools, `linstor sp l` reports the free capacity cached at the satellite's last report rather than asking the satellite, and that report can land while `zfs destroy` is still freeing blocks in the background. A cleanup that had removed every volume it created could then wait out its whole budget against a figure that never moved. That residue has reached about 0.9 GiB, with no known upper bound, against 1 GiB for the smallest volume a suite is known to have leaked, so a byte tolerance has no safe value between the two. Record the datasets under each node's pool before the suite, read with `zfs list` on that node's satellite, and wait until no dataset outside that set remains. A leaked volume is then reported by name and node, and a destroy still freeing space in the background does not count. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bc400dff15 |
test(e2e): hold the wait-budget guard to a start margin across four suites (#4468)
## What this PR does `hack/mariadb-first-boot-waits.bats` let two regressions of the wait budgets it guards pass green. This closes both. The change is test and doc only: no wait, op or chart value moves. A wait now has to clear its floor by a named `MBW_START_MARGIN` of 60s, not by one second. The wait's clock starts before the install's by the Application reconcile, the chart artifact fetch and the helm-controller queue. Nothing in the tree bounds those three, so the value is a policy choice and the header says so. It equals the headroom every current wait already carries (660 against a 600s floor), which is why no value has to change. A fixture pins 660 against 600 as passing and 659 and 601 as refused. The sizing guide in `docs/agents/e2e-testing.md` asked only for "strictly above" the floor, which the guard now rejects, so it now asks for a start margin and points at the constant instead of restating the number. The mongodb, rabbitmq and clickhouse walkthrough waits and their Chainsaw ops are now held to the same contract as mariadb, through one table with a row per suite. Each suite gets its own floor test and its own op test, so a regression turns only that suite's tests red. The floor comes from the api-server's install-timeout default, and each suite's ApplicationDefinition is checked for an override. The startup-probe half is computed only for mariadb, since the other three charts set no `startupProbe`. Each op must keep the remainder it had before its waits were raised, and that raise preserved it: 900s for mariadb and rabbitmq, 1200s for mongodb, 2700s for clickhouse. The remainder is taken over every HelmRelease wait the op reaches, the bucket wait included, so raising a wait together with its op keeps the remainder instead of turning the raise into slack. The mongodb target wait sits behind the `SKIP_RESTORE=1` its op passes, so it is not reached and is left out; the test turns red once the op stops passing it. The Chainsaw HR-Ready asserts of mongodb, rabbitmq and clickhouse (6m) stay out of scope. They sit under the floor by per-suite choice, and the header says so. Verification: - The file goes from 22 to 28 tests. Four mariadb-only tests are replaced by the eight per-suite ones: the example floor, the both-waits op slack, the check that both applications are waited on and the ApplicationDefinition override. Two fixture tests drive the per-suite floor and op checks through directory, suite-file and RD overrides. They cover each way of falling short that the live tree cannot produce, since every live wait sits at exactly 660 and every remainder at exactly its constant. That includes an RD that overrides or disables the install wait, and a startup budget large enough to bind over the install timeout. - `make bats-unit-mariadb-first-boot-waits` passes, and the same file passes under dash through `hack/cozytest.sh`, as CI runs it. - Mutations, each confirmed applied and each run test by test, because the runner stops at the first failure: - Any suite's database wait lowered to 601: only that suite's floor test fails. That includes the mongodb target wait the op never reaches. - Each op lowered back to its old value (30m, 30m, 30m, 60m): only that suite's op tests fail. For mariadb that is also the existing prefix-remainder test. - `SKIP_RESTORE=1` dropped from the mongodb op: only the mongodb op test fails. - Any suite's bucket wait raised to 660, or the rabbitmq one indented out of the flow: only that suite's op tests fail. Raising a bucket wait to 660 together with its op by 6m keeps all 28 tests green. - A third clickhouse wait added in a new file under `examples/backups/clickhouse/`: the clickhouse floor and op tests fail, because the waits are read from every script in the directory rather than from a listed subset. - Any suite's target wait renamed to wait on the source again, in mariadb and rabbitmq: only that suite's floor test fails. Dropping the distinct-release check, or its `sort -u`, fails only the floor fixture test. - Inside the helpers themselves, one mutation per branch of every new or changed helper, 37 in all: each turns its own fixture test red, or the live suite checks where the branch selects which waits are read at all. Two branches whose deletion changed no output were removed rather than pinned. - `MBW_START_MARGIN` set to 0: only the two fixture tests that encode the margin fail. Deleting the constant fails every test that reads it, as an unbound variable. - `hack/select-e2e.sh` selects no suite for this diff. Left for later: - The file name. `hack/mariadb-first-boot-waits.bats` now guards four suites. I kept the name because renaming it changes its make target and every reference to it, which belongs in a change of its own. - The SKIP_RESTORE stop works within one file. A suite whose waits span several scripts, as clickhouse's do, would still count the waits in later files if its op ever passed SKIP_RESTORE=1. That errs toward a false red, not a silent pass, and no such op exists today. - Measuring the margin. The delay from Application creation to the start of the install is measurable from CI reports and would turn the 60s policy value into a bound. That delay is not bounded anywhere today. Closes #4418 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): the wait-budget guard now requires a 60s margin over each floor and covers the mongodb, rabbitmq and clickhouse backup walkthroughs alongside mariadb. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified that HR-Ready waits should exceed both the app’s first-boot budget and the release install timeout, with a start margin for controller processing. * **Tests** * Expanded wait-budget and timeout validation across configured suites, including checks for missing or invalid settings, operation slack, restore-skip behavior, and bucket waits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
68ee97300b |
docs(monitoring): say a central tenant's collector still needs enabling
The tracingStorages description said a tenant with shared-central tracing gets the collector without a local entry, which reads as if tracingCentral alone deploys it. tracingCollector.enabled still decides whether it renders. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
69a16bd2da |
docs(tenant): shorten the tracingCentral description
The description ships in the godoc, the schema, the README, the ApplicationDefinition and the dashboard's field help, and at about 2,800 characters it buried what an operator needs to decide on the field. It now says what the field does, what tenant-root has to host first, and who is inside the shared store's trust boundary; the lifecycle details move to the user guide. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
a0af2fd2f8 |
test(e2e): hold the postgres and bucket walkthrough waits to the floor
The postgres backup walkthrough's HelmRelease waits and every table walkthrough's bucket wait now clear the install-timeout floor, and the Chainsaw ops grew to contain them, but nothing checked it: a wait dropped back under the floor, or raised without its op, would pass. The op check cannot catch the bucket case on its own, since a lower wait only widens the slack. Add postgres to the table the other walkthroughs are held by, with the remainder its op kept through the raise, and hold each table suite's bucket wait to the bucket release's floor, read from the api-server default with bucket-rd checked for an override. This holds the budgets only; what a CNPG first boot costs on this platform is still unmeasured. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4386014cf4 |
test(e2e): restate the wait guard's figures for the raised bucket waits
The mariadb op and its bucket wait both grew by six minutes, so the worked arithmetic in the remainder test and the note that the bucket wait sits under its floor no longer describe the tree. The header also said the other suites' HR-Ready asserts sit under the floor by choice, which stopped being true once they were raised. The checks and their constants are unchanged: every remainder still sits at exactly its constant. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
67e42bbbc0 |
docs(agents): stop describing the HR-Ready waits as under their floor
The sizing guidance said most of the tree sat under the install-timeout floor, that mariadb was the only suite raised, that the walkthrough bucket waits were under it, and that harbor's asserts were on and under theirs. None of that holds once the waits clear their floors, and a reader would take the old band as the norm to copy. Name kafka as the floor no budget can clear, keep harbor as the example of two floors in one suite, and note that a walkthrough wait passing no budget takes its helper's default, which a grep for the number misses. The list of definitions that disable the install wait also missed vm-instance-rd, which is why its assert has no floor to clear, and the section never said where a Package release's floor comes from. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
04d469fc85 |
fix(tests): let the backup walkthrough waits outlive their installs
The backup round-trips drive examples/backups, whose bucket waits gave up at 300s and whose postgres waits at 360s, under the 600s install timeout cozystack-api gives each of those releases. The kafka example's bucket wait passed no budget and took the helper's 300s default. A failed install there reported only that the wait ran out. Raise each to 660s, and grow the Chainsaw op around every walkthrough by what its waits gained, so a failure at any later wait keeps the headroom it had and is still reported by the script rather than cut off by the op. The kafka and kafka-metadata Kafka waits stay: their 7h install timeout cannot be cleared inside the E2E job cap. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
49cefe0dff |
fix(tests): size HR-Ready asserts on nested and Package releases
A nested <name>-system release takes its install timeout from the template that renders it, not from cozystack-api, and a Package release takes it from cozystack-operator. The external-dns, openbao and qdrant templates set 10m and harbor's sets 15m; the operator gives 10m. Their asserts sat at 5m or 10m, so a failed install reported as a bare timeout. Raise each above its own floor: 11m, and 16m for harbor-test-system. harbor-test was on its 600s floor exactly, which expires first because its clock starts before the install's; it goes to 11m with the generated parents here. kafka keeps its 6m: kafka-rd sets a 7h install timeout that no budget inside the E2E job cap can clear, and the assert now says so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9e6cb7d0cc |
fix(tests): let HR-Ready asserts outlive the install they wait on
cozystack-api gives every release it generates a 600s install timeout, and these asserts gave up at 5-6m. A failed install was therefore reported as a bare assert timeout while the release was still installing and had not yet recorded why it failed. Raise them to 11m, above that floor, so the release's own reason is on record before Chainsaw gives up. The cost is up to six more minutes on a red run of each suite; a green run is unchanged. vm-instance-test stays: vm-instance-rd disables the install wait, so its release goes Ready as soon as helm has applied. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
94b328c6eb |
test(e2e): hold every raised backup wait to a start margin
The wait-budget guard accepted a wait one second over the install timeout, although the wait's clock starts before the install's by the Application reconcile, the chart fetch and the helm-controller queue. A wait lowered from 660 to 601 passed every test and brought back the bare assert timeout the budgets exist to avoid. A wait now has to clear its floor by a named 60s margin. Nothing in the tree bounds those three delays, so the value is policy: the headroom every current wait already carries. The sizing guide in docs/agents/e2e-testing.md asked only for "strictly above", which the guard now rejects; it points at the margin instead of restating a number. The mongodb, rabbitmq and clickhouse walkthrough waits were raised for the same reason as mariadb's, and their Chainsaw ops grown with them, but nothing held either. Each suite is now a row in one table and gets its own floor and op tests, so a regression reds only its own suite. Floors come from the api-server install-timeout default and, for mariadb, the chart's startup probe; each suite's RD is checked for an override instead of being assumed not to carry one. Each op keeps the remainder it had before the raise over every HelmRelease wait it reaches, the bucket one included, so raising a wait and its op together keeps the remainder instead of widening the slack. The mongodb target behind SKIP_RESTORE=1 is not reached and is left out until the op stops passing it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
889bb79236 |
feat(harbor): restrict the portal and API to listed source ranges (#4300)
## What this PR does Adds `managementWhitelist` to the Harbor app, a list of IP addresses or CIDR ranges allowed to reach the web portal and the Harbor API, and I made it a typed list on purpose. The obvious alternative, a free-form `ingress.annotations` map, gives tenants a way into the shared ingress controller. The platform ingress-nginx runs with `allowSnippetAnnotations: true` (the kubernetes app needs a `configuration-snippet`), and v1.11.5 defaults `annotations-risk-level` to `Critical`, so it accepts any annotation, `server-snippet` included. Tenants can't create Ingress objects themselves, but they can write any field of an app spec. An annotation map on the Harbor spec would let any tenant put arbitrary nginx config into a controller that serves every tenant. A typed list is also what the repo already does for this: `filer.whitelist` in `extra/seaweedfs`, `whitelist` in `extra/ingress` and `tcp-balancer`. The use case is a registry that serves customers: pulls and pushes come from anywhere, the admin UI and API should answer only to an office or VPN range. Right now the portal, `/api/v2.0` and `/v2/` sit on one Ingress and one public host, and a tenant can't restrict any of them. With a non-empty list the existing Ingress gets `nginx.ingress.kubernetes.io/whitelist-source-range`, and `/v2` and `/service/token` move to a second Ingress, `<release>-registry-ingress`, on the same host. ingress-nginx applies that annotation only to the locations of the Ingress that carries it, so those two stay open. They are what a registry client needs: Harbor answers an anonymous `/v2/` request with a Bearer challenge whose realm is `/service/token` on the same host. Everything else under `/service/` is `/service/notifications/*`, internal job callbacks that the bundled nginx already answers with 404, so it stays on the restricted side. The second Ingress uses the same TLS secret and has no cert-manager annotations, otherwise ingress-shim gets two owners for one Certificate. The HTTP-01 solver is its own Ingress and is not restricted. With the default empty list the chart renders exactly what it renders today. An HTTPRoute can't express a source restriction, so in Gateway API mode a non-empty list fails the render. Dropping it silently would leave the portal public while the spec says it is restricted. Each entry has to be an IP address or CIDR range, and a bad one fails the render. ingress-nginx reads an empty `whitelist-source-range` as no restriction and turns an unparseable one into a 503 for the whole location, and the values schema generator can't validate list items. Three things users should know. Harbor-to-Harbor replication that pulls from this registry goes through its API, so the other registry's address has to be on the list. The check uses the client address that ingress-nginx sees: with `externalTrafficPolicy: Local` or PROXY protocol that is the real client, behind another proxy it is the proxy. And for the portal and API the list replaces the tenant ingress `whitelist` rather than narrowing it. Disabling the portal is not here. The vendored Harbor chart (1.19.1) always renders the portal Deployment and its internal nginx always proxies `/` to it, so there is no small switch to wire. The new Ingress is added to the dashboard Role and to the ingress selector in the ApplicationDefinition. ### Screenshots No UI code change. The field reaches the dashboard form through the regenerated schema and `keysOrder`. ### Downstream repositories The terraform provider models Harbor by hand, so the new field needs a follow-up there: cozystack/terraform-provider-cozystack#46. The website reference page is generated from `README.md` on release. Nothing else in the trigger map is touched: no package added or renamed, no platform, installer or `hack/` change. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/46 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(harbor): add `managementWhitelist`, a list of IP addresses or CIDR ranges allowed to reach the Harbor web portal and API. Only `/v2/` and `/service/token` stay reachable from any address. The default is an empty list, which keeps the current behaviour. Not supported with Gateway API publishing: a non-empty list fails the render there. Each entry must be an IP address or CIDR range; for the portal and API the list replaces the tenant ingress `whitelist`. ``` |
||
|
Myasnikov Daniil
|
7fe713580d |
test(e2e): drop readiness asserts that HelmRelease Ready already covers
The qdrant and external-dns apps render their workload in a nested -system HelmRelease, and both suites already assert that HelmRelease Ready, which helm-controller reports only once the workload is up. Asserting readyReplicas on the same Deployment or StatefulSet again adds nothing, so those asserts keep only the instance-label pin. The vminstance PVC asserts follow a DataVolume Ready assert, and a DataVolume is not Ready before its PVC is Bound. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
3303cc59c7 |
fix(e2e): delete the kafka suite's broker and controller PVCs
The kafka chart sets deleteClaim: false on both node pools, so deleting an app leaves its volumes behind on purpose. The suite deleted its apps and never the volumes. The 35 GiB a run leaves in tenant-test, 20 GiB of it on one node, kept that LINSTOR node short of space for the kubernetes suites that run after it. The backup round-trip leaves its two clusters' volumes in tenant-root the same way. Each test now deletes the PVCs by strimzi.io/cluster once its apps are gone, and waits for them to go. The kafka test does it from a step cleanup, so a failed run reclaims them too. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
64b3c9619f |
fix(e2e): hold replicated-lane LINSTOR cleanup to a pre-suite baseline
The kubernetes suites' cleanup in the replicated lane demanded 90 GiB free on every LINSTOR node. Volumes that earlier suites and the platform legitimately keep can hold a node under that figure even when the suite reclaimed everything it used, so the cleanup failed suites that had leaked nothing. Both lanes now record the per-node baseline before the suite creates anything and compare against it on cleanup. The 90 GiB figure is what a replicated suite needs to place its workers and CDI scratch volumes, so it stays as a precondition checked before the baseline, and a shortfall there is reported as the sandbox's state rather than the suite's. A run that fails before recording its baseline used to leave the previous suite's file in place for its cleanup to compare against; the file is now removed when the suite starts. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
54159736bc |
feat(talos)!: bump Talos to v1.14.2 (#4598)
## What this PR does Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4. ZFS moves from 2.4.3 to 2.4.4. On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning on a failed send (EPIPE). What followed was RCU stalls, processes stuck in D-state, a hanging `drbdsetup disconnect` and two-phase-commit timeouts. The [9.3.4 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has fixes for a sender thread pinning a CPU while its connection is down, for two-phase-commit issues and a hanging `drbdadm disconnect`, and for two nodes ending UpToDate with different data after a reconnect. None of them are in 9.3.3. I matched the trace to those entries by their descriptions only, not against the commits. The base installer image had to change. Talos v1.14 no longer publishes `ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses `ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager profiles. I built the amd64 installer from the new profile with `imager:v1.14.2` locally and it builds without errors. PR CI will not boot the new image. It builds the installer and matchbox images, but the e2e runs on the upstream `talos:v1.13.5` container from `hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD and ZFS is the nightly after merge, which builds the nocloud disk from these profiles and runs it in QEMU. On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on demand. After upgrading a node from v1.13.6, `drbd` was loaded and the transport module was not, `drbdsetup new-peer` failed with "Failed to create transport (drbd_transport_xxx module missing?)", and every DRBD resource on the node stayed in Connecting. Listing the module in `machine.kernel.modules` fixed it without a reboot. The cause is [siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565), in Talos since v1.14.0: the kernel is built with an empty modprobe path, so it no longer loads modules on `request_module()`, which is how DRBD asks for its transport. Any module that used to load that way now has to be listed explicitly (reported upstream as siderolabs/talos#14501). Outside DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2 kernel builds as modules, so LVM-thin or multipath users on Talos need them listed too. The e2e node config now loads `drbd_transport_tcp` explicitly, and the talm preset (cozystack/talm#252) and the install docs (cozystack/website#720) get the same line. The module ships in the drbd extension on v1.13 too, so the extra line is safe before the upgrade. The order matters for host nodes. Either update the cozystack preset in the talm project to a version with cozystack/talm#252 and re-render and apply the node config, or add `drbd_transport_tcp` to `machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2. A node upgraded first loses DRBD replication until the module is added. The host network policy now also denies port 2383 to `world`. Talos v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from 2379 to a separate listener on 2383, and the policy only blocked 2379 and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes would be reachable from outside, protected only by client mTLS. Two smaller changes come with the bump. The multus `talos-cni-plugins-checked-against` marker moves to v1.14.2, because the pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as multus. The system memory limits doc now describes the v1.14 OOM trigger, since v1.14.0 dropped its global memory PSI clause ([siderolabs/talos#13895](https://github.com/siderolabs/talos/pull/13895)). Before upgrading hosts, operators should know a few things from the [v1.14.0 release notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the Talos compatibility code: - Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a cluster on Kubernetes 1.31 has to upgrade Kubernetes first. - etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to 2383. The Cozystack etcd scrape proxy reads a separate metrics listener on `127.0.0.1:2381`, which this change does not move. Anything else that scraped or health-checked etcd on 2379 has to switch to 2383, and a firewall outside the cluster that blocked 2379 should block 2383 too. - etcd and kube-apiserver now require TLS 1.3, so a client that can only speak TLS 1.2 stops connecting. - Workload isolation (`SecurityProfileConfig`) stays off on upgraded clusters, but `talosctl gen config` turns it on for new ones. I haven't tested Cozystack with it enabled. - `talosctl apply-config --mode=reboot` is gone. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#720 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: cozystack/talm#252 - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The website PR refreshes the `next` version pins for v1.14.2 and adds the module to the Talos install pages. The talm PR adds it to the cozystack preset. ### Release note ```release-note feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Upgraded host node support to Talos v1.14.2 across machine images, with refreshed firmware and storage extensions. * Updated memory-pressure guidance for Talos v1.14, including version-specific behavior and configuration recommendations. * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383. * Updated cluster preparation to load the TCP transport module required for DRBD. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
de570a3806 |
feat(platform): add slim installation variants (#4595)
## What this PR does This adds three minimal platform variants for small installs, such as arm64 labs, where isp-full is too heavy. They are `isp-slim` for Talos, `isp-slim-generic` for k3s, kubeadm or RKE2, and `isp-hosted-slim` for clusters where the host provides CNI and storage. A slim variant installs only the base platform: the engine, API and dashboard, tenants, ingress and Gateway API. The two non-hosted variants also get LINSTOR. objectstorage-controller stays on too, because cozystack-controller watches BucketClaim unconditionally and crash-loops without that CRD. Everything else is opt-in through `bundles.enabledPackages`, including every paas and naas application and its operator, monitoring, backups, etcd, SeaweedFS, metrics-server, VPA, Multus and MetalLB. The iaas bundle is refused, so there are no VMs and no managed Kubernetes. With the stock preset a slim variant emits 20 Packages (15 for hosted), where isp-full emits 77. On the two non-hosted slim variants, networking is Cilium alone. Without VMs nothing needs Kube-OVN, and Cilium L2 announcements take the place of MetalLB. The admin creates a `CiliumLoadBalancerIPPool` and a `CiliumL2AnnouncementPolicy`, or uses `publishing.externalIPs`. Pod ranges come from `node.spec.podCIDR`, which Talos and k3s allocate by default. `networking.encryption` only drives Kube-OVN IPsec, so the slim variants refuse it instead of silently ignoring it. The tenant application needs its own slim variant. The default one waits on the monitoring, etcd and SeaweedFS applications, so on slim cozystack-basics and tenant-root would never become ready. tenant-rd and cozystack-basics still reference the default variant's artifacts by name. Those artifacts exist anyway, because the PackageSource builds them for every variant. The full variants don't change. I rendered isp-full, isp-full-generic and isp-hosted against main and got byte-identical Packages. The only new output there is the slim variant in the tenant-application PackageSource. Things to know before using it: - An opt-in package does not pull in its dependencies. The presets and the docs list the chains. A package enabled without its chain stays `DependenciesNotReady`. - Slim is for new installs. Switching a live isp-full cluster to isp-slim moves the networking Package from `kubeovn-cilium` to `cilium`, the operator removes the Kube-OVN release, and running pods lose networking. The other Packages are kept by `helm.sh/resource-policy: keep`, so the switch doesn't make the cluster smaller either. - No e2e suite runs a slim variant yet. Helm unit tests cover the presets, opt-in, OIDC, the networking values and both refusals, and a dependency check over every emitted Package passes for all three presets. I installed isp-slim from a build of this branch on a fresh three-node Talos 1.13 cluster (amd64). All Packages and HelmReleases went Ready, with no Kube-OVN, MetalLB, monitoring, backup or KubeVirt pods. From outside, the dashboard and the API answered by name with valid certificates. A LoadBalancer Service got an address from a Cilium pool and answered through L2 announcements. Pod-to-pod traffic across nodes, cluster DNS and a replicated LINSTOR volume worked. Opting in postgres-operator and postgres-application gave a Ready Postgres. That run found the BucketClaim crash-loop above, which is fixed here. The base platform used about 215m CPU and 3.2 GiB of memory, not counting the Kubernetes control plane. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#718 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [x] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: cozystack/ansible-cozystack#79 - [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: cozystack/ccp#23 - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(platform): add the isp-slim, isp-slim-generic and isp-hosted-slim variants. They install only the base platform (engine, dashboard, tenants, ingress, gateway, and LINSTOR outside hosted) with Cilium-only networking and no MetalLB; applications, operators, monitoring, backups, etcd and SeaweedFS are opt-in through bundles.enabledPackages, and the iaas bundle is not available. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added the `isp-slim`, `isp-slim-generic`, and `isp-hosted-slim` platform variants, with support for PaaS and NaaS bundles. * Slim variants use Cilium networking and include a reduced set of packages by default. Additional packages and their dependencies can be enabled as needed. * **Limitations** * IaaS and the platform encryption toggle are not supported on slim variants. Cilium-native encryption is not configured. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
312409ecd2 |
test(e2e): fail the suite whose pods outlive its cleanup (#4492)
## What this PR does A Chainsaw suite could pass while the workloads it created kept running after its own cleanup, and nothing noticed until a later suite failed. On one E2E run the brokers of four Kafka releases from suites that had passed and deleted their applications (#4488), plus a TenantControlPlane sitting on its finalizer with two apiserver pods (#4489), pushed pod memory requests to the node limit. Harbor then failed on `Insufficient memory` two suites later, under the wrong name. This makes the suite that leaked fail, and name what it left behind. - Chainsaw v0.2.15 has no hook that runs after a test's cleanup: its `Configuration` carries `error.catch`, which runs only on failure. So `make test-chainsaw` now runs `hack/e2e-chainsaw-run.sh`, which invokes `chainsaw test` once per suite directory and checks the cluster between invocations with `hack/e2e-chainsaw/_lib/leak-guard.sh`. - Attribution: a pod is charged to a suite when all three hold. It is not terminal. Its UID was not in the pod list taken before the suite. The root of its ownerReferences chain was created at or after the suite started. A platform pod replaced mid-suite by a rollout, an eviction or a crash traces back to an object the install created, so it is not charged. A pre-existing pod adopted by a controller recreated mid-suite is excluded by the UID baseline. A leak is charged once, to its own suite. - The suite start is read off an apiserver, from the `creationTimestamp` of a server-side dry-run create, rather than off the sandbox. With several control-plane nodes the reading and a root's timestamp can come from different apiservers, so the rule assumes their clocks agree more closely than the gap between the reading and the suite's first object; no skew margin is added. Source for that behaviour: the generic registry `Store.create` stamps it through `rest.FillObjectMetaSystemFields`, then `DryRunnableStorage.Create` copies it back without persisting (`staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go` and `dryrun.go` at Kubernetes v1.33.0). - Budget: charged pods get 300s to disappear, the default `timeouts.cleanup` in `.chainsaw.yaml`; a unit test keeps the two equal. A suite that overrides its own cleanup timeout (kafka sets 8m) is covered the same way, because the check runs only after `chainsaw test` has returned, which is after every cleanup in the suite has finished under its own timeout; the 300s start from there. The leaks this targets sat about 110 minutes (Kafka brokers) and 30+ minutes (the TenantControlPlane) past their suites. Every green suite logs `leak-guard: suite=<dir> clear after Ns`, so the number can be checked against later runs. - Past the budget the run prints `LEAKED-PODS suite=<dir> budget=300s count=N`, then one line per pod with its phase, deletion state, root and `app.kubernetes.io/instance`. It raises a `leaked-pods` error annotation and adds a failed `leak-guard` test case to `chainsaw-report.xml`. The run continues through the remaining suites and exits non-zero at the end, as the single invocation did on a red test. - A suite that already failed on its own is checked once without the wait. Its leftovers are reported as a `::warning` with `LEAKED-PODS suite=<dir> checked-once=chainsaw-failed`, since some may still be tearing down. - Each pod gets one `kubectl wait` inside the remaining budget, and nothing is retried. A timeout or a spent budget gives the verdict from the last good pod list, also when the re-list after it fails, so a leak fails the suite. A wait that fails only after running for its whole `--timeout` counts as a timeout whatever it printed: kubectl maps an expired context to "timed out waiting for the condition" only inside the watch, so a deadline that passes during its initial lookup surfaces as "context deadline exceeded" (`wait.go` and `delete.go` in kubectl v1.33.0). A spent budget is caught before calling kubectl and never passed as `--timeout=0`, which means "check once". - Any other error on a wait or a re-list, before the budget is spent and before any timeout, stops the waiting and is reported as `LEAKED-PODS suite=<dir> count=N inconclusive="<error>"`. That is a warning, not a leak failure, so an apiserver blip cannot become a leak red. - A failed first reading after a suite, including a pod list that cannot be parsed, leaves the suite unchecked. It fails with a `leak-guard` annotation and a report case saying the cluster could not be read, not that it leaked. Errors that are not leaks carry the annotation title `leak-guard`, not `leaked-pods`. - Known leaks are listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt` as `<suite-dir> <issue-ref>`: `kubernetes-latest` and `kubernetes-previous` against #4489 (both suites create a TenantControlPlane through `_lib/run-kubernetes.sh`). `kafka` and `kafka-metadata` were listed against #4488 until #4490 fixed it; those entries are gone. An allowlisted suite prints its leak as a `::warning` and does not fail. An entry that names a directory that is not a suite stops the run before the first suite, so a rename cannot keep an exemption alive. The PR fixing each issue deletes its lines. Suite selection is unchanged. The runner runs the named directories when there are any, and otherwise every directory directly under `hack/e2e-chainsaw/`, each handed to Chainsaw's own discovery; it refuses a tree with a test at its top level, where that split would stop being exact. I verified this with Chainsaw v0.2.15's own `discovery.DiscoverTests`, built from source at that tag. For the full tree, for `kafka redis harbor` and for `kubernetes-latest`, the union of the per-suite discoveries equals what a single `chainsaw test [suites]` discovers: 57, 7 and 2 tests, identical lists. 57 matches the test count in an earlier full run's report. A unit test pins that the runner passes chainsaw the suites and flags it was given, in order. The order changes too: suites used to run in Chainsaw's own order and now run one directory at a time in alphabetical order, which moves the kubernetes suites after the kafka ones. Before #4490 fixed #4488 this starved them (see the E2E criterion below). With the fix merged the order has no effect, and a future leak that lands before a memory-heavy suite is named at the suite that leaked. One behaviour does change: a selected name that is not a directory used to fail the single invocation before any test ran; now only that suite fails and the others run. `chainsaw-report.xml` keeps its path and shape: one `<testsuites>` root with the summed counts, one `<testsuite>` per suite directory. Its consumers are the `Collect chainsaw report` / `Upload chainsaw report` steps in `pull-requests.yaml`, `e2e-fork.yaml`, `nightly.yaml` and `e2e-tag.yaml`, which `docker cp` the file and upload it as an artifact. Nothing in the repository parses it. The merge is unit-tested on two per-suite reports in Chainsaw's real shape, one of them failed, and the result is checked to parse as XML. Tests: `hack/chainsaw-leak-guard_test.bats` (31 tests, in `make unit-tests`; named without the `e2e-` prefix, which the unit lane excludes) covers the attribution rule, the budget, the allowlist, the merge and the runner, all against stub `kubectl` and `chainsaw`. Every rule was mutation-checked: removing the guard call, having the Makefile bypass the runner, inverting the allowlist match, flipping or tightening the root-timestamp comparison, removing the UID baseline, removing the terminal-phase filter, skipping the budget wait, changing the budget, lowering or raising the depth cap, accepting a stale allowlist entry, reading an unreadable pod list as empty, dropping the leak case from the report, skipping directories in the selection, treating any failed owner read as a missing owner, dropping the separate report for a suite that already failed, dropping the not-run reason from the report, reading any wait error as a timeout, removing the inconclusive branch, letting a failed re-list after a timeout turn a leak inconclusive, ignoring a failed re-list, reporting an unreadable cluster as a leak, reading an unparsable pod list as clean, reporting a kubectl warning instead of the error, passing the report name without its extension, wording a failed re-list as a failed wait, dropping an allowlist's last line when it has no newline, dropping the check that `--set-string` carries a value, and reading a wait that ran its whole timeout but printed a context error as inconclusive. Each one turns its named test red, and restoring turns it green. E2E criterion, registered before the run: in the `Run E2E tests` log of this PR, every suite that ran prints `leak-guard: suite=<dir> clear after`, a `LEAKED-PODS suite=<dir>` line carrying `allowlisted=`, or one carrying `checked-once=` (a suite that failed on its own) or `inconclusive=` (waiting stopped on a failed read), and no line matches `LEAKED-PODS suite=.* budget=`. With #4490 merged, `kafka` and `kafka-metadata` are no longer allowlisted and must print `leak-guard: suite=kafka clear after` and `leak-guard: suite=kafka-metadata clear after`, which is the live check that #4490 fixed #4488. The first run of this PR (before #4490, run 36223896793) is the positive control: it printed `LEAKED-PODS suite=kafka count=15` and `suite=kafka-metadata count=8`, allowlisted, and every other suite cleared in 0-37s. That run also went red on both kubernetes suites, because those 23 leaked brokers left no memory for the second tenant worker VM (`0/3 nodes are available: 3 Insufficient memory`), which the order change described above made possible. The guard has not yet run against a real cluster, so the first run of this PR should carry the `full-e2e` label; its `clear after` lines are the first measured teardown times, and any unexpected `LEAKED-PODS` line is either a real leak or a false positive to understand before merge. Follow-ups, known and out of scope here: - The allowlist exempts a whole suite, so a new leak in an allowlisted suite stays hidden until the issue's fix removes the line. - Attribution is per suite directory, not per Chainsaw `Test`; the leak lines carry the release instance, which usually names the test. - Only pods are checked. A leaked PVC, LoadBalancer IP or bucket holds no node memory and is not seen. - The 300s budget rests on Chainsaw's cleanup timeout, not on a measured pod-gone latency; the `clear after` lines are where that measurement will come from. - A platform object recreated mid-suite (for example a HelmRelease uninstall/reinstall remediation, or a force-replace on an immutable field) has a root newer than the suite start, and its pods would be charged to whichever suite was running as a leak; the first CI runs are where to look for this. - The per-pod wait recomputes the remaining budget before each pod, and no unit test pins that. - `kubernetes-previous` is allowlisted for #4489 although the stuck finalizer was seen only in `kubernetes-latest`; both suites create a TenantControlPlane the same way, and the exemption hides any new leak there until #4489 is fixed. - Each clean suite now waits for its own pods to finish terminating before the next starts, where the tail used to overlap the next suite. The first full run's `clear after` lines say what that costs against the E2E job's 215-minute cap. - After a timeout followed by a failed re-list, the leak lines come from the list taken before the wait, so a pod whose own wait had already succeeded is still named and the count can be too high; the verdict is unaffected. - Likewise, when every wait succeeded and the re-list then fails, the inconclusive report still lists the pods whose deletion was just confirmed. In the boundary case where the budget runs out during the last successful wait and the re-list then fails, the result is inconclusive rather than a verdict from the last good list, because the guard only treats the budget as spent when it finds it exhausted at a check. - A mistyped `COZY_LEAK_ALLOWLIST` path passes allowlist validation; it fails closed, since no suite is then exempt. - An allowlisted suite is not waited on, so `kubernetes-latest` and `kubernetes-previous` will print `LEAKED-PODS ... allowlisted=` on most runs even for pods still in normal teardown; waiting out the budget first and applying the allowlist only to what remains would stop that noise. - A slow teardown in each of several suites can add up to 300s apiece, against the E2E job's 215-minute cap. On the first full run of this PR (run 36223896793) the `clear after` values of the suites that were waited on summed to 210s; the last full run before this change took 158 of the 215 minutes. - A pod whose owner's resource type is no longer served gets "the server doesn't have a resource type" rather than NotFound, so its suite is reported unchecked instead of leaked; it still fails. - `kubectl wait --for=delete` waits for deletion, while the guard skips pods in `Succeeded` or `Failed`; a pod that finishes but is never deleted (a Job under a leaked root) spends the rest of the budget before the suite shows `clear`. The verdict is right; only job time is lost. - The unit tests do not pin the one-line empty report Chainsaw writes for a directory with no tests (`_lib`), nor the branch that catches a spent budget before calling `kubectl wait`; both are verified against the Chainsaw and kubectl sources rather than by a test. - The runner's per-suite report directory from `mktemp -d` is not removed; the sandbox is discarded after the job. - Each non-suite directory under `hack/e2e-chainsaw/` (today `_lib`) costs one empty Chainsaw invocation, because the runner leaves the decision of what is a test to Chainsaw. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff adds files under `hack/`, changes the `test-chainsaw` target, and rewords one comment and the failure line in the four E2E workflows. The trigger map's `hack/` entry is for cozystack/ccp, whose skills anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`, none of which change here. A GitHub code search for `chainsaw` across the cozystack organisation finds hits only in this repository. ### Release note ```release-note test(e2e): the E2E app suite now fails the suite whose pods outlive its own cleanup by more than 300s, naming each leftover pod and its owner; suites with a filed leak are listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt and only warn. ``` |
||
|
Aleksei Sviridkin
|
42b7239198 |
test(e2e): wait for the opensearch handoff before the first API call (#4462)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The opensearch suite sends its first OpenSearch API call, the index create in `make-the-cluster-permanently-yellow`, as soon as the nodes StatefulSet reports one ready replica. Two operator actions are released at that moment and neither has finished: the bootstrap pod handing over the cluster-manager role, and a rolling restart of the only data node. #4412 is the first of them hitting the create with a 503. This PR makes the step wait for both before its first call: an `error:` assert that `opensearch-test-bootstrap-0` no longer exists, then an assert that the nodes StatefulSet has settled on its update revision, in the same form the suite already uses after the listener move. Anchors below are opensearch-k8s-operator v2.8.0 with the in-tree patches applied, paths under `opensearch-operator/`. **Bootstrap handoff.** - The operator starts retiring the bootstrap pod only once the cluster is `Initialized` (`pkg/reconcilers/cluster.go:110-111`), and that flag is set as soon as every cluster-manager pool has all its replicas ready (`controllers/opensearchController.go:257`, `pkg/builders/cluster.go:1146-1164`). A Ready data node therefore starts the handoff and does not finish it. - The voting exclusion is a cluster-manager action, so it goes through the bootstrap pod and fails while that pod's security is not initialized. The operator then keeps the pod and requeues after 10s (`pkg/reconcilers/cluster.go:36`, `:186-190`). - The pod is deleted only after the bootstrap node has left the last committed voting configuration (`pkg/reconcilers/cluster.go:195-205`, `opensearch-gateway/services/os_client.go:760-771`). Once the pod object is gone, including its Terminating phase, the bootstrap process has exited and holds no vote, and the only cluster-manager-eligible node left is `opensearch-test-nodes-0`. - The assert cannot pass vacuously. The cluster forms around the bootstrap pod (`cluster.initial_master_nodes` names only that pod, `pkg/builders/cluster.go:807`), and the pod is created unconditionally while the cluster is not initialized (`pkg/reconcilers/cluster.go:112-113`). **Rolling restart on first boot.** The failing run's own operator log and StatefulSet dump show it: - `10:27:52.850` the reconcile in which `Initialized` became true. - `10:27:53.251` "Failed to exclude the bootstrap node from the voting configuration". - `10:27:53.981` "Starting rolling restart of the StatefulSet opensearch-test-nodes" (`pkg/reconcilers/rollingRestart.go:155`). - `10:27:54.079` "Couldn't proceed with rolling restart … because enabled shard allocation" (`:182`), the first step of the yellow path, after which the next 10s requeue moves on to `PreparePodForDelete` (`:195`) and deletes the pod. The StatefulSet in the same dump is at generation 2 with `currentRevision` `…744cc7b944` and `updateRevision` `…677c65d7fb`: the pod template was rewritten after `opensearch-test-nodes-0` was created, and the restart onto it waits only on `Initialized` (`rollingRestart.go:79`, `:123`). With one replica that restart takes down the node the step's requests go to, so a create that got past the handoff could still land on a terminating or restarting node. Once the StatefulSet reports `currentRevision == updateRevision` with its one replica updated and ready, the node the step talks to is on its final template and the operator has nothing pending. The suite runs one topology: one replica in a single pool with the cluster_manager, data and ingest roles, on a PVC. The bootstrap pod comes back only through the emptyDir recovery path (`pkg/reconcilers/cluster.go:452-478`), which a PVC-backed cluster never takes, and a restart that was not pending leaves the second assert true immediately. I checked the rest of the suite for the same window. Apart from this step, only the catch script (diagnostics, `|| true`) calls the OpenSearch HTTP API. `verify-endpoint-serves-the-chart-chain` makes a TLS handshake only, and it runs after the suite has already waited for the listener roll to complete, on a single node that is its own cluster-manager. Budgets: 3m for the handoff, which took about two seconds in the failing run; that leaves room for several 10s retire retries, a 30s requeue and the default 30s termination grace period the bootstrap pod runs with. 6m for the restart, shorter than the 12m the suite gives the listener roll because nothing is pulled here: the health steps requeue every 10s and the pod then restarts on an image the node already holds, where the first boot, bootstrap included, took under two and a half minutes. Residual risk, unproven: if the pod template were rewritten again after the rollout assert passed, that assert would have passed on a template that is not the final one. I found no mechanism for it. The StatefulSet builder reads no cluster status (the only status read in `pkg/builders/cluster.go` is the replica check in `AllMastersReady`), so the transition to `Initialized` cannot rewrite the template by itself; a rewrite would need a configuration or secret input to change after that point. Which input changed on first boot in the failing run I could not tell from the log, because the ControllerRevisions were not captured. What I ran: `chainsaw lint test` with Chainsaw v0.2.15, the version the e2e sandbox pins, passes on the changed suite. As a positive control, lint rejects each new operation when I rename one of its keys. `hack/select-e2e.sh` on this diff selects `opensearch` only. What I did not run: the E2E suite itself. The race is timing-dependent (one hit in six failed in-tree runs checked, per the issue), so a single green run would not prove the fix either. The argument rests on the operator code and the failing run's log cited above. Closes #4412 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The only file changed is `hack/e2e-chainsaw/opensearch/chainsaw-test.yaml`, which no downstream repository consumes. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Cluster health checks now wait for bootstrap completion and verify that all nodes are ready and up to date before proceeding. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
4d7f9e485a |
docs(ci): say why the overlay's ancestry tie goes to main (#4469)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does This fixes the comment on the overlay's ancestry tie, which described the wrong reason for the tie. No behaviour changes. The comment said a line fresh off main is the same as main. That holds only for the graph. `cozystack-packages:main` is built from main's tip, so once main moves past the cut, a stack on such a line gets newer images than the line. The comment now says so. It also says why the tie still goes to main: a branch forked from main before the cut ties the same way, and that case is far more common. This follows up on a review note in #4460 and is stacked on it. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note docs(ci): explain why the base-branch overlay sends an ancestry tie to main. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified how artifact selection works for stacked pull requests on release lines without line-specific commits, including why `main` is preferred in a tie and how that may affect image freshness. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a66907f535 |
fix(backups): share the walkthrough wait helpers and fix the S3 preflight cleanup (#4635)
## What this PR does This fixes the S3 access preflight and a few backup walkthroughs, and moves the walkthrough wait helpers into one shared file. The access preflight now keeps its evidence and cleans up after itself. When `mc alias set` failed, it dropped the port-forward log, and now it prints it like the other exits do. The exit trap used to kill the port-forward before `mc rm`, so the delete always hit a closed port and the warning gave no reason. The delete now runs first, and the warning carries mc's own error, or says the delete timed out when mc was killed before writing one. A PUT that timed out (rc 124) may have landed, so it counts as uploaded. A PUT the server refused does not, so access that never went live gets no delete and no warning. The wait helpers were copied into every walkthrough and had drifted. `wait_for_field`, `wait_hr_ready` and `wait_deleted` had 12, 8 and 2 copies with 5, 3 and 2 different bodies. They now live in `examples/backups/_lib/wait-helpers.sh`, and each `00-helpers.sh` sources it. Call sites and wait budgets are unchanged. Where copies differed, the shared body keeps every diagnostic any copy had. What changes per walkthrough: - `wait_for_field` in every walkthrough but redis now shows kubectl's stderr, so a Forbidden or transport error is visible instead of reading as an empty field. `--ignore-not-found` keeps a resource that does not exist yet from printing an error on every poll. Redis already kept stderr. - `wait_for_field` in foundationdb, nats and vmi gains the optional 7th terminal-value argument. None of their callers pass it, so nothing changes at runtime. - vmi loses its progress dots. - `wait_hr_ready` in redis and kafka-metadata now prints every condition and the release history on Stalled or timeout. It also says whether the release never appeared, was deleted, or could not be looked up. - The kafka `wait_deleted` timeout message reads "it is still present" instead of "still present". `hack/backup-examples-shared-helpers.bats` goes red if a walkthrough defines one of these helpers again or stops providing them, and it pins both sides of the `wait_for_field` read: kubectl's reason for a failed read is shown, a missing resource prints nothing. The byte-identity pin between copies in `hack/mariadb-first-boot-waits.bats` is gone, and its errexit test runs through the mariadb walkthrough's helpers. The kafka topic-data demo had the empty-read race from #4400. `kafka_run` read topic state through a throwaway `kubectl run -i` Pod. It now execs into a long-lived CLI Pod that keeps the restricted security context, and `cleanup.sh` removes it by label. The Pod name comes from its own `KAFKA_TOPIC_CLI_POD`, so setting the kafka-metadata demo's `KAFKA_CLI_POD` does not move it. #4400 itself was already fixed on main for nats, kafka-metadata and rabbitmq. The nats and kafka-metadata demos could exec into a Pod they had just refused. They read CLI output inside `$(...)`, where bash does not apply errexit, so the ownership refusal was ignored. A failed read of the Pod's phase was treated as no Pod, and the demo went on to delete and replace it. Every setup step now returns on failure. Their "exec fails with kubectl's own error" tests could never fail: the runner's trace wrote the expected message into the same stderr the test grepped. The stub now prints the message itself. Each fix has a test that fails on main. For each fix I also reverted it and checked that only its own test goes red. One gap stays open outside this PR. `hack/select-e2e.sh` maps `examples/backups/<app>/` paths by app name, so a later PR that changes only `examples/backups/_lib/wait-helpers.sh` would select no suite. This PR is fine because it touches every walkthrough. The rule belongs to the select-e2e owners and is tracked in #4626. While reviewing I also found that the kafka README puts steps that create and exec into Pods under the tenant persona, and the tenant role can do neither. That is #4625, left for its own change. Left out on purpose, because they are outside what this diff changes: - The kafka topic demo and the kafka-metadata demo each carry a CLI Pod helper, and the kafka-metadata one runs without the restricted security context. - The walkthrough READMEs do not say that a directory now needs `../_lib` next to it. - The errexit test in `hack/mariadb-first-boot-waits.bats` loops over one file. I kept it so the rebase against #4471 stays mechanical. Closes #3757, closes #4243, closes #4244 ### Screenshots No UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup walkthroughs share one set of wait helpers that keep kubectl errors visible, the kafka topic-data demo no longer reads empty replies from a throwaway Pod, the nats and kafka-metadata demos no longer exec into a CLI Pod they refused, and the e2e S3 access preflight says why client setup or cleanup failed and deletes its test object reliably. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Kafka topic backup walkthroughs now reuse a named CLI Pod, with restricted security settings and checks to prevent using a Pod owned by another demo. * Backup walkthroughs can wait for resources to reach a desired state, Helm releases to become ready, or resources to be deleted. * **Bug Fixes** * Kafka and NATS walkthroughs now stop and report errors when CLI Pod lookups, creation, deletion, or command execution fail. * S3 preflight cleanup now attempts removal after a timed-out upload and reports cleanup failures, including timeouts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a9dc106a81 |
fix(e2e): bound post-install-prep calls and make its failures name their cause (#4632)
## What this PR does The post-install-prep step of the e2e install could hang without failing, and several of its failure paths gave up without saying why. The script runs in the background of the install test, which waits on it, so a kubectl call that never returns keeps the install open until the job's own timeout. This PR fixes both problems in `hack/e2e-post-install-prep.sh`. - Every call that does work now has a wall-clock bound: the pool listing and creates, both applies, the CDI patch, the reads in the LINSTOR node loop, and the diagnostic reads on every failure path. Plain requests also get a smaller `--request-timeout`, so kubectl can say why a request failed before the outer timeout kills it. When a bound fires, the log names the call. - `kubectl wait --timeout` turned out to be no bound at all for the GET it runs first, so each wait attempt got an outer timeout too. If that timeout cuts an attempt, the timeout line says so. - The wait loop keeps kubectl's stderr now. An expired credential or an RBAC denial used to look the same as "not created yet" and ended in a timeout on the object. Each error is printed once when it changes, and the timeout line repeats the last one. - When the linstor wait times out, the script lists every HelmRelease first. The linstor release can be just waiting for a release further up the dependency chain, and the Ready messages name which one. - The MetalLB CRD wait and the CDI StorageProfile wait used to exit silently. Now they print a timeout line and one last read of the missing object with its error. The MetalLB wait also prints the metallb HelmRelease. - A failed or killed storage pool listing stops the script before any pool is created. Before, it read as "no pools exist", and the creates that followed failed with an unrelated error. All of this is tested in `hack/post-install-prep.bats` with the existing stub PATH and virtual clock. One test checks that every kubectl call on the success path and on each timeout path runs under a bound. The literal grep in `hack/linstor-storage-mode_test.bats` is replaced with a test of the behaviour. The rest of #3680 lives outside this script and moves to #4628: the empty `{"error":{}}` in helm-controller's Helm log, previous-container logs on passing runs, and node and pod resource state during install. Closes #3708, closes #3707, closes #3713, closes #3714, closes #3680. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): bound every call in the post-install-prep step and make its failures name their cause, so a hung call fails the install instead of holding it until the job timeout ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Reliability** * Cluster operations now have bounded wait times, with clearer diagnostics when resources or endpoints are delayed or unavailable. * Storage setup stops when pool listings fail instead of masking the error. * **Tests** * Expanded coverage for timeout handling, error reporting, and failures during storage and resource setup. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d8ec8f0af1 |
fix(ci): run every bats unit file after a failure and cover build-*.sh in the cross-compile check (#4633)
## What this PR does Two fixes to the tooling of the unit lane. A local run of the bats unit tests stopped at the first red file and never ran the files after it, while the output looked like a finished suite. Each bats file is its own make target, and without `--keep-going` make stops scheduling targets after the first failure. CI passes `-k`, so it was not affected. The target now runs the files in a `--keep-going` sub-make. That sub-make still exits non-zero if any file failed, and it shares the caller's `-j` slots and `--output-sync`. The new `hack/bats-unit-keep-going.bats` drives the real Makefile against two fixture files. It checks that both run and the target fails when the first one fails, that the target exits 0 when both pass, and that the sub-make still finds the root Makefile when `MAKEFILES` names another file. I split the exact-line check on this target in `hack/bats-runner-coverage.bats` in two to match the new recipe. A workflow comment and a test name that still called each file a prerequisite now call it a target. The cross-compile check never looked at the multus-cni builder. It treated only a script named exactly `*build.sh` as a compile step, and multus runs upstream `./hack/build-go.sh`. The pattern now allows a suffix after `build` but no `/`, and each case has a fixture test. Among the Dockerfiles the check scans under `packages/`, this matches exactly one new line, the multus stage, and that stage passes. Closes #3692 for the make-level truncation. The in-file half, where `hack/cozytest.sh` ends a file at its first failing test, is tracked in #4620. Closes #4500 Things I left out: - The multus stage passes because a `curl ... TARGETARCH` step sits in the same stage. The check does not see that `build-go.sh` reads `TARGETPLATFORM`. The test header already documents this per-stage limit. - The sub-make reads only the root Makefile, so an extra `make -f other.mk` given to the parent no longer reaches the bats targets. Command-line variables and `MAKEFILES` still do. Nothing in the repo calls it that way. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): `make bats-unit-tests` runs every bats file after a failing one and still fails at the end, and the image cross-compile check now covers build scripts such as `build-go.sh`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Unit tests now continue running across individual test-file failures while still reporting an overall failure if any file fails. * Build-stage detection now recognizes build scripts with names such as `build-go.sh`. * **Tests** * Added coverage for unit-test results and build-script detection. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
cdc4b88a90 |
fix(ci): close e2e suite selection gaps in select-e2e and select-install (#4634)
## What this PR does The e2e test selector picked the wrong Chainsaw suites for some diffs. In one case it lost an escalation, in another it ran the full suite where a subset was enough, and in two more it missed a suite or ran nothing. The fixes are in `hack/select-e2e.sh`, with the `.yml` part also in `hack/select-install.sh`. - A changed file in a directory under `hack/e2e-chainsaw/` that holds no suite escalated only when it was the whole diff. Next to any path that selected a suite, the escalation was dropped with nothing on stderr. The directory name is now checked against the suite list right where the path is classified, like the `hack/e2e-apps/` rule already did. The empty-selection backstop can't be reached anymore. I kept it as a guard, and the doc says its line has no test. - A seaweedfs change ran the full suite, because `cozystack.seaweedfs-application` mapped to a `seaweedfs` suite that does not exist. Now it selects the suites that use tenant-root's object storage: bucket, the backup round-trips that write through the in-cluster `seaweedfs-s3` endpoint, and harbor, which keeps its registry in a BucketClaim served by the seaweedfs COSI driver. The issue says this mapping would break the round-trip test. It doesn't, since that test only walks suite to source to suite, and `vm-disk-application` already maps to `vminstance` the same way. The test takes the expected set from the suites' own yaml. - A suite written as `chainsaw-test.yml` was invisible to both selectors. Chainsaw v0.2.15 loads that name when there is no `chainsaw-test.yaml` (`pkg/discovery/load.go`), so such a suite runs but never gets selected. Both scripts list it now. `select-install.sh --validate` also passed when `find` failed, and now it fails. - An edit under `examples/backups/_lib/` was read as an app named `_lib` and selected nothing. It escalates now, same as `hack/e2e-chainsaw/_lib/`. - The selector comments quoted bats file counts and escalation-branch counts. All of them were stale, so I removed them. The doc half of #4120 was already fixed on main. Each fix has a test that fails on main. I also mutated each fix: its own test goes red and no other test does. Three related gaps outside these lines are tracked in #4630: the `packages/core/testing/Makefile` comment that names only `chainsaw-test.yaml`, suites built from numbered step files, and a file placed directly in `examples/backups/`. Two things worth knowing before merge: - Any file except the `.disabled` one in a parked suite's directory, like `hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml`, now always runs the full suite. That is the safe direction. If it gets expensive, those directories could be made inert. - The `examples/backups/_lib/` rule and its doc line point at a directory another PR adds. If this PR merges first, the rule just has nothing to match yet. Closes #3816, closes #3710, closes #4283, closes #4120, closes #4205, closes #4626. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): e2e test selection no longer drops an escalation for a non-suite path under hack/e2e-chainsaw/, selects the suites that use seaweedfs instead of the full run, sees suites written as chainsaw-test.yml, and escalates edits to examples/backups/_lib/. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * End-to-end test selection now recognizes suites configured with either supported YAML filename and reports discovery failures instead of silently proceeding. * Changes to shared backup walkthrough helpers and unrecognized test-suite directories now trigger broader test coverage. * Storage-related changes in SeaweedFS now select all relevant test suites. * **Documentation** * Updated testing guidance to describe suite discovery, file-change matching, and when the full test suite is selected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
031ded71ee |
feat(seaweedfs): alert before an instance stops taking writes (#4573)
## What this PR does A SeaweedFS instance that runs out of room stops taking writes without anything paging. A volume server below its `minFreeSpacePercent` reserve turns every volume it holds read-only; before that, it can run out of slots, and a bucket whose volumes are all full then cannot get a new one, even with half the disk free. Backups archiving into the instance fail in turn, and a Postgres cluster doing continuous WAL archiving keeps the segments on its own volume until that fills too. The generic `KubePersistentVolumeFillingUp` alert does not catch this: it pages below 3% free, while a volume server stalls at its 5% reserve or, when slots run out, with the disk still half empty; and its early warning needs the disk to keep filling, which stops once writes fail. This PR adds per-instance rules to the SeaweedFS system chart, on signals the servers already publish and the platform already scrapes, rendered under the same `monitoring.enabled` gate as the chart's ServiceMonitors: - `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less than 10% of its disk left above the reserve. - `SeaweedFSVolumeServerFull` (warning): a volume server cannot host another volume, because it is at the reserve or has no slot left. - `SeaweedFSWritesFailing` (critical): the master has failed to place a write in every 5-minute window for 15 minutes. Writes may still land in between; the usual cause is a bucket with no writable volume left that the master cannot grow. A new bucket's first writes fail a few picks while its volumes grow, which the 15-minute hold does not reach unless new buckets keep arriving. Every expression is pinned to the release namespace, because the root vmalert evaluates each instance's rules against every namespace's metrics. It also adds `docs/operations/seaweedfs-capacity-for-backups.md`: how to size an instance for continuous WAL archiving, what the rules mean, and how to grow an instance. The expressions were validated live rather than by unit tests, since the repository has no rule-evaluation tooling: on a test cluster, a standalone SeaweedFS 4.45 on a 1 GiB volume was scraped by the platform's vmagent and the rules rendered from this chart were evaluated by the root vmalert. All three loaded healthy and each reached `firing`: `SeaweedFSVolumeServerDiskLow` with 5.7% of the disk left above the reserve, `SeaweedFSVolumeServerFull` both at the reserve and on slot exhaustion, which stopped one bucket at 50% disk use, and `SeaweedFSWritesFailing` under continuous failing writes, resolving once they stopped. A new bucket's first successful writes on a free disk did fail a few picks, which is why that rule holds for 15 minutes. On the healthy platform instance all three stay quiet. helm-unittest pins what a render can break: the namespace selector of each rule, the shared label set of the two arms of the full-server rule, the gate, the critical rule's hold, and that alert templating reaches vmalert unrendered; each assertion was checked to fail when its line is mutated. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches a system chart template, its test and a document under `docs/operations/`; no app values, platform values, ApplicationDefinition, tooling or `hack/` path from the trigger map. ### Release note ```release-note feat(seaweedfs): alert when a SeaweedFS volume server nears its free-space reserve, cannot host another volume, or keeps failing to place writes, and document how to size an instance for continuous WAL archiving. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added namespace-scoped SeaweedFS monitoring alerts for low or exhausted disk capacity, unavailable volume slots, and write-placement failures. Capacity alerts use a 10% disk threshold, and alerts trigger after 15 minutes; write failures are marked critical. - **Documentation** - Added guidance on estimating SeaweedFS storage needs for Postgres backups, interpreting capacity alerts, and expanding storage when supported by the StorageClass. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
97fc4359ad |
feat(scheduler): set scheduling-class label on mutated pods (#2621)
## What this PR does
Extends the lineage-controller-webhook so that, on every Pod it mutates
with a `SchedulingClass`, it also sets a label of the same name (in
addition to the existing annotation).
Today, when a SchedulingClass is in effect on a tenant, the webhook
injects:
- `spec.schedulerName: cozystack-scheduler`
- annotation `scheduler.cozystack.io/scheduling-class: <name>`
This is enough for the cozystack-scheduler to look the class up and
apply its `nodeAffinity`, `nodeSelector`, and topology rules. However,
it is **not** enough to write a `SchedulingClass` whose
`spec.podAffinity` rules `labelSelector` targets pods across multiple
applications, because `labelSelector` matches labels and not
annotations.
This PR adds the label, leaving the annotation untouched for backward
compatibility:
```go
podLabels := obj.GetLabels()
if podLabels == nil {
podLabels = make(map[string]string)
}
podLabels[schedulerapi.SchedulingClassLabel] = schedulingClass
obj.SetLabels(podLabels)
```
The constants `SchedulingClassLabel` and `SchedulingClassAnnotation`
exported by `github.com/cozystack/cozystack-scheduler/pkg/apis/v1alpha1`
already share the same value
(`scheduler.cozystack.io/scheduling-class`), so no new key is
introduced.
**Why this is useful**: a single SchedulingClass can now express
"co-locate every pod that uses me at the same topology" with a rule
like:
```yaml
apiVersion: cozystack.io/v1alpha1
kind: SchedulingClass
metadata:
name: co-region-prod
spec:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
scheduler.cozystack.io/scheduling-class: co-region-prod
topologyKey: topology.kubernetes.io/region
```
When several applications share this class, the second-and-later pods
will be constrained to the region the first one landed on — without
hard-coding any region in the class itself.
### Tests
Adds a `webhook_test.go` (none existed in the package). Five unit tests,
using `clientfake.NewClientBuilder` +
`dynamicfake.NewSimpleDynamicClientWithCustomListKinds`, cover:
- Label, annotation, and `schedulerName` are all set when a class
exists.
- Existing pod labels are preserved (no clobbering).
- Non-Pod objects are a no-op.
- Pods in a namespace without a scheduling-class label are a no-op.
- Pods whose referenced `SchedulingClass` CR does not exist are a no-op
(the existing defensive behavior, preserved).
### Release note
```release-note
feat(scheduler): the lineage-controller-webhook now sets a scheduler.cozystack.io/scheduling-class label on mutated pods (in addition to the existing annotation), enabling SchedulingClass authors to write podAffinity rules that match pods across applications.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Pods with a valid scheduling class now receive its label alongside the
scheduler configuration and annotation.
- Existing Pod labels are preserved when the scheduling-class label is
added.
- **Bug Fixes**
- Scheduling-class settings are applied only when the namespace
references a scheduling class that exists. Non-Pods and namespaces
without a scheduling-class label are left unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|