717 Commits

Include renames

Author	SHA1	Message	Date
Aleksei Sviridkin	c5f352829c	ci(tests): gate the shell scripts this repository owns on shellcheck ... shellcheck ran nowhere: not from the Makefile, not from a workflow, not from pre-commit. Seven files under hack/ nevertheless carried `# shellcheck disable=` directives, twelve of the twenty-eight with a rationale beside them, so the tree was written as though the check existed. hack/shellcheck.sh lints every shell script this repository owns and compares the result with hack/shellcheck-baseline.txt. Enumeration is a union of shebang and `.sh` suffix, and neither half is redundant. The suffix alone misses fifty-seven files with no extension, fifty-three of them the platform migrations run-migrations.sh runs on a cluster upgrade and which no linter had seen before this; the shebang alone misses thirteen held by the suffix, the sourced libraries that open with a `# shellcheck shell=` directive and four with no marker at all. Narrowing to either one loses files, and what the run then reports never names the enumeration as the cause. Measured, and it differs by half: a script that leaves scope carrying baseline entries goes STALE, one carrying none -- sixty-six of the hundred and ninety -- leaves in silence, and dropping the suffix branch also removes the sourced libraries from the command line, so their sourcers gain SC1091 as NEW. The reader is shown a fixed finding to regenerate away, or a new finding to go fix, and in neither case a narrower gate. The baseline records the findings that predate the gate as `path check count` triples. Among the scripts in scope no file is skipped and nothing is excluded on the command line, though in-file `# shellcheck disable=` directives are still honoured, so a finding suppressed that way is absent from the list rather than recorded in it; two classes sit outside the scan and say so in the baseline header, the vendored chart scripts that make update regenerates and the .bats files, whose bats shebang the enumeration does not match. shellcheck parses those perfectly well, with a dialect and checks of its own for them; they are not baselined yet, and the .bats arm of the filter is a guard for one that acquires a shell shebang rather than an exclusion that does anything today. Dropping line and column numbers keeps an unrelated edit from reddening the ratchet, while the count still catches a new finding in a script already listed. An entry whose findings are gone fails the run too, so fixing a script forces the list to shrink rather than leaving a standing licence to reintroduce what was removed. The regenerate advice is withheld whenever the same run also reports a NEW finding. Printed together, the last line read wins, and regenerating a mixed run records the new findings rather than reporting them -- the cure offered for one half launders the other. Three ways the gate could have passed without checking anything are closed. A missing binary fails rather than skips. The version is pinned on both sides and the script refuses to run when they disagree, because findings differ between shellcheck releases and an unpinned binary reports against a baseline not written for it. And shellcheck's own exit status is kept instead of being piped through xargs, which collapses it: a status above 1 means an input was never analysed, and that now fails rather than reading as a clean file. A path holding a space or a tab is refused instead of linted. A baseline entry is three space-separated fields, so such a path cannot be recorded: the summary splits it and the entry names a file that does not exist. Refusing is also what licenses the word splitting the enumeration relies on, which until now was a comment asserting the tree had no such name rather than a check that it does not. hack/shellcheck.bats drives the gate through a mutation that introduces a violation and its removal, both directions of the ratchet, each of the four refusals above, and the enumeration each half of the union catches, the shrink half in the case where a count drops without reaching zero as well as the one where an entry vanishes, and the whitespace refusal on a tab as well as on a space. Every red path asserts the exit status before it asserts the message: `! cmd` alone proves nothing, because `set -e` never fires on an inverted command, and a gate that printed each ERROR line while exiting 0 satisfied all thirteen output greps when that was measured. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-07 16:28:42 +03:00
Aleksei Sviridkin	6c82988570	fix(tests): freeze the EXIT-trap count for multus-install-cni-plugins.bats (#3584) ... <!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does `hack/multus-install-cni-plugins.bats` arrived on main in 7739d1e29 carrying 12 EXIT traps, one day before the EXIT-trap ratchet in `hack/cozyreport.bats` was written, so the frozen list never learned about it. The guard therefore fails on main itself, and on every PR whose CI builds a merge with main — `make bats-unit-tests` exits 1 with: ``` FAIL: the set of unconverted EXIT-trap files changed. frozen: ... nightly-mirror_test.bats=5 ... found: ... multus-install-cni-plugins.bats=12 nightly-mirror_test.bats=5 ... ``` This records the count, which is what the guard's own comment prescribes for exactly this case: *"a file that did not exist when this guard was written arrives carrying its own \[traps\] ... Counts are updated rather than the files converted: all of them are owned by other branches, and a conflict there costs more than an uncovered trap."* Converting the file is not in scope here and belongs to whoever owns the multus branch. Verified by reproducing the guard's logic against a pristine checkout of `main` at 879d0f6db: the `found` set differs from `frozen` by exactly this one entry before the change, and matches after it. Found while investigating a red `Unit & controller tests` on #3571, which carries none of this. ### Screenshots No UI changes. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated the expected EXIT-trap inventory to include `multus-install-cni-plugins.bats`. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-07 10:42:43 +03:00
IvanHunters	318be935ca	fix(e2e): capture Talos logs for tenant worker join failures (#3548) ... ## What this PR does Addresses #3513. When tenant Kubernetes workers miss the 18-minute Ready deadline, the E2E runner now mints a one-hour `os:reader` Talos client certificate from the tenant's existing issuer and runs authenticated `talosctl` from a hardened same-namespace diagnostics Pod. It captures bounded `dmesg` and kubelet output for every worker VMI in the cozyreport snapshot, records missing IPs and command exit codes, continues across unreachable workers, and preserves the original test failure. The diagnostics Pod has no service-account token or Secret volume; credentials are streamed into an `emptyDir`, and the Certificate, Secret, and Pod are labeled for cleanup. Workers that have not reached Talos `apid` can only yield a bounded connection error, while workers that reached `apid` provide guest logs. This adds the evidence needed to distinguish the remaining guest boot failure but does not claim to resolve its underlying cause. The readiness budget and success path are unchanged. Because this changes a shared E2E helper, test-impact analysis selects the full E2E suite for this PR. ### Screenshots Not applicable; no UI changes. ### Testing - `hack/cozytest.sh hack/run-kubernetes-talos-diagnostics_test.bats` (10 tests) - `hack/cozytest.sh hack/run-kubernetes-drain_test.bats` - `hack/cozytest.sh hack/cozyreport-talos.bats` - `bash -n hack/e2e-chainsaw/_lib/run-kubernetes.sh` - `git diff --check origin/main...HEAD` The full `make bats-unit-tests` run reached the unrelated `hack/migration-seaweedfs-db-adopt.bats` suite and could not continue because Docker is unavailable in the development environment. A live failure-path run was not performed because the available development environment has no Talos tenant fixture; this PR is intended to exercise that path in Cozystack CI. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): capture authenticated Talos kernel and kubelet diagnostics when tenant Kubernetes workers miss the Ready deadline ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved troubleshooting when worker nodes fail to join by collecting per-node operating system and kubelet diagnostics. * Diagnostics now continue across partial failures and clean up temporary resources afterward. * Extended Kubernetes test operation timeouts to allow failure diagnostics to complete reliably. * **Tests** * Added regression coverage for diagnostic collection, timeout handling, multi-worker scenarios, missing node addresses, and cleanup. * **Chores** * Ensured the required diagnostic image is preloaded on nodes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-07 10:40:03 +03:00
Aleksei Sviridkin	0dae3ca95a	fix(e2e): budget tenant scheduling separately from workload readiness (#3579) ... ## What this PR does The tenant backend Deployment in the kubernetes suites had one 300s budget to reach `condition=Available`, and two unrelated variable costs shared it. The first is scheduling. What the suite establishes before that point is two tenant nodes Ready, which is weaker than schedulable: a Ready node still carries `node.cilium.io/agent-not-ready` until the tenant cilium agent claims it, and a node the bringup has not finished with is `SchedulingDisabled`. In the run recorded in #3577 scheduling took 2m18s and 1m57s in the two suites, which left the image pull to finish inside what remained. It did not, and both suites reported the same `timed out waiting for the condition` for what were two different shortfalls. So this waits for a node that actually accepts a Pod before creating the workload, on its own budget and its own failure message. The gate encodes the scheduler's rule for a Pod that tolerates nothing (Ready, not unschedulable, no `NoSchedule` or `NoExecute` taint) over a `custom-columns` probe, prints the node table on both outcomes so a timeout names the taint that held it, and treats a failed probe as not-schedulable so an API blip cannot release it. On the happy path it adds no wall time: it spends the seconds the Pod would otherwise spend Pending, plus at most one 5s poll interval. On a failing run the two budgets stack, so a run that exhausts both now gives up at ~600s where it used to give up at 300s, inside the 40m Chainsaw script op the suites document as a ~25m bringup. The readiness wait keeps its 300s, now starting from a schedulable node, and dumps deployment, pod and event state when it runs out. The workload image is also pinned by digest. The tenant workers reach no Docker Hub mirror (`hack/e2e-talos-image-cache.yaml` serves the Talos worker OS disk image over HTTP and is not a registry mirror), so nginx is pulled from Docker Hub on every run either way. The digest does not take the pull off the critical path; it fixes what that pull returns instead of leaving a floating tag free to change size and content under a fixed deadline. Preloading the image would need infrastructure this tree does not have, and is not attempted here. Nothing will bump the pin: no renovate manager reads `hack/`, and the comment says so, because for a throwaway test workload the freeze is the point. What this does not claim: it removes two measured consumers from a fixed budget, both taken from the events of a single run. Whether that budget was the only thing making those suites red is not established from one run, so this is not offered as the fix for a red pipeline. `hack/run-kubernetes-schedulable_test.bats` covers the new logic: every branch of the predicate, the multi-node scan, the poll-again path, the deadline, and a probe that fails. Each test was verified by mutating the helper and checking that the intended test, and no other, went red. Note for whoever reviews alongside #3575: that branch mirrors `ghcr.io` for tenant worker pulls, not `docker.io`, so it does not change the Docker Hub pull described here. `git merge-tree` reports no conflict between the two, nor with #3548. Observed in #3577. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: The trigger map was walked against the diff. The change is confined to `hack/e2e-chainsaw/_lib/run-kubernetes.sh` and one new `hack/*.bats` unit test. It moves and renames nothing under `hack/`, changes no make target, and does not touch `hack/e2e-prepare-cluster.bats`, any package, any CRD or any namespace name, so none of the listed repositories are reached. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes deployment readiness by waiting for a suitable, schedulable node before timing backend startup. * Added clearer diagnostics when nodes are unavailable or backend readiness fails. * Replaced the floating backend container image tag with a fixed, verified version. * **Tests** * Added coverage for node readiness, cordoning, taints, polling, timeouts, probe failures, and deployment sequencing. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-06 23:46:50 +03:00
Mattia Eleuteri	3aef42b993	fix(tests): freeze the EXIT-trap count for multus-install-cni-plugins.bats ... hack/multus-install-cni-plugins.bats arrived on main in 7739d1e29 carrying 12 EXIT traps, one day before the ratchet in hack/cozyreport.bats was written, so the frozen list never learned about it. The guard therefore fails on main itself, and on every PR whose CI builds a merge with it: FAIL: the set of unconverted EXIT-trap files changed. frozen: ... nightly-mirror_test.bats=5 ... found: ... multus-install-cni-plugins.bats=12 nightly-mirror_test.bats=5 ... Record the count, which is what the guard's own comment prescribes for a file that did not exist when it was written: "Counts are updated rather than the files converted: all of them are owned by other branches, and a conflict there costs more than an uncovered trap." Verified by reproducing the guard's logic against a pristine checkout of main: the difference is exactly this one entry, before the change and none after it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Mattia Eleuteri <mattia@hidora.io>	2026-08-06 16:58:08 +02:00
Aleksei Sviridkin	879d0f6db6	fix(installer): make operator readiness honest in the chart and the install gate (#3466) ... ## What this PR does The operator Deployment ran `replicas: 1` with `maxUnavailable: 1`, so the Deployment controller took the minimum available replica count as `1 - 1 = 0` and the `Available` condition reported `True` with no running pod at all. Measured against an operator whose image tag does not exist: the condition was `True` with `reason: MinimumReplicasAvailable` while the only pod sat in `ImagePullBackOff`, `unavailableReplicas: 1`, and no `availableReplicas` key in the status. That broke two separate things, so this fixes both. The E2E install gate asked `kubectl wait --for=condition=Available` and passed in 156 ms against that dead operator. The install then carried on and died about two minutes later on the `packages.cozystack.io` CRD wait further down the file, which sends you after CRD installation when the operator never started. It now uses `kubectl rollout status`, which reads updated and available replicas directly, and dumps the deployment, pods, operator logs and namespace events when it fails. This half is deliberately independent of the chart: `rollout status` is correct under any rollout strategy, so a later change to that strategy cannot quietly make the gate vacuous again. The chart is the other half, and fixing the gate does not cover it. `Available` is read by more than this test, and until the strategy changes an alert on that condition, a health dashboard, or a person reading `kubectl get deploy` all show green while the operator is dead. The strategy moves to `maxSurge: 1` / `maxUnavailable: 0`, which restores the ordinary meaning of the condition. Kubernetes refuses `maxSurge: 0` and `maxUnavailable: 0` together, so the surge has to move for the unavailability to reach zero. That pair is worth reading twice, because it is not a configuration I picked. At `replicas: 1` the Kubernetes default of `25%` / `25%` resolves to exactly `maxSurge: 1` and `maxUnavailable: 0`, since the surge rounds up and the unavailability rounds down. What this PR does is stop overriding the default, not introduce a tuning of its own, and the risk of the change should be judged on that basis. Moving the surge is safe here, and I checked rather than assumed, because the pair came from somewhere. It was carried over from the earlier `cozystack` Deployment, where a sidecar published host port 8123 and `hostNetwork` left the surge pod unschedulable on a single-node cluster. It got here in two steps, which is worth knowing if you go looking for the reason: by the commit before this file existed the sidecar was already gone and no port was declared, yet the pair stayed. It had outlived its own reason before it was ever copied, which is why there is no justification to find in this file's own history. On a single-node cluster the surge pod is scheduled next to the outgoing one and the rollout completes; adding a declared container port reproduces the original `didn't have free ports for the requested pod ports` failure exactly. Nothing else in the pod holds a port either: metrics and health probes are bound to `0`, and the configured webhook server never starts because the operator registers no webhook and never asks the manager for the server. The surge makes leader election load-bearing, though it covers less of the overlap than it first appears. While `maxSurge: 0` capped total pods at `replicas` a second manager process could not exist, so `--leader-elect=true` was defence in depth. With a surge pod the lease is what keeps the incoming manager's reconcile loops off the cluster while the outgoing one is still leader, and because there is no readiness probe the incoming pod counts as available as soon as its container starts, so that overlap covers the whole image pull. It does not cover everything: the operator installs CRDs, installs Flux and creates the platform PackageSource before `mgr.Start()`, and the lease is only taken inside `mgr.Start()`, so those three privileged installs run outside leader election entirely. They are safe because both pods write the same objects through server-side apply under the same field manager, which is a different guarantee from the lease. The `k8s-await-election` wrapper this operator replaced did hold the whole process until it won leadership, so if that property sounds familiar it belongs to the predecessor rather than to this binary. The sibling `opensearch-operator` chart turns leader election off at one replica for a real reason: a transient lease-renewal blip makes controller-runtime exit the process, which at one replica reads as a crashlooping Deployment. This operator installs first, when the apiserver is at its shakiest, so that scenario applies here too. I am accepting the risk knowingly, because after `maxSurge: 1` the alternative is two managers reconciling the same cluster, which is worse. The flag is pinned in the unit test so it cannot be turned off without the test going red. One consequence is permanent, so I am writing it down now instead of leaving it to be found in a year. Because any bound port breaks the surge pod under `hostNetwork`, an `httpGet` readiness probe is off the table for this Deployment for good and only an `exec` probe could ever work. So `Available` still answers "a container started", not "the operator is healthy": an operator that starts and then dies partway through installing CRDs is caught by the CRD waits below the gate, not by the gate itself. The surge also opens one new window. The operator installs CRDs and Flux before the manager starts, which puts that work outside leader election, so the incoming pod runs those applies while the outgoing pod is still leader. Both apply the same objects server-side and the outgoing pod exits moments later. `--wait` is dropped from the helm step in the same change. Helm decides a Deployment is ready with the same `replicas - maxUnavailable` arithmetic, so on the old strategy it was satisfied at zero ready pods and gated nothing, which is why the broken install got past it as well. Once the strategy is honest it would gate for real, but on its own 2m budget, and it would fail the install before the rollout gate ran and hide the diagnostics behind a bare Helm timeout. Hooks are waited for with or without the flag, and the release otherwise carries only the Deployment and its RBAC objects, so the rollout gate is the better single place to decide the operator is up. A helm unittest suite pins the strategy and all four preconditions the surge makes load-bearing: no container or initContainer declares a port, both listener arguments stay disabled, leader election stays on, and no probe uses `httpGet`. That last one follows from the two disabled listeners leaving no port for a probe to reach, and it is the worst of the four to get wrong, because the others wedge an upgrade while a `livenessProbe` against a disabled listener crashloops the pod on every cluster, fresh install included. Only the `httpGet` field is forbidden rather than the probes themselves, so an `exec` probe stays available. All four run against all three chart variants, because the template branches on variant inside the container spec and a port added under one branch would render for that variant alone. I checked the suite fails for the right reasons instead of trusting it. Reverting the strategy, declaring a `containerPort` on the operator container, declaring one under a single variant branch, declaring one on an initContainer, pointing a bind-address at a real address, turning leader election off, and adding a readiness, liveness or startup probe with `httpGet` each turns it red on its own assert. Inserting a container ahead of the operator trips the container-name pin before the argument asserts can start describing the wrong container. An `exec` probe correctly stays green. The rationale is split by audience. What an operator applying the rendered manifest has to respect stays in plain comments and travels into `_out/assets/cozystack-operator-*.yaml`: why the strategy is what it is, that no port may be declared or bound, and that leader election has to stay on. Everything that is only about this repository moves into a `{{/* */}}` template comment that stops at the chart boundary. That covers the pointer to the test and which assert pins what, where the strategy was copied from, why the webhook server on 9443 is not a third port binder, and why `opensearch-operator` reaches the opposite conclusion on leader election. The other templates in this chart already keep maintainer notes that way. ### Screenshots Not applicable, no UI changes. ### Downstream repositories I left the boxes empty on purpose rather than to skip the question, because the honest answer is not one of them and someone else should make the call. Walking the trigger map in `docs/agents/contributing.md`, nothing fires. The `hack/` triggers name `hack/e2e-prepare-cluster.bats`, `hack/package.mk`, `hack/update-crd.sh` and moves or renames under `hack/`, and this PR only edits the body of `hack/e2e-install-cozystack.bats`. The `ansible-cozystack` trigger is renaming an object the role waits on, and `deployment/cozystack-operator` keeps its name. The behaviour that role sees does change, though. It runs `kubectl wait deployment/cozystack-operator --for=condition=Available`, the same vacuous check this PR replaces here, so the chart change turns its wait from one that returns instantly into one that means something. Its `cozystack_operator_wait_timeout` defaults to 300 seconds, the same budget I gave the gate here, so a cold image pull fits and I do not think anything needs to change over there. No follow-up PR seemed warranted for a change that only makes an existing check work, but that is a judgement call rather than a trigger, which is why the boxes are empty. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(installer): the cozystack-operator Deployment no longer reports Available with zero running pods. It ran replicas: 1 with maxUnavailable: 1, which made the minimum available replica count 0, so the condition stayed True while the operator was dead and anything reading it, including the E2E install gate, saw green. The rollout strategy is now maxSurge: 1 / maxUnavailable: 0 and the install gate waits for the rollout instead of the condition. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved operator installation checks by monitoring deployment rollouts and providing deployment, pod, log, and event details when a rollout fails. * Updated operator upgrades to maintain availability during rollout, including in single-node environments. * Documented rollout requirements and behavior across supported installation configurations. * **Tests** * Added coverage for rollout settings and upgrade compatibility across supported installation variants. * Clarified Helm unit-test directory coverage documentation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-06 15:25:39 +03:00
Aleksei Sviridkin	2527e915bd	fix(e2e): collect diagnostics for unready pods and record every truncated capture (#3567) ... ## What this PR does The e2e diagnostic tarball exists so a red run can be explained from the uploaded artifact alone. Several of its collectors answered "nothing is wrong" instead — some because they selected on a column that never carries the value they filtered for, others because they stopped early and left nothing saying they had. Both failure modes look identical to a healthy cluster from the outside, which is the worst possible shape for a diagnostic. Four selectors in `hack/cozyreport.sh` and `hack/cozyreport-summary.sh` were reading the wrong thing: - Per-pod evidence was chosen by the STATUS column, so a pod left at `0/1 Running` by a readiness probe that never passes was skipped. That is a common way for a suite to time out, and the artifact carried no trace of the pod the run died on. Selection now compares the two halves of the READY column; terminal `Succeeded`/`Completed` pods stay excluded. - The pending-LoadBalancer section filtered `$4`, which is CLUSTER-IP. CLUSTER-IP holds an address, `None` or `<none>` and never `<pending>`, so that loop had never executed. `<pending>` only ever appears in EXTERNAL-IP, `$5`. - The cozystack-apps section gated all three of its kinds on `kubectl get crd <kind>.apps.cozystack.io`. ApplicationDefinition is a CRD but in a different group, and Tenant comes from an aggregated APIService with no CRD to ask about, so the probe failed and the section never ran. - The kamaji section was gated on `cozy-linstor`'s `linstor-controller` deployment rather than kamaji's own, so a cluster without LINSTOR got no kamaji logs and no TenantControlPlanes — on exactly the runs that need them. The summary had three of its own: the HelmRelease listing printed `$5` where STATUS is the Ready condition's whole message, reducing every failed release to one word like "Helm"; the Flux sources listing used a jsonpath with a nested filter that kubectl rejects outright, so it printed nothing on every report ever taken; and pod AGE was read as `$6`, which shifts whenever RESTARTS prints the `7 (4m12s ago)` form — that is, for exactly the rows worth reading. The second half is bounding. The pod, VM, VMI and services walks now run under a count cap and a wall-clock budget, and every read is individually time-boxed. The point is not the bounds themselves but that each one names itself when it fires: a walk stopped by the cap or the clock leaves `COLLECTION-TRUNCATED.txt`, a pod list that never returned leaves `COLLECTION-FAILED.txt` instead of the empty tree a healthy cluster produces, a log cut short says so on its own last line, and a `--tail` bound is recorded beside the logs rather than inside them, since the platform's controllers log JSON per line and a trailing prose line costs a parser the whole file. A knob value the collector could not use is reported too, so a truncation note quoting `600` is never mistaken for the `10m` somebody actually set. Where `timeout` is absent the reads run unbounded rather than not at all, and the section says which of the two happened, because a missing local binary is not a statement about the cluster. One behaviour change worth stating plainly, because it is a trade and not a pure gain: the per-pod section used to be uncapped and read container logs whole, and it is now bounded by `COZY_REPORT_PODS_MAX` (40), `COZY_REPORT_PODS_BUDGET` (600s) and `COZY_REPORT_POD_TAIL` (2000). Below those thresholds the new tree is a strict superset of the old one — the old selector missed `Running 0/1` entirely and skipped logs for `Pending` pods. Above them the artifact is smaller than before, every drop is named in the tree, and `COZY_REPORT_POD_TAIL=-1` with a larger `COZY_REPORT_PODS_MAX` restores the old reach. Because widening the selection and capping the walk in one change could otherwise let a newly-in-scope pod evict one the uncapped walk always collected, the ordering splits by evidence class before namespace: pods that are not `Running` — the class the collector took before this — all rank ahead of pods that are `Running` but unready, so the cap can only ever drop a pod the old collector did not keep either. `hack/e2e-capture-previous-logs.sh` gets the same treatment: its notes now go to `capture-notes.txt` in the artifact as well as to the job log, and a pod list that never returned is reported as such rather than as "nothing restarted". That script now creates its own output directory before its first line of output, which makes the comment in `hack/e2e-chainsaw/.chainsaw.yaml` wrong about why the once-per-test guard there holds — the only change to that file is the corrected comment, saying the `-d` check keys on whether the catch block already ran rather than on what the helper found. The rest is test coverage — a new `hack/cozyreport.bats`, an extended `hack/capture-previous-logs.bats` — and the removal of `EXIT`-trap cleanup from the bats files those tests now cover. That trap is worth its own note: inside an `@test` body it replaces the one the `bats` binary installs for its own bookkeeping, so a test that then fails prints no TAP line at all. It does not appear as `not ok`; it disappears, and only the `# bats warning: Executed N instead of expected M tests` line at the end distinguishes the run from a green one. A guard test enforces the ban over a named list of files that grows as the remaining ones are converted. ### Testing Every collector this touches is now covered by unit tests that run without a cluster: a new `hack/cozyreport.bats` (166 tests) and an extended `hack/capture-previous-logs.bats` (34), driven through `hack/cozytest.sh`. Ten `hack/*.bats` files run green locally, 265 tests in total, with each file's `@test` count reconciled against its passing count rather than trusted to the exit status — that reconciliation is the point of the trap change, since an exit code answers "did anything fail" and never "did anything run". The guard test that bans `EXIT`-trap cleanup was checked by committing the violation rather than by reading it: a folded `trap 'rm -rf "$tmp"' \` + `EXIT` added to a file on its list turns the guard red and names that file, and removing it turns the guard green again. `shellcheck --shell=sh` over the three changed scripts reports strictly fewer findings than before the change (`hack/cozyreport.sh` SC2086 170 to 116, SC2162 15 to 13; the three SC2102 in `hack/cozyreport-summary.sh` are gone), and `hack/e2e-capture-previous-logs.sh` is clean. The parts that need a cluster — that the bounds fire, and that each one lands in the artifact — are exercised by the e2e run itself, since these collectors run on every failed suite. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. This PR touches `hack/cozyreport.sh`, `hack/cozyreport-summary.sh`, `hack/e2e-capture-previous-logs.sh`, `hack/e2e-chainsaw/.chainsaw.yaml`, ten `hack/*.bats` files and `docs/agents/e2e-testing.md`. Nothing is moved or renamed under `hack/`, no make target changes what it does, and the files downstream consumers key on (`hack/package.mk`, `hack/common-envs.mk`) are untouched. No package, values key, CRD, variant, namespace or release asset is involved. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): the diagnostic report now collects per-pod evidence for pods that are Running but not Ready, which it previously skipped, and several collector sections that silently produced nothing now report correctly. Collection that stops early on a bound now says so inside the artifact, so a truncated report can be told apart from a healthy cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Diagnostic reports now prioritize relevant resources and collect bounded pod, VM, service, storage, certificate, and Flux data. - Reports preserve partial results and clearly identify timeouts, unavailable resources, truncation, warnings, and collection failures. - Previous-container logs now include bounded capture, namespace prioritization, failure details, and partial-read evidence. - Reports target the operator workload and safely handle missing dependencies or temporary storage failures. - **Documentation** - Expanded guidance for reliable end-to-end testing and diagnostic report collection. - **Tests** - Added comprehensive coverage for report generation, log capture, failure handling, limits, and portability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-06 15:25:20 +03:00
Aleksei Sviridkin	a20f3620b1	fix(e2e): select catch pod logs by the nested release instance label (#3564) ... ## What this PR does The harbor, openbao and qdrant catch blocks selected pod logs on the app release name. Each of these charts deploys its workload through a nested HelmRelease named `<release>-system` and passes `fullnameOverride: <release>` down to it, so the inner chart stamps `app.kubernetes.io/instance` from its own release name, with the suffix, while object names come from the override, without it. The selector matched no pods, and a failed run collected nothing. - points the three `podLogs` catch collectors at the `-system` instance label - adds a `podLogs` collector for Harbor's CNPG database, which no collector reached before, selected on `cnpg.io/cluster` rather than on the instance label CNPG merely derives from the Cluster name - pins the instance label on a resource in an assert each suite already runs, so a chart renaming that label fails a step instead of quietly emptying a capture - prints the instance label of every pod in the namespace from each test-wide catch, because `podLogs` cannot report that it matched nothing - keeps the unsuffixed value as a second collector in qdrant only, because the parent chart's PVC-cleanup Job labels its pod template with the parent release name and that suite deletes the parent inside a `try:` Harbor renders the same hook and deliberately does not get the same collector: it has no `delete` op, so its parent release is uninstalled only in Chainsaw's cleanup phase, which no catch can observe: catch is a defer inside the step, while the cleaner runs from `t.Cleanup`, after the test function returns. A comment records that asymmetry so the next reader does not restore the collector for symmetry. Each comment states what the pin does and does not cover: it does not catch a wrong value typed into a selector, and it catches nothing when a run fails before reaching the assert. The qdrant comment is equally explicit about how narrow its collector's window is, because the hook always exits 0, so Helm calls it succeeded and `hook-succeeded` takes the pod away; what remains is an uninstall that hangs with Helm still waiting. Telling a lookup's outcomes apart is one helper under `hack/e2e-chainsaw/_lib/` rather than a copy per call site, with the outcome logic split from the kubectl call so it can be pinned without a cluster — the shape `_lib/etcd-probe.sh` already uses. Both the census and the qdrant cleanup listing report through it, differing only in wording. Each lookup keeps stderr apart from the value it captures: kubectl exits 0 while logging that it could not reach an API group, and an aggregated APIService in that state is what a failed run usually looks like, so folding the streams would make the "nothing found" branch unreachable and print that log line as though it were a result. Note what the extraction costs in CI: touching the shared `_lib/` and adding a top-level `hack/*.bats` escalates test-impact analysis from the three suites this change edits to the full Chainsaw suite, so this PR runs everything and is exposed to the whole spread of known flakes. A red suite unrelated to harbor, openbao or qdrant is that exposure, not a defect in this diff. ## Testing - `hack/pod-label-census_test.bats`, 9 cases over the helper's outcomes, run via `hack/cozytest.sh` and picked up by `make unit-tests` through the existing `BATS_UNIT_FILES` glob - mutation-checked rather than assumed: folding stderr back into the captured value, propagating kubectl's exit code out of the helper, and collapsing the failed/empty distinction each turn the suite red, and each is green again on revert - `chainsaw lint test --file` on each of the three suites, and `chainsaw test --no-cluster --config hack/e2e-chainsaw/.chainsaw.yaml --include-test-regex '^$'` over all three, both with v0.2.15, the version the sandbox image pins - `shellcheck --shell=sh` on the helper and on the catch scripts extracted from the YAML - the qdrant step catch driven against a stub kubectl that fails: it reports the failed lookup rather than dying on it - every selector, pin and comment claim checked against the chart that stamps it ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): collect harbor, openbao and qdrant pod logs on a failed run ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added pod-label census diagnostics to improve visibility into workload labels during test failures. * Enhanced failure diagnostics for Harbor, OpenBao, and Qdrant by collecting relevant workload and database logs. * Added bounded reporting for missing pods, lookup failures, and incomplete cleanup resources. * **Bug Fixes** * Updated diagnostic selectors and readiness checks to match chart-generated instance labels. * Improved PVC cleanup diagnostics with clearer reporting of remaining resources. * **Tests** * Added coverage for successful, empty, failed, and partial diagnostic results. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-06 15:25:09 +03:00
Aleksei Sviridkin	5542ae52b7	fix(e2e): budget tenant scheduling separately from workload readiness ... The tenant backend Deployment had a single 300s budget to reach condition=Available, and two unrelated variable costs shared it. The first is scheduling. What the suite establishes before that point is two tenant nodes Ready, which is weaker than schedulable: a Ready node still carries node.cilium.io/agent-not-ready until the tenant cilium agent claims it, and a node the bringup has not finished with is SchedulingDisabled. Scheduling took 2m18s and 1m57s in the two tenant suites of one run, leaving the image pull to finish inside what was left. It did not, and both suites failed with the same message for two different shortfalls. Wait for a node that actually accepts a toleration-free Pod before creating the workload, on its own budget and its own failure message. The gate encodes the scheduler's rule for such a Pod (Ready, not unschedulable, no NoSchedule or NoExecute taint) over a custom-columns probe, prints the node table on both outcomes so a timeout names the taint that held it, and treats a failed probe as not-schedulable so an API blip cannot release it. The readiness wait keeps its 300s, now starting from a schedulable node, and dumps deployment, pod and event state when it runs out. Pin the workload image by digest. The tenant workers reach no registry mirror, so nginx is pulled from Docker Hub on every run either way; the digest fixes what that pull returns instead of leaving a floating tag free to change size and content under a fixed deadline. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-06 12:20:50 +03:00
Aleksei Sviridkin	115d9bbaaf	fix(installer): let the operator Deployment report Available honestly ... At replicas: 1 with maxUnavailable: 1 the Deployment controller takes the minimum available replica count as 1 - 1 = 0, so the Available condition reports True with no running pod at all. That reading is what every consumer of the status sees: an alert on Available, a health dashboard, or a person reading kubectl get deploy all show green while the operator is dead. The replacement is not a tuned pair, it is the platform default written down. At replicas: 1 the default 25% / 25% resolves to maxSurge 1 and maxUnavailable 0, because the surge rounds up and the unavailability rounds down. So this change stops overriding the default rather than introducing a configuration of its own, and Kubernetes rejects maxUnavailable: 0 together with maxSurge: 0, so the surge could not have gone the other way regardless. The overridden pair was carried over from the earlier cozystack Deployment, where a darkhttpd sidecar published a host port and hostNetwork left a surge pod unschedulable on a single-node cluster. It arrived here in two steps: by the commit before this file was created the sidecar was already gone and no port was declared, yet the pair stayed, so it had outlived its own reason before it was ever copied. That is why looking for a justification in this file's history finds none. With no port declared anywhere in the pod there is no hostNetwork port conflict left, so the surge pod schedules beside the outgoing one; declaring one reintroduces exactly the scheduling failure the pair was added for. Not exercised on a live cluster in this rework. Stay on RollingUpdate rather than move to Recreate, which would fix the condition too, and by a shorter route: MaxUnavailable() reports 0 for any strategy that is not RollingUpdate, so the minimum available replica count becomes replicas - 0 = 1 and the condition is honest without any surge pod at all. The transition was not attempted, and the failure this project fixed in ouroboros 0.8.1 does not transfer to it. There the rollingUpdate block was the one the apiserver defaults onto a Deployment created without one, so it belonged to no applier and outlived the switch to Recreate. This Deployment has carried an explicit block since the commit that created it, and nothing applies it server-side, so whether Recreate would upgrade cleanly is an open question about the field-ownership history of the live object rather than a known rejection. What keeps RollingUpdate here is availability rather than apply semantics: maxUnavailable: 0 leaves the outgoing operator running when the replacement never starts, where Recreate deletes it first and leaves none. Editing the numbers inside the block is upgrade-safe from every state the field can already be in. The surge introduces new windows, and leader election covers less of them than it looks. The operator runs four privileged install steps before mgr.Start(), and the lease is only acquired inside mgr.Start(): the CRD install, the Flux install, the platform source resource and the platform PackageSource. All four therefore run in the incoming pod outside leader election entirely. What the lease does cover is the reconcile loops, the part that would otherwise conflict continuously instead of once at startup. That leaves three seams, and only the first two happen on a healthy upgrade. The routine one is the incoming pod's install steps against the outgoing pod's live reconcile loops, since the outgoing pod ran its own installs at its startup and has been inside mgr.Start() ever since. Those do not contend for the same fields: the installs write as field manager cozystack-operator, while the reconcilers write as cozystack-packagesource-controller and cozystack-package-controller and go mostly to status subresources. Different owners, different objects, so this seam is safe by separation rather than by any shared manager. The second is version skew rather than field contention, and maxSurge: 1 widens it rather than opening it. Under maxSurge: 0 the surge budget counts desired replicas rather than live pods, so the incoming pod is created while the outgoing one may still be Terminating - the overlap existed. What it did not have was a live counterparty: scale-down sends SIGTERM before the new pod is created, and that cancels the manager context, so the old loops were already unwinding. Here the outgoing pod is not signalled until the incoming one is available, so the overlap runs against a fully live reconciler for a termination grace period instead of a vestigial one. The incoming pod reports Ready the instant its container starts, because there is no readiness probe, so its CRD install runs while the outgoing binary's reconcile loops are still live and now writing against a schema the incoming pod has already moved. Old code against a new schema, which no field manager separates. It is bounded rather than benign: the outgoing pod exits on SIGTERM within seconds, a rejected write is transient and the next reconcile repairs it, and no object is left corrupt. Widening the skew, by adding a probe that delays Ready or by a migration the old code cannot round-trip, is what would make this seam matter. The third seam is install against install, and it needs the outgoing pod to be restarting, since a crashlooping pod re-runs its four steps on every restart, which is exactly the broken operator this gate exists to catch. Three of the four steps use server-side apply under the same field manager, so they never conflict and resolve last-writer-wins whenever the two pods carry different manifests. That ordering matters in one direction, and it is accepted here rather than guarded against: the CRD install and the Flux install both apply with Force: true, so an outgoing pod that writes last silently overwrites what the incoming pod has just applied, with no conflict and no error to exit on. The incoming pod is inside mgr.Start() by then and will not apply again until it restarts, so that revert of the upgrade's install phase persists rather than healing itself. Reaching it needs a restart landing in the seconds between the incoming pod's installs and the outgoing pod's SIGTERM. installPlatformSourceResource is the remaining exception, a read-modify-write of a Get followed by Create or by Update with the observed resourceVersion, so two pods running it at once can collide on AlreadyExists or on a conflict; that failure is loud and self-healing, since any error there exits the process and the pod restarts, and the object is not left corrupt. Pin the strategy and all four preconditions the surge makes load-bearing in a unit test: no container or initContainer declares a port, both listener arguments stay disabled, leader election stays on, and no probe uses httpGet. They fail differently and none is visible to the others, since a declared port stops the scheduler, a bound one only fails once the incoming process starts, and a second manager without a lease fails nothing at all until it writes. The httpGet precondition follows from the two disabled listeners leaving no port to target, and it is the worst of the four to get wrong: the other three wedge an upgrade, while a livenessProbe against a disabled listener crashloops the pod on every cluster, including a fresh install where no surge happens. Only the httpGet field is forbidden rather than the probes themselves, because an exec probe stays legitimate. All four are checked against all three variants, because the template branches on variant inside the container spec and a port added under one branch renders for that variant alone. The strategy type is pinned alongside its two values, since maxSurge and maxUnavailable say nothing on their own once the type is no longer RollingUpdate. The listener preconditions are pinned less tightly than the other two, and the test says so rather than implying coverage it does not have. A membership assert has no notion of position while Go's flag package keeps the last occurrence, so a duplicate after the disabled one wins and the assert still passes, while a duplicate before it loses and the assert passes just the same - the test cannot tell those two apart. Moving the disabled flag last fixes the process without making the assert catch anything. Rejecting the two addresses this binary defaults to closes the shape a values-gated block to enable metrics would take, and nothing available here closes an arbitrary address, since helm-unittest cannot express that a flag appears at most once. Note in hack/helm-unit-tests.sh that the loop covers packages/core, which its comment omitted. The suite added here lives there, so a later edit reconciling the loop with the stale comment would have silently stopped running it. Split the rationale by audience rather than keeping it all in shipped comments. What an operator applying the rendered manifest has to respect stays in plain comments and travels into the install artifacts: that the pair is the default, why maxUnavailable must stay 0, that no port may be declared or bound, that no probe may use httpGet, and that leader election has to stay on. Most of what stays in the template comment is what the chart's own suite pins, so it cannot drift out of step with the chart without a test going red: the four preconditions and which assert covers each, and why the variant loop is needed. Three things there are deliberate exceptions, kept because anyone editing the strategy has to know them and no assert can carry them - that the webhook server configured on 9443 is not a port binder, since controller-runtime adds it to the runnables only inside GetWebhookServer() and this binary never calls it; that two overlaps between the pods sit outside the lease; and why RollingUpdate is kept rather than Recreate. Those name what exists. The seam analysis above, which is how each one works, deliberately stays here instead. It describes another file's internals, nothing pins it, and a commit message is fixed in time, so it cannot quietly fall out of date the way a comment can. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 18:15:41 +03:00
Aleksei Sviridkin	e80b8e370f	fix(e2e): collect logs for unready pods and record truncated captures ... The failure snapshot selected pods by their STATUS column, so a pod whose container runs while its readiness probe never passes -- STATUS Running, READY 0/1 -- was dropped from the report entirely. That is a common way for a suite to time out: the replica never joins, its owning HelmRelease sits in InProgress, and the artifact carries no trace of the pod the run died on. Select on the READY column instead, which also covers the partially ready multi-container case, and keep every phase that was collected before. On one degraded cluster that is four of the seven broken pods the old filter reported nothing about. The VMI and VM sections had the same blind spot and now share one selector that reads READY as the last field, because a Pending object prints fewer columns than a Running one and the two kinds differ only in how many columns precede READY. READY counts ready containers rather than the pod's Ready condition, so a pod held back by an unsatisfied readiness gate is out of scope; a test pins that rather than leaving it to be found in an empty directory. Logs are requested for every container of a selected pod rather than skipped for Pending ones, and the per-pod collection moved into a helper so it can be driven against a stub kubectl. kubectl's refusal is the finding, and an empty result is ambiguous on its own: a read against an unscheduled pod exits 0 having printed nothing, which is what a container that started and stayed quiet also leaves. Every site that classifies a failed read now orders its branches the same way, naming the timeout first where both a cutoff and an uncapturable stderr apply. The CRD gate in the summary had no such branch at all: it returned in silence, which is byte-identical to the CRD being absent, so every section behind that gate vanished from a report whose only real problem was that the machine writing it had nowhere to put a temp file. Collection used to stop silently. A walk that hit its cap said so in the job log and nowhere in the artifact. Now the pod walk leaves COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off part way through, COLLECTION-FAILED.txt when the pod list never came back rather than the empty tree a healthy cluster produces, and COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a fact about the machine writing the report and not about the cluster. Tenant namespaces are collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform pods while the pod the suite died on falls off the end. Container logs are read with --tail=2000 where they were previously read whole, and every bounded file states the bound it was written under, because --tail drops the oldest lines and nothing else in the output would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log. The bounded whole-object read no longer merges stderr into pod.yaml, vm.yaml and vmi.yaml, which are read by yq rather than by eye: a warning on an otherwise complete read left an unparseable file. stderr goes to a scratch file and is copied in only when kubectl returned no object, and the truncation markers those files carry are written as YAML comments, since a bare marker line after a complete mapping is a second top-level node. Six latent faults surfaced while making these reads report. The kamaji module was gated on the LINSTOR controller, so a cluster without LINSTOR got no kamaji evidence at all, which is what a tenant-kubernetes bootstrap timeout needs most. The Flux sources section nested one jsonpath filter inside another, which kubectl rejects outright, so it printed nothing on every report ever taken and read as "all sources Ready". The HelmRelease listing took a single field as STATUS where STATUS is the Ready condition's whole message, so every failed release rendered as one word. The services section filtered for a pending LoadBalancer address on the CLUSTER-IP column, which never carries that value. The cozystack module read its image from a Deployment the installer stopped shipping when it moved to the operator, putting kubectl's NotFound in the first file of the report. And the cozystack apps section gated all three of its kinds on a CRD in an aggregated API group, so it never ran once; two of those names were wrong, and both are collected again. The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats test that installs one and then fails prints no TAP line at all, so output grepped for `not ok` reads as green, while hack/cozytest.sh reports the same failure correctly. A guard enforces that over a named list of files rather than a glob with exceptions, because a list of exceptions is the same claim written so that forgetting to shorten it makes it false. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 17:37:04 +03:00
Aleksei Sviridkin	3fc8e524b2	fix(e2e): select catch pod logs by the nested release instance label ... The harbor, openbao and qdrant app charts deploy their workload through a nested HelmRelease named `<release>-system`, and pass `fullnameOverride: <release>` down to it. The inner chart therefore stamps app.kubernetes.io/instance from its own release name, with the suffix, while object names come from the override, without it. The catch blocks selected on the outer release name, so podLogs matched no pods and every failed run collected an empty result. A pod name says nothing about its instance label; the two come from different sources on purpose. In qdrant the unsuffixed value is therefore not retired but kept as a collector of its own: the parent chart's PVC-cleanup Job labels its pod template with the parent release name, and that suite deletes the parent inside a `try:`, so the Job is in scope while the test still runs. The comment there is explicit about how narrow that window is -- the hook always exits 0, so Helm calls it succeeded and `hook-succeeded` takes the pod away; what is left is an uninstall that hangs with Helm still waiting on it. Harbor renders the same hook and does not get the same collector. It has no `delete` op, so its parent release is uninstalled only in Chainsaw's cleanup phase, which no catch can observe: catch is a defer inside the step, while the cleaner runs from t.Cleanup, after the test function returns. That collector would be the same permanently-empty capture this commit removes. A comment records the asymmetry so the next reader does not restore it for symmetry. openbao renders no such hook at all. Each workload value is now also pinned on a resource in an assert the suite already runs, so the chart renaming a label fails a step instead of quietly emptying a capture. The comments state what that does and does not cover: it does not catch a wrong value typed into a selector, and it catches nothing when the run fails before the assert. Harbor's database was reached by no collector at all. It is selected on `cnpg.io/cluster`, CNPG's own primary label, rather than on the instance label CNPG merely derives from the Cluster name -- the primary label has nothing to drift from, and it is what the postgres suite and the WorkloadMonitors already use. Each lookup keeps stderr apart from the value it captures. kubectl exits 0 while logging a klog line about an API group it could not reach, and an aggregated APIService in that state is what a failed run usually looks like; folded into the value, that line makes the "nothing found" branch unreachable and prints as though it were a result. Telling those outcomes apart is one helper in hack/e2e-chainsaw/_lib/ rather than a copy per call site, separated from the kubectl call so it can be pinned without a cluster, the way etcd-probe.sh already is. Both the census and the qdrant cleanup listing report through it, differing only in wording, so the discrimination is implemented and tested once. The tests cover each outcome for both wordings, including the zero exit with a klog line on stderr -- the case a folded stream turned into a row. A collector that matches nothing cannot say so: podLogs is silent, and `kubectl get -l` exits zero on an empty match. Each test-wide catch prints the instance label of every pod in the namespace, and the qdrant cleanup catch reports its three outcomes apart from a single call -- printing what that call returned rather than running it again, which could put a healthy listing under a failure heading. That census runs on a degraded cluster, where kubectl otherwise retries discovery against an unreachable apiserver until something kills the script, so every lookup carries `--request-timeout`. Measured against an unreachable apiserver, a bounded call costs about five times the flag, since kubectl retries discovery five times before it gives up. Each script then carries an explicit op timeout sized above its own calls -- 2m for the single-call censuses, 5m for the qdrant cleanup listing, which makes three. An inner budget is what should fire: a SIGKILL at the op boundary yields nothing, where a kubectl that gives up leaves an error to print. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 16:46:50 +03:00
Aleksei Sviridkin	d3ae21b7f0	fix(e2e): name every way the previous-instance capture falls short ... The capture wrote its reasons to the job log only. A reader holding the uploaded artifact saw N logs in a directory with no way to tell N from all of them: the container cap that dropped the rest, a pod list that never returned, a previous instance that produced nothing, and a read the outer backstop cut short all looked alike from outside. Every one of those lines now goes to capture-notes.txt as well, and the directory is created before the first of them, so a capture that found nothing is distinguishable from one that never ran. A pod list that failed is no longer reported as nothing having restarted. That is a claim about the cluster which the script never observed, and it is the claim most likely to talk a triager out of the crash-loop hypothesis on exactly the run where it is right. Exit 137 no longer asserts whose signal ended the read. 124 comes from timeout and from nothing else, but 137 is 128+SIGKILL, which the kill grace produces and so does the OOM killer or a teardown signalling the process group. Both belong in the cut-off branch, since the capture ends early either way, but reported flatly as the script's own timeout a read killed at second two reads as one that waited the full eighteen. COZY_PREVLOG_TAIL rejects a value kubectl cannot use and says so. A zero tail asks for no lines while kubectl still exits 0, which the empty-output branch would otherwise report as the container having logged nothing, discarding the only remaining copy of a log the kubelet has already collected. A leading zero is rejected because `$(( ))` reads it as octal while `[` reads it as decimal, so one string means two numbers. Each kept log names the bound it was written under, since --tail drops the oldest lines and nothing else in the output says so, and -1 suppresses that note because no bound was applied. An output directory that cannot be written is reported as the local condition it is. It arrives as an argument, so it can be a file, or missing, or unwritable; the redirection then fails before kubectl runs and hands the shell's own status to the branch that names kubectl, which reported "no previous instance retrieved (kubectl exit 2)" once per container. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 16:03:57 +03:00
Aleksei Sviridkin	ceccac27b0	test(hack): clean up bats temp dirs without EXIT traps ... A `trap ... EXIT` inside an `@test` replaces the one the bats binary installs for its own bookkeeping, so a test that then fails prints no TAP line at all. Not `not ok`, nothing: the run ends with a warning that N of M tests executed, and anyone judging it by grepping for `not ok` reads a green suite. Cleanup moves to the last line of each body instead. Both runners set -e, so a failed test leaves its scratch directory behind, which is what you want to look at anyway. hack/promote-retag_test.bats was failing this way already, reporting five passes and no failures while six of its eleven tests printed nothing. Those six invoke the script for real, and it requires sha256sum; they pin PATH to the stub directory plus /usr/bin and /bin, which is where coreutils installs that binary and where macOS does not, so the failure was of the platform rather than of the code and CI never saw it. The stub directory already supplied yq and skopeo for the same reason and now supplies sha256sum too, resolved from absolute candidates because the stub itself is first on PATH. Two file headers described the trap arrangement they no longer use and are corrected rather than left to outlive it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 16:03:57 +03:00
IvanHunters	0612a538a3	fix(e2e): bound the Talos-capture failure-branch forensic kubectl calls ... The describe/get-events/get-certificaterequests calls that run when the reader Certificate or the diagnostics Pod misses its readiness wait were unbounded, yet those branches fire precisely when the management apiserver is wedged. A stalled forensic call there can burn minutes of the 50m step budget before the pre-existing (a2)/(c) diagnostics and the EXIT-trap tenant snapshot run. Add --request-timeout=30s to them, matching the happy-path capture calls. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-08-05 15:13:56 +03:00
IvanHunters	cfbfd770b8	fix(e2e): bound the tenant Talos capture kubectl calls ... The in-guest diagnostics chain runs only when the sandbox is already degraded enough to miss the 18m node-join deadline (the documented management-etcd fsync-storm regime), yet its kubectl legs were unbounded while every talosctl leg is wrapped in timeout. A stalled apiserver call in the (b) capture could consume the remaining step budget before (a2), (c) and the EXIT-trap tenant crust-gather run, so a diagnostics feature could reduce total diagnostic yield under the exact conditions it targets. Add --request-timeout=30s to the raw Certificate/Pod apply, the three reader-Secret reads and the worker VMI list, and route the Certificate Ready wait through kubectl_wait_retry so a single transient etcd-leader flap retries instead of aborting the whole capture. Update the bats assertion for the reworded wait call. Also correct a stale comment: the chainsaw step timeout is 50m, not 40m. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-08-05 14:47:49 +03:00
IvanHunters	5e3c286e20	fix(e2e): read files with cat in the Talos diagnostics bats helper ... assert_file_contains used $(<file), a bash-only construct. cozytest.sh is run under /bin/sh (dash on the Ubuntu CI runner via Makefile), where $(<file) expands to an empty string instead of the file contents, so every assertion after it failed and reddened the unit-tests job, which in turn skips the e2e gate and leaves the diagnostics path unexercised. Read the file with $(cat ...), which is POSIX and behaves identically under bash and dash. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-08-05 14:19:50 +03:00
Aleksei Sviridkin	7739d1e294	feat(multus)!: stage reference CNI plugins from the multus image ... Multus-backed NetworkAttachmentDefinitions delegate to the upstream containernetworking reference plugins (bridge, macvlan, ipvlan, ...), and nothing guarantees a node has them in the directory Multus searches. On k3s they live under /var/lib/rancher/k3s/data/cni while Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails there with `failed to find plugin "bridge" in path [/opt/cni/bin]`. Bundle 14 of the release's plugins into the multus image and add an install-cni-plugins init container that installs them into the host /opt/cni/bin, mirroring the existing install-multus-binary container. loopback, dhcp, tap and dummy are left out; the Dockerfile records why. Gate it on a new chart value, stageCniPlugins, resolved per bundle and on for both. Talos bundles only bridge, firewall, flannel, host-local, loopback and portmap, so the other ten are missing there exactly as on generic Linux; the four that overlap are rewritten with the same upstream version, v1.9.1 on both sides, rather than an older one. A marker in the image Dockerfile records the Talos release that pin was compared against, and a test holds it to the release the platform ships. multus therefore moves out of system.common-packages into the two variant branches, the way linstor already does for talos.enabled. networking.stageCniPlugins overrides the variant default in either direction. Where staging is on the install is unconditional: a plugin is replaced on every pod recreation, not only when the destination is missing. A destination that is already a directory is reported and left alone, since replacing it is not something this can do safely. Turning the value off afterwards does not undo it; nothing here removes or restores a plugin. The container is not privileged and takes no mount propagation. It runs in the spc_t SELinux domain, without which an enforcing node refuses the write. Each plugin is installed under a temporary name and swapped in with a rename: /opt/cni/bin is shared with cilium and kube-ovn. The plugin tarball is checksummed before extraction. A plugin that cannot be installed is reported on stderr and to /dev/termination-log, and the container still exits 0: Multus stays the node's primary CNI and nothing removes its config, so an init container that never completes would leave the shim with no daemon behind it. The manifest change lives in customize-deployment.patch so it survives make update. BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI plugins in /opt/cni/bin on existing nodes, Talos included -- four of the names it writes are ones the Talos node image also carries. Opt out before upgrading: declining afterwards does not restore what was replaced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 14:19:47 +03:00
Aleksei Sviridkin	517dd11d1a	fix(tests): wait for operator rollout instead of condition=Available ... condition=Available answers whether the Deployment sits within the slack its rollout strategy allows, not whether the operator came up. That is the wrong question for an install gate, and it stays wrong whatever the strategy is: any maxUnavailable equal to replicas drives the minimum available replica count to 0 and the condition then reports True with no running pod. Measured against an operator stuck in ImagePullBackOff, the old check passed in 156 ms, and the install went on to fail two minutes later on the CRD wait, which pointed the investigation at the wrong component. rollout status reads updated and available replicas directly and only succeeds once the updated pod is available, which holds under any rollout strategy. Dump the deployment, pods, operator logs and events on failure so the cause is legible where it is detected. Drop --wait from the helm step. Helm 3 decides a Deployment is ready with the same replicas - maxUnavailable arithmetic, so on the current strategy it is satisfied at zero ready pods and gates nothing; were the strategy tightened it would instead fail the install on its own 2m budget before this gate runs, hiding the diagnostics behind a bare Helm timeout. That arithmetic is specific to Helm 3, since Helm 4 reworked --wait into a wait strategy with a different default, so treat the reason as version-bound. The conclusion is not: one gate that reports what it saw is better than two that can disagree about whose budget expired first, whichever major is installed. Hooks are waited for with or without the flag. The 5m budget here covers scheduling and a cold pull of an image that is not in the prepull set. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 13:13:26 +03:00
Myasnikov Daniil	60e44b24f1	fix(e2e): make tenant Talos diagnostics reliable ... Fix admission, capture setup, timeout headroom, and portability. Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-08-05 14:52:31 +05:00
Myasnikov Daniil	c8ec60cfe3	test(e2e): remove optional search tool dependencies ... Use Bash and awk assertions in the Talos diagnostics regression tests so they run in the standard E2E sandbox without optional search utilities. Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-08-05 14:20:33 +05:00
Myasnikov Daniil	80152b1a07	fix(e2e): capture Talos logs for tenant worker join failures ... Tenant Kubernetes worker join failures currently stop at host-side VMI and CSR state because the worker-only topology does not materialize a tenant talosconfig. Mint a short-lived os:reader client certificate and run talosctl from a hardened same-namespace Pod after the Ready deadline. Preserve bounded dmesg and kubelet captures for every worker without masking the original test failure. Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-08-05 13:18:20 +05:00
Myasnikov Daniil	83fce03839	test(release): scope the freeze pins to their steps and cover the guard ... Three assertions extracted a step block, checked it was non-empty, and then ran their grep over the whole workflow. An unrelated step carrying the same condition would keep them green after the protected step lost its own guard, which is precisely the slip they exist to catch. Each now greps its own block, and `id: freeze` moved into the freeze step's test where it belongs. The force-detection greps looked for `+refs/heads/` and `+refs/tags/`, which is not how git spells a forced refspec: the `+` leads the refspec, so `git push origin "+HEAD:refs/heads/$BRANCH"` force-moves the branch and passed both checks. Match a leading `+` on any push refspec instead. Adds a test for the new build-release dispatch guard: the pattern anchored at both ends, the ref_type comparison and not merely its env wiring, a hard failure rather than a skip, and the guard ordered before the checkout. Every pin mutation-tested — forcing either push with a `+` refspec, parking the freeze condition on another step, dropping `id: freeze`, removing or unanchoring the dispatch guard, demoting it to exit 0, and relocating it after the checkout all turn the suite red. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-08-04 14:23:51 +05:00
Myasnikov Daniil	e8675dd321	docs(release): reword the changelog-backstop comments for the freeze model ... Three comments still described the changelog reaching main by default, with a release-X.Y-targeted promote PR as a patch-release corner case. After the freeze that is inverted: promote-rc.yaml bases the promote PR on release-X.Y whenever the branch exists, and cutting vX.Y.0-rc.1 creates it, so every release merges its changelog onto the maintenance line and none reach main on the way. The tags.yaml backstop is now the normal route, and its self-skip on `exists` is the exception. These comments are the first thing anyone debugging a missing changelog reads, so pointing at the wrong model of which path is normal costs real time. Reworded in place: the generate-changelog job header, the three-paths comment above check_changelog, its inline at_tag note, finalize's explanation of why update-releasenotes.yaml is not trusted with the release body, and the contract test that pins the porting behaviour. Prose only, no behaviour change. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-08-04 14:23:32 +05:00
Myasnikov Daniil	e532844007	test(release): pin the freeze and backport-target contracts ... Neither cut-prerelease.yaml nor backport.yaml had any test coverage, and neither can be exercised by a PR lane: the first runs only on workflow_dispatch and its whole purpose is a one-way write-once side effect, the second only on a merged main-targeted PR. Every invariant here is one that would otherwise first report a slip at a real release. Pinned, following promote-gate-contract.bats so a guard demoted to a comment cannot satisfy a pin: - the freeze is gated to kind == 'rc' && patch == '0'. Widening either half re-opens a frozen line with no error surfacing. - neither the branch create nor the tag push is forced, in any spelling including a `+refs/` refspec. A force-push on the branch would drag in everything merged since the freeze. - an existing release-X.Y is left untouched rather than reused. - the refuse gate is gated on a main dispatch and runs BEFORE the tag push, so a mistaken dispatch cannot burn a write-once tag name; the freeze runs after it, at the commit that was actually tagged. - backport targets come from enumerated branches, with no getLatestRelease and no min-1 arithmetic, and `previous` is the second-newest existing line. - the comparator sorts numerically descending, which is the only reason release-1.10 outranks release-1.9. A slip to the default lexicographic sort aims every backport at a stale line silently. - freezing a line dispatches its first build-release run. backport.yaml's logic is a github-script block whose comments name getLatestRelease and min-1, so the assertions over it strip `//` comments; the YAML `#` filter alone would let a comment satisfy a pin. The Makefile discovers hack/*.bats, so this runs under `make unit-tests` with no wiring. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-08-03 21:38:41 +05:00
myasnikovdaniil	f1f383626e	fix(ci): overlay images from the PR base branch, and publish per-line artifacts (#3471) ... ## What this PR does The PR finalize job overlays image refs for every package a PR did not rebuild, so e2e and the installer do not test last-release images for everything outside the PR's build matrix. It could only ever read `cozystack-packages:main`, which meant a **release-line PR was handed main's binaries to run against its own line's charts**. cozystack/cozystack#3437 is the demonstration: a one-line change on `release-1.6` deactivating an app failed install deterministically, twice, with ``` helmrelease/backupstrategy-controller: Helm install failed … error validating data: SchemaError(github.com/cozystack/cozystack/pkg/apis/core/v1alpha1.Option.spec): unknown model in reference: "github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec" ``` main's `cozystack-controller` served an aggregated OpenAPI that `release-1.6`'s charts could not validate against. Nothing in that PR was broken; the lane was. Both branches carry `option_types.go` and key `OptionSpec` identically in the committed generated OpenAPI, so this is a generation mismatch at runtime, not a codegen drift. This PR fixes it by giving every base branch its own artifact to overlay from, rather than by turning the overlay off. ### The change 1. **`pull-requests.yaml`** reads `cozystack-packages:${BASE_REF}` (`github.base_ref`) instead of a hardcoded `:main`. 2. **`build-release.yaml`** (new) publishes that artifact for maintained `release-<major>.<minor>` branches exactly as `build-main.yaml` does for main: every image tagged with the branch, and the whole packages tree pushed with each reference digest-pinned to what the run just built. An earlier revision of this branch simply skipped the overlay for non-main bases. That fixed the wrong-images problem but left release-line PRs testing their line's *last release*: for any package the PR did not rebuild, the committed ref is the released digest, so a component changed by an earlier backport was still exercised as its pre-backport binary until the next rc. Per-line artifacts remove that gap too, which is why the guard was replaced rather than kept. ### Three deliberate choices **The trigger matches line branches only** (`release-[0-9]+.[0-9]+`). The per-release and rc staging branches `promote-rc.yaml` and `tags.yaml` create — `release-1.6.1`, `release-1.6.0-rc.4` — must not trigger a full rebuild; their images come from the tag build and rebuilding them is waste. **`WRITE_CACHE` stays `0`.** `CACHE_REGISTRY/<img>:buildcache` is a single ref per image and `build-main.yaml` is deliberately its only, serialized writer so concurrent builds cannot race on the cache manifest — the 409 class #2711 fixed for image tags. A line build can overlap a main build, so writing here would reintroduce that race. Line builds read the cache. **A missing artifact still degrades to committed refs, but says so.** On a release line it emits a `::warning::` naming the branch. Silent degradation is indistinguishable from a working overlay, which is how a mis-specified branch filter could hide for a whole release cycle. ### Cost One `make build` per push to a maintained line — in practice per merged backport. That is the price of release-line PRs testing their line's tip instead of its last release. ### Verification `hack/overlay-main-images_test.bats` pins the artifact tag to the base branch, rejects a hardcoded `:main` in either overlay step, and pins `build-release.yaml`'s branch filter, image tag and `WRITE_CACHE: '0'`. Mutation-checked: restoring `:main`, setting `WRITE_CACHE: '1'`, and broadening the filter to `release-*` each fail a test. 13/13 green; `actionlint` and `zizmor` clean. Worth an explicit ack in review: the branch-filter pattern is the one thing no local test can prove, since only GitHub evaluates it. If it does not match, `build-release` never runs and the new `::warning::` is what surfaces it on the next release-line PR. ### Backport `release-1.6` needs this too — for `pull_request` events GitHub builds the workflow from the merge ref, so a release-line PR only stops receiving main's images once the change is on its base branch. cozystack/cozystack#3472 carried the interim guard and is closed in favour of backporting this instead. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a “Build release line” workflow that builds and publishes images and the packages artifact for maintained `release-<major>.<minor>` branches. * **Bug Fixes** * Updated PR workflow finalization to use base-branch–specific package overlays, avoiding incorrect main-branch image references when targeting release branches. * Improved fallback behavior when the base-branch packages artifact is unavailable. * **Tests** * Added workflow wiring tests to verify base-branch artifact usage and that each maintained release line publishes its own packages artifact. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-03 18:22:47 +05:00
Andrei Kvapil	0f0204a6c2	fix(e2e): disable guest fsync on ephemeral CI VM disks (#3455) ... ## What this PR does The e2e QEMU guests are launched in `hack/e2e-prepare-cluster.bats` with three `-drive … if=virtio,format=raw` lines that carry no `cache=` option, so QEMU falls back to its `cache=writeback` default, which honors every guest fsync against the host-backed raw disk files. The management-cluster Talos etcd — which holds every operator's leader-election lease — fsyncs on each Raft commit. During the VM-heavy suites, containerd is simultaneously unpacking dozens of images and CDI is importing multi-GB VM disks onto the same host-backed raw files. The resulting fsync storm serializes etcd's writes into multi-second stalls, so the management apiserver returns `context deadline exceeded` on lease PUTs, and several operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji) lose leadership at the same moment. Whichever chainsaw suite is mid-flight then fails — a different one each run (vmcluster, kubernetes-latest, vmdisk), which is the fingerprint of shared-resource contention rather than a bug in any single suite. This adds `cache=unsafe` to each guest drive. That makes guest flushes no-ops against the host page cache, which removes the fsync-gated etcd stalls at the root of the flakiness. It is safe here precisely because these CI VMs are ephemeral and destroyed at the end of every run — a crash before writeback loses only throwaway state — and it must never be used for real data. The change is minimal and uniform across all three guest drives; no non-guest drive is touched (there are none in this file). Runner sizing alone (#3251) did not resolve this, because more CPU and memory do not change the I/O path: the guest fsyncs still propagate through `cache=writeback` to the host, and etcd still stalls behind the same fsync storm. This addresses the I/O path directly. `git log -S 'cache='` on this file is empty, so a cache mode has never previously been set here. ### Screenshots Not applicable — no UI changes. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against the actual diff. The only entries that reference `hack/e2e-prepare-cluster.bats` (ansible-cozystack and ccp) are scoped to node prerequisites — kernel modules, containerd settings, the LVM `global_filter`, and the cluster domain. This change touches none of those: it adds a QEMU host-side hypervisor flag (`cache=unsafe`) to the CI VM `-drive` options, which is invisible to the guest OS and to any real node. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): boot e2e QEMU guests with cache=unsafe so ephemeral CI VM fsyncs no longer stall the management-cluster etcd, removing the cross-operator leader-election flakiness that failed a different chainsaw suite each run ``` --- This PR was prepared with the assistance of an AI agent (Claude). A human maintainer reviewed the diff and the reasoning before submission and remains responsible for the change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated end-to-end virtual machine startup testing to use optimized QEMU disk caching settings for improved reliability and performance. * Improved end-to-end Kubernetes cluster readiness checks under load by extending the deadline for reaching “at least 2 nodes Ready,” and aligned related messaging and failure diagnostics accordingly. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-29 19:32:45 +02:00
Andrei Kvapil	3dc0f56477	feat(kube-ovn): build the chart and image in-tree, retiring the wrapper repo (#3460) ... ## What this PR does Brings the `kubeovn-chart` wrapper into the tree, so `cozystack/kubeovn-chart` can be archived. Until now `packages/system/kubeovn` pulled a chart tarball from the latest `cozystack/kubeovn-chart` tag at `make update` time, and that repository also built and pushed the `kubeovn` image. The wrapper existed only to cache the upstream chart; with the current caching approach that reason is gone, and the split cost a second repository, a second release step, and a package whose provenance a reader cannot follow without leaving the tree. The package is now shaped like `packages/system/cilium`: - `update` pulls the upstream `kubeovn/kube-ovn` chart directly and applies `patches/mtu.diff`, then syncs the version into the Dockerfile. The MTU patch already carried monorepo-relative paths, so it applies unchanged. - `image` builds `images/kubeovn` (carrying the existing `fix-keepvmip-portgroup` patch) and pins the digest into `values.yaml`, using the shared `image-tags` / `cache-args` helpers. It also carries `cozystack/kubeovn-chart#5`, which never got a tag: `OVN_IPSEC_KEY_DIR` is pinned under `/var/lib`. The kube-ovn default sits in `/etc/origin`, which a Talos node cannot create, so enabling IPsec left the node without ESP security associations. ### The nightly host rewrite had to be fixed first Moving the image build here activates a gap recorded in `docs/agents/image-refs.md`. The nightly host rewrite is a literal `<src-registry>/` substring replace, and kubeovn keeps its host in `global.registry.address` with the bare repository in a sibling key — no trailing slash, no match. While the image came from a separate repository the baked reference carried the public host and the ownership filter skipped it; once CI builds the image here it stamps the build-registry host into that key, and the result is an image mirrored to the public registry while the published tree keeps pointing at the private build registry. The second commit adds an end-of-line-anchored expression that fires only when the source registry is the complete scalar value, so a contiguous reference — which continues past the host — is still handled by the first expression and the two can never both fire on one reference. `keycloak-operator` splits at a different boundary and stays unfixed; that is documented, and dormant while no CI path rebuilds it. Verified: `hack/nightly-mirror_test.bats` passes with the new case and the case fails without the fix; `promote-retag` and `promote-rewrite-tags` suites unaffected; `helm template` and `helm unittest` on the package pass. The pre-existing `helm lint` complaint about a missing `kube-ovn` dependency reproduces on `main` and is left alone. ### Follow-ups this PR does not do - `cozystack/kubeovn-chart` should be archived once this merges, with a pointer here. Its open PR #5 is carried by this change and needs no separate landing. - `cozystack/website` links the Kube-OVN license at `cozystack/kubeovn-chart` in `operations/configuration/licenses.md` and `guides/platform-stack/_index.md` (all versions, including `next`). Archived repositories stay readable so nothing breaks, but the link should move upstream or in-tree. **No follow-up has been opened yet** — flagging rather than ticking the box. ### Screenshots N/A — build and packaging change, no user-facing or UI impact. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walked the trigger map against the diff. The website row matches — not through the app-list or platform-component triggers, but because two of its pages link the repository being archived. Left unticked because no follow-up exists there yet; a human should decide whether the license link moves to `kubeovn/kube-ovn` or to this package. No other row matches: the change touches no `packages/apps/` or `packages/extra/` package, no `values.schema.json`, no `ApplicationDefinition`, no platform variant, and no node prerequisite. ### Release note ```release-note feat(kube-ovn): the Kube-OVN chart and image are now built in this repository instead of cozystack/kubeovn-chart. OVN_IPSEC_KEY_DIR now defaults to /var/lib/ovs_ipsec_keys, so enabling IPsec on Talos nodes no longer leaves them without ESP security associations. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated Kube-OVN image build to use upstream chart metadata and keep registry/version alignment during `update`. - Added conditional MTU support for the OVN CNI DS via chart values. - Introduced `OVN_IPSEC_KEY_DIR` to standardize the OVN IPsec keys directory. - **Bug Fixes** - Preserved VM networking port-group assignments during live migration when an alive virt-launcher sibling pod exists. - Improved nightly mirror host rewriting for split and standalone registry host formats. - **Tests** - Added coverage for the end-of-line “whole scalar value” host rewrite case. - **Documentation** - Refreshed “Known gaps” guidance for nightly mirror host rewriting behavior and limitations. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-29 18:19:41 +02:00
Myasnikov Daniil	2330d6f3ca	fix(ci): overlay images from the PR base branch, and publish per-line artifacts ... Supersedes the interim guard in this branch`s first commit, which skipped the overlay entirely for release-line PRs. Skipping fixed the wrong images but left those PRs testing their line`s last release: for any package the PR did not rebuild, the committed ref is the released digest, so a component changed by an earlier backport was exercised as its pre-backport binary until the next rc. The overlay now reads `cozystack-packages:<base branch>` instead of always `:main`, and build-release.yaml publishes that artifact for every maintained `release-<major>.<minor>` branch the way build-main.yaml does for main: images tagged with the branch, and the whole packages tree pushed with each reference digest-pinned to what the run just built. Each base branch therefore has its own generation to overlay from, which is what the original bug was really about — #3437 failed install because main`s cozystack-controller served an aggregated OpenAPI release-1.6`s charts could not validate against. Three deliberate choices: * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The per-release and rc staging branches promote-rc.yaml and tags.yaml create (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their images come from the tag build, and rebuilding them would be waste. * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per image and build-main.yaml is deliberately its only, serialized writer so concurrent builds cannot race on the cache manifest (the 409 class #2711 fixed for image tags). A line build can overlap a main build, so writing here would reintroduce that race. Line builds read the cache. * A missing artifact still degrades to committed refs, but on a release line it now emits a ::warning:: naming the branch. Silent degradation is indistinguishable from a working overlay, which is how a mis-specified branch filter would hide for a whole release cycle. Cost: one `make build` per push to a maintained line, i.e. per merged backport. hack/overlay-main-images_test.bats pins the artifact tag to the base branch, rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main, setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test. 13/13 green; actionlint and zizmor clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-29 17:29:43 +05:00
myasnikovdaniil	cc44f6c981	ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449) ... ## What this PR does Moves release validation from promote time to rc time, making promotion itself a pure retag operation. This is the follow-up to the v1.6.0 retrospective: the promote flow's checks ran on the promote PR (coupled to promotion, skippable by never promoting), the changelog and website docs were produced only after the release went public, and the finalize tag push silently failed to trigger `tags.yaml` at all (persisted-credential bug, root-caused below). The release lifecycle after this PR: 1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes the staging branch — and now runs a **mandatory full e2e** for every `-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and any published tag) get the same validation as a **manual button** (`workflow_dispatch`). 2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**: the release changelog is generated from the rc tag (reusable `changelog-generate.yaml`) and committed to the rc staging branch, ready for review long before promotion. 3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence: the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages the release, picks up the rc-time changelog (or generates one), and **opens the website docs PR** on cozystack/website from the staging branch — parked with an explicit DO-NOT-MERGE-until-finalize contract. 4. The promote PR **no longer gets `full-e2e` automatically** — its tree is the rc's already-validated digests with only tag strings rewritten; a maintainer can still add the label manually to force the suite. 5. Finalize performs the irreversible side effects (tag, publish, retag, chart) — and with `persist-credentials: false` its tag push now actually authenticates as the CI app, so the `tags.yaml` backstops (`generate-changelog`, `update-website-docs`) fire for promoted tags. This is the one-line fix for the v1.6.0 incident where the tag triggered nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader` silently defeats the app token injected via `git remote set-url`, and `GITHUB_TOKEN` pushes create no workflow runs. Everything is pinned by executable contract tests (`hack/promote-gate-contract.bats`, extended `hack/release-changelog-contract.bats` / `release-changelog-behaviour.bats` — 48 tests total, including behaviour tests that execute the new `hack/parse-rc-tag.sh` and `hack/select-changelog-source.sh` for real), and `docs/release.md` is rewritten to describe the new lifecycle end-to-end. ### Commit map (review commit-by-commit) - `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing changelog-before-publication rework (changelog generated at promote, committed to the staging branch before the PR opens, finalize publishes it as the release body; tags.yaml job becomes a verbatim-port backstop), rebased onto main. - `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane (prerelease → staging branch, stable → tag; draft releases fail fast; dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts). - `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut rc tags (also on `release_exists` re-runs); fixes the `cut-prerelease.yaml` comment that falsely claimed e2e already existed. - `2dfbeea77` — `changelog-generate.yaml` (reusable core) + `changelog-rc.yaml` (rc-time button committing to the staging branch); promote-rc gains a top-level `parse` job with a strict no-whitespace tag policy (`hack/parse-rc-tag.sh`); source selection is validate-then-fallthrough (`hack/select-changelog-source.sh`) so an invalid higher-priority changelog can never silently win. - `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override + removal of the automatic `full-e2e` label (pull-requests.yaml release-path e2e now requires an explicit label; a post-open label now actually starts the run). - `936190d4b` — `website-docs` job at promote time behind a FETCH_REF-support guard (see dependency below), with the DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before `release-next`; `tags.yaml::update-website-docs` stays as the backstop. - `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0 root cause). - `717c8860f` — `docs/release.md` coherence pass over the whole new lifecycle. ### Dependency The promote-time website docs job needs cozystack/website#631 (`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard detects the missing support and degrades to a ⚠️ line in the promote PR body — promotion is unaffected and the tag-time backstop backfills, so this PR is safe to merge first. ### Merge instructions Please **merge-commit, do not squash**: the 13 commits are deliberately atomic (the credential root-cause fix is separable from the feature work) and squashing would collapse ~3k lines into one commit, hurting bisectability. ### First live run The e2e lane is verified by lint, contract tests and dry-traces, but has not yet executed against a real tag; the intended first live run is a manual `e2e-tag.yaml` dispatch against an existing published rc tag before the gate is relied on for a real promotion. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added reusable changelog generation with validation, “present” signaling, and safe artifact upload behavior (including RC-time support). * Added an E2E release-tag workflow with tag parsing, deterministic sandboxing, and comprehensive report collection. * Improved RC promotion with a stricter parse front-door, verified E2E green gate (optional bypass), and earlier parallel changelog/asset staging. * **Documentation** * Refreshed release-process docs to match the updated RC-to-stable, changelog, and website-docs timing rules. * **Bug Fixes** * Improved PR finalization to source release bodies from verified changelogs and warn/fallback on missing or empty content. * Tightened label gating so E2E runs only under the intended conditions. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-29 17:14:29 +05:00
Andrei Kvapil	6cb579338b	fix(nightly): rewrite an image host that is the whole scalar value ... The nightly host rewrite is a literal "<src-registry>/" substring replace, so it only reaches a host sitting contiguously in front of the repository. kubeovn keeps its host in global.registry.address with the bare repository in a sibling key: no trailing slash, no match. The image was mirrored to the public registry and the tree kept pointing at the private build registry. The gap was dormant while kubeovn's image came from a separate repository, because the baked ref carried the public host and the ownership filter skipped it. Building the image in-tree stamps the build-registry host into that key and turns it live, so fix it here. Add an end-of-line-anchored expression that fires only when the source registry is the complete value; a contiguous ref continues past the host, so the two expressions cannot both fire on one reference. keycloak-operator splits at a different boundary and stays unfixed -- neither of its keys holds the source registry as a whole. Documented, and dormant while no CI path rebuilds it. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>	2026-07-28 18:07:34 +02:00
Andrei Kvapil	8b461f0c0b	feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins (#3378) ... ## What this PR does Cozystack builds and publishes `ghcr.io/cozystack/cozystack/grafana` — a rebuilt Grafana image — and uses it by default in the monitoring stack. Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a redistributor of an AGPL Grafana build. This is a governance concern for CNCF Incubation (there is no precedent for a CNCF project redistributing an AGPL binary). The only reason the image was ever rebuilt was to bundle the VictoriaLogs datasource plugin, which was not in the official Grafana plugin catalog at the time. It is now published there as a signed plugin (`victoriametrics-logs-datasource`), so the rebuild is no longer necessary. This PR moves the deployment to the upstream `grafana/grafana` image (digest-pinned, routed through `cozy-lib.image` so mirrored-registry installs keep working) and installs the plugins at container startup via `GF_INSTALL_PLUGINS`, with versions pinned: - `victoriametrics-logs-datasource 0.29.0` — the VictoriaLogs datasource (signed; the reason the image was rebuilt) - `natel-discrete-panel 0.1.1`, `grafana-worldmap-panel 1.0.6`, `marcusolsson-dynamictext-panel 6.2.0` — reproduce the panel set the removed Dockerfile vendored **Air-gapped installs keep working.** The plugin zips are not downloaded from the internet at pod startup: they are mirrored from the official catalog into the existing `grafana-dashboards` image at build time and served by that in-cluster static file server, next to the dashboards it already serves. `GF_INSTALL_PLUGINS` points at `http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…`, so the runtime dependency stays inside the cluster and the artifact is mirrored into private registries together with every other first-party image. A new `allow-to-grafana-dashboards` CiliumNetworkPolicy (mirroring `allow-to-keycloak`) opens tenant egress to that server, since tenant egress is default-deny. As a result, `ghcr.io/cozystack/cozystack/grafana` is no longer built or published: the Dockerfile, the pinned image tag, the `image` targets in both monitoring Makefiles, and the build step in the root Makefile are removed. Notes: - The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored plugin zips are Apache-2.0 licensed and are served unmodified from the official catalog builds (signatures intact). - `allow_loading_unsigned_plugins` and the custom `/var/lib/grafana-plugins` path are dropped: the catalog datasource is signed, and plugins install into the writable default `GF_PATHS_PLUGINS` (`/var/lib/grafana/plugins`). - The VictoriaLogs datasource provisioning (the `GrafanaDatasource` CRs pointing at `vlselect`) is unchanged. - Plugins are re-installed on every pod start from the in-cluster server (no PVC involved, ~77 MB per start per replica over the cluster network); a failed download fails the pod loudly instead of silently dropping the datasource. - `natel-discrete-panel` and `grafana-worldmap-panel` are Angular plugins; Angular is disabled by default in Grafana 11, so they do not load — this is unchanged from the current rebuilt image (same Grafana 11.6.15, same config). They are kept for parity and can be dropped as a follow-up. ### How this was verified - `helm unittest` across the repo passes, including new regression tests: the monitoring chart asserts the exact upstream image reference (with and without `images-registry` set) and the exact `GF_INSTALL_PLUGINS` value; the tenant chart asserts the new egress policy. - Local end-to-end of the full chain in Docker: built the modified `grafana-dashboards` image, served it on a shared network, and started upstream `grafana/grafana:11.6.15` with the rendered `GF_INSTALL_PLUGINS` value. All four plugins download from the in-cluster-style URL (no internet egress from Grafana), and `GET /api/plugins` reports `victoriametrics-logs-datasource` v0.29.0 as `enabled: true` with `signature: valid`. ### Screenshots Not applicable — no UI change. Grafana renders the same datasources and panels; only the image source and plugin delivery mechanism change. ### Downstream repositories Walked the trigger map against the diff. This is an internal change to how the monitoring stack sources the Grafana image and its plugins (`packages/system/monitoring`, `packages/system/grafana-operator`, `packages/apps/tenant` network policy). No package is added, renamed or removed; no `values.schema.json`, `ApplicationDefinition`, platform values, release asset, shared `hack/` tooling or `package.mk` contract changes; the added `grafana.image` key is an internal system-chart value, not a user-facing app field. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Grafana in monitoring is now based on a digest-pinned upstream image for more consistent deployments. * Required Grafana plugins are installed automatically at startup via `GF_INSTALL_PLUGINS`; unsigned-plugin configuration was removed. * Tenant networking now includes a new egress rule specifically for Grafana dashboards. * Grafana dashboards image now includes required plugin assets prepared during image build. * **Tests** * Added/updated checks covering digest pinning, mirrored-registry behavior, plugin-installation behavior, and the new Grafana dashboards network policy. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-28 18:06:57 +02:00
Andrei Kvapil	330acfe6e2	fix(e2e): widen tenant node-join deadline to 18m ... The kubernetes-latest and kubernetes-previous suites flaked on the worker node-join wait under host load. In run 30260770694 both failed at exactly 12m with zero tenant Nodes registered and the tenant cilium HR still mid-install: a transient drbd.linbit.com/lost-quorum taint delayed the worker DataVolume imports, so the KubeVirt VMIs were only created ~2m into this wait and the guests then had ~10m to import Talos, boot, register a kubelet and let cilium turn the nodes Ready. This single deadline absorbs the entire worker bring-up (the machinedeployment replicas=2 gate clears before the VMIs even exist), so under storage pressure 12m is too tight. A less-loaded fleet run passed both suites unchanged, confirming load-induced slowness rather than a stuck bring-up. Widen the deadline to 18m, which restores margin and still sits well inside the 40m chainsaw step timeout. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>	2026-07-28 17:48:40 +02:00
Andrei Kvapil	ad35afa534	fix(e2e): disable guest fsync on ephemeral CI VM disks ... The e2e QEMU guests boot with three raw virtio -drive lines that carry no cache= option, so QEMU defaults to cache=writeback and honors every guest fsync. The management-cluster Talos etcd fsyncs on each Raft commit, while containerd unpacks dozens of images and CDI imports multi-GB VM disks onto the same host-backed raw files during the VM-heavy suites. The fsync storm serializes etcd's writes into multi-second stalls, the mgmt apiserver times out lease PUTs, and many operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji) lose leadership at once — failing whichever chainsaw suite is mid-flight, a different one each run. Add cache=unsafe to each guest drive so guest flushes become no-ops against the host page cache, removing the fsync-gated etcd stalls. This is safe only because these CI VMs are ephemeral and destroyed every run; never use it for real data. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>	2026-07-28 17:48:40 +02:00
Andrei Kvapil	b22b7b2431	test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race (#3430) ... ## What this PR does The split-horizon-routing-proof step ran two external-dns probes as bare Pods and polled them by fixed name. When the Cilium host-scope IPAM plus Gateway API restore race (cilium/cilium#46799) hands a probe the per-node reserved:ingress IP, the sandbox is rejected with "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting a controller to reschedule it onto a free IP. A bare Pod has no controller, so that delete was terminal: the probe vanished and the step failed on a NotFound pod. It surfaces as an intermittent kuberture E2E failure with a `FailedCreatePodSandBox` event on a probe pod. This runs the probes as Jobs. The Job re-creates its pod on a fresh IP after the healer deletes it, so the proof survives the race. Each Job is asserted to reach Complete instead of polling a fixed pod name, and its logs are read by the job-name label. The failure path also gains a catch podLogs selector for the probe pods, so their stdout is captured when a probe fails for a different reason. ### Screenshots None. No UI changes. ### Downstream repositories The diff touches only the kuberture E2E suite under `hack/e2e-chainsaw/kuberture/`. No package, values schema, chart, CRD or platform manifest changed. Nothing in the downstream trigger map is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved end-to-end test reliability by hardening the split-horizon routing proof against probe workload churn. * Strengthened route isolation validation between public and internal probes. * Enhanced failure diagnostics by capturing logs from the completed probe Jobs. * **Testing** * Updated DNS EDNS probes to run as Kubernetes Jobs (instead of bare Pods) to support retries and rescheduling. * Added explicit assertions that both public and internal probe Jobs reach **Complete** before running hostname/route isolation checks. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-28 17:43:26 +02:00
myasnikovdaniil	2c453778f1	fix(seaweedfs): make naming audit fail closed on kubectl and payload errors (#3436) ... ## What this PR does Fixes #3431. `hack/seaweedfs-naming-audit.sh` was fail-open: any kubectl failure produced an empty table indistinguishable from an honestly clean fleet — a failed namespace LIST meant zero namespaces walked, failed secret/PVC/STS LISTs meant zero findings, all silenced by `2>/dev/null` with no error handling anywhere in the script. The runbook uses this output as the gate before PVC deletion, as post-deletion verification, and as an upgrade precondition, so a transient API error could green-light destroying data. Every kubectl call now routes through a `run_kubectl` helper: a non-zero exit prints a FATAL line naming the failed query and the code propagates up the whole chain (enumerations restructured to capture-then-check, since an `exit` inside `$(...)` dies with the subshell). By-name GETs separate existence from extraction: a `--ignore-not-found -o name` probe keeps legitimate absence (Helm-pruned revision, unbound PVC) a clean skip, while everything else fails loudly — including a release Secret that exists but has no decodable payload or decodes without a chart name, which previously dropped the whole tenant from the audit. Incomplete PV-age evidence now marks the generation incomplete and falls to the safe "direction cannot be established" branch, so a failed GET can no longer flip an OVERLAP verdict into a wrong Step-3 deletion candidate. A successful audit prints the same bytes as before. Tests: 14 new cozytest cases — failure injection for every LIST and by-name GET site, corrupt-payload variants (missing `.data.release`, undecodable base64/gzip, chartless `{}`), and full-output golden diffs proving the success paths stay byte-identical to the pre-change script. 25/25 green via `hack/cozytest.sh`. ### Release note ```release-note fix(seaweedfs): the naming-audit script fails loudly when a kubectl query or a Helm release payload read fails, instead of printing an empty table indistinguishable from a clean fleet ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Audits now fail clearly (non-zero exit + fatal messaging) when Kubernetes queries or Helm release data is unavailable or corrupt, instead of producing incomplete results. * Missing release revisions are skipped safely; missing/unreadable PV age evidence no longer leads to incorrect deletion candidates, and MIXED now reports conservative “direction cannot be established” when evidence is incomplete. * Non-SeaweedFS releases remain silently ignored. * **Tests** * Added end-to-end acceptance coverage for fail-closed behavior and byte-for-byte stable output across key naming scenarios. * **Documentation** * Updated the runbook to require relying on the audit script exit code before proceeding, with a dedicated MIXED re-check step. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-28 16:56:05 +05:00
myasnikovdaniil	49222768ec	fix(release): make promote-retag digest verification media-type-agnostic (#3435) ... ## What this PR does The finalize post-copy verification runs `skopeo inspect --format '{{.Digest}}'` on every freshly retagged stable ref. For OCI artifacts that command prints nothing and exits non-zero — `cozystack-packages` is a flux artifact with `config.mediaType: application/vnd.cncf.flux.config.v1+json` — so the v1.6.0 finalize (run 29932878820) aborted right after both copies of that artifact had already succeeded, leaving 32 of 43 repos without stable tags and skipping the cozy-installer publish. `--dry-run` never reaches this branch, which is why rehearsals did not catch it. This PR computes the digest as sha256 over the raw manifest instead (`skopeo inspect --raw`), which is how registries define the manifest digest and works identically for container images and OCI artifacts. The helper gates the hash on skopeo's exit status via a temp file rather than a pipeline, so a missing tag still resolves to an empty string (the pre-check reads that as "tag absent, proceed to copy") and never to the empty-input hash `sha256:e3b0c442…`, and never aborts under `set -eu`. Verified against live GHCR: the raw-manifest hash of `cozystack-packages:v1.6.0` matches the digest finalize expected (`bf68208730860fa8…`), and the old and new methods agree byte-for-byte on regular multi-arch images (`cozystack-operator`, `kubeovn`, `cozystack-controller`, `kubevirt-cloud-provider`). New cozytest cases cover the OCI-artifact digest resolution, a post-copy mismatch, and the missing-tag path; the mock skopeo asserts `--raw`, so the suite fails against the old code. ### Release note ```release-note fix(release): promote's post-copy digest verification no longer breaks on OCI artifacts (cozystack-packages), which aborted the v1.6.0 finalize mid-retag with 32 of 43 repos left untagged ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved OCI artifact retagging by deriving stable manifest digests from raw manifests to ensure accurate “already correct” and mismatch detection. * Prevented incorrect skips when the stable tag points to an unexpected digest. * Added safer handling for missing or unknown stable tags and clearer failure behavior for unexpected empty manifest responses and transient registry errors. * **Tests** * Expanded promotion/retag test coverage for digest-based resolution, post-copy verification mismatches, missing-tag behavior, and transient registry failures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-28 16:55:42 +05:00
Aleksei Sviridkin	436a19df1b	test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race ... The split-horizon-routing-proof step ran two external-dns probes as bare Pods and polled them by fixed name. When the Cilium host-scope IPAM + Gateway API restore race (cilium/cilium#46799) hands a probe the per-node reserved:ingress IP, the sandbox is rejected with "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting a controller to reschedule it. A bare Pod has no controller, so the delete was terminal: the probe vanished and the step failed polling a NotFound pod. Run the probes as Jobs so the pod is re-created on a fresh IP after the healer's delete, assert each Job reaches Complete instead of polling a fixed Pod name, and read the logs by job-name label. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-27 22:59:49 +03:00
Aleksei Sviridkin	a7ce4371c2	docs(agents): correct the chainsaw podLogs capability note ... The e2e conventions, the previous-logs capture script, and the global chainsaw catch all claimed the v0.2.15 podLogs collector exposes only container/tail/selector. It also accepts name, namespace, timeout and the cluster fields; only the previous option is genuinely missing. The wrong list steers catch authors toward imperative scripts for cross-namespace log capture that podLogs handles declaratively. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-27 22:48:25 +03:00
Aleksei Sviridkin	824a799c6a	test(e2e): capture redis convergence diagnostics in harbor suite ... When harbor's redis sentinels never converge, the suite times out on the HelmRelease assert with no evidence of why: the catch pod-log selector matches neither the redis-operator (it runs in another namespace) nor the rfr-/rfs- pods, and the RedisFailover CRD ships no status subresource to describe-assert on. Add a RedisFailover describe plus a best-effort script that dumps the operator log and the rfr-/rfs- pod states and logs, so the stalled reconcile gate is visible in the failure output. Also widen the harbor-test HR wait from 6m to 10m: readiness is gated on the operator electing a redis master, which normally takes 2-3m but runs long on a CPU-committed runner. This matches the harbor-test-system budget in the same suite and the justified longer waits allowed by the e2e conventions. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-27 22:48:24 +03:00
Myasnikov Daniil	fea40d6314	fix(ci): only overlay current-main images on main-based PRs ... The PR finalize job pulls cozystack-packages:main and repoints every package the PR did not rebuild at current-main images. On a main-based PR that is the whole point: it stops e2e from testing last-release images for everything outside the build matrix. On a release-line PR it is wrong — the committed refs there are not stale, they are that line`s released digests, and the charts are written against exactly those. So a release-line PR was installing main`s binaries onto its own charts, and the mismatch grows with every commit main gains. #3437 is the demonstration: a one-line change on release-1.6 that deactivates an app failed install deterministically, twice, with SchemaError(...core/v1alpha1.Option.spec): unknown model in reference: "...core~1v1alpha1.OptionSpec" from main`s cozystack-controller serving an aggregated OpenAPI that branch`s charts cannot validate against. Nothing in the PR was broken; the lane was. Left alone this makes every 1.6 backport look red, which is when release-branch PRs are busiest. Both overlay steps are now gated on `github.base_ref == main`, so a release-line PR keeps its committed digests — the behaviour that predates the overlay. Retargeting the overlay at a per-line artifact would be better but is not possible today: build-main.yaml publishes only cozystack-packages:main, and the registry carries no release-* equivalent (verified against the packages repo`s tag list: `main` plus per-PR tags, nothing else). hack/overlay-main-images_test.bats pins the wiring per step, so adding a third overlay step without the guard fails the suite. Mutation-checked by removing one guard. 12/12 green; actionlint clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-27 21:41:57 +05:00
Andrei Kvapil	a5a0449c9b	test(hack): re-point the shape-2 image-ref exemplar at grafana-dashboards ... The promote-retag and image-refs enumeration tests used the monitoring grafana.tag as their canonical example of a reference living outside a values.yaml. That file is gone now that the Grafana rebuild is dropped, so use the grafana-dashboards tag — the same storage shape, still in the tree — as the exemplar. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>	2026-07-27 13:08:50 +02:00
Myasnikov Daniil	1eded27900	test(release): scope the trim pin to the parse step, fix two stale references ... Three review nits, all verified against the tree, plus a tighter scope than the first attempt at them. The behaviour suite asserted "promote-rc does not trim the rc tag" against the whole file. promote-rc.yaml has no .trim() today, so it passed, but any unrelated .trim() added anywhere in the workflow would have failed a test about tag policy. Scoping it to the parse JOB was not enough either: that job is ~180 lines, most of them the e2e gate's github-script, whose string handling has nothing to do with tag policy. Both greps are now scoped to the parse STEP, and run through a code_lines filter copied from the contract suite so a YAML comment mentioning .trim() cannot fail them. Mutation-checked in both directions: a .trim() injected into the parse step fails the test, one injected into the e2e-gate step of the same job does not. - The Step 2 checklist entry still said "must be main" while Step 2 itself now lists three valid starting points (main, detached at the release tag, detached at the rc tag being promoted). The checklist is what an agent ticks off, so it was the half that mattered; it now points at Step 2 rather than re-encoding the count, so a fourth configuration cannot desync it. The heading said "Checking current branch", which two of the three states do not have. - hack/testdata/needs-semantics.yaml named release-changelog-contract.bats as its consumer; the only consumer is release-changelog-behaviour.bats. 50/50 tests green across the three suites (11 + 24 + 15). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-27 14:03:49 +05:00
Myasnikov Daniil	a2667c58ca	fix(ci): grant rc-e2e the checks:write ceiling e2e-tag.yaml declares ... A caller job's permissions are the ceiling for every job in the workflow it calls, and GitHub validates that ceiling when it CREATES the run — before the calling job's `if:` is evaluated, let alone before a step runs. e2e-tag.yaml's e2e job declares `checks: write` for its workflow_dispatch-only breakpoint, so granting only `contents: read` here made the run fail validation with "requesting 'checks: write', but is only allowed 'checks: none'". A startup failure takes the whole run with it, and an `if:` cannot gate it, so this was not limited to rc cuts: EVERY tag push — rc and stable, including the one finalize pushes to publish a release — would have produced no build, no draft release and no staging branch. The comment claimed the opposite: that because workflow_call can never reach the breakpoint branch, the grant was unnecessary. Runtime reachability has no bearing on a static check. e2e-tag.yaml now states the contract for its callers, next to the permission block that was being read as "contents: read is all this needs". hack/promote-gate-contract.bats pins both halves of the pair, and separately pins the `E2E <tag> (full suite)` job name the gate correlates evidence by. Neither is reachable from any PR lane — tags.yaml runs only on tag pushes — so a regression here would otherwise surface at the next release. Mutation-checked: dropping the grant fails the first pin, renaming the e2e job fails the second. changelog-rc.yaml's reusable-workflow call now pins `contents: read` explicitly, matching promote-rc.yaml's identical call. The outcome is the same either way, but the docs describe an omitted block as the repository default rather than the workflow's own top-level block, and this is the file class where that ambiguity just cost a release lane. Reported by CodeRabbit on #3449; blast radius and the missing pins found by an adversarial re-check. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-27 14:03:49 +05:00
Myasnikov Daniil	ae39475789	fix(release): prove a stable tag is absent before copying over it ... Review notes from #3435, and a correction to my first attempt at them. manifest_digest gated on skopeo`s exit status alone, so a zero exit with no bytes hashed the empty string to sha256:e3b0c442... — a digest that looks real and belongs to no manifest. The first fix simply required non-empty bytes, which was worse than it looked: the caller turns an empty result into "not published, safe to copy", so a registry answering 200 with an empty body went from an accidental refusal (the bogus digest mismatched, so the write-once check refused) to an actual copy over whatever the tag really held. Verified against real skopeo 1.23: a 200 with Content-Length 0 does exit 0 with no stdout. The helper is now explicitly three-state. A digest, or empty output for a tag the registry PROVED absent, or non-zero for anything indeterminate — which under set -e aborts the promotion before it writes. Absence has to be reported as "manifest unknown"; 429, 5xx, auth and network failures all exit non-zero with no bytes too, and reading those as "unpublished" is the same fail-open one level down. finalize retags ~42 refs with no retry wrapper, so a single rate-limit was enough to reach that path. skopeo`s stderr is captured instead of discarded, so the diagnostics now name the real failure. Tests: the write-once refusal branch the review found untested (an existing stable tag at a different digest must be refused before any write), the empty- body case (refuses, writes nothing, and the empty-input hash never appears in the diagnostics), and a 429 that must not read as an absent tag. The mock now emits realistic registry errors, since a bare non-zero exit is no longer proof of absence. Both new guards are mutation-checked: accepting any non-zero exit as absence fails the 429 test, dropping the empty-bytes check fails the empty-body test. 11/11 tests green, hack/image-pin-consistency.bats still 2/2, sh -n clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-27 13:56:45 +05:00
Myasnikov Daniil	e02adad38f	fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls ... Review findings from #3436, plus what an adversarial re-check of the first attempt at them turned up. The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals "config" (the release`s values) after "chart", so a values subtree that spells chart.metadata.name shadowed the real chart name: the release then read as non-SeaweedFS, the tenant silently vanished from the report, and the script exited 0 -- the precise false clean this PR exists to prevent, reachable without any corruption at all. It now takes the FIRST match. The path stays adjacent on purpose: a looser "any name after metadata" matches chart.templates[].name, which Helm emits immediately after metadata on every healthy release, so the review`s suggested relaxation would have returned a template path for every tenant. Over-strictness fails loudly and recoverably; over-looseness deletes data quietly. Newlines are now folded before matching, so a pretty-printed payload parses at all rather than aborting, and whitespace around the punctuation is tolerated. first_deployed got the same treatment -- it had the identical greedy, single- line-only shape, and a spaced payload silently dropped the PV-vintage row from the report. The FATAL message names the path it read and the shape it expected, so a future Helm format change is diagnosable instead of looking like real corruption. Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy) each byte-compared against the same golden as the compact payload, so the shape must make no difference to the report. Mutation-checked -- restoring last-match extraction fails the decoy test on its own, restoring the single-line matcher fails the pretty test. The test fake answered any unmodelled kubectl invocation with exit 0 and empty stdout, the fail-open shape this script was rewritten to reject, and enough to let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97 naming the invocation. Verified inert first by instrumenting the fake: 88 invocations across the suite, none unmodelled. Also added the missing test for the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it. The runbook now tells the operator to read the exit code: a non-zero exit means the table is incomplete and no step may be taken on it. That contract lived only in the script header, while the runbook is what the operator follows. 29/29 tests green; sh -n clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-27 13:52:59 +05:00
Myasnikov Daniil	ee838261b1	test(release): pin the promote gate with POSIX grep, not ripgrep ... The rc-E2E gate contract used ripgrep for every pin. The unit-test runner has no ripgrep, so `code_lines()` hit "rg: not found", the `|| true` on it swallowed the 127, and every pin degraded to "0 matches" — the suite failed on the first assertion with no hint that the tool, not the workflow, was missing. No other hack/*.bats uses ripgrep. Swap all pins to POSIX grep and make the filter fail closed: grep exits 1 when nothing is selected (legitimate for an empty block) and 2 on an actual error, so only the latter propagates instead of silently yielding no lines. Verified 13/13 green both normally and with `rg`/`fd` shimmed out of PATH to reproduce the runner, plus the eleven suites the aborted CI run never reached (all green, zero shim hits). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-27 12:02:53 +05:00
Myasnikov Daniil	01e1e7188e	ci(release): stop finalize persisting credentials so the tag push triggers tags.yaml ... pull-requests-release.yaml's finalize job checked out the repo with the default persist-credentials, leaving GITHUB_TOKEN embedded as http.extraheader. Every later tag/branch push re-injects the CI app token via `git remote set-url`, but a persisted extraheader silently wins over the URL credential — so those pushes authenticated as GITHUB_TOKEN. A GITHUB_TOKEN-authenticated push creates no workflow run (GitHub anti-recursion), which is exactly why v1.6.0's stable tag never triggered tags.yaml: the generate-changelog and update-website-docs backstops never ran. Set persist-credentials: false on the finalize checkout, so the set-url app token actually authenticates the pushes — the same handling promote-rc.yaml already uses. Audited every git/gh step in the job: both tag steps set-url the app token before pushing, and Ensure-maintenance-branch / Publish-draft-release / the retag+chart steps use the app token via github-script or a separate registry login — none depends on the persisted GITHUB_TOKEN extraheader. Pin it in hack/promote-gate-contract.bats. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-24 18:16:26 +05:00
Myasnikov Daniil	936190d4b6	ci(release): open the website docs PR at promote time ... The cozystack/website 'update managed apps reference' PR used to be opened only at tag time by tags.yaml::update-website-docs — after finalize pushed the stable tag, i.e. after the release was already published (too late to review) — and it fetched README content from that stable tag (BRANCH=RELEASE_TAG), which finalize only creates post-merge. Add a promote-time website-docs job to promote-rc.yaml (needs [parse, promote]) that opens the same update-docs-vX.Y.Z PR from the release-X.Y.Z staging branch via the website Makefile's FETCH_REF variable, so the docs are reviewable alongside the promote PR while the stable tag is still unborn. It refreshes the trunk pins from the staging ref before release-next so a new minor inherits fresh Talos/cozystack pins (the v1.6 talos-pin incident class), then runs update-all with FETCH_REF pinned to the staging branch. The job is non-blocking: a FETCH_REF-support guard fails loudly (warn + exit 1) when cozystack/website main predates the FETCH_REF Makefile variable — otherwise make would silently ignore the unknown FETCH_REF override, fall back to the not-yet-created stable tag, and update_apps.sh would warn-and-continue into STUB docs. Any failure surfaces as a website-docs warning line in the promote PR body (plumbed through open-pr via needs.website-docs.result) and never blocks promotion; the tag-time job stays as the backstop and no-ops when promote already did the work. MERGE-TIMING CONTRACT: the website PR must NOT be merged before finalize publishes the release — doing so flips the site's latest-version pointer to a version whose docs and openapi are not on GitHub yet, 404ing every version-pin. The website PR body and the promote PR body both say so in bold. Pin the contract in hack/promote-gate-contract.bats (needs shape, FETCH_REF invocation, the guard, non-persisted checkout credentials, the DO-NOT-MERGE body wording, the promote-PR status line, and the tags.yaml backstop comment) and relax the release-changelog-contract needs pin to tolerate the appended website-docs dependency. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-24 18:16:26 +05:00
Myasnikov Daniil	ff7ddd3c9a	ci(release): gate promotion on a green rc e2e and drop the automatic full-e2e label ... Promotion now assumes checks passed at rc time instead of re-running them. At dispatch, before any staging happens, promote-rc verifies the rc tag has a green full e2e via either evidence path: the rc's tags.yaml push run containing a successful 'rc-e2e / E2E <tag> (full suite)' workflow_call job, or a successful manual e2e-tag.yaml dispatch correlated by its exact tag-bearing job name (immutable rc digests make past evidence valid). No evidence fails the dispatch with both paths and the remediation named. A skip_e2e_gate boolean input bypasses the gate loudly for emergencies; the promote PR body always carries the verification state (verified or bypassed), and 'verified' can only originate in the gate's green branch. The promote PR keeps the release label but no longer gets full-e2e automatically: its tree is the rc's already-validated digests with only tag strings rewritten, so pull-requests.yaml now runs e2e on the release path only when a maintainer opts in by adding full-e2e. The regular-PR path and the ordinary full-e2e forcing are untouched, and the new labeled trigger makes a post-open label actually start the run. A new promote-gate contract suite pins the gate's DAG position, the override default, the label set, the body lines, and the label-gate condition with comment-proof executable-line counts. Assisted-By: GPT-5 <noreply@openai.com> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-24 18:16:26 +05:00