64 Commits

Include renames

Author	SHA1	Message	Date
Aleksei Sviridkin	c5f352829c	ci(tests): gate the shell scripts this repository owns on shellcheck ... shellcheck ran nowhere: not from the Makefile, not from a workflow, not from pre-commit. Seven files under hack/ nevertheless carried `# shellcheck disable=` directives, twelve of the twenty-eight with a rationale beside them, so the tree was written as though the check existed. hack/shellcheck.sh lints every shell script this repository owns and compares the result with hack/shellcheck-baseline.txt. Enumeration is a union of shebang and `.sh` suffix, and neither half is redundant. The suffix alone misses fifty-seven files with no extension, fifty-three of them the platform migrations run-migrations.sh runs on a cluster upgrade and which no linter had seen before this; the shebang alone misses thirteen held by the suffix, the sourced libraries that open with a `# shellcheck shell=` directive and four with no marker at all. Narrowing to either one loses files, and what the run then reports never names the enumeration as the cause. Measured, and it differs by half: a script that leaves scope carrying baseline entries goes STALE, one carrying none -- sixty-six of the hundred and ninety -- leaves in silence, and dropping the suffix branch also removes the sourced libraries from the command line, so their sourcers gain SC1091 as NEW. The reader is shown a fixed finding to regenerate away, or a new finding to go fix, and in neither case a narrower gate. The baseline records the findings that predate the gate as `path check count` triples. Among the scripts in scope no file is skipped and nothing is excluded on the command line, though in-file `# shellcheck disable=` directives are still honoured, so a finding suppressed that way is absent from the list rather than recorded in it; two classes sit outside the scan and say so in the baseline header, the vendored chart scripts that make update regenerates and the .bats files, whose bats shebang the enumeration does not match. shellcheck parses those perfectly well, with a dialect and checks of its own for them; they are not baselined yet, and the .bats arm of the filter is a guard for one that acquires a shell shebang rather than an exclusion that does anything today. Dropping line and column numbers keeps an unrelated edit from reddening the ratchet, while the count still catches a new finding in a script already listed. An entry whose findings are gone fails the run too, so fixing a script forces the list to shrink rather than leaving a standing licence to reintroduce what was removed. The regenerate advice is withheld whenever the same run also reports a NEW finding. Printed together, the last line read wins, and regenerating a mixed run records the new findings rather than reporting them -- the cure offered for one half launders the other. Three ways the gate could have passed without checking anything are closed. A missing binary fails rather than skips. The version is pinned on both sides and the script refuses to run when they disagree, because findings differ between shellcheck releases and an unpinned binary reports against a baseline not written for it. And shellcheck's own exit status is kept instead of being piped through xargs, which collapses it: a status above 1 means an input was never analysed, and that now fails rather than reading as a clean file. A path holding a space or a tab is refused instead of linted. A baseline entry is three space-separated fields, so such a path cannot be recorded: the summary splits it and the entry names a file that does not exist. Refusing is also what licenses the word splitting the enumeration relies on, which until now was a comment asserting the tree had no such name rather than a check that it does not. hack/shellcheck.bats drives the gate through a mutation that introduces a violation and its removal, both directions of the ratchet, each of the four refusals above, and the enumeration each half of the union catches, the shrink half in the case where a count drops without reaching zero as well as the one where an entry vanishes, and the whitespace refusal on a tab as well as on a space. Every red path asserts the exit status before it asserts the message: `! cmd` alone proves nothing, because `set -e` never fires on an inverted command, and a gate that printed each ERROR line while exiting 0 satisfied all thirteen output greps when that was measured. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-07 16:28:42 +03:00
Aleksei Sviridkin	2527e915bd	fix(e2e): collect diagnostics for unready pods and record every truncated capture (#3567) ... ## What this PR does The e2e diagnostic tarball exists so a red run can be explained from the uploaded artifact alone. Several of its collectors answered "nothing is wrong" instead — some because they selected on a column that never carries the value they filtered for, others because they stopped early and left nothing saying they had. Both failure modes look identical to a healthy cluster from the outside, which is the worst possible shape for a diagnostic. Four selectors in `hack/cozyreport.sh` and `hack/cozyreport-summary.sh` were reading the wrong thing: - Per-pod evidence was chosen by the STATUS column, so a pod left at `0/1 Running` by a readiness probe that never passes was skipped. That is a common way for a suite to time out, and the artifact carried no trace of the pod the run died on. Selection now compares the two halves of the READY column; terminal `Succeeded`/`Completed` pods stay excluded. - The pending-LoadBalancer section filtered `$4`, which is CLUSTER-IP. CLUSTER-IP holds an address, `None` or `<none>` and never `<pending>`, so that loop had never executed. `<pending>` only ever appears in EXTERNAL-IP, `$5`. - The cozystack-apps section gated all three of its kinds on `kubectl get crd <kind>.apps.cozystack.io`. ApplicationDefinition is a CRD but in a different group, and Tenant comes from an aggregated APIService with no CRD to ask about, so the probe failed and the section never ran. - The kamaji section was gated on `cozy-linstor`'s `linstor-controller` deployment rather than kamaji's own, so a cluster without LINSTOR got no kamaji logs and no TenantControlPlanes — on exactly the runs that need them. The summary had three of its own: the HelmRelease listing printed `$5` where STATUS is the Ready condition's whole message, reducing every failed release to one word like "Helm"; the Flux sources listing used a jsonpath with a nested filter that kubectl rejects outright, so it printed nothing on every report ever taken; and pod AGE was read as `$6`, which shifts whenever RESTARTS prints the `7 (4m12s ago)` form — that is, for exactly the rows worth reading. The second half is bounding. The pod, VM, VMI and services walks now run under a count cap and a wall-clock budget, and every read is individually time-boxed. The point is not the bounds themselves but that each one names itself when it fires: a walk stopped by the cap or the clock leaves `COLLECTION-TRUNCATED.txt`, a pod list that never returned leaves `COLLECTION-FAILED.txt` instead of the empty tree a healthy cluster produces, a log cut short says so on its own last line, and a `--tail` bound is recorded beside the logs rather than inside them, since the platform's controllers log JSON per line and a trailing prose line costs a parser the whole file. A knob value the collector could not use is reported too, so a truncation note quoting `600` is never mistaken for the `10m` somebody actually set. Where `timeout` is absent the reads run unbounded rather than not at all, and the section says which of the two happened, because a missing local binary is not a statement about the cluster. One behaviour change worth stating plainly, because it is a trade and not a pure gain: the per-pod section used to be uncapped and read container logs whole, and it is now bounded by `COZY_REPORT_PODS_MAX` (40), `COZY_REPORT_PODS_BUDGET` (600s) and `COZY_REPORT_POD_TAIL` (2000). Below those thresholds the new tree is a strict superset of the old one — the old selector missed `Running 0/1` entirely and skipped logs for `Pending` pods. Above them the artifact is smaller than before, every drop is named in the tree, and `COZY_REPORT_POD_TAIL=-1` with a larger `COZY_REPORT_PODS_MAX` restores the old reach. Because widening the selection and capping the walk in one change could otherwise let a newly-in-scope pod evict one the uncapped walk always collected, the ordering splits by evidence class before namespace: pods that are not `Running` — the class the collector took before this — all rank ahead of pods that are `Running` but unready, so the cap can only ever drop a pod the old collector did not keep either. `hack/e2e-capture-previous-logs.sh` gets the same treatment: its notes now go to `capture-notes.txt` in the artifact as well as to the job log, and a pod list that never returned is reported as such rather than as "nothing restarted". That script now creates its own output directory before its first line of output, which makes the comment in `hack/e2e-chainsaw/.chainsaw.yaml` wrong about why the once-per-test guard there holds — the only change to that file is the corrected comment, saying the `-d` check keys on whether the catch block already ran rather than on what the helper found. The rest is test coverage — a new `hack/cozyreport.bats`, an extended `hack/capture-previous-logs.bats` — and the removal of `EXIT`-trap cleanup from the bats files those tests now cover. That trap is worth its own note: inside an `@test` body it replaces the one the `bats` binary installs for its own bookkeeping, so a test that then fails prints no TAP line at all. It does not appear as `not ok`; it disappears, and only the `# bats warning: Executed N instead of expected M tests` line at the end distinguishes the run from a green one. A guard test enforces the ban over a named list of files that grows as the remaining ones are converted. ### Testing Every collector this touches is now covered by unit tests that run without a cluster: a new `hack/cozyreport.bats` (166 tests) and an extended `hack/capture-previous-logs.bats` (34), driven through `hack/cozytest.sh`. Ten `hack/*.bats` files run green locally, 265 tests in total, with each file's `@test` count reconciled against its passing count rather than trusted to the exit status — that reconciliation is the point of the trap change, since an exit code answers "did anything fail" and never "did anything run". The guard test that bans `EXIT`-trap cleanup was checked by committing the violation rather than by reading it: a folded `trap 'rm -rf "$tmp"' \` + `EXIT` added to a file on its list turns the guard red and names that file, and removing it turns the guard green again. `shellcheck --shell=sh` over the three changed scripts reports strictly fewer findings than before the change (`hack/cozyreport.sh` SC2086 170 to 116, SC2162 15 to 13; the three SC2102 in `hack/cozyreport-summary.sh` are gone), and `hack/e2e-capture-previous-logs.sh` is clean. The parts that need a cluster — that the bounds fire, and that each one lands in the artifact — are exercised by the e2e run itself, since these collectors run on every failed suite. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. This PR touches `hack/cozyreport.sh`, `hack/cozyreport-summary.sh`, `hack/e2e-capture-previous-logs.sh`, `hack/e2e-chainsaw/.chainsaw.yaml`, ten `hack/*.bats` files and `docs/agents/e2e-testing.md`. Nothing is moved or renamed under `hack/`, no make target changes what it does, and the files downstream consumers key on (`hack/package.mk`, `hack/common-envs.mk`) are untouched. No package, values key, CRD, variant, namespace or release asset is involved. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): the diagnostic report now collects per-pod evidence for pods that are Running but not Ready, which it previously skipped, and several collector sections that silently produced nothing now report correctly. Collection that stops early on a bound now says so inside the artifact, so a truncated report can be told apart from a healthy cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Diagnostic reports now prioritize relevant resources and collect bounded pod, VM, service, storage, certificate, and Flux data. - Reports preserve partial results and clearly identify timeouts, unavailable resources, truncation, warnings, and collection failures. - Previous-container logs now include bounded capture, namespace prioritization, failure details, and partial-read evidence. - Reports target the operator workload and safely handle missing dependencies or temporary storage failures. - **Documentation** - Expanded guidance for reliable end-to-end testing and diagnostic report collection. - **Tests** - Added comprehensive coverage for report generation, log capture, failure handling, limits, and portability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-08-06 15:25:20 +03:00
Aleksei Sviridkin	e80b8e370f	fix(e2e): collect logs for unready pods and record truncated captures ... The failure snapshot selected pods by their STATUS column, so a pod whose container runs while its readiness probe never passes -- STATUS Running, READY 0/1 -- was dropped from the report entirely. That is a common way for a suite to time out: the replica never joins, its owning HelmRelease sits in InProgress, and the artifact carries no trace of the pod the run died on. Select on the READY column instead, which also covers the partially ready multi-container case, and keep every phase that was collected before. On one degraded cluster that is four of the seven broken pods the old filter reported nothing about. The VMI and VM sections had the same blind spot and now share one selector that reads READY as the last field, because a Pending object prints fewer columns than a Running one and the two kinds differ only in how many columns precede READY. READY counts ready containers rather than the pod's Ready condition, so a pod held back by an unsatisfied readiness gate is out of scope; a test pins that rather than leaving it to be found in an empty directory. Logs are requested for every container of a selected pod rather than skipped for Pending ones, and the per-pod collection moved into a helper so it can be driven against a stub kubectl. kubectl's refusal is the finding, and an empty result is ambiguous on its own: a read against an unscheduled pod exits 0 having printed nothing, which is what a container that started and stayed quiet also leaves. Every site that classifies a failed read now orders its branches the same way, naming the timeout first where both a cutoff and an uncapturable stderr apply. The CRD gate in the summary had no such branch at all: it returned in silence, which is byte-identical to the CRD being absent, so every section behind that gate vanished from a report whose only real problem was that the machine writing it had nowhere to put a temp file. Collection used to stop silently. A walk that hit its cap said so in the job log and nowhere in the artifact. Now the pod walk leaves COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off part way through, COLLECTION-FAILED.txt when the pod list never came back rather than the empty tree a healthy cluster produces, and COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a fact about the machine writing the report and not about the cluster. Tenant namespaces are collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform pods while the pod the suite died on falls off the end. Container logs are read with --tail=2000 where they were previously read whole, and every bounded file states the bound it was written under, because --tail drops the oldest lines and nothing else in the output would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log. The bounded whole-object read no longer merges stderr into pod.yaml, vm.yaml and vmi.yaml, which are read by yq rather than by eye: a warning on an otherwise complete read left an unparseable file. stderr goes to a scratch file and is copied in only when kubectl returned no object, and the truncation markers those files carry are written as YAML comments, since a bare marker line after a complete mapping is a second top-level node. Six latent faults surfaced while making these reads report. The kamaji module was gated on the LINSTOR controller, so a cluster without LINSTOR got no kamaji evidence at all, which is what a tenant-kubernetes bootstrap timeout needs most. The Flux sources section nested one jsonpath filter inside another, which kubectl rejects outright, so it printed nothing on every report ever taken and read as "all sources Ready". The HelmRelease listing took a single field as STATUS where STATUS is the Ready condition's whole message, so every failed release rendered as one word. The services section filtered for a pending LoadBalancer address on the CLUSTER-IP column, which never carries that value. The cozystack module read its image from a Deployment the installer stopped shipping when it moved to the operator, putting kubectl's NotFound in the first file of the report. And the cozystack apps section gated all three of its kinds on a CRD in an aggregated API group, so it never ran once; two of those names were wrong, and both are collected again. The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats test that installs one and then fails prints no TAP line at all, so output grepped for `not ok` reads as green, while hack/cozytest.sh reports the same failure correctly. A guard enforces that over a named list of files rather than a glob with exceptions, because a list of exceptions is the same claim written so that forgetting to shorten it makes it false. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 17:37:04 +03:00
Aleksei Sviridkin	7739d1e294	feat(multus)!: stage reference CNI plugins from the multus image ... Multus-backed NetworkAttachmentDefinitions delegate to the upstream containernetworking reference plugins (bridge, macvlan, ipvlan, ...), and nothing guarantees a node has them in the directory Multus searches. On k3s they live under /var/lib/rancher/k3s/data/cni while Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails there with `failed to find plugin "bridge" in path [/opt/cni/bin]`. Bundle 14 of the release's plugins into the multus image and add an install-cni-plugins init container that installs them into the host /opt/cni/bin, mirroring the existing install-multus-binary container. loopback, dhcp, tap and dummy are left out; the Dockerfile records why. Gate it on a new chart value, stageCniPlugins, resolved per bundle and on for both. Talos bundles only bridge, firewall, flannel, host-local, loopback and portmap, so the other ten are missing there exactly as on generic Linux; the four that overlap are rewritten with the same upstream version, v1.9.1 on both sides, rather than an older one. A marker in the image Dockerfile records the Talos release that pin was compared against, and a test holds it to the release the platform ships. multus therefore moves out of system.common-packages into the two variant branches, the way linstor already does for talos.enabled. networking.stageCniPlugins overrides the variant default in either direction. Where staging is on the install is unconditional: a plugin is replaced on every pod recreation, not only when the destination is missing. A destination that is already a directory is reported and left alone, since replacing it is not something this can do safely. Turning the value off afterwards does not undo it; nothing here removes or restores a plugin. The container is not privileged and takes no mount propagation. It runs in the spc_t SELinux domain, without which an enforcing node refuses the write. Each plugin is installed under a temporary name and swapped in with a rename: /opt/cni/bin is shared with cilium and kube-ovn. The plugin tarball is checksummed before extraction. A plugin that cannot be installed is reported on stderr and to /dev/termination-log, and the container still exits 0: Multus stays the node's primary CNI and nothing removes its config, so an init container that never completes would leave the shim with no daemon behind it. The manifest change lives in customize-deployment.patch so it survives make update. BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI plugins in /opt/cni/bin on existing nodes, Talos included -- four of the names it writes are ones the Talos node image also carries. Opt out before upgrading: declining afterwards does not restore what was replaced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-08-05 14:19:47 +03:00
Andrei Kvapil	3dc0f56477	feat(kube-ovn): build the chart and image in-tree, retiring the wrapper repo (#3460) ... ## What this PR does Brings the `kubeovn-chart` wrapper into the tree, so `cozystack/kubeovn-chart` can be archived. Until now `packages/system/kubeovn` pulled a chart tarball from the latest `cozystack/kubeovn-chart` tag at `make update` time, and that repository also built and pushed the `kubeovn` image. The wrapper existed only to cache the upstream chart; with the current caching approach that reason is gone, and the split cost a second repository, a second release step, and a package whose provenance a reader cannot follow without leaving the tree. The package is now shaped like `packages/system/cilium`: - `update` pulls the upstream `kubeovn/kube-ovn` chart directly and applies `patches/mtu.diff`, then syncs the version into the Dockerfile. The MTU patch already carried monorepo-relative paths, so it applies unchanged. - `image` builds `images/kubeovn` (carrying the existing `fix-keepvmip-portgroup` patch) and pins the digest into `values.yaml`, using the shared `image-tags` / `cache-args` helpers. It also carries `cozystack/kubeovn-chart#5`, which never got a tag: `OVN_IPSEC_KEY_DIR` is pinned under `/var/lib`. The kube-ovn default sits in `/etc/origin`, which a Talos node cannot create, so enabling IPsec left the node without ESP security associations. ### The nightly host rewrite had to be fixed first Moving the image build here activates a gap recorded in `docs/agents/image-refs.md`. The nightly host rewrite is a literal `<src-registry>/` substring replace, and kubeovn keeps its host in `global.registry.address` with the bare repository in a sibling key — no trailing slash, no match. While the image came from a separate repository the baked reference carried the public host and the ownership filter skipped it; once CI builds the image here it stamps the build-registry host into that key, and the result is an image mirrored to the public registry while the published tree keeps pointing at the private build registry. The second commit adds an end-of-line-anchored expression that fires only when the source registry is the complete scalar value, so a contiguous reference — which continues past the host — is still handled by the first expression and the two can never both fire on one reference. `keycloak-operator` splits at a different boundary and stays unfixed; that is documented, and dormant while no CI path rebuilds it. Verified: `hack/nightly-mirror_test.bats` passes with the new case and the case fails without the fix; `promote-retag` and `promote-rewrite-tags` suites unaffected; `helm template` and `helm unittest` on the package pass. The pre-existing `helm lint` complaint about a missing `kube-ovn` dependency reproduces on `main` and is left alone. ### Follow-ups this PR does not do - `cozystack/kubeovn-chart` should be archived once this merges, with a pointer here. Its open PR #5 is carried by this change and needs no separate landing. - `cozystack/website` links the Kube-OVN license at `cozystack/kubeovn-chart` in `operations/configuration/licenses.md` and `guides/platform-stack/_index.md` (all versions, including `next`). Archived repositories stay readable so nothing breaks, but the link should move upstream or in-tree. **No follow-up has been opened yet** — flagging rather than ticking the box. ### Screenshots N/A — build and packaging change, no user-facing or UI impact. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walked the trigger map against the diff. The website row matches — not through the app-list or platform-component triggers, but because two of its pages link the repository being archived. Left unticked because no follow-up exists there yet; a human should decide whether the license link moves to `kubeovn/kube-ovn` or to this package. No other row matches: the change touches no `packages/apps/` or `packages/extra/` package, no `values.schema.json`, no `ApplicationDefinition`, no platform variant, and no node prerequisite. ### Release note ```release-note feat(kube-ovn): the Kube-OVN chart and image are now built in this repository instead of cozystack/kubeovn-chart. OVN_IPSEC_KEY_DIR now defaults to /var/lib/ovs_ipsec_keys, so enabling IPsec on Talos nodes no longer leaves them without ESP security associations. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated Kube-OVN image build to use upstream chart metadata and keep registry/version alignment during `update`. - Added conditional MTU support for the OVN CNI DS via chart values. - Introduced `OVN_IPSEC_KEY_DIR` to standardize the OVN IPsec keys directory. - **Bug Fixes** - Preserved VM networking port-group assignments during live migration when an alive virt-launcher sibling pod exists. - Improved nightly mirror host rewriting for split and standalone registry host formats. - **Tests** - Added coverage for the end-of-line “whole scalar value” host rewrite case. - **Documentation** - Refreshed “Known gaps” guidance for nightly mirror host rewriting behavior and limitations. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-29 18:19:41 +02:00
myasnikovdaniil	cc44f6c981	ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449) ... ## What this PR does Moves release validation from promote time to rc time, making promotion itself a pure retag operation. This is the follow-up to the v1.6.0 retrospective: the promote flow's checks ran on the promote PR (coupled to promotion, skippable by never promoting), the changelog and website docs were produced only after the release went public, and the finalize tag push silently failed to trigger `tags.yaml` at all (persisted-credential bug, root-caused below). The release lifecycle after this PR: 1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes the staging branch — and now runs a **mandatory full e2e** for every `-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and any published tag) get the same validation as a **manual button** (`workflow_dispatch`). 2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**: the release changelog is generated from the rc tag (reusable `changelog-generate.yaml`) and committed to the rc staging branch, ready for review long before promotion. 3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence: the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages the release, picks up the rc-time changelog (or generates one), and **opens the website docs PR** on cozystack/website from the staging branch — parked with an explicit DO-NOT-MERGE-until-finalize contract. 4. The promote PR **no longer gets `full-e2e` automatically** — its tree is the rc's already-validated digests with only tag strings rewritten; a maintainer can still add the label manually to force the suite. 5. Finalize performs the irreversible side effects (tag, publish, retag, chart) — and with `persist-credentials: false` its tag push now actually authenticates as the CI app, so the `tags.yaml` backstops (`generate-changelog`, `update-website-docs`) fire for promoted tags. This is the one-line fix for the v1.6.0 incident where the tag triggered nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader` silently defeats the app token injected via `git remote set-url`, and `GITHUB_TOKEN` pushes create no workflow runs. Everything is pinned by executable contract tests (`hack/promote-gate-contract.bats`, extended `hack/release-changelog-contract.bats` / `release-changelog-behaviour.bats` — 48 tests total, including behaviour tests that execute the new `hack/parse-rc-tag.sh` and `hack/select-changelog-source.sh` for real), and `docs/release.md` is rewritten to describe the new lifecycle end-to-end. ### Commit map (review commit-by-commit) - `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing changelog-before-publication rework (changelog generated at promote, committed to the staging branch before the PR opens, finalize publishes it as the release body; tags.yaml job becomes a verbatim-port backstop), rebased onto main. - `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane (prerelease → staging branch, stable → tag; draft releases fail fast; dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts). - `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut rc tags (also on `release_exists` re-runs); fixes the `cut-prerelease.yaml` comment that falsely claimed e2e already existed. - `2dfbeea77` — `changelog-generate.yaml` (reusable core) + `changelog-rc.yaml` (rc-time button committing to the staging branch); promote-rc gains a top-level `parse` job with a strict no-whitespace tag policy (`hack/parse-rc-tag.sh`); source selection is validate-then-fallthrough (`hack/select-changelog-source.sh`) so an invalid higher-priority changelog can never silently win. - `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override + removal of the automatic `full-e2e` label (pull-requests.yaml release-path e2e now requires an explicit label; a post-open label now actually starts the run). - `936190d4b` — `website-docs` job at promote time behind a FETCH_REF-support guard (see dependency below), with the DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before `release-next`; `tags.yaml::update-website-docs` stays as the backstop. - `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0 root cause). - `717c8860f` — `docs/release.md` coherence pass over the whole new lifecycle. ### Dependency The promote-time website docs job needs cozystack/website#631 (`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard detects the missing support and degrades to a ⚠️ line in the promote PR body — promotion is unaffected and the tag-time backstop backfills, so this PR is safe to merge first. ### Merge instructions Please **merge-commit, do not squash**: the 13 commits are deliberately atomic (the credential root-cause fix is separable from the feature work) and squashing would collapse ~3k lines into one commit, hurting bisectability. ### First live run The e2e lane is verified by lint, contract tests and dry-traces, but has not yet executed against a real tag; the intended first live run is a manual `e2e-tag.yaml` dispatch against an existing published rc tag before the gate is relied on for a real promotion. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added reusable changelog generation with validation, “present” signaling, and safe artifact upload behavior (including RC-time support). * Added an E2E release-tag workflow with tag parsing, deterministic sandboxing, and comprehensive report collection. * Improved RC promotion with a stricter parse front-door, verified E2E green gate (optional bypass), and earlier parallel changelog/asset staging. * **Documentation** * Refreshed release-process docs to match the updated RC-to-stable, changelog, and website-docs timing rules. * **Bug Fixes** * Improved PR finalization to source release bodies from verified changelogs and warn/fallback on missing or empty content. * Tightened label gating so E2E runs only under the intended conditions. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-29 17:14:29 +05:00
Andrei Kvapil	6cb579338b	fix(nightly): rewrite an image host that is the whole scalar value ... The nightly host rewrite is a literal "<src-registry>/" substring replace, so it only reaches a host sitting contiguously in front of the repository. kubeovn keeps its host in global.registry.address with the bare repository in a sibling key: no trailing slash, no match. The image was mirrored to the public registry and the tree kept pointing at the private build registry. The gap was dormant while kubeovn's image came from a separate repository, because the baked ref carried the public host and the ownership filter skipped it. Building the image in-tree stamps the build-registry host into that key and turns it live, so fix it here. Add an end-of-line-anchored expression that fires only when the source registry is the complete value; a contiguous ref continues past the host, so the two expressions cannot both fire on one reference. keycloak-operator splits at a different boundary and stays unfixed -- neither of its keys holds the source registry as a whole. Documented, and dormant while no CI path rebuilds it. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>	2026-07-28 18:07:34 +02:00
Aleksei Sviridkin	a7ce4371c2	docs(agents): correct the chainsaw podLogs capability note ... The e2e conventions, the previous-logs capture script, and the global chainsaw catch all claimed the v0.2.15 podLogs collector exposes only container/tail/selector. It also accepts name, namespace, timeout and the cluster fields; only the previous option is genuinely missing. The wrong list steers catch authors toward imperative scripts for cross-namespace log capture that podLogs handles declaratively. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-27 22:48:25 +03:00
Myasnikov Daniil	1eded27900	test(release): scope the trim pin to the parse step, fix two stale references ... Three review nits, all verified against the tree, plus a tighter scope than the first attempt at them. The behaviour suite asserted "promote-rc does not trim the rc tag" against the whole file. promote-rc.yaml has no .trim() today, so it passed, but any unrelated .trim() added anywhere in the workflow would have failed a test about tag policy. Scoping it to the parse JOB was not enough either: that job is ~180 lines, most of them the e2e gate's github-script, whose string handling has nothing to do with tag policy. Both greps are now scoped to the parse STEP, and run through a code_lines filter copied from the contract suite so a YAML comment mentioning .trim() cannot fail them. Mutation-checked in both directions: a .trim() injected into the parse step fails the test, one injected into the e2e-gate step of the same job does not. - The Step 2 checklist entry still said "must be main" while Step 2 itself now lists three valid starting points (main, detached at the release tag, detached at the rc tag being promoted). The checklist is what an agent ticks off, so it was the half that mattered; it now points at Step 2 rather than re-encoding the count, so a fourth configuration cannot desync it. The heading said "Checking current branch", which two of the three states do not have. - hack/testdata/needs-semantics.yaml named release-changelog-contract.bats as its consumer; the only consumer is release-changelog-behaviour.bats. 50/50 tests green across the three suites (11 + 24 + 15). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-27 14:03:49 +05:00
Myasnikov Daniil	717c8860f6	docs(release): make the release playbook coherent with the promote flow ... Final coherence pass over docs/release.md (and the releasing.md pointer) after several incremental patches to the release-cycle rework left the whole-document narrative uneven. - Rewrite the regular and patch release sequences so a maintainer reading top-to-bottom performs the actual flow: cut rc -> watch mandatory rc E2E -> [optional Generate RC Changelog button] -> promote dispatch (E2E gate; changelog generated or picked up; website docs PR opened and parked) -> review the promote PR (no E2E by default; E2E/changelog/website-docs lines) -> merge -> finalize -> merge the parked website PR, then announce. - Purge the last old-model sentence: "vX.Y.0 is tagged from the last RC or a later commit in main" -> promoted from the rc by digest, tag cut write-once at the promote PR merge commit. - State the alpha/beta contract where prereleases are discussed: built and published, button-only E2E (no mandatory rc-e2e), and no promotion path (promote-rc accepts vX.Y.Z-rc.N alone). - Fix the Phase 4 heading (was "chore(release): cut vX.Y.Z"; the PR is "chore(release): promote ... -> ...") and add the missing website-docs line to the reviewer checklist; add a keep-parked-until-publish item to the pre-release checklist. - Add cut-prerelease.yaml, changelog-rc.yaml, and changelog-generate.yaml to "See also" so the workflow inventory is complete and consistent with the "seven workflows + reusable changelog-generate" count. - releasing.md: replace the stale "typically involves ... creating git tags / building artifacts" list (invalidated by the promote-not-rebuild model) with the real high-level sequence. Every workflow/job/step name cited was verified against the YAML on this branch. hack/release-changelog-contract.bats stays green (24/24); no test pins docs/release.md strings, so no test change was needed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-24 18:16:26 +05:00
Myasnikov Daniil	34e45295fe	ci(release): fix second-round review findings ... Two more review passes (Claude + Codex) on the previous commit. Codex found the one that would have broken the very next promotion. The agent reads its own instructions from the working tree, which is the rc tag. Every rc cut before this change carries the old §2 of docs/agents/changelog.md — two valid configurations, "stop" on anything else — and a detached rc-tag checkout is exactly that "anything else". So promoting v1.6.0-rc.4 would have presented a compliant agent with instructions telling it to refuse, silently, producing no changelog on every promotion of an existing rc. Verified against the tag. The same applies to hack/validate-changelog.sh, absent from those trees entirely, which would have failed the Verify step with "command not found" and, in open-pr, taken the promote PR down with it. Both are now overlaid from the dispatch ref, best-effort, in both jobs. The structural validation added last round rejected every PATCH changelog. It asserted a `# Cozystack vX.Y.Z` header and a 20-line floor; patch releases use `# vX.Y.Z (date)` and v1.5.1 is a complete 19-line release. Good AI output would have been discarded and the promote PR mislabelled "generation failed". Measured across the tree: the header form holds for 23 of 115 changelogs, so it was never an invariant. The predicates now live in hack/validate-changelog.sh, tested directly against real changelogs rather than through YAML greps — the reason the defect shipped is that the suite pinned finalize's guard but nothing on the promote-side validator, and stayed green when the validator was mutated away. It accepts both header conventions, has no line floor, and checks the version in the header, the release-link comment and the compare link, so a changelog generated for the wrong version is caught. All 16 current-era changelogs pass; whitespace, empty, truncated and wrong-version inputs are rejected. Also fixed: - A fatal artifact download in open-pr would abort the job before the PR step — the same wedge as a skipped job, one step lower. Download is now best-effort and the commit and PR body key off what is on disk, not an upstream job output that says nothing about whether the transfer worked. - A pre-existing changelog (hand-written, or preserved across the force-push) reached the PR without ever being validated, was advertised with ✅ and published verbatim. It now goes through the same validator, as does the tags.yaml backstop. - `gh pr view <branch>` matches closed PRs, so re-dispatching after closing an abandoned promote PR edited the dead PR and opened nothing. Branch on state explicitly. - `gh pr edit --body` unconditionally overwrote the PR body, destroying the upstream breaking-change notes docs/release.md instructs reviewers to paste there. Refresh only while the body is still the generated one. - open-pr consumed raw dispatch input while its siblings used the validated parse; the changelog job's bash parse did not trim, so an input with a stray space passed promote and failed there. Both fixed. - The prompt and §2 said "trailing release-link comment"; it is the first thing in every changelog. Tests grow to 16, each mutation-checked: dropping the overlay, making the download fatal, reverting the existing-changelog check to `-s`, and removing the backstop validator each fail the suite. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-24 11:34:17 +05:00
Myasnikov Daniil	f49d54a689	ci(release): fix review findings; keep website docs at tag time ... Dual-model review (Claude + Codex) of the previous two commits found seven substantive defects, two of them destructive. Both models independently reached three of them; the split is noted per item. Descope the website half (Claude, verified by reproduction). The docs job was moved to promote time, but the website Makefile pins the fetch ref with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that cannot exist at promote time, since promote's own validate step hard-fails if it does. update_apps.sh logs the failed fetch and exits 0 AFTER truncating each destination file, so every app doc would become a stub, `changed=true` would open the PR, and the new auto-merge would land it unreviewed. Not fixable from this repo. update-website-docs returns to tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize auto-merge is dropped. Opening it earlier needs a cozystack/website change first and is left as a follow-up. This also removes the unordered-sibling race Codex found, where the release PR could merge before the docs job opened its PR, leaving finalize nothing to merge and no retry. Fix the non-blocking guarantee (both models). Only the AI step carried continue-on-error; the changelog JOB did not, so a flaky `npm i -g @github/copilot` failed it and `needs` skipped open-pr entirely — staging branch pushed, release drafted, no promote PR. Gate open-pr on promote alone. Generate from a checkout the contract permits (both models). The job checked out the rc staging branch, which docs/agents/changelog.md §2 rejects — a compliant agent stops and writes nothing, so the normal path produced no changelog at all. Check out the rc tag detached and document that configuration in §2. The checkout ref now comes from the validated parse output rather than raw dispatch input. Port, don't regenerate, in the backstop (both models). For a patch release promoted onto release-X.Y the changelog never reaches main, so the backstop fired and ran a second AI pass; once its PR merged, update-releasenotes.yaml overwrote the published body with unreviewed text. Detect the file at the tag and port it verbatim; the AI runs only when it is absent from both main and the tag. Survive re-dispatch (Codex). promote force-pushes the staging branch, erasing a hand-written changelog — the documented recovery when generation fails. Preserve an existing one across the rebuild, and stop open-pr overwriting it with generated output. Validate structurally, not by size (Codex). A truncated or whitespace-only AI run passed `[ -s ]`, was committed, advertised as valid in the PR body, and published verbatim as release notes. Assert the header, the compare link and a plausible line count; finalize additionally refuses whitespace-only content rather than blanking the notes. Add hack/release-changelog-contract.bats — nine tests pinning each link in the chain, mutation-checked: removing open-pr's if:, breaking finalize's body read, and restoring paths-ignore each fail the suite. Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which described the old flow throughout. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-24 11:34:16 +05:00
myasnikovdaniil	55195ded0c	test(e2e): capture previous container logs and stop killing the snapshot (#3346) ... ## What this PR does Root-causing a recent E2E failure was nearly blocked because the diagnostics captured on failure had collected almost nothing. Two independent mechanisms were responsible, and both had been silently in place for a long time. **The failing container's previous logs were never captured.** The pod had crash-looped, so the decisive evidence lived in the first container instance, reachable only through `kubectl logs --previous`. Chainsaw's `podLogs` action has no `previous` field, so this could not be expressed per-suite; the conclusion had to be reconstructed from indirect evidence instead. A new capture runs from the global catch and from the BATS failure hook, selecting containers at runtime by `restartCount > 0` across both init and regular containers, so a healthy run stays quiet and future suites are covered without per-suite wiring. **The catch was structurally guaranteed to be killed.** A Chainsaw `script` op is bounded by `timeouts.exec`, which is 5m, while the catch's own inner budgets summed to roughly 960s. It reported `SCRIPT | ERROR | signal: killed` and produced nothing — and its second leg, the data-plane capture, had therefore never run at all. The kill signature was itself misleading: `timeout` exits 124 and cannot produce that message, which comes from Go's `exec.Cmd` killing the process group. Each catch op now carries an explicit timeout above the budgets it contains, so the inner one fires and leaves a partial, usable artifact instead of losing everything. **The snapshot was truncated at 60 seconds.** `crust-gather --duration` defaults to 60s and was never passed, so every snapshot ever taken was cut short, with nothing in the artifact saying so. The API discovery it performs first is not covered by that budget at all, which is where the outer wall-clock was going on a cluster with a degraded aggregated APIService — precisely the state a failed run is in. Both budgets are now set explicitly, and the collector's output goes to a log inside the snapshot with a completeness verdict, so "is this snapshot complete?" is answerable from the artifact. Failure paths are reported rather than swallowed: an unreachable API says so instead of implying nothing had restarted, a truncated capture is kept and marked, and every dropped item is logged, since silent truncation reads as full coverage. ### What is not verified here The mechanism claims are read from the pinned versions of Chainsaw and crust-gather. Everything cluster-facing is not verified by this PR: the go-template's output shape against a live API server, `--previous` returning the first instance for a real crash-looper, and whether the new collection budgets suffice on a degraded cluster. The bats suite covers the selection helpers only, and those cases are mutation-tested. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Diagnostics** * Improved end-to-end failure evidence by capturing “previous instance” logs for crash-looping pods (including restarted init containers) and prioritizing relevant namespaces. * Added time-bounded diagnostics collection with explicit outer timeouts; collector output is saved to per-snapshot log files. * Classifies collector outcomes as complete, timeout-truncated, or failed, while preserving the original test failure signal. * **Documentation** * Expanded E2E failure-diagnostics guidance with new timeout and evidence-capture conventions. * **Tests** * Added unit-style tests for previous-log selection, prioritization, row limiting, and deterministic log artifact naming. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-22 17:22:11 +05:00
Myasnikov Daniil	1f302d0d78	docs(agents): correct issue ref, mirror claim and example in image-refs ... The build-list violations bullet pointed at #3143, which tracks the adjacent gap (packages in the list going stale between releases) and never covers packages absent from the list; they are now tracked in the newly filed #3416. The nightly split-host paragraph claimed both affected images are mirrored while the published tree points at the build registry — today neither ref carries the source-registry host (keycloak-operator is never rebuilt by CI, kubeovn is built externally by cozystack/kubeovn-chart), so the ownership filter skips them and the gap is latent, not live; reworded to say exactly that. The opening example paired a v1.6.0 tag with the real v1.5.0 grafana digest — a ref that will never exist — and grafana is slated to stop being built; replaced with the literal committed cozystack-api ref. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-22 12:23:29 +05:00
Aleksei Sviridkin	76f7267ff2	test(e2e): describe --previous as the preceding instance, not the first ... kubectl logs --previous returns the immediately preceding terminated container instance, not the original first one — after multiple restarts the first instance is already gone. Reword the helper, the BATS and Chainsaw failure handlers, and the diagnostics doc to match the actual --previous semantics. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-21 18:57:01 +03:00
Myasnikov Daniil	9bd8c17971	fix(ci): shape-filter the promote postcondition and close test gaps ... Review of the previous commit found the postcondition itself could abort a release. It grepped the bare "X.Y.Z-rc.N" substring across the tree, and three live files carry such strings without being image refs: kubevirt-csi-driver's go.sum pins github.com/golang/protobuf v1.4.0-rc.2 -- a version cozystack actually cut, so promoting v1.4.0 would have failed on a Go checksum line -- metallb_test.yaml names v1.5.0-rc.2 in a comment, and the console pnpm-lock.yaml carries rolldown@1.0.0-rc.15. The remediation the error message gave would have made things worse: declaring go.sum as a ref source would sed a checksum file. Match the version only in image-reference position now: after an image/repository/tag key with no intervening '#', or immediately before an @sha256 digest. Narrowing it to "${RC}@sha256:" alone would have been wrong -- cilium keeps `tag: v1.5.0` and `digest:` under separate keys, so the version is not always adjacent to the digest. Verified against four real rc versions that previously aborted. The suite also only tested the fail-closed direction, which is why that defect survived, and the collector had no completeness guard at all: a first-party ref added at an unenumerated path on a `latest@` tag was mirrored by nothing and passed all eighteen tests. Since the mirror's failure mode is the worst of the three -- the published nightly points at the private build registry -- that gap mattered most where coverage was thinnest. Added: a fail-open test using the three real rc versions above; a collector completeness diff that discovers refs by content and fails on any the enumeration misses; coverage for the OCI-artifact shape, whose deletion previously left every suite green; and an unreadable-file test, since a read error was being treated as "no match" and silently skipped. The component-versioned test was vacuous -- its rc version shared no substring with the kamaji tag it claimed to protect, so it passed by construction. It now uses an rc version sharing kamaji's 0.19.0 prefix, which a loosely-anchored implementation would corrupt. Round-trip fixture discovery matched a combined 'ghcr.io/cozystack/ cozystack/' prefix, which skips keycloak-operator (host in a sibling registry: key) and kubeovn (host in global.registry.address). It now matches 'cozystack/cozystack' and stamps split `tag:` keys too. Also documented two pre-existing gaps rather than half-fixing them: the nightly host rewrite is a contiguous-substring replace and cannot reach a split host, and a ref inside a gzip is invisible to every tool including the postcondition, which skips binaries. Both are host-dimension defects; the version dimension this change owns is unaffected, because the version always lives in a tag key regardless of where the host sits. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-21 20:42:28 +05:00
Aleksei Sviridkin	07c67a8913	test(e2e): capture previous container logs and stop killing the snapshot ... Root-causing a crash-loop failure was blocked twice over by the diagnostics collected on failure. `kubectl logs` shows only the current container instance, so for a crash-looping pod the suites' `podLogs` collector captured the Nth restart -- a fresh replay of the same startup path -- while the evidence that explained the failure sat in the first, interrupted instance. Chainsaw's podLogs collector has no `previous` option, so add e2e-capture-previous-logs.sh and drive it from the global catch and from the BATS failure hook. It discovers containers with restartCount > 0 at runtime (init containers included, where an interrupted bootstrap lives), orders the app suites' namespace first, and dumps each to the job log and the snapshot tree. Containers that never restarted are skipped, so a healthy run emits one line rather than a page of "previous terminated container not found". Every way that capture can come up short is named rather than inferred, because kubectl exits 1 for a garbage-collected container, an RBAC denial and an unreachable node alike: the container cap, a pod list that never returned (which is not the same as nothing having restarted), a read that timed out versus one kubectl refused and the reason it gave, and a previous instance that produced no output. A partial capture is kept and marked truncated rather than deleted -- it is the only copy of a log already gone from the cluster. The crust-gather snapshot was killed by the Chainsaw op containing it. A script op is bounded by timeouts.exec (5m), and the catch's own wall-clock wrappers summed to 960s, so the script was SIGKILLed partway through -- taking out the process group mid-write and skipping the data-plane capture entirely. Give each catch op an explicit timeout above the budgets it contains, so an inner timeout fires instead and leaves a partial, still-servable snapshot behind. The data-plane leg drops from 600s to 300s at that call site so both legs fit one envelope; the BATS hook keeps the wider budget, having no containing op. Set crust-gather --duration explicitly as well. It is the collection phase's own budget, it defaults to 60s, and on elapse it abandons the collection and skips its finish step, leaving a truncated tree with no indication anything is missing. The outer wall-clock has to exceed it because the API discovery that runs first is not covered by that budget at all -- with an unhealthy aggregated APIService, the state a failed run is usually in, discovery alone can consume the whole timeout before a single object is written. Send each collector's output to a log beside its snapshot instead of /dev/null, with a complete/truncated/failed verdict, so that question is answerable from the artifact. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-21 18:37:07 +03:00
Myasnikov Daniil	09323fff13	fix(ci): cover every image-ref storage shape in promote, retag and mirror ... Promotion rewrote the rc version substring across the depth-2 package values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the premise that the kubernetes app was the only one whose .tag files carry the cozystack version. Nine other .tag files do, and system/multus seds its ref straight into a vendored upstream manifest. v1.6.0 was therefore staged with 33 of 55 first-party refs still reading v1.6.0-rc.4. promote-retag.sh and nightly-mirror.sh had the same blind spot from the other direction: both scanned only the depth-2 values.yaml, so those images were never retagged to the stable version (30 refs selected before, 42 after) and never mirrored to the public registry for a nightly. A retag miss merely leaves an image short of a tag, since the digest still resolves inside one registry; a mirror miss is worse, because the host rewrite walks the same file list and a ref that is neither mirrored nor rewritten leaves the published tree pointing at the private build registry. The three call sites each carried their own idea of where a ref can live, which is how they drifted apart. Move that knowledge into hack/lib/image-refs.sh as the single enumeration -- image_ref_files() for the files, collect_image_refs() for the refs inside them -- and have all three source it, so a new storage location is declared once and reaches every consumer. The rewrite itself moves out of promote-rc.yaml into hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only observable by cutting a release: there was nothing to unit test. It now also asserts its own postcondition, scanning wider than it rewrites and failing the promotion when any ref survives in a location the enumeration does not know about, rather than shipping it quietly. Component-versioned images (kamaji's v0.19.0-cozystack.N) and third-party pass-through images do not ride the cozystack version line and are deliberately left alone; a test pins that so a future fix here cannot over-reach. capi-providers-cpprovider is a documented known gap: its ref is also embedded in a gzipped copy that the chart actually ships, and rewriting only the readable copy would diverge the two. docs/agents/image-refs.md records the contract -- three tag classes, three storage shapes, the invariants, and which consumer does what. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-21 20:14:24 +05:00
Andrey KolkovandClaude Fable 5	f1627354b4	docs(contributing): point the ansible migration note at loadBalancerClass ... The downstream-coordination checklist still directed contributors at an "externalIPs to exposureClass" migration; exposureClass was removed in 3dbc7d366 and the native replacement key is publishing.loadBalancerClass. Last leftover of the ExposureClass removal this PR sweeps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-07-17 14:35:40 +04:00
Aleksei Sviridkin	44507d2cd0	docs(agents): add downstream repository checklist to the PR template ... Cozystack is upstream for repositories that nothing keeps in sync with it, and no CI job compares the two sides, so a change here breaks them in silence. Only two narrow paths are automated, and both fire on a release rather than on a PR. It has already happened. The website docs generator works from an app list hardcoded in its own Makefile, and packages/apps/opensearch was never added to it, so an application we ship has no reference page at all. The Terraform provider still offers a Kubernetes version this repo removed. The plugin repo still sends operators to hack/ scripts that were deleted, one of them rewritten as a Go command. Add a Downstream Repositories trigger map to the contributing guide, naming per repository which change here forces a change over there and which file to touch, and a checklist in the PR template that links to it. The template addresses AI agents directly: they author a large share of PRs and will otherwise tick a box without opening the follow-up it claims. The coupling is wider than the obvious ones, and mostly invisible. talm restates the node contract, overlapping ours but already diverged, so it must be diffed rather than overwritten. cozyhr reads an annotation the operator writes and silently merges the wrong values if it is renamed. cozy-proxy is vendored in from its own repo, yet consumes labels produced here and quietly stops handling external VM services if they move. The telemetry server queries metric names emitted here, and a rename just empties its dashboards. None of these fail loudly. Guard the map with a unit test, because a map about silent drift must not drift silently itself. It pins every in-repo path the map cites to a file that exists, pins the map and the checklist to the same repository list, and pins the workflow carve-out below to the map's location, to the job that reads the flag, and to the output that carries it: cutting any one of those three leaves the other two looking wired up. The header states plainly what the suite cannot check, so a green tick is not mistaken for a correct map: whether a coupling is real, and whether it still holds over there, stays a human's job. Editing the map alone is a docs-only PR, which skips the unit tests and would exempt the map from its own guard. Rather than widen the docs-only rule, which also gates the Talos image build, the plan step exports a separate flag that only the unit-test job reads: a docs PR now runs the tests and still builds nothing. It detects the map by its old path too, since a rename is reported as the new path alone. Document that gh pr create --body and --body-file replace the body wholesale, which would otherwise drop the checklist from every CLI-authored PR. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-07-14 17:07:47 +03:00
Myasnikov DaniilandClaude Opus 4.8	18715f8c63	docs(agents): document Chainsaw v0.2.15 gotchas in the e2e-testing guide ... The v0.2.15 syntax/semantics traps were tribal knowledge scattered across PR review notes (the number-literal caveat lived only in #2826's body, which evaporates on merge). Consolidate them into a durable subsection so the next author finds them before hitting each one fresh: filter-as-list conditions, no number-literals/backticks in JMESPath, error: fails hard on an unknown GVK, no conditional Test-level skip, step-level-only finally, bounded/blocking cleanup surfacing latent teardown bugs, projection-style failure diffs, and the catch-script cwd. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-10 16:16:18 +05:00
Myasnikov DaniilandClaude Fable 5	9297e51170	Merge origin/main into feat/e2e-chainsaw-example ... Reconcile the Chainsaw e2e migration with ~500 commits of main drift: - keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's _lib/ rename with the Chainsaw finally/snapshot deltas re-applied - resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted; their post-branch assertion drift is re-ported into the Chainsaw suites in follow-up commits - drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup, serviceexposure) from _lib/ relocation; ported as Chainsaw suites in follow-up commits - docs/agents/e2e-testing.md: merge main's plan-job TIA wording with the Chainsaw suite phrasing Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-07-08 18:04:08 +05:00
Myasnikov DaniilandClaude Opus 4.8	902cbe2d74	ci: address CodeRabbit review on build-matrix workflow ... - scope create-github-app-token to the current repo with read-only contents permission (was org-wide, inheriting the app's blanket installation permissions); both token steps only read releases/assets - fail-fast the Talos image upload when the disk is missing (if-no-files-found: error) so drift fails in build-talos, not later as a confusing e2e download error - download the flux install script to a tempfile before executing it, so a failed curl is not masked by the piped `sudo bash` exiting 0 - fix the regular-PR artifact-source comment (Talos disk comes from build-talos, pr.patch from finalize) and the TIA scope wording in docs/agents/e2e-testing.md Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-25 17:51:38 +05:00
Myasnikov DaniilandClaude Opus 4.8	2940093102	test(e2e): capture per-failed-test crust-gather snapshots in Chainsaw harness ... The Chainsaw app suite runs via `chainsaw test` directly, bypassing hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot per failed BATS test. As a result a failing Chainsaw run uploaded a cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic for failures like the gateway/etcd/bucket/openbao ones on this branch. Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the kubernetes-latest/previous suites so the existing tenant snapshot from run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh already folds snapshots/ into the uploaded artifact. The e2e-testing guide and the sandbox Dockerfile comment are updated to record the convention. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-25 13:43:50 +05:00
Myasnikov DaniilandClaude Opus 4.8	c82b55c621	docs(agents): align E2E guide and references with the Chainsaw migration ... Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative asserts replace the until-get/wait backstops, Chainsaw cleanup replaces test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom, and the TIA/diagnostics sections describe suites + `make test-chainsaw CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it is now the standard. Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/ in the gemini styleguide and the nats/etcd backup READMEs. The backup-postgres.bats / backup-foundationdb.bats files cited by the postgres examples, the postgres README, and the foundationdb controller comment never existed -- drop the false 'CI e2e drives this' claims and describe those backup flows as the manual reference flows they are. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-25 13:43:50 +05:00
Myasnikov DaniilandClaude Opus 4.8	e22d84f2c8	test(e2e): migrate app suite from BATS to Chainsaw and wire into CI ... Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot (postgres + bucket) to all 18 app suites. - Declarative ports assert on status/conditions and concrete fields; the repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate collapses into a single polling assert with structured failure diffs and automatic events/describe/podLogs capture on failure. - Inherently imperative suites keep their logic in script steps: openbao init/unseal, kuberture split-horizon probes, vminstance, the gateway admission/impersonation cases, and kubernetes-latest/previous, which wrap the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim. - Gateway tests derive the tenant apex from the namespace host label at runtime, so they are host-independent. CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw binary is added to the e2e-sandbox image. install-cozystack and test-openapi stay BATS. The per-app 3-retry loop is dropped, since assertion polling replaces the fixed-timeout flakiness it papered over. Verified against a dev cluster (DB/app, storage and VM suites pass). Service port assertions use the ports[*].port projection form because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal port filters. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-25 13:42:34 +05:00
Myasnikov Daniil	fa49dbaa33	Merge origin/main into refactor/build-matrix-2937 ... Reconcile the parallel-matrix PR build (#2983) with main, which since this branch diverged landed a competing single-VM build restructure (#2939) and test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build supersedes #2939: - .github/workflows/pull-requests.yaml: keep #2983's plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact interface (pr-patch + talos-image) is identical on both sides, so e2e depends on finalize/build-talos with no input rewiring. main's top-level permissions and SHA-pinned action refs applied across all jobs. - All other files merged cleanly (only pull-requests.yaml conflicted). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-25 13:10:36 +05:00
myasnikovdaniil	a036cfd1e9	ci(tests): remove redundant Release E2E workflow (#2978) ... ## What this PR does Removes the `Release E2E` workflow (`.github/workflows/release-e2e.yaml`), which ran the full bats suite on every release tag. **Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm — its own header states *"a failure here is an alarm, not a release blocker."* The coverage it provided is already delivered by the release-cut PR: the "Prepare release" commit bakes image digests into `packages/core/` (installer, platform, testing) and other packages, and `select-e2e.sh`'s full-suite pattern escalates any `packages/core/` change to the **full** suite. So the release PR already runs the whole suite against the commit the tag will point at. **Why now.** The workflow has never been green since it was added (2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed at the build step. Even after #2973 fixed the registry login earlier today, the next run still failed at *Build Talos image*. A perpetually-red, non-gating duplicate is pure noise. **Changes** - Delete `.github/workflows/release-e2e.yaml`. - Drop `release-e2e` from the full-suite pattern in `hack/select-e2e.sh`. - Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats` to `pull-requests.yaml` (the remaining E2E workflow), preserving "workflow edit → full suite" coverage. All 11 `select-e2e_test.bats` cases pass. - Update `docs/agents/e2e-testing.md` to describe the release-cut PR as where the full suite runs. Refs #2397. ### Release note ```release-note ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Removed automated release end-to-end testing workflow * Updated E2E test selection logic to reflect testing infrastructure changes * **Documentation** * Updated E2E testing procedures documentation to reflect current testing approach <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-06-25 10:26:29 +05:00
Aleksei Sviridkin	a1b88cfa6e	docs(tests): drop the merged k8s-version pin from the in-flight e2e list ... The "Pin the e2e management cluster Kubernetes version" item is no longer in-flight — it is merged and live in hack/e2e-prepare-cluster.bats (--kubernetes-version v1.33.12). Remove it from the "not yet the merged standard" section; its rationale lives in the bats comment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-06-23 05:51:07 +03:00
Myasnikov DaniilandClaude Opus 4.8	35d8024b8e	docs(agents): update e2e-testing for the matrix build pipeline ... The "What TIA does and does not do" section described the old single Build job and the `detect-changes` paths-filter. Update it for the new `plan` job (docs gate + build matrix), note that image builds are now diff-scoped by hack/build-matrix.sh (distinct from TIA's app-loop trim), and point the "widen the skip" advice at the plan docs-gate. Refs #2937 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-22 20:08:52 +05:00
Myasnikov DaniilandClaude Opus 4.8	b81e92ac9c	ci(tests): remove redundant Release E2E workflow ... The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on every release tag as a non-blocking alarm — its own header notes "a failure here is an alarm, not a release blocker." It duplicates coverage the release-cut PR already provides: the "Prepare release" commit bakes image digests into `packages/core/` (and other packages), which TIA escalates to the full suite, so the release PR exercises the whole suite against the commit the tag will point at. It has also never been green since it was added (2026-05-02): every tag run failed at the build step. Even after #2973 fixed the registry login today, the next run still failed (Build Talos image). Removing a perpetually-red, non-gating duplicate removes noise without losing coverage. Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s full-suite pattern, repoints its unit test to `pull-requests.yaml`, and updates the e2e-testing doc. Refs #2397 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-22 18:05:53 +05:00
Aleksei Sviridkin	a8c3a57fa2	docs(agents): consolidate build, test, and verification guidance (#2933) ... ## What this PR does Promotes universal project guidance that previously lived only in a local (gitignored) `CLAUDE.md` into the committed agent docs, so every contributor and every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh clone. - `overview.md`: Build & Development Commands (root + per-package make targets, build env vars), Values schema generation, a Testing section, Domain references - `contributing.md`: a "Verify Before Considering Done" matrix + Review Expectations - Dedup: `overview.md`'s Git Commits block now points to `contributing.md` > Note: `overview.md` links to `docs/agents/e2e-testing.md` added in #2932 — merge > that one first or together. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Expanded contribution guidance with a “Verify Before Considering Done” pre-submission checklist and clearer expectations for including behavioral regression tests in bug-fix PRs. * Broadened build/development documentation with a “Build and Development Commands” guide, including required environment variables, values-schema generation, and test entry points. * Updated contribution workflow notes by consolidating commit-convention guidance and adding pointers to task-specific design references. * Added a Claude entry-point document that directs to the main agents guide and explains local override placement. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-06-22 11:50:14 +03:00
Myasnikov DaniilandClaude Opus 4.8	be716a7e0e	docs(agents): drop broken dashboard doc reference ... Address review feedback from coderabbitai on docs/agents/overview.md:145: internal/controller/dashboard/README.md does not exist (no such directory), and there is no dashboard reference doc to point to. Remove the dead bullet; the backups DESIGN.md reference is valid and kept. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-19 11:22:14 +05:00
Myasnikov DaniilandClaude Opus 4.8	680e3cc47b	docs(agents): clarify PR description template wording ... Address review feedback from gemini-code-assist on docs/agents/contributing.md:145: reword the awkward "on the template" phrasing to "aligned with the template". Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-19 11:22:02 +05:00
Myasnikov DaniilandClaude Opus 4.8	71d76ec3d3	docs(agents): distinguish make helm-unit-tests from full suite ... Address review feedback from gemini-code-assist on docs/agents/overview.md:103: the Helm-only target is make helm-unit-tests; make unit-tests runs the full unit suite. Point the bullet at the precise target and note the difference. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-19 11:21:51 +05:00
Myasnikov DaniilandClaude Opus 4.8	aaa2688b68	docs(agents): correct make unit-tests description in overview ... Address review feedback from gemini-code-assist on docs/agents/overview.md:72: make unit-tests runs the full suite (Helm, BATS, Go, presets, readiness), not only Helm unit tests. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-19 11:21:40 +05:00
Myasnikov DaniilandClaude Opus 4.8	943d5932c0	docs(agents): consolidate build, test, and verification guidance ... Promote the universal project guidance that previously lived only in a local CLAUDE.md into the committed agent docs, so every contributor and every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh clone. overview.md gains Build and Development Commands (root + per-package make targets, build env vars), Values schema generation, a Testing section, and Domain references. contributing.md gains a "Verify Before Considering Done" matrix and Review Expectations. Dedup: overview's Git Commits block now points to contributing.md instead of restating it. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-16 17:12:34 +05:00
Myasnikov DaniilandClaude Opus 4.8	d522c5b5c5	docs(agents): add E2E and CI testing conventions ... Capture the e2e stabilization conventions established over the last few months as a routed agent doc: no retries on deterministic CI steps, event-driven backstops over fixed sleeps, no EXIT/RETURN traps, fail-fast HelmRelease-readiness gating, the status.history remediation guard, first-class failure diagnostics, and a reviewer checklist. Also documents Test-Impact Analysis (TIA) and its real scope — it trims only the per-app loop, runs after install, and most PRs escalate to the full suite. Wire it into AGENTS.md so every agent reads it before e2e/CI work. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-06-16 17:12:16 +05:00
Myasnikov Daniil	39605f213f	docs(agents,ci): finish "tag-not-HEAD" sweep and fix changelog-exists gate ... - tags.yaml: the check_changelog step now looks at origin/main rather than the working tree. After Round 2 we check out the release tag, which never has the changelog file (the PR opened by this job merges to main after the tag is cut), so the previous `[ -f ]` test always returned false and the gate was effectively dead on reruns. - changelog.md: Step 6's `RELEASE_END=$(git log -1 --format=%ai HEAD)` (lines 179, 222) was missed in the previous "never HEAD" sweep — now uses v<new_version>, matching Step 5. - changelog.md: Step 2 ("on main branch") was incompatible with the CI flow which now runs detached at the release tag. Relaxed it to list both configurations (interactive on main; CI detached at v<version>), while the "Scope and boundaries" section continues to forbid the agent from switching branches. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-04-30 15:01:48 +05:00
Myasnikov Daniil	ce63bfcdb0	docs(agents): fix new-contributor detection and tags.yaml/doc drift ... - tags.yaml: drop "(or HEAD)" from the changelog prompt; the doc says "never HEAD" and the workflow now matches. - docs/agents/changelog.md: exclude the current release file from the "previous contributors" set used for first-time-contributor detection. The previous globbing on `docs/changelogs/v*.md` swept up the file being generated, so `comm -23` would never report any newcomers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-04-30 13:37:24 +05:00
Myasnikov Daniil	0f9198d510	docs(agents): address changelog rule review feedback ... - Drop the "orphan backport, drop entry" rule: a backport whose original is on main but hasn't shipped in a stable tag is the first delivery and must stay in the changelog. - Add `v` prefix to `<previous_version>`/`<new_version>` placeholders in all four `git log` invocations for consistency with the rest of the doc. - Apply the bot/CI account filter to the new-contributors detection (and the standalone previous-contributors extraction) so bot accounts can no longer leak into the human Contributors list. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-04-30 10:32:54 +05:00
Myasnikov Daniil	2420d8949b	fix(ci): generate changelog from the release tag, not from main ... The generate-changelog job in .github/workflows/tags.yaml previously checked out main and ran the AI agent there. The agent followed docs/agents/changelog.md, which instructed it to compute the release range with `git log <previous_version>..HEAD`. For minor releases (vX.Y.0) cut from main this happens to give the right answer; for patch releases (vX.Y.Z, Z>0) cut from a release-X.Y branch, HEAD-on- main is a strict superset of the release tag and contains commits that were merged to main both before and after the tag. The v1.3.1 changelog generated by this workflow (#2480) demonstrated the failure mode: it included 8 PRs merged to main but never shipped in v1.3.1, 6 backport PRs that landed on release-1.3 *after* v1.3.1 was tagged, both originals and their backports as separate entries, a hallucinated 2024 PR, and the cozystack-ci bot in the contributors list. The corrected v1.3.1 changelog contains the single fix that v1.3.1 actually ships. This PR fixes the root cause and tightens the agent guardrails: * tags.yaml: check out the release tag commit (`ref: ${{ steps.tag. outputs.tag }}`) instead of `main`, so HEAD == the release commit and `git log v<prev>..HEAD` corresponds to the actual release. The Create-changelog-branch step still creates the PR branch from origin/main, so PRs continue to merge cleanly into main. * tags.yaml: update the AI prompt to state explicitly that HEAD is the release commit and that the upper bound for the release range is the new tag, not main. * docs/agents/changelog.md: replace `..HEAD` with `..v<new_version>` in every example so the instruction is unambiguous regardless of which branch is checked out. * docs/agents/changelog.md: add a hard rule that backport PRs MUST be merged into the original entry (single line, two PR numbers), never listed as a second entry. Document the edge case where the original PR is outside the range — drop the entry. * docs/agents/changelog.md: forbid using the brief description verbatim as the detailed description, and forbid inventing entries for PRs that are not in the release range. * docs/agents/changelog.md: filter bot/CI accounts (`app/*`, `*[bot]`, cozystack-ci, github-actions, dependabot, renovate) out of the human Contributors list. Bot attribution on individual entries is still valid. The corresponding correction for the v1.3.1 changelog itself is in PR #2480 on the changelog-v1.3.1 branch. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>	2026-04-29 12:22:34 +05:00
Aleksei Sviridkin	bc1eca10cb	chore(ci): adopt CNCF/k8s label conventions (#2495) ... ## What this PR does Adopt CNCF/Kubernetes label conventions for issues and PRs and add automated labeling. **Canonical label file**: `.github/labels.yml`. Synced into the repository by `.github/workflows/labels.yaml` (EndBug/label-sync@v2) on push to `main`, weekly cron, and manual dispatch. UI-only label edits are overwritten — propose changes via PR to this file. ### Label namespaces Following the [Kubernetes label scheme](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md): - `kind/*` — issue or PR type (bug, feature, documentation, support, cleanup, regression, flake, failing-test, api-change, breaking-change) - `priority/*` — urgency (critical-urgent, important-soon, important-longterm, backlog) - `triage/*` — review state (needs-triage, accepted, needs-information, not-reproducible, duplicate, unresolved) - `lifecycle/*` — issue/PR lifecycle (active, frozen, stale, rotten) - `area/*` — subsystem; 15 seeded plus `area/uncategorized` fallback. Extensible — propose a new area when no existing one fits. - `do-not-merge/*` — PR merge blockers (work-in-progress, hold) Cozystack-specific labels preserved: `epic`, `community`, `help wanted`, `good first issue`, `quality-of-life`, `upstream-issue`, `backport`, `backport-previous`, `release`, `automated`, `debug`, `sponsored`, `lgtm`, `ok-to-test`, `security/*`, `size:*`. ### `area/*` set 15 areas seeded by activity in open issues and PRs: `area/ai`, `area/api`, `area/build`, `area/ci`, `area/dashboard`, `area/database`, `area/extra`, `area/kubernetes`, `area/monitoring`, `area/networking`, `area/platform`, `area/release`, `area/storage`, `area/testing`, `area/virtualization`. Plus `area/uncategorized` as the auto-labeler fallback. ### Migration safety Existing labels are renamed via `aliases:` in `labels.yml`. GitHub preserves the label ID, so all currently tagged issues and PRs keep their tags under the new name without losing references: | Old | New | |---|---| | `bug` | `kind/bug` | | `enhancement` | `kind/feature` | | `documentation` | `kind/documentation` | | `question` | `kind/support` | | `frozen` | `lifecycle/frozen` | | `stale` | `lifecycle/stale` | | `duplicate` | `triage/duplicate` | | `do-not-merge` | `do-not-merge/work-in-progress` | | `do not merge` | `do-not-merge/work-in-progress` | `delete-other-labels: false` on the initial rollout. Generic GitHub-default labels (`wontfix`, `invalid`) are preserved untouched and will be removed in a follow-up cleanup PR. EndBug processes aliases sequentially, so the second of the two `do-not-merge*` aliases hits a name collision and logs a warning — the legacy label survives that one sync and is cleaned up in the same follow-up. ### PR auto-labeling `.github/workflows/pr-labeler.yaml` parses each PR title on `opened`, `edited`, `reopened`, and `synchronize` and applies labels additively (never removes): - **type → `kind/*`**: feat, fix, docs, chore, refactor (others get no kind) - **scope → `area/*`**: scope mapping covers all current cozystack components (full table in `docs/agents/contributing.md`) - **`!` after type or `BREAKING CHANGE:` footer**: applies `kind/breaking-change` - **`[Backport release-1.x]` prefix**: stripped before parsing; `area/release` and `backport` labels added - **Composite scope** (`feat(platform, system, apps): …`): each part mapped independently - **Bracket fallback** (`[scope] description`): maps `area/*` but cannot infer `kind/*` - **Unmapped scope or non-conventional title**: applies `area/uncategorized` for human review ### Schema validation `.github/workflows/labels.yaml` runs a `validate` job on every PR touching `labels.yml` or its workflow. Asserts: - description ≤ 100 chars (GitHub REST API limit) - color is 6-char hex without leading `#` - unique top-level names - aliases do not collide with top-level names Sync runs only on push to main, weekly cron, and manual dispatch; PR runs validate-only. ### Hardcoded label/title references updated - `.github/ISSUE_TEMPLATE/bug_report.md`: `labels: 'bug'` → `labels: 'kind/bug'` - `.github/workflows/tags.yaml`: - changelog PR labels `['documentation', 'automated']` → `['kind/documentation', 'automated']` - release PR title `Release v${version}` → `chore(release): cut v${version}` (so the auto-labeler applies `kind/cleanup` + `area/release`) - changelog PR title `docs: add changelog for v${version}` → `docs(release): add changelog for v${version}` (so the auto-labeler applies `kind/documentation` + `area/release`) ### Documentation - `AGENTS.md`: Activation entry pointing agents to `labels.yml` and the PR title auto-labeling rules. States explicitly that `area/*` accuracy outweighs reuse — propose a new area when none fits, do not shoehorn into a wrong one. - `docs/agents/contributing.md`: PR Title Auto-Labeling section with type→kind and scope→area tables. ### Out of scope (follow-up PRs) - Removal of redundant labels (`wontfix`, `invalid`, plus the surviving legacy `do not merge` if alias-rename collision keeps it) - Org-wide sync from `cozystack/.github/labels.yml` - Dosu bot configuration update for `lifecycle/stale` (requires dashboard access) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Automated PR labeling from Conventional Commits (type → kind/*, scope → area/*), plus a comprehensive namespaced label taxonomy. * **Chores** * Workflows to validate, sync, and auto-apply labels (including scheduled/manual runs and validation checks). * Added repository-wide label configuration and normalized bug label metadata to namespaced form. * Updated release PR titling/labeling conventions. * **Documentation** * Contributor and agent guidance on PR title conventions, label mappings, and triage procedures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-04-27 13:16:12 +03:00
Aleksei Sviridkin	f527ce683b	docs(agents): clarify that Scopes list is illustrative ... The Scopes section was read as an exhaustive enumeration, which led to review feedback flagging any scope outside the list as invalid. The intent has always been that contributors pick the most specific scope for the change and extend the list when a genuinely new area appears. Reword the section accordingly and add operator as an example scope. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-04-27 11:20:12 +03:00
Aleksei Sviridkin	d86bc7760a	docs(agents): document PackageSource upgradeCRDs field ... Describe when to set upgradeCRDs: CreateReplace (operators that evolve their CRD set additively between versions) and the data-loss risk of enabling it on operators that drop fields. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-04-27 11:19:30 +03:00
Aleksei Sviridkin	738762994e	docs(agents): fix markdown table cell spacing in contributing.md ... Address review feedback from gemini-code-assist on docs/agents/contributing.md:73: add space before closing pipe in the type to kind mapping table for consistency with other rows. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-04-27 03:30:52 +03:00
Aleksei Sviridkin	31f4435eb0	docs(agents): replace Unicode ellipsis with ASCII in contributing.md ... Address review feedback from gemini-code-assist on docs/agents/contributing.md:88: … replaced with ... for compatibility across editors and tools. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-04-27 03:30:29 +03:00
Aleksei Sviridkin	c8ed1c652c	chore(ci): adopt CNCF/k8s label conventions ... Add .github/labels.yml as the canonical label set, synced into the repository by .github/workflows/labels.yaml using EndBug/label-sync. Conventions follow the Kubernetes scheme: https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/, do-not-merge/. Cozystack-specific labels preserved (epic, community, security/*, size:*). Migration via aliases keeps references on existing issues and PRs: - bug -> kind/bug - enhancement -> kind/feature - documentation -> kind/documentation - question -> kind/support - frozen -> lifecycle/frozen - stale -> lifecycle/stale - do-not-merge -> do-not-merge/work-in-progress delete-other-labels is false on the initial rollout; redundant labels ("do not merge", duplicate, invalid, wontfix) stay until a follow-up PR removes them after stabilisation. The labels workflow has a validate job (python3 schema check) that runs on PR. Sync runs only on push to main, weekly cron, and manual dispatch. Schema invariants: - description <= 100 chars (GitHub REST API limit) - color is 6-char hex without leading # - unique top-level names - aliases do not collide with top-level names PR auto-labeling (.github/workflows/pr-labeler.yaml): - Parses PR title as Conventional Commits header (type, scope, !). - type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs -> kind/documentation, chore/refactor -> kind/cleanup; style, perf, test, build, ci, revert -> no kind label). - scope -> area/* via embedded mapping; composite scopes split on comma. Bracket-style fallback ([scope] description) maps area/* but cannot infer kind/*. - '[Backport release-1.x]' prefix is stripped; area/release and backport labels are added. - '!' after type or 'BREAKING CHANGE:' footer in body adds kind/breaking-change. - Unmapped scope or non-conventional title adds area/uncategorized to flag for human review. - Additive only — never removes existing labels. Hardcoded label references updated: - .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug) - .github/workflows/tags.yaml (documentation -> kind/documentation) AGENTS.md gains an Activation entry pointing agents to labels.yml as the source of truth and to contributing.md for the title auto-labeling table. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-04-27 03:15:09 +03:00
Aleksei Sviridkin	4813566a30	docs(agents): mark scopes list as illustrative examples ... Address review feedback from gemini-code-assist on docs/agents/contributing.md:11: Scope linters kept flagging valid scopes like 'agents' as unknown because the list read as exhaustive. Annotate it as examples (not exhaustive) and add 'agents' to the Other group so both humans and review bots stop tripping on scopes that are already in regular use across the repo history. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-04-23 18:02:04 +03:00
Aleksei Sviridkin	41fd80711b	docs(agents): fix grammar in regen discovery hint ... Address review feedback from gemini-code-assist on docs/agents/contributing.md:30: Reword "likely to need regenerated" (regional construction) to "likely needs to be regenerated" for standard technical prose. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>	2026-04-23 18:00:07 +03:00