mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
905249fb39c19c34c8dce74d528c0cef700a91d7
298
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
8e273a7341 |
docs(ci): record what escalating on the non-PR workflows costs
Three of the four workflows on the escalation list cannot be exercised by the run they trigger. `e2e-fork` runs on `workflow_run`, which always takes the default-branch copy; `e2e-tag` runs on `workflow_call` from a tag push; `nightly` runs on a schedule. Only `pull-requests.yaml` is executed from the pull request's own head, so only there does escalating test the edit rather than the tree around it. That argument existed in the reference doc and was removed by this branch rather than answered, and the list it argued against is what replaced it. Measured over six months of main, 26 commits touch one of the three with no other escalating path; each now pays a full Chainsaw run it used to skip, on a suite whose flake rate is the reason people avoid running it for nothing. Keeping them is defensible: the rule stays one idea rather than two, and dropping them leaves those files inert, which reads as an oversight rather than a decision. What was not defensible was paying it silently. The trade now sits next to the pattern that makes it, with the number, so the next person weighing it starts from measurement instead of rediscovering the mechanism. The reference doc points there rather than carrying its own copy. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
55f3c79e9f |
docs(ci): stop pointing at the inert list for paths it does not hold
Four places said an empty selection means the inert pattern matched, and sent the reader to `inert_config_pattern` to see which paths those are. Documentation, dashboards and `*.md` are dropped before that pattern is ever consulted, and a backups example whose app has no suite is dropped where backup examples are matched. A reader checking the pattern for any of them finds nothing and has to reconstruct the order by reading the script. Name the rules that produce an empty selection instead, and keep the claim to the one the lanes rely on: empty is a decision some rule reached, not a path nothing looked at. The same bullet claimed every path is classified. One is not. The inert pattern makes `.github/` inert as a whole directory and the workflows that run the suite are exempted from it by name, so a workflow added under `.github/workflows/` reads as inert and the fall-through never sees it. Today's enumeration is correct and I checked it against `rg -l test-chainsaw`, so this is not a live hole, but the sentence saying it cannot exist is what makes the reminder next to that list look decorative. Record it as the guard it is. Separately, the fork lane's rationale for appending a newline described a mechanism this branch removes: the reader now guards its own loop, so the terminator is no longer what keeps the last path alive. Say that it is now the text-file contract and a second line of defence across two files, which is a reason to keep it and a different reason than the one written there. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
ae4e9e55cb |
fix(ci): classify every TIA path and escalate the unclassified
`select-e2e.sh` silently selected nothing for a changed path it did not recognise outside `packages/`. Both e2e lanes read an empty selection as "skip Chainsaw" and then post the required "E2E Tests" status green, so an unrecognised path produced a green merge gate with no suite run. The worked example is `hack/common-envs.mk`, which supplies the tag, push and output flags of every image in the tree and matched nothing: `full_suite_pattern` covered `hack/*.sh` and `hack/*.bats` but not `hack/*.mk`. `pkg/`, `tools/`, `go.mod`, `hack/lib/` and `hack/buildkitd.toml` were in the same position, as was `.github/workflows/e2e-fork.yaml` while `pull-requests.yaml` was already escalated. A blanket "empty means full suite" fallback would be the wrong fix: it escalates the inert paths the design deliberately skips. Instead every path now lands in one of three classes by an explicit rule — the full-suite list for shared build inputs and the e2e harness, an inert list for paths that cannot affect what e2e exercises, and the PackageSource graph for `packages/(apps|system|extra)/`. A path matching none of them escalates and logs why, so the next path someone adds fails safe instead of failing open. The two lists are checked in that order, so a specific escalation beats a broad inert directory: `.github/` is inert, the workflows that run the suite are not. `*.md` and `docs/`/`dashboards/` stay ahead of both, so a README under `packages/core/` does not inherit that tree's escalation. Ten tests cover both sides of the classification. Mutation-checked: reverting the fall-through fails the unclassified-path test, dropping `.github/` from the inert list fails the non-e2e-workflow test, and swapping the two list checks fails the existing pull-requests.yaml test. Note the per-entry escalations are belt-and-braces — removing `hack/*.mk` from the pattern still escalates via the fall-through, which is the point of having it. Closes #3392 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
7dca642894 |
fix(ci): restore e2e for fork PRs and close the merge-gate bypass (#3262)
## What this PR does Fixes #3257 — fork PRs never ran e2e, and the skipped required check let them merge anyway. Fork PRs get no secrets on `pull_request`, so `make image` pushed anonymously and died with `denied`. That failed every build, collapsed the `build → finalize → e2e` chain, and left the required **E2E Tests** job *skipped* — which branch protection treats as satisfied, so an approved fork PR could merge with e2e never having run. This implements **Option A** from the issue (privileged `workflow_run` split), pushing fork e2e images to the existing CI registry (OCIR): **Build primitive (`hack/common-envs.mk`)** - New `OCI_EXPORT_DIR` mode: build each image to a per-image OCI archive instead of pushing, and force `PUSH`/`LOAD` off. The digest is captured via `--metadata-file` regardless of output type, so the refs baked into `pr.patch` match what the privileged run later pushes. Threaded through the `image-tags` macro every package already uses — no per-package Makefile change. **`pull-requests.yaml` (unprivileged, fork branch)** - `build` / `build-talos` / `finalize` export images to OCI archives and upload them as artifacts instead of pushing → the red `denied` build wall disappears and the chain no longer collapses. - The in-tree e2e job is renamed off `"E2E Tests"` and guarded to same-repo PRs. A new `e2e-report` job concludes the required `"E2E Tests"` context as an **explicit check-run** (success on a docs-only PR or when e2e passed, failure otherwise). A skipped job can no longer satisfy the required check — this also closes the same hole for same-repo PRs whose build fails. **`e2e-fork.yaml` (new, privileged `workflow_run` from the default branch)** - Resolves the fork PR, opens a pending `"E2E Tests"` check-run, and fails-closed if the fork build failed. - Pushes the pre-built image archives to OCIR **by digest** and flux-pushes the digest-pinned packages tree. The OCIR token is live only for trusted binaries (`git`/`skopeo`/`flux`/`yq`) over data — never for `make` or any fork-authored script. - Runs the e2e suite against the pushed closure with **no push token** and concludes the check-run on the PR head SHA. Fork build code never receives registry credentials; the fork's images and test scripts run only in the credential-less, ephemeral e2e job — a stronger posture than a `pull_request_target` run that would hand fork code the secrets directly. ### Validation Local: buildx OCI-archive output + digest capture, macro expansion for single- and multi-image packages, non-fork push path unchanged, `actionlint` clean. Needs live validation (why this is a **draft**): `workflow_run` artifact download across runs; that the `"E2E Tests"` commit status satisfies the required context and keeps a fork PR non-mergeable until the privileged run concludes it; `refs/pull/<N>/merge` fetch + `git apply --3way`; `skopeo --preserve-digests` end-to-end. `workflow_run` only runs from the default branch, so the privileged half is exercised by mirroring the workflow onto a personal fork's default branch and opening a throwaway fork PR into it — not by merging first. _Since review_: the fork `e2e` job ran the pre-#2826 BATS app loop that the merged Chainsaw migration deleted (already broken); it now runs the same Chainsaw + TIA path as the in-tree job, with suite selection and the `full-e2e` override sourced from `resolve`'s trusted base-repo file list / labels (not a merge-ref diff). Publish job hardened per the non-blocking follow-ups: base-tree image allowlist, `skopeo copy --all`, pinned flux CLI. The required `E2E Tests` gate is now published as a commit **status** (default token + `statuses:write`), not a COZYSTACK_CI app check-run — the app lacked `checks:write` (`checks.create` 403'd) and a default-token check-run floated under a labeler suite; a commit status has neither problem, and forks can't post one (their `pull_request` token is read-only). Two review-bot findings fixed too: fork `OCI_EXPORT_DIR` is now absolute (a relative path misplaced archives under `make -C <pkg>` and every fork build would fail at upload), and the "≥1 archive" guard no longer no-ops under `nullglob`. ### Repo-admin prerequisites One repository setting lives outside this diff: 1. **Branch protection: keep `E2E Tests` required.** The gate is now a commit **status** named `E2E Tests` (posted with the default `GITHUB_TOKEN` — see `docs/agents/e2e-testing.md` §10), not a job or an app check-run. Branch protection already requires `E2E Tests` (currently the e2e job's name, renamed here to `E2E (in-tree)`), so the required context now resolves to the status — **no GitHub App, no `checks:write` grant, no app-id pin needed**. Just don't pin the context to a specific app. 2. **Actions → "Require approval for all outside collaborators".** Each fork run must be a deliberate maintainer click before it executes, since the fork's unprivileged run produces the artifacts the privileged `e2e-fork.yaml` consumes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (7 files: two CI workflows, `docs/agents/e2e-testing.md`, `hack/common-envs.mk` + its bats suite, and two package `Makefile`s). The one trigger worth naming is "change developer tooling (`cozyvalues-gen`, `cozypkg`, the package `Makefile`s) → `development.md`" on the website. It does not fire here: `OCI_EXPORT_DIR` is a CI-only build mode that defaults off, and the local developer workflow is unchanged — `make image` still builds and pushes exactly as before, which the new default-path case in `hack/common-envs_test.bats` now pins. No chart values, no API surface, no app list, no platform variant, no Talos version bump and no release asset rename are touched. - [x] No downstream repository is affected by this change ### Release note ```release-note fix(ci): fork pull requests now run e2e via a privileged workflow, and a fork PR can no longer merge on a skipped (never-run) required E2E Tests check ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a dedicated fork E2E workflow that runs after the base PR and reports results via the required **E2E Tests** commit status. - Fork PRs now export build images as OCI archive artifacts for later publishing. - Added a same-repo E2E reporting job and renamed the in-tree E2E flow to **E2E (in-tree)**. - **Bug Fixes** - Added tighter, fail-closed gating to ensure merge blocking when **E2E Tests** isn’t correctly finalized. - **Documentation** - Documented fork vs same-repo E2E split, OCI export behavior, and required fork security settings. - **Tests** - Added Bats coverage validating `OCI_EXPORT_DIR` and preventing unintended registry pushes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
806b66f25c |
docs(ci): stop describing a one-workflow E2E CI
Two sentences predate the second workflow and were left describing the world before it. The scope line at the top is the pointer AGENTS.md routes agents through before they touch the E2E CI, and it named only the in-tree workflow. A reader sent there to change the fork lane would not learn that the section holding its invariants exists, which is the one thing that line has to do. It now names both workflows and says which pull requests each serves. The TIA escalation list said edits to "the E2E workflows" escalate to the full suite. That was true while there was one. The selector's list names `pull-requests.yaml` and not `e2e-fork.yaml`, so a change touching only the fork workflow selects nothing: the Chainsaw step is skipped, while the platform install and the OpenAPI tests still run. Say that plainly, and say why it is tolerable rather than pretending it is designed. A `workflow_run` always executes the default-branch copy of the file, so running the suite against a change to it would not exercise that change anyway. It is the same shape as the fail-open the section already cites. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
cc8b01d5cd |
docs(ci): say when each lane's status opener actually lands
The reference section claimed both lanes open the required context when their run starts, and enumerated the triggers it protects against: a re-run, a reopen, a `full-e2e` label. Every one of those starts a Pull Request run, and on a fork that run cannot open the status at all. The opener is guarded same-repo only, because a fork's `pull_request` token is read-only, and the fork's opener lives in the privileged workflow, which starts only once the unprivileged build has finished. So on the lane the sentence was most likely to be read about, an already-green head SHA keeps reading green for the whole build. The gap is structural. A read-only token cannot post a status at any point in the run, and that same read-only token is what makes publishing the gate with the default token safe in the first place. It is bounded by the build rather than open-ended. What was wrong was only the claim that it does not exist. Say so in all three places that carried the overstatement: the section itself, the opener's own comment, and the contract test, which pins that each lane has an opener and cannot express when it lands. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9c68b1f83e |
fix(ci): stop a cancelled fork run from publishing a verdict
The invariant that a cancelled run publishes nothing was written down in three places and held in two of them. The fork lane's terminal reporter ran under `always()`, which includes cancellation. When a second privileged run supersedes the first, the dying one still posted `failure` on a head SHA it no longer owned, for a suite that never finished. It self-corrected once the superseding run reported, but that can be two hours later, and until then a legitimate pull request reads red. The same-repo lane was moved off `always()` for exactly this reason. Move this one too: a failed publish or e2e still reports, since neither is a cancellation, and a cancelled run now leaves the `pending` it opened standing until its successor concludes. The same-repo guard's rationale cited a scenario that no longer exists. It described a label event cancelling the run that opened the pull request, which the concurrency split made impossible for every label the workflow discards. The guard is still needed, for a push, for a `full-e2e` label superseding the head run, and for a manual cancel, so the comment now names those instead of an extinct one. The reference section listed the pieces this invariant rests on and omitted the reporters, which is the half a reader is most likely to touch, and counted them, which was already wrong and would go wrong again on the next addition. It now enumerates without counting. The contract pins both reporters, because both lanes have had this defect at different times in the same shape. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8e097488c2 |
docs(ci): name every writer of the "E2E Tests" status, and why it opens
The section that tells the next editor to keep exactly one run per head SHA responsible for the status listed only two of the three jobs that write it, and described the same-repo lane as posting once at the end. Both stopped being true when the same-repo lane gained an opener. The section also spent a paragraph on the concurrency invariants without saying why an opener has to exist at all, which is the part a reader would otherwise remove as redundant. Say that each lane opens with `pending` and concludes, that a status never expires so the opener is what stops a re-run inheriting the previous run's green, and that `plan`'s opener is `continue-on-error` because every build and test job hangs off that job. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
fd06fb32e0 |
docs(ci): say what fork TIA hardening buys, and what it does not
Four comments and a documentation bullet claimed that routing the changed-file list through the base-repo API stops a fork from making test selection pick nothing. It does not. The `e2e` job runs from the fork's own tree, so `hack/select-e2e.sh` is the fork's script: a PR that edits it to print nothing gets `skip=true`, the suite step never runs, the job succeeds and the required status goes green — the exact outcome the paragraph presented as prevented. The same holds for the suites under `hack/e2e-chainsaw/`, the install bats and the testing package's recipes, all of which the fork authors and this job executes. Hardening the inputs is still worth doing, since a merge-ref diff would be a second bypass on top of the one that exists, so the code does not change. What changes is the claim: the inputs are trusted, the decision is not, and the fork lane is a correctness gate against an honest contributor's regression rather than a boundary against a hostile fork. State plainly what does contain a hostile fork — the maintainer approval required before any fork run executes, the absence of any credential in that job, and a diff editing the harness being visible in review. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
e2643ef4b9 |
docs(ci): explain the fork image allowlist and fix a test comment
Two statements that were not true of the code next to them. The fork `publish` job accepts only archive names it can derive from the base parent, so a fork cannot name an artifact into an arbitrary registry repository. The consequence is that a fork PR adding a new image fails on `image '<name>' is not in the base-tree allowlist` with no way to pass, and nothing said so — including the E2E section that AGENTS.md points a reader at when a fork run fails. Record the constraint, the exact log line, and the two ways around it, so the failure reads as the guard working rather than as a broken PR. The concurrency contract's comment on `code_lines` claimed a real grep error propagates out of the helper. It does under the bats binary, and not under `hack/cozytest.sh`, whose translator appends `return 0` to every line that is exactly `}` — file-level helpers included — so a helper's last command never decides its status there. Both runners still fail closed, because each test asserts its extraction is non-empty before comparing, so a swallowed error surfaces as empty input rather than as a pass. Say that instead. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7c5e96947e |
fix(ci): stop two runs from publishing "E2E Tests" on one SHA
Giving every `labeled` event its own concurrency group stopped a discarded label from cancelling a live run, but it also stopped a `full-e2e` label from doing so — and that label is not discarded. `plan` proceeds for it and `e2e-report` publishes with it, so the labelled run and the head run it was meant to widen both post the `E2E Tests` context, on the same head SHA, with no ordering between them. That was a deliberate trade at the time: commit statuses are latest-wins per context, so the reasoning went, the full-suite verdict lands last whenever its suite finishes last, and the label is an explicit opt-in to a longer run. This commit reverses that decision, because the premise only holds while the full run succeeds. The full suite is the superset, so a full-e2e run that fails early — a build flake, a suite failing at minute twenty — finishes before the narrower head run it was meant to widen, and that run's later green erases the failure. The required status then reads success for a SHA whose suite did not pass. One publisher per head SHA is worth more than the minutes of runner time a cancelled narrow run costs. Move aside only the label events `plan` discards, by making the concurrency condition the exact complement of `plan`'s guard. A `full-e2e` run stays in the main group and supersedes the head run the way a push does, so exactly one publisher of the context is alive per head SHA. Opting into the full suite now costs one suite instead of two, and a second label landing afterwards can no longer cancel the full-e2e run, because the two are no longer grouped together. The contract test gains the complement — the group key must exclude the publishing label rather than every label — and a check that the label name is one and the same string in all three guards that read it, since renaming it in one place splits the decision without touching a line that reads load-bearing. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
3c50a09457 |
fix(ci): keep a discarded run out of the fork e2e gate's group
The privileged fork e2e run groups by fork repo + branch with cancel-in-progress, so every completed Pull Request run for that branch lands in the same group -- including a run that built nothing. Labelling an open fork PR produces exactly that: pull-requests.yaml triggers on `labeled` and `plan` discards every label other than full-e2e, so all jobs skip. The privileged run for that event cancels the one that is mid-suite on the same head SHA, then `resolve` recognises the discarded trigger and posts no status at all. The required "E2E Tests" context is a commit status, so the head SHA is left at "Expected" with nothing left to flip it -- the run that replaced the owner is the very one that decided to stay silent -- and the PR stays unmergeable until the contributor pushes again. Key the group by the triggering run's id when that run concluded `cancelled` or `skipped`, so a run that will post nothing can cancel only itself. Real runs keep superseding each other on fork repo + branch. The contract test ties the conclusions in the key to the ones `resolve` stays silent for. Those two lists sit in different languages in different halves of the file, and a third conclusion added to the guard alone would restore the wedge without touching a line a reviewer reads as load-bearing. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7e2db4da7e |
fix(e2e): bound every remaining read in the report collector
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and the rest -- nodes, namespaces, events, HelmReleases, packages, packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads and every module gate -- read with none. The tarball is written on the collector's last line, so a single read that never returns does not truncate the report, it loses it, on the run where it is the only evidence left. Each read now goes through the shape that fits it: the existing bounded reader for a read whose output is a file, the existing selector for one that feeds a loop, and a new probe for the `if kubectl get deploy ...` gates. A gate is a read like any other, and an unbounded one is worse than the reads it guards, since the module below it never starts. Its cut-off is recorded, because skipping the module leaves the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. The reads that stay on the bare wrapper say so where they sit; what they share is that the output is not the reader's to write, going into a variable or through a pipeline that transforms it, so there is no moment at which a marker could attach. The header claimed a per-section list of what was bounded, and the tests froze that list plus a per-section count of the unbounded reads. Both are gone. The list had already gone stale, naming four sections while the file had ten, and the counts were an undercount by construction -- they matched a redirect to `$VAR/`, which no gate, selector or command substitution has. One test replaces both: no read below the library guard runs without a ceiling, checked by looking for a counterexample. A ceiling per read is not a ceiling on the report, and the header now says so where it used to imply the opposite. The pod, VM, VMI and services walks carry a count cap and a time budget; the walks over nodes, namespaces, HelmReleases, packages, PVCs and certificates carry neither, so their cost is the cluster's to set. What this removes is the read that never returns. The bounded reader now names the tool it ran and describes what that tool returned. It served kubectl alone until this change gave it talosctl and the four host commands, so a `df` that hit a stale mount left a marker reading "kubectl exited 1 ... not because the object ended" -- two false statements in one line, in an artifact whose premise is that its notes can be believed. The exec reader keeps the remote command's whole message in the listing it belongs to. `kubectl exec` appends its own `command terminated with exit code N` after whatever the command said, so quoting the last line there drops the cause and keeps the trailer, and sending the message away from the file leaves a zero-byte listing -- the "returned nothing, or never ran" question this collector exists to answer. Its cut-off note also stops asserting that the archive is beside it: a stream that got members across is kept and truncated, one killed before its first byte leaves zero bytes and the call site's emptiness check removes it, and the reader runs before that check, so it names both. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
920ad3e30d |
fix(e2e): bound the report reads that run inside a pod
The LINSTOR module runs six reads through `kubectl exec`: three listings, the error-report index, and the controller and per-satellite ErrorReport bundles. They are the only reads in the collector whose duration is set by a workload rather than by the apiserver -- a wedged controller, a satellite that stops answering, or a broken SPDY upgrade holds the stream open with no clock anywhere in the path. The tarball is written on the collector's last line, so one of those never returning does not truncate the report, it loses it. They go through a reader of their own rather than the existing one. `kubectl exec` returns the remote command's exit status, so a 124 there is either this collector's deadline expiring or a command that itself exited 124, and the status does not separate them; describing it the way every `get`, `logs` and `describe` is described would state a mechanism about the machine writing the report for something that may have happened inside a pod. The note says what is open instead of resolving it, and goes beside the output rather than into it, since two of the six stream a gzipped tarball that a line of prose turns into an archive nothing can open. kubectl's own message travels with the note for the same reason -- until now it was merged into the listings, so every run put the "Defaulted container" line at the top of a table. The guard that held the old assumption asserted only that no bounded read was an exec, which leaving every exec unbounded satisfies. It now also requires each exec to be bounded and requires the reader not to borrow the classification written for reads whose 124 is unambiguous. Matching on the verb rather than on the string `kubectl exec` is part of the fix: half of these calls spell it `kubectl -n cozy-linstor exec`, which that string never saw. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
96e33fe656 |
ci(e2e): give the report collection step a ceiling of its own
The step that runs hack/cozyreport.sh carried no timeout-minutes in any of the three workflows that run it, so it inherited only the job's 180. The collector writes its tarball on its last line and the upload is the next step, so a read that never returns spends what is left of the job budget, the job is cancelled, and the upload never runs -- the artifact is lost entirely rather than truncated, on exactly the run where it is the only evidence left. `|| true` guards against a non-zero exit, not against a read that does not return. 30 minutes is measured against the step rather than divided out of the job: it takes 3-42s across sampled runs, on failed e2e jobs as well as green ones, and 30 leaves five sixths of the job budget for the steps that upload what this one produced. A test pins the ceiling in all three workflows and requires it to stay under the job's own, since a step ceiling at or above the job's cap is the same as no ceiling at all. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2527e915bd |
fix(e2e): collect diagnostics for unready pods and record every truncated capture (#3567)
## What this PR does The e2e diagnostic tarball exists so a red run can be explained from the uploaded artifact alone. Several of its collectors answered "nothing is wrong" instead — some because they selected on a column that never carries the value they filtered for, others because they stopped early and left nothing saying they had. Both failure modes look identical to a healthy cluster from the outside, which is the worst possible shape for a diagnostic. Four selectors in `hack/cozyreport.sh` and `hack/cozyreport-summary.sh` were reading the wrong thing: - Per-pod evidence was chosen by the STATUS column, so a pod left at `0/1 Running` by a readiness probe that never passes was skipped. That is a common way for a suite to time out, and the artifact carried no trace of the pod the run died on. Selection now compares the two halves of the READY column; terminal `Succeeded`/`Completed` pods stay excluded. - The pending-LoadBalancer section filtered `$4`, which is CLUSTER-IP. CLUSTER-IP holds an address, `None` or `<none>` and never `<pending>`, so that loop had never executed. `<pending>` only ever appears in EXTERNAL-IP, `$5`. - The cozystack-apps section gated all three of its kinds on `kubectl get crd <kind>.apps.cozystack.io`. ApplicationDefinition is a CRD but in a different group, and Tenant comes from an aggregated APIService with no CRD to ask about, so the probe failed and the section never ran. - The kamaji section was gated on `cozy-linstor`'s `linstor-controller` deployment rather than kamaji's own, so a cluster without LINSTOR got no kamaji logs and no TenantControlPlanes — on exactly the runs that need them. The summary had three of its own: the HelmRelease listing printed `$5` where STATUS is the Ready condition's whole message, reducing every failed release to one word like "Helm"; the Flux sources listing used a jsonpath with a nested filter that kubectl rejects outright, so it printed nothing on every report ever taken; and pod AGE was read as `$6`, which shifts whenever RESTARTS prints the `7 (4m12s ago)` form — that is, for exactly the rows worth reading. The second half is bounding. The pod, VM, VMI and services walks now run under a count cap and a wall-clock budget, and every read is individually time-boxed. The point is not the bounds themselves but that each one names itself when it fires: a walk stopped by the cap or the clock leaves `COLLECTION-TRUNCATED.txt`, a pod list that never returned leaves `COLLECTION-FAILED.txt` instead of the empty tree a healthy cluster produces, a log cut short says so on its own last line, and a `--tail` bound is recorded beside the logs rather than inside them, since the platform's controllers log JSON per line and a trailing prose line costs a parser the whole file. A knob value the collector could not use is reported too, so a truncation note quoting `600` is never mistaken for the `10m` somebody actually set. Where `timeout` is absent the reads run unbounded rather than not at all, and the section says which of the two happened, because a missing local binary is not a statement about the cluster. One behaviour change worth stating plainly, because it is a trade and not a pure gain: the per-pod section used to be uncapped and read container logs whole, and it is now bounded by `COZY_REPORT_PODS_MAX` (40), `COZY_REPORT_PODS_BUDGET` (600s) and `COZY_REPORT_POD_TAIL` (2000). Below those thresholds the new tree is a strict superset of the old one — the old selector missed `Running 0/1` entirely and skipped logs for `Pending` pods. Above them the artifact is smaller than before, every drop is named in the tree, and `COZY_REPORT_POD_TAIL=-1` with a larger `COZY_REPORT_PODS_MAX` restores the old reach. Because widening the selection and capping the walk in one change could otherwise let a newly-in-scope pod evict one the uncapped walk always collected, the ordering splits by evidence class before namespace: pods that are not `Running` — the class the collector took before this — all rank ahead of pods that are `Running` but unready, so the cap can only ever drop a pod the old collector did not keep either. `hack/e2e-capture-previous-logs.sh` gets the same treatment: its notes now go to `capture-notes.txt` in the artifact as well as to the job log, and a pod list that never returned is reported as such rather than as "nothing restarted". That script now creates its own output directory before its first line of output, which makes the comment in `hack/e2e-chainsaw/.chainsaw.yaml` wrong about why the once-per-test guard there holds — the only change to that file is the corrected comment, saying the `-d` check keys on whether the catch block already ran rather than on what the helper found. The rest is test coverage — a new `hack/cozyreport.bats`, an extended `hack/capture-previous-logs.bats` — and the removal of `EXIT`-trap cleanup from the bats files those tests now cover. That trap is worth its own note: inside an `@test` body it replaces the one the `bats` binary installs for its own bookkeeping, so a test that then fails prints no TAP line at all. It does not appear as `not ok`; it disappears, and only the `# bats warning: Executed N instead of expected M tests` line at the end distinguishes the run from a green one. A guard test enforces the ban over a named list of files that grows as the remaining ones are converted. ### Testing Every collector this touches is now covered by unit tests that run without a cluster: a new `hack/cozyreport.bats` (166 tests) and an extended `hack/capture-previous-logs.bats` (34), driven through `hack/cozytest.sh`. Ten `hack/*.bats` files run green locally, 265 tests in total, with each file's `@test` count reconciled against its passing count rather than trusted to the exit status — that reconciliation is the point of the trap change, since an exit code answers "did anything fail" and never "did anything run". The guard test that bans `EXIT`-trap cleanup was checked by committing the violation rather than by reading it: a folded `trap 'rm -rf "$tmp"' \` + `EXIT` added to a file on its list turns the guard red and names that file, and removing it turns the guard green again. `shellcheck --shell=sh` over the three changed scripts reports strictly fewer findings than before the change (`hack/cozyreport.sh` SC2086 170 to 116, SC2162 15 to 13; the three SC2102 in `hack/cozyreport-summary.sh` are gone), and `hack/e2e-capture-previous-logs.sh` is clean. The parts that need a cluster — that the bounds fire, and that each one lands in the artifact — are exercised by the e2e run itself, since these collectors run on every failed suite. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. This PR touches `hack/cozyreport.sh`, `hack/cozyreport-summary.sh`, `hack/e2e-capture-previous-logs.sh`, `hack/e2e-chainsaw/.chainsaw.yaml`, ten `hack/*.bats` files and `docs/agents/e2e-testing.md`. Nothing is moved or renamed under `hack/`, no make target changes what it does, and the files downstream consumers key on (`hack/package.mk`, `hack/common-envs.mk`) are untouched. No package, values key, CRD, variant, namespace or release asset is involved. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): the diagnostic report now collects per-pod evidence for pods that are Running but not Ready, which it previously skipped, and several collector sections that silently produced nothing now report correctly. Collection that stops early on a bound now says so inside the artifact, so a truncated report can be told apart from a healthy cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Diagnostic reports now prioritize relevant resources and collect bounded pod, VM, service, storage, certificate, and Flux data. - Reports preserve partial results and clearly identify timeouts, unavailable resources, truncation, warnings, and collection failures. - Previous-container logs now include bounded capture, namespace prioritization, failure details, and partial-read evidence. - Reports target the operator workload and safely handle missing dependencies or temporary storage failures. - **Documentation** - Expanded guidance for reliable end-to-end testing and diagnostic report collection. - **Tests** - Added comprehensive coverage for report generation, log capture, failure handling, limits, and portability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
e80b8e370f |
fix(e2e): collect logs for unready pods and record truncated captures
The failure snapshot selected pods by their STATUS column, so a pod whose container runs while its readiness probe never passes -- STATUS Running, READY 0/1 -- was dropped from the report entirely. That is a common way for a suite to time out: the replica never joins, its owning HelmRelease sits in InProgress, and the artifact carries no trace of the pod the run died on. Select on the READY column instead, which also covers the partially ready multi-container case, and keep every phase that was collected before. On one degraded cluster that is four of the seven broken pods the old filter reported nothing about. The VMI and VM sections had the same blind spot and now share one selector that reads READY as the last field, because a Pending object prints fewer columns than a Running one and the two kinds differ only in how many columns precede READY. READY counts ready containers rather than the pod's Ready condition, so a pod held back by an unsatisfied readiness gate is out of scope; a test pins that rather than leaving it to be found in an empty directory. Logs are requested for every container of a selected pod rather than skipped for Pending ones, and the per-pod collection moved into a helper so it can be driven against a stub kubectl. kubectl's refusal is the finding, and an empty result is ambiguous on its own: a read against an unscheduled pod exits 0 having printed nothing, which is what a container that started and stayed quiet also leaves. Every site that classifies a failed read now orders its branches the same way, naming the timeout first where both a cutoff and an uncapturable stderr apply. The CRD gate in the summary had no such branch at all: it returned in silence, which is byte-identical to the CRD being absent, so every section behind that gate vanished from a report whose only real problem was that the machine writing it had nowhere to put a temp file. Collection used to stop silently. A walk that hit its cap said so in the job log and nowhere in the artifact. Now the pod walk leaves COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off part way through, COLLECTION-FAILED.txt when the pod list never came back rather than the empty tree a healthy cluster produces, and COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a fact about the machine writing the report and not about the cluster. Tenant namespaces are collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform pods while the pod the suite died on falls off the end. Container logs are read with --tail=2000 where they were previously read whole, and every bounded file states the bound it was written under, because --tail drops the oldest lines and nothing else in the output would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log. The bounded whole-object read no longer merges stderr into pod.yaml, vm.yaml and vmi.yaml, which are read by yq rather than by eye: a warning on an otherwise complete read left an unparseable file. stderr goes to a scratch file and is copied in only when kubectl returned no object, and the truncation markers those files carry are written as YAML comments, since a bare marker line after a complete mapping is a second top-level node. Six latent faults surfaced while making these reads report. The kamaji module was gated on the LINSTOR controller, so a cluster without LINSTOR got no kamaji evidence at all, which is what a tenant-kubernetes bootstrap timeout needs most. The Flux sources section nested one jsonpath filter inside another, which kubectl rejects outright, so it printed nothing on every report ever taken and read as "all sources Ready". The HelmRelease listing took a single field as STATUS where STATUS is the Ready condition's whole message, so every failed release rendered as one word. The services section filtered for a pending LoadBalancer address on the CLUSTER-IP column, which never carries that value. The cozystack module read its image from a Deployment the installer stopped shipping when it moved to the operator, putting kubectl's NotFound in the first file of the report. And the cozystack apps section gated all three of its kinds on a CRD in an aggregated API group, so it never ran once; two of those names were wrong, and both are collected again. The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats test that installs one and then fails prints no TAP line at all, so output grepped for `not ok` reads as green, while hack/cozytest.sh reports the same failure correctly. A guard enforces that over a named list of files rather than a glob with exceptions, because a list of exceptions is the same claim written so that forgetting to shorten it makes it false. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7739d1e294 |
feat(multus)!: stage reference CNI plugins from the multus image
Multus-backed NetworkAttachmentDefinitions delegate to the upstream containernetworking reference plugins (bridge, macvlan, ipvlan, ...), and nothing guarantees a node has them in the directory Multus searches. On k3s they live under /var/lib/rancher/k3s/data/cni while Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails there with `failed to find plugin "bridge" in path [/opt/cni/bin]`. Bundle 14 of the release's plugins into the multus image and add an install-cni-plugins init container that installs them into the host /opt/cni/bin, mirroring the existing install-multus-binary container. loopback, dhcp, tap and dummy are left out; the Dockerfile records why. Gate it on a new chart value, stageCniPlugins, resolved per bundle and on for both. Talos bundles only bridge, firewall, flannel, host-local, loopback and portmap, so the other ten are missing there exactly as on generic Linux; the four that overlap are rewritten with the same upstream version, v1.9.1 on both sides, rather than an older one. A marker in the image Dockerfile records the Talos release that pin was compared against, and a test holds it to the release the platform ships. multus therefore moves out of system.common-packages into the two variant branches, the way linstor already does for talos.enabled. networking.stageCniPlugins overrides the variant default in either direction. Where staging is on the install is unconditional: a plugin is replaced on every pod recreation, not only when the destination is missing. A destination that is already a directory is reported and left alone, since replacing it is not something this can do safely. Turning the value off afterwards does not undo it; nothing here removes or restores a plugin. The container is not privileged and takes no mount propagation. It runs in the spc_t SELinux domain, without which an enforcing node refuses the write. Each plugin is installed under a temporary name and swapped in with a rename: /opt/cni/bin is shared with cilium and kube-ovn. The plugin tarball is checksummed before extraction. A plugin that cannot be installed is reported on stderr and to /dev/termination-log, and the container still exits 0: Multus stays the node's primary CNI and nothing removes its config, so an init container that never completes would leave the shim with no daemon behind it. The manifest change lives in customize-deployment.patch so it survives make update. BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI plugins in /opt/cni/bin on existing nodes, Talos included -- four of the names it writes are ones the Talos node image also carries. Opt out before upgrading: declining afterwards does not restore what was replaced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
24e9c5396c |
docs(release): resync the release doc with the freeze model
The freeze changed three things the prose still described the old way. The backport table claimed `backport-previous` resolves to `release-X.(Y-1)` and that resolution runs through `getLatestRelease`. Neither is true any more: the job enumerates the real `release-X.Y` branches, sorts them numerically descending, and takes the newest and second-newest existing lines. The `Y-1` derivation is the min-1 bug this branch dropped. Line 155 already carried the correct description, so the file contradicted itself. The freeze window's effect on backport targets lived only in a code comment. During it `backport` aims at the line being stabilised and `backport-previous` at the last published stable, so the line one step further back has no label for the duration. Document the trade-off where the labels are described. The changelog backstop and Phase 6 both framed a promote PR targeting `release-X.Y` as "a patch release". After the freeze every promote PR targets the release line, minor and patch alike, which makes the backstop the only path a `.0` changelog takes to `main`. Also note that re-cutting a line after an unusable rc.1 means deleting `release-X.Y` by hand — the refuse step blocks every other route, by design. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
b55e822eaf |
docs(release): align the patch-release diagrams with the freeze model
Address review feedback from coderabbitai on docs/release.md:150: the patch-release gitGraphs still branched release-1.2 off main after the v1.2.0 commit, depicting the branch as created at the stable release. Under the rc freeze the line forks at vX.Y.0-rc.1 and the release commit never lands on main, as the surrounding prose already states. Fork release-1.2 at the rc.1 commit and place 'Release v1.2.0' on the release line in all three patch-section diagrams. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
3f15229795 |
docs(release): fix the Backports link fragment
Address review feedback from lexfrei and coderabbitai on docs/release.md:39: the anchor pointed at #backporting, but the heading is '## Backports' (slug #backports), so the link resolved nowhere and tripped markdownlint MD051. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e5428caaa6 |
docs(release): document the rc freeze
The Release Candidates section said the opposite of current behaviour — "new features and changes can still be added before the regular release" — so it is rewritten around the freeze, along with the Regular Release step list, both gitGraph diagrams, and the workflow reference sections. The promote PR now targets release-X.Y rather than main, so the diagrams show main forking away at the freeze and never receiving the release commit. Patch Releases collapses to "same as above, the branch already exists", since after the freeze both flows are identical. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
6a73a6caa9 |
docs(agents): call the E2E gate a commit status, not a check-run
The §10 heading, four workflow comments and both TOCTOU comments still called the required "E2E Tests" context a check-run, while the body of that same section — and the implementation — insist it is a commit status. The distinction is the whole reason the section exists: a check-run made with the default token floats into an arbitrary check-suite, which is why the gate is a status. Leaving the wrong primitive in the heading steers a maintainer wiring branch protection at exactly the wrong thing. The paragraph explaining why it is NOT a check-run keeps that wording, since there the term is the rejected alternative. Also adds the blank line missing before the "Chainsaw v0.2.15 gotchas" heading. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
c46037ce7b |
fix(ci): stop a cancelled run from wedging the "E2E Tests" gate
Now that the required "E2E Tests" context is a commit status rather than a job, a run that dies before posting leaves the head SHA at "Expected" — which blocks the merge instead of passing it. Two paths reached that state. Same-repo: a `labeled` event shared the PR's concurrency group, so labelling an open PR cancelled the live opened/synchronize run, and the replacement run was then discarded by `plan`'s full-e2e guard -- so `e2e-report` skipped with it and nothing posted the status the cancelled run was going to. Observed on live CI: run 30614258694 (cancelled) followed by 30614280603 (skipped) on the same SHA, 24s apart, from a maintainer adding kind/* labels just after a push. Label events now get their own concurrency group, so they can only supersede each other. Release promotion is unaffected: `gh pr create --label` attaches `release` at creation, so a promote PR gets a single run that already sees the label (verified on #3397 and #3425, one run each, build/finalize gated off). Fork: `resolve` had no equivalent, so a cancelled triggering run fell through to the `CONCLUSION != 'success'` branch and posted `failure` — a false red on a SHA whose e2e never ran, which the replacement run for that same SHA would not flip back (it takes the discarded-label path and posts nothing). It now returns early on a cancelled or wholly-skipped conclusion. Checked on the conclusion rather than the job list because a cancelled run can report no jobs at all, which would make the `Plan build` lookup miss. Both stay fail-closed: declining to post can only preserve a verdict a real run reached, never invent one. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
3dc0f56477 |
feat(kube-ovn): build the chart and image in-tree, retiring the wrapper repo (#3460)
## What this PR does Brings the `kubeovn-chart` wrapper into the tree, so `cozystack/kubeovn-chart` can be archived. Until now `packages/system/kubeovn` pulled a chart tarball from the latest `cozystack/kubeovn-chart` tag at `make update` time, and that repository also built and pushed the `kubeovn` image. The wrapper existed only to cache the upstream chart; with the current caching approach that reason is gone, and the split cost a second repository, a second release step, and a package whose provenance a reader cannot follow without leaving the tree. The package is now shaped like `packages/system/cilium`: - `update` pulls the upstream `kubeovn/kube-ovn` chart directly and applies `patches/mtu.diff`, then syncs the version into the Dockerfile. The MTU patch already carried monorepo-relative paths, so it applies unchanged. - `image` builds `images/kubeovn` (carrying the existing `fix-keepvmip-portgroup` patch) and pins the digest into `values.yaml`, using the shared `image-tags` / `cache-args` helpers. It also carries `cozystack/kubeovn-chart#5`, which never got a tag: `OVN_IPSEC_KEY_DIR` is pinned under `/var/lib`. The kube-ovn default sits in `/etc/origin`, which a Talos node cannot create, so enabling IPsec left the node without ESP security associations. ### The nightly host rewrite had to be fixed first Moving the image build here activates a gap recorded in `docs/agents/image-refs.md`. The nightly host rewrite is a literal `<src-registry>/` substring replace, and kubeovn keeps its host in `global.registry.address` with the bare repository in a sibling key — no trailing slash, no match. While the image came from a separate repository the baked reference carried the public host and the ownership filter skipped it; once CI builds the image here it stamps the build-registry host into that key, and the result is an image mirrored to the public registry while the published tree keeps pointing at the private build registry. The second commit adds an end-of-line-anchored expression that fires only when the source registry is the complete scalar value, so a contiguous reference — which continues past the host — is still handled by the first expression and the two can never both fire on one reference. `keycloak-operator` splits at a different boundary and stays unfixed; that is documented, and dormant while no CI path rebuilds it. Verified: `hack/nightly-mirror_test.bats` passes with the new case and the case fails without the fix; `promote-retag` and `promote-rewrite-tags` suites unaffected; `helm template` and `helm unittest` on the package pass. The pre-existing `helm lint` complaint about a missing `kube-ovn` dependency reproduces on `main` and is left alone. ### Follow-ups this PR does not do - `cozystack/kubeovn-chart` should be archived once this merges, with a pointer here. Its open PR #5 is carried by this change and needs no separate landing. - `cozystack/website` links the Kube-OVN license at `cozystack/kubeovn-chart` in `operations/configuration/licenses.md` and `guides/platform-stack/_index.md` (all versions, including `next`). Archived repositories stay readable so nothing breaks, but the link should move upstream or in-tree. **No follow-up has been opened yet** — flagging rather than ticking the box. ### Screenshots N/A — build and packaging change, no user-facing or UI impact. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walked the trigger map against the diff. The website row matches — not through the app-list or platform-component triggers, but because two of its pages link the repository being archived. Left unticked because no follow-up exists there yet; a human should decide whether the license link moves to `kubeovn/kube-ovn` or to this package. No other row matches: the change touches no `packages/apps/` or `packages/extra/` package, no `values.schema.json`, no `ApplicationDefinition`, no platform variant, and no node prerequisite. ### Release note ```release-note feat(kube-ovn): the Kube-OVN chart and image are now built in this repository instead of cozystack/kubeovn-chart. OVN_IPSEC_KEY_DIR now defaults to /var/lib/ovs_ipsec_keys, so enabling IPsec on Talos nodes no longer leaves them without ESP security associations. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated Kube-OVN image build to use upstream chart metadata and keep registry/version alignment during `update`. - Added conditional MTU support for the OVN CNI DS via chart values. - Introduced `OVN_IPSEC_KEY_DIR` to standardize the OVN IPsec keys directory. - **Bug Fixes** - Preserved VM networking port-group assignments during live migration when an alive virt-launcher sibling pod exists. - Improved nightly mirror host rewriting for split and standalone registry host formats. - **Tests** - Added coverage for the end-of-line “whole scalar value” host rewrite case. - **Documentation** - Refreshed “Known gaps” guidance for nightly mirror host rewriting behavior and limitations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
0f6e99eff8 |
fix(ci): publish the "E2E Tests" gate as a commit status, not a check-run
Route the required "E2E Tests" merge gate through a GitHub commit status
posted with the default GITHUB_TOKEN (+ statuses:write), replacing the
COZYSTACK_CI app check-run. The app path needed the app granted checks:write
(it isn't — checks.create 403s "Resource not accessible by integration"), and
the earlier default-token check-run floated under a random check-suite
("PR size label"). A commit status has neither problem: branch protection
matches it by context name, it carries no check-suite grouping, and it is safe
with the default token because a fork's pull_request token is read-only and
cannot post one — only same-repo runs (e2e-report, guarded !fork) and the
privileged workflow_run in e2e-fork.yaml can set it.
The security property is unchanged: a never-posted required status stays
"Expected" and blocks the merge, so a skipped job can't smuggle a green
(#3257). Removes all COZYSTACK_CI app-token machinery from e2e-fork.yaml —
`resolve` posts the status directly, no `check_run_id` threading, no `open`
step.
Branch protection must require the "E2E Tests" status context (no GitHub App
and no app-id pin needed).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
03b519dc75 |
fix(ci): fork e2e — Chainsaw+TIA run and image-push hardening
The fork e2e path in e2e-fork.yaml still ran the pre-#2826 BATS app loop (`hack/e2e-apps/*.bats` + `test-apps-%`), which the merged Chainsaw migration deleted — so the fork run was already broken. Rewrite it to the same Chainsaw + Test-Impact-Analysis path as the in-tree job, and clear the remaining review follow-ups on the publish job. - Run E2E via a single `make test-chainsaw` invocation (no per-app retry — e2e-testing.md §1). Suite selection and the full-e2e override come from `resolve`'s TRUSTED base-repo listFiles/labels (new `changed_b64` / `full_e2e` outputs), never a git diff of the fork-controlled merge ref, so TIA cannot be shaped into skipping the suite (keeps the #3257 property). Collect + upload the Chainsaw JUnit report like the in-tree job. - Restrict fork image pushes to a trusted allowlist derived from the base parent (HEAD^1) Makefiles, so a fork cannot mint an arbitrary repo path under the CI registry by renaming an image. The charset guard stays as defence in depth. - `skopeo copy --all` so a multi-platform build's index digest (the one baked into pr.patch) is preserved. - Pin the flux CLI (FLUX_VERSION) in the privileged publish job instead of install.sh's rolling latest. Docs: e2e-testing.md §10 records that the fork path now runs Chainsaw + TIA and that fork suite selection must stay sourced from trusted data. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
06d734cf53 |
docs(agents): document the fork-PR e2e split and check-run gate (#3257)
docs/agents/e2e-testing.md is the mandated reference for E2E CI changes but described a single pull-requests.yaml pipeline. Document the split (same-repo e2e in pull-requests.yaml; fork e2e in the privileged e2e-fork.yaml via workflow_run), that the required "E2E Tests" context is a synthetic check-run rather than a job (so renaming the e2e job must preserve it), that fork builds export OCI archives instead of pushing, and the repo prerequisite Actions → "Require approval for all outside collaborators". Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
cc44f6c981 |
ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449)
## What this PR does Moves release validation from promote time to rc time, making promotion itself a pure retag operation. This is the follow-up to the v1.6.0 retrospective: the promote flow's checks ran on the promote PR (coupled to promotion, skippable by never promoting), the changelog and website docs were produced only after the release went public, and the finalize tag push silently failed to trigger `tags.yaml` at all (persisted-credential bug, root-caused below). The release lifecycle after this PR: 1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes the staging branch — and now runs a **mandatory full e2e** for every `-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and any published tag) get the same validation as a **manual button** (`workflow_dispatch`). 2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**: the release changelog is generated from the rc tag (reusable `changelog-generate.yaml`) and committed to the rc staging branch, ready for review long before promotion. 3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence: the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages the release, picks up the rc-time changelog (or generates one), and **opens the website docs PR** on cozystack/website from the staging branch — parked with an explicit DO-NOT-MERGE-until-finalize contract. 4. The promote PR **no longer gets `full-e2e` automatically** — its tree is the rc's already-validated digests with only tag strings rewritten; a maintainer can still add the label manually to force the suite. 5. Finalize performs the irreversible side effects (tag, publish, retag, chart) — and with `persist-credentials: false` its tag push now actually authenticates as the CI app, so the `tags.yaml` backstops (`generate-changelog`, `update-website-docs`) fire for promoted tags. This is the one-line fix for the v1.6.0 incident where the tag triggered nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader` silently defeats the app token injected via `git remote set-url`, and `GITHUB_TOKEN` pushes create no workflow runs. Everything is pinned by executable contract tests (`hack/promote-gate-contract.bats`, extended `hack/release-changelog-contract.bats` / `release-changelog-behaviour.bats` — 48 tests total, including behaviour tests that execute the new `hack/parse-rc-tag.sh` and `hack/select-changelog-source.sh` for real), and `docs/release.md` is rewritten to describe the new lifecycle end-to-end. ### Commit map (review commit-by-commit) - `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing changelog-before-publication rework (changelog generated at promote, committed to the staging branch before the PR opens, finalize publishes it as the release body; tags.yaml job becomes a verbatim-port backstop), rebased onto main. - `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane (prerelease → staging branch, stable → tag; draft releases fail fast; dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts). - `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut rc tags (also on `release_exists` re-runs); fixes the `cut-prerelease.yaml` comment that falsely claimed e2e already existed. - `2dfbeea77` — `changelog-generate.yaml` (reusable core) + `changelog-rc.yaml` (rc-time button committing to the staging branch); promote-rc gains a top-level `parse` job with a strict no-whitespace tag policy (`hack/parse-rc-tag.sh`); source selection is validate-then-fallthrough (`hack/select-changelog-source.sh`) so an invalid higher-priority changelog can never silently win. - `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override + removal of the automatic `full-e2e` label (pull-requests.yaml release-path e2e now requires an explicit label; a post-open label now actually starts the run). - `936190d4b` — `website-docs` job at promote time behind a FETCH_REF-support guard (see dependency below), with the DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before `release-next`; `tags.yaml::update-website-docs` stays as the backstop. - `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0 root cause). - `717c8860f` — `docs/release.md` coherence pass over the whole new lifecycle. ### Dependency The promote-time website docs job needs cozystack/website#631 (`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard detects the missing support and degrades to a ⚠️ line in the promote PR body — promotion is unaffected and the tag-time backstop backfills, so this PR is safe to merge first. ### Merge instructions Please **merge-commit, do not squash**: the 13 commits are deliberately atomic (the credential root-cause fix is separable from the feature work) and squashing would collapse ~3k lines into one commit, hurting bisectability. ### First live run The e2e lane is verified by lint, contract tests and dry-traces, but has not yet executed against a real tag; the intended first live run is a manual `e2e-tag.yaml` dispatch against an existing published rc tag before the gate is relied on for a real promotion. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added reusable changelog generation with validation, “present” signaling, and safe artifact upload behavior (including RC-time support). * Added an E2E release-tag workflow with tag parsing, deterministic sandboxing, and comprehensive report collection. * Improved RC promotion with a stricter parse front-door, verified E2E green gate (optional bypass), and earlier parallel changelog/asset staging. * **Documentation** * Refreshed release-process docs to match the updated RC-to-stable, changelog, and website-docs timing rules. * **Bug Fixes** * Improved PR finalization to source release bodies from verified changelogs and warn/fallback on missing or empty content. * Tightened label gating so E2E runs only under the intended conditions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
6cb579338b |
fix(nightly): rewrite an image host that is the whole scalar value
The nightly host rewrite is a literal "<src-registry>/" substring replace, so it only reaches a host sitting contiguously in front of the repository. kubeovn keeps its host in global.registry.address with the bare repository in a sibling key: no trailing slash, no match. The image was mirrored to the public registry and the tree kept pointing at the private build registry. The gap was dormant while kubeovn's image came from a separate repository, because the baked ref carried the public host and the ownership filter skipped it. Building the image in-tree stamps the build-registry host into that key and turns it live, so fix it here. Add an end-of-line-anchored expression that fires only when the source registry is the complete value; a contiguous ref continues past the host, so the two expressions cannot both fire on one reference. keycloak-operator splits at a different boundary and stays unfixed -- neither of its keys holds the source registry as a whole. Documented, and dormant while no CI path rebuilds it. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
myasnikovdaniil
|
2c453778f1 |
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors (#3436)
## What this PR does Fixes #3431. `hack/seaweedfs-naming-audit.sh` was fail-open: any kubectl failure produced an empty table indistinguishable from an honestly clean fleet — a failed namespace LIST meant zero namespaces walked, failed secret/PVC/STS LISTs meant zero findings, all silenced by `2>/dev/null` with no error handling anywhere in the script. The runbook uses this output as the gate before PVC deletion, as post-deletion verification, and as an upgrade precondition, so a transient API error could green-light destroying data. Every kubectl call now routes through a `run_kubectl` helper: a non-zero exit prints a FATAL line naming the failed query and the code propagates up the whole chain (enumerations restructured to capture-then-check, since an `exit` inside `$(...)` dies with the subshell). By-name GETs separate existence from extraction: a `--ignore-not-found -o name` probe keeps legitimate absence (Helm-pruned revision, unbound PVC) a clean skip, while everything else fails loudly — including a release Secret that exists but has no decodable payload or decodes without a chart name, which previously dropped the whole tenant from the audit. Incomplete PV-age evidence now marks the generation incomplete and falls to the safe "direction cannot be established" branch, so a failed GET can no longer flip an OVERLAP verdict into a wrong Step-3 deletion candidate. A successful audit prints the same bytes as before. Tests: 14 new cozytest cases — failure injection for every LIST and by-name GET site, corrupt-payload variants (missing `.data.release`, undecodable base64/gzip, chartless `{}`), and full-output golden diffs proving the success paths stay byte-identical to the pre-change script. 25/25 green via `hack/cozytest.sh`. ### Release note ```release-note fix(seaweedfs): the naming-audit script fails loudly when a kubectl query or a Helm release payload read fails, instead of printing an empty table indistinguishable from a clean fleet ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Audits now fail clearly (non-zero exit + fatal messaging) when Kubernetes queries or Helm release data is unavailable or corrupt, instead of producing incomplete results. * Missing release revisions are skipped safely; missing/unreadable PV age evidence no longer leads to incorrect deletion candidates, and MIXED now reports conservative “direction cannot be established” when evidence is incomplete. * Non-SeaweedFS releases remain silently ignored. * **Tests** * Added end-to-end acceptance coverage for fail-closed behavior and byte-for-byte stable output across key naming scenarios. * **Documentation** * Updated the runbook to require relying on the audit script exit code before proceeding, with a dedicated MIXED re-check step. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a7ce4371c2 |
docs(agents): correct the chainsaw podLogs capability note
The e2e conventions, the previous-logs capture script, and the global chainsaw catch all claimed the v0.2.15 podLogs collector exposes only container/tail/selector. It also accepts name, namespace, timeout and the cluster fields; only the previous option is genuinely missing. The wrong list steers catch authors toward imperative scripts for cross-namespace log capture that podLogs handles declaratively. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
1eded27900 |
test(release): scope the trim pin to the parse step, fix two stale references
Three review nits, all verified against the tree, plus a tighter scope than the first attempt at them. The behaviour suite asserted "promote-rc does not trim the rc tag" against the whole file. promote-rc.yaml has no .trim() today, so it passed, but any unrelated .trim() added anywhere in the workflow would have failed a test about tag policy. Scoping it to the parse JOB was not enough either: that job is ~180 lines, most of them the e2e gate's github-script, whose string handling has nothing to do with tag policy. Both greps are now scoped to the parse STEP, and run through a code_lines filter copied from the contract suite so a YAML comment mentioning .trim() cannot fail them. Mutation-checked in both directions: a .trim() injected into the parse step fails the test, one injected into the e2e-gate step of the same job does not. - The Step 2 checklist entry still said "must be main" while Step 2 itself now lists three valid starting points (main, detached at the release tag, detached at the rc tag being promoted). The checklist is what an agent ticks off, so it was the half that mattered; it now points at Step 2 rather than re-encoding the count, so a fourth configuration cannot desync it. The heading said "Checking current branch", which two of the three states do not have. - hack/testdata/needs-semantics.yaml named release-changelog-contract.bats as its consumer; the only consumer is release-changelog-behaviour.bats. 50/50 tests green across the three suites (11 + 24 + 15). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e02adad38f |
fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls
Review findings from #3436, plus what an adversarial re-check of the first attempt at them turned up. The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals "config" (the release`s values) after "chart", so a values subtree that spells chart.metadata.name shadowed the real chart name: the release then read as non-SeaweedFS, the tenant silently vanished from the report, and the script exited 0 -- the precise false clean this PR exists to prevent, reachable without any corruption at all. It now takes the FIRST match. The path stays adjacent on purpose: a looser "any name after metadata" matches chart.templates[].name, which Helm emits immediately after metadata on every healthy release, so the review`s suggested relaxation would have returned a template path for every tenant. Over-strictness fails loudly and recoverably; over-looseness deletes data quietly. Newlines are now folded before matching, so a pretty-printed payload parses at all rather than aborting, and whitespace around the punctuation is tolerated. first_deployed got the same treatment -- it had the identical greedy, single- line-only shape, and a spaced payload silently dropped the PV-vintage row from the report. The FATAL message names the path it read and the shape it expected, so a future Helm format change is diagnosable instead of looking like real corruption. Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy) each byte-compared against the same golden as the compact payload, so the shape must make no difference to the report. Mutation-checked -- restoring last-match extraction fails the decoy test on its own, restoring the single-line matcher fails the pretty test. The test fake answered any unmodelled kubectl invocation with exit 0 and empty stdout, the fail-open shape this script was rewritten to reject, and enough to let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97 naming the invocation. Verified inert first by instrumenting the fake: 88 invocations across the suite, none unmodelled. Also added the missing test for the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it. The runbook now tells the operator to read the exit code: a non-zero exit means the table is incomplete and no step may be taken on it. That contract lived only in the script header, while the runbook is what the operator follows. 29/29 tests green; sh -n clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
717c8860f6 |
docs(release): make the release playbook coherent with the promote flow
Final coherence pass over docs/release.md (and the releasing.md pointer) after several incremental patches to the release-cycle rework left the whole-document narrative uneven. - Rewrite the regular and patch release sequences so a maintainer reading top-to-bottom performs the actual flow: cut rc -> watch mandatory rc E2E -> [optional Generate RC Changelog button] -> promote dispatch (E2E gate; changelog generated or picked up; website docs PR opened and parked) -> review the promote PR (no E2E by default; E2E/changelog/website-docs lines) -> merge -> finalize -> merge the parked website PR, then announce. - Purge the last old-model sentence: "vX.Y.0 is tagged from the last RC or a later commit in main" -> promoted from the rc by digest, tag cut write-once at the promote PR merge commit. - State the alpha/beta contract where prereleases are discussed: built and published, button-only E2E (no mandatory rc-e2e), and no promotion path (promote-rc accepts vX.Y.Z-rc.N alone). - Fix the Phase 4 heading (was "chore(release): cut vX.Y.Z"; the PR is "chore(release): promote ... -> ...") and add the missing website-docs line to the reviewer checklist; add a keep-parked-until-publish item to the pre-release checklist. - Add cut-prerelease.yaml, changelog-rc.yaml, and changelog-generate.yaml to "See also" so the workflow inventory is complete and consistent with the "seven workflows + reusable changelog-generate" count. - releasing.md: replace the stale "typically involves ... creating git tags / building artifacts" list (invalidated by the promote-not-rebuild model) with the real high-level sequence. Every workflow/job/step name cited was verified against the YAML on this branch. hack/release-changelog-contract.bats stays green (24/24); no test pins docs/release.md strings, so no test change was needed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
01e1e7188e |
ci(release): stop finalize persisting credentials so the tag push triggers tags.yaml
pull-requests-release.yaml's finalize job checked out the repo with the default persist-credentials, leaving GITHUB_TOKEN embedded as http.extraheader. Every later tag/branch push re-injects the CI app token via `git remote set-url`, but a persisted extraheader silently wins over the URL credential — so those pushes authenticated as GITHUB_TOKEN. A GITHUB_TOKEN-authenticated push creates no workflow run (GitHub anti-recursion), which is exactly why v1.6.0's stable tag never triggered tags.yaml: the generate-changelog and update-website-docs backstops never ran. Set persist-credentials: false on the finalize checkout, so the set-url app token actually authenticates the pushes — the same handling promote-rc.yaml already uses. Audited every git/gh step in the job: both tag steps set-url the app token before pushing, and Ensure-maintenance-branch / Publish-draft-release / the retag+chart steps use the app token via github-script or a separate registry login — none depends on the persisted GITHUB_TOKEN extraheader. Pin it in hack/promote-gate-contract.bats. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
936190d4b6 |
ci(release): open the website docs PR at promote time
The cozystack/website 'update managed apps reference' PR used to be opened only at tag time by tags.yaml::update-website-docs — after finalize pushed the stable tag, i.e. after the release was already published (too late to review) — and it fetched README content from that stable tag (BRANCH=RELEASE_TAG), which finalize only creates post-merge. Add a promote-time website-docs job to promote-rc.yaml (needs [parse, promote]) that opens the same update-docs-vX.Y.Z PR from the release-X.Y.Z staging branch via the website Makefile's FETCH_REF variable, so the docs are reviewable alongside the promote PR while the stable tag is still unborn. It refreshes the trunk pins from the staging ref before release-next so a new minor inherits fresh Talos/cozystack pins (the v1.6 talos-pin incident class), then runs update-all with FETCH_REF pinned to the staging branch. The job is non-blocking: a FETCH_REF-support guard fails loudly (warn + exit 1) when cozystack/website main predates the FETCH_REF Makefile variable — otherwise make would silently ignore the unknown FETCH_REF override, fall back to the not-yet-created stable tag, and update_apps.sh would warn-and-continue into STUB docs. Any failure surfaces as a website-docs warning line in the promote PR body (plumbed through open-pr via needs.website-docs.result) and never blocks promotion; the tag-time job stays as the backstop and no-ops when promote already did the work. MERGE-TIMING CONTRACT: the website PR must NOT be merged before finalize publishes the release — doing so flips the site's latest-version pointer to a version whose docs and openapi are not on GitHub yet, 404ing every version-pin. The website PR body and the promote PR body both say so in bold. Pin the contract in hack/promote-gate-contract.bats (needs shape, FETCH_REF invocation, the guard, non-persisted checkout credentials, the DO-NOT-MERGE body wording, the promote-PR status line, and the tags.yaml backstop comment) and relax the release-changelog-contract needs pin to tolerate the appended website-docs dependency. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ff7ddd3c9a |
ci(release): gate promotion on a green rc e2e and drop the automatic full-e2e label
Promotion now assumes checks passed at rc time instead of re-running them. At dispatch, before any staging happens, promote-rc verifies the rc tag has a green full e2e via either evidence path: the rc's tags.yaml push run containing a successful 'rc-e2e / E2E <tag> (full suite)' workflow_call job, or a successful manual e2e-tag.yaml dispatch correlated by its exact tag-bearing job name (immutable rc digests make past evidence valid). No evidence fails the dispatch with both paths and the remediation named. A skip_e2e_gate boolean input bypasses the gate loudly for emergencies; the promote PR body always carries the verification state (verified or bypassed), and 'verified' can only originate in the gate's green branch. The promote PR keeps the release label but no longer gets full-e2e automatically: its tree is the rc's already-validated digests with only tag strings rewritten, so pull-requests.yaml now runs e2e on the release path only when a maintainer opts in by adding full-e2e. The regular-PR path and the ordinary full-e2e forcing are untouched, and the new labeled trigger makes a post-open label actually start the run. A new promote-gate contract suite pins the gate's DAG position, the override default, the label set, the body lines, and the label-gate condition with comment-proof executable-line counts. Assisted-By: GPT-5 <noreply@openai.com> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
2dfbeea776 |
ci(release): dispatch changelog generation at rc time
Extract the promote-time changelog generation into a reusable workflow (changelog-generate.yaml, workflow_call) and add a dispatchable wrapper (changelog-rc.yaml) so a maintainer can generate the release changelog the moment an rc goes green, while there is still time to review it. The reusable core keeps promote-rc's observable behaviour: detached rc-tag checkout, read-only app token for the AI step, the validate-changelog.sh gate, non-blocking AI failure, and the same changelog-vX.Y.Z artifact. .release-tooling is overlaid from github.ref_name, which under workflow_call is the caller's dispatch ref (and coincides with the reusable workflow's own ref for a local uses:), so "current tooling against an old rc tag" is preserved. Secrets are declared explicitly rather than inherited, so a caller that forgets one fails at parse time. The rc tag is parsed by a shared hack/parse-rc-tag.sh that REJECTS whitespace outright (a stripped 'v1. 6.0-rc.2' would silently become a different tag). promote-rc parses it ONCE in a cheap `parse` job that both `promote` and `changelog` depend on — so they still run in parallel with each other while consuming one validated tag, and a whitespace tag fails the whole promotion rather than letting promote succeed while the changelog job rejects the same input and the PR opens empty. The reusable core mirrors the reject-whitespace policy inline because its parse must run before any checkout. changelog-rc.yaml verifies the rc staging branch release-X.Y.Z-rc.N exists UP FRONT (in parse, which generate depends on), so a missing branch fails in seconds rather than after a 30-minute AI run, then commits the result there as docs/changelogs/vX.Y.Z.md idempotently (compare-before-push). promote-rc's changelog job is a thin uses: call. Source selection — validate-then-fall-through across the rc tag tree, the rc staging branch (the rc-time pickup), then AI generation, discarding any invalid source with a loud warning so it never suppresses a valid lower-priority one — lives in hack/select-changelog-source.sh, executed after the tooling overlay. open-pr carries the picked-up file onto release-X.Y.Z and finalize publishes it. Contract and behaviour bats are repointed and extended: the shared parser, the source-selection script, and promote's parse policy are all exercised by execution on adversarial inputs; the remaining grep pins match executable lines with exact counts so a comment or dead block cannot satisfy them. docs/release.md lists the new workflow and the rc-time entry point. actionlint and zizmor (default and pedantic) are clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
6fc9708514 |
docs(release): align validator description with validate-changelog.sh checks
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
34e45295fe |
ci(release): fix second-round review findings
Two more review passes (Claude + Codex) on the previous commit. Codex found the one that would have broken the very next promotion. The agent reads its own instructions from the working tree, which is the rc tag. Every rc cut before this change carries the old §2 of docs/agents/changelog.md — two valid configurations, "stop" on anything else — and a detached rc-tag checkout is exactly that "anything else". So promoting v1.6.0-rc.4 would have presented a compliant agent with instructions telling it to refuse, silently, producing no changelog on every promotion of an existing rc. Verified against the tag. The same applies to hack/validate-changelog.sh, absent from those trees entirely, which would have failed the Verify step with "command not found" and, in open-pr, taken the promote PR down with it. Both are now overlaid from the dispatch ref, best-effort, in both jobs. The structural validation added last round rejected every PATCH changelog. It asserted a `# Cozystack vX.Y.Z` header and a 20-line floor; patch releases use `# vX.Y.Z (date)` and v1.5.1 is a complete 19-line release. Good AI output would have been discarded and the promote PR mislabelled "generation failed". Measured across the tree: the header form holds for 23 of 115 changelogs, so it was never an invariant. The predicates now live in hack/validate-changelog.sh, tested directly against real changelogs rather than through YAML greps — the reason the defect shipped is that the suite pinned finalize's guard but nothing on the promote-side validator, and stayed green when the validator was mutated away. It accepts both header conventions, has no line floor, and checks the version in the header, the release-link comment and the compare link, so a changelog generated for the wrong version is caught. All 16 current-era changelogs pass; whitespace, empty, truncated and wrong-version inputs are rejected. Also fixed: - A fatal artifact download in open-pr would abort the job before the PR step — the same wedge as a skipped job, one step lower. Download is now best-effort and the commit and PR body key off what is on disk, not an upstream job output that says nothing about whether the transfer worked. - A pre-existing changelog (hand-written, or preserved across the force-push) reached the PR without ever being validated, was advertised with ✅ and published verbatim. It now goes through the same validator, as does the tags.yaml backstop. - `gh pr view <branch>` matches closed PRs, so re-dispatching after closing an abandoned promote PR edited the dead PR and opened nothing. Branch on state explicitly. - `gh pr edit --body` unconditionally overwrote the PR body, destroying the upstream breaking-change notes docs/release.md instructs reviewers to paste there. Refresh only while the body is still the generated one. - open-pr consumed raw dispatch input while its siblings used the validated parse; the changelog job's bash parse did not trim, so an input with a stray space passed promote and failed there. Both fixed. - The prompt and §2 said "trailing release-link comment"; it is the first thing in every changelog. Tests grow to 16, each mutation-checked: dropping the overlay, making the download fatal, reverting the existing-changelog check to `-s`, and removing the backstop validator each fail the suite. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f49d54a689 |
ci(release): fix review findings; keep website docs at tag time
Dual-model review (Claude + Codex) of the previous two commits found seven substantive defects, two of them destructive. Both models independently reached three of them; the split is noted per item. Descope the website half (Claude, verified by reproduction). The docs job was moved to promote time, but the website Makefile pins the fetch ref with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that cannot exist at promote time, since promote's own validate step hard-fails if it does. update_apps.sh logs the failed fetch and exits 0 AFTER truncating each destination file, so every app doc would become a stub, `changed=true` would open the PR, and the new auto-merge would land it unreviewed. Not fixable from this repo. update-website-docs returns to tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize auto-merge is dropped. Opening it earlier needs a cozystack/website change first and is left as a follow-up. This also removes the unordered-sibling race Codex found, where the release PR could merge before the docs job opened its PR, leaving finalize nothing to merge and no retry. Fix the non-blocking guarantee (both models). Only the AI step carried continue-on-error; the changelog JOB did not, so a flaky `npm i -g @github/copilot` failed it and `needs` skipped open-pr entirely — staging branch pushed, release drafted, no promote PR. Gate open-pr on promote alone. Generate from a checkout the contract permits (both models). The job checked out the rc staging branch, which docs/agents/changelog.md §2 rejects — a compliant agent stops and writes nothing, so the normal path produced no changelog at all. Check out the rc tag detached and document that configuration in §2. The checkout ref now comes from the validated parse output rather than raw dispatch input. Port, don't regenerate, in the backstop (both models). For a patch release promoted onto release-X.Y the changelog never reaches main, so the backstop fired and ran a second AI pass; once its PR merged, update-releasenotes.yaml overwrote the published body with unreviewed text. Detect the file at the tag and port it verbatim; the AI runs only when it is absent from both main and the tag. Survive re-dispatch (Codex). promote force-pushes the staging branch, erasing a hand-written changelog — the documented recovery when generation fails. Preserve an existing one across the rebuild, and stop open-pr overwriting it with generated output. Validate structurally, not by size (Codex). A truncated or whitespace-only AI run passed `[ -s ]`, was committed, advertised as valid in the PR body, and published verbatim as release notes. Assert the header, the compare link and a plausible line count; finalize additionally refuses whitespace-only content rather than blanking the notes. Add hack/release-changelog-contract.bats — nine tests pinning each link in the chain, mutation-checked: removing open-pr's if:, breaking finalize's body read, and restoring paths-ignore each fail the suite. Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which described the old flow throughout. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
8aeedde088 |
feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported, tested capability for the new `backups.cozystack.io` API. Fixes #2774. Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the `recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime` plumbing; #3313 has since merged and this branch is rebased on `main`. This PR adds the reliability + verification layer #2774 asked for. ### Changes - **Precise failure reason for an unreachable target.** The restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the sole authority for "this restore is stuck" — the client waits the full window either way. What changes is the *diagnosis*: once the deadline has elapsed, if `recoveryTime` was set and the recovery Cluster never went healthy, the CNPG driver reads the `full-recovery` pod logs and — when several distinct attempts exited with PostgreSQL's `FATAL: recovery ended before configured recovery target was reached` — fails the RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason `RecoveryTargetUnreachable`) and an actionable message, instead of a bare generic timeout. Timing is unchanged; only the reason/message differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.) Design points: - **We deliberately do NOT trip early off the FATAL.** A valid near-now target emits that same FATAL transiently — repeatedly, for many attempts, on a slow archiver — before it converges, so any earlier trip would wrongly reject a recoverable restore (observed twice in CI). The FATAL is used only to *explain* a failure the deadline has already declared. - **Health wins unconditionally** — the deadline/classification check precedes the healthy check, but a recovery that converges within the window reaches success and is never classified as failed (a live dev7 restore did exactly this). - **Counting distinct FATAL-ing attempts** (not pod failures) tolerates a one-off archive-lag miss on a near-now target and ignores transient recovery-pod crashes (a node blip / brief API-server unreachability never emits that FATAL). - Adds the `pods/log` RBAC grant the log read needs (distinct subresource from `pods`); a `Forbidden` is logged loudly and surfaced in the failure message rather than silently disabling the classification. - **e2e** (`examples/backups/postgres/run-all.sh`, wired into the Chainsaw `postgres-2-backup-roundtrip` suite): - Step 45 — write `before`, capture server timestamp (µs precision), write `after`, restore to that timestamp, assert `before` present and `after` absent. - Step 46 — a `recoveryTime` an hour ahead must fail with reason `RecoveryTargetUnreachable` once the deadline elapses (asserts the *reason*, so it proves the classification fired, not just that a timeout occurred). - Chainsaw budget raised 25m → 40m to cover three sequential restores + the post-deadline classification. - **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a `docs/operations/backup-classes.md` PITR section (recoverable window, discovering earliest/latest restorable time from the backup catalog since `firstRecoverabilityPoint` is unreliable under the plugin, archive-lag tolerance, idempotency under GitOps). ### Verification - `go build` / `go vet` clean; unit tests pass (`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`, `recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level success-path test asserting `Succeeded` + `RecoveryConverged=True`, and — added per review — `TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`, a reconcile-level test driving the deadline branch through all three sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL absent → generic timeout, `Forbidden` log read → generic timeout + missing-`pods/log`-RBAC note. It fails if the call-site reason is reverted to the generic one). - Reviewed via adversarial branch-review; fixes include a critical missing-RBAC dead-on-arrival, a classification false-positive on transient recovery-pod crashes, and a sub-second e2e flake. - **dev7**: backup + WAL archiving + recovery convergence verified live on the #3313 build. The load-bearing assumption of the classification — CNPG **accumulates** failed `full-recovery` pods rather than reusing one — was confirmed (observed 7 simultaneous failed recovery pods). A fully-green live step-46 run was blocked by intermittent COSI bucket-provisioning flakiness on dev7; the negative path is covered by the CI Chainsaw e2e. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The only user-facing surface is the new `docs/operations/backup-classes.md` PITR section; the website keeps a hand-mirrored copy per docs version. - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: PITR docs mirror warranted (not yet opened; same pending mirror as #3313's backup-classes changes) - [x] No other downstream repository is affected by this change ```release-note feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added PostgreSQL point-in-time recovery (PITR) support via RFC3339 `recoveryTime`. * Introduced a PITR RestoreJob example, plus helper variables and an unreachable-target scenario. * **Bug Fixes** * Improved restore deadline handling with deterministic, log-based fail-fast classification for unreachable recovery targets, including clearer conditions. * **Documentation** * Added detailed PITR operational documentation (recovery windows, failure behavior, and GitOps/idempotent behavior). * Expanded README and example comments for PITR verification and failure expectations. * **Tests** * Extended unit tests and end-to-end chainsaw workflow to cover PITR success and unreachable failures. * **Chores** * Updated RBAC to allow pod log reads; refined cleanup and run scripts for additional RestoreJob artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
a2acfcde1a |
docs(release): add changelog for v1.6.0 (#3429)
## What this PR does Adds `docs/changelogs/v1.6.0.md` (667 lines), generated locally with the process in `docs/agents/changelog.md`. Normally `tags.yaml`'s `generate-changelog` job produces this on the stable tag push, but that job never ran for `v1.6.0` — see the note below. **Coverage boundary.** The changelog runs through #3386, i.e. the release period ending at the `v1.6.0-rc.4` cut. Roughly eight PRs merged to `main` after that cut are deliberately absent, because `v1.6.0` is a promotion rather than a rebuild: every shipped artifact is pinned to an rc.4 digest (`cozystack-packages@sha256:bf682087…`, `cozystack-operator@sha256:93b305f7…`), so that later code is in the tagged tree but not in anything users install. The changelog describes what shipped. **Why this is a manual PR.** `generate-changelog` and `update-website-docs` are jobs in `tags.yaml`, triggered by `push: tags`. The `v1.6.0` tag was pushed by `pull-requests-release.yaml`'s finalize job using the `cozystack-ci` app token, and no workflow run was created for that ref — every rc tag, pushed by `cut-prerelease.yaml` with the same app, did trigger it. `v1.6.0` is the first stable tag created through the promote flow, so this path had never run before. The cause is not yet established and needs a repro rather than a guess; the website docs versioning is still outstanding for the same reason. ### Screenshots N/A — documentation only. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, which is a single new file under `docs/changelogs/`. No trigger matches: the map covers package layout, release assets, CRDs, workflow behaviour and naming contracts, none of which a changelog file touches. The separate repository list in `docs/agents/changelog.md` was already consumed while generating this file — it feeds the notes, it is not a code coupling. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added comprehensive release notes for version 1.6.0. * Documented breaking changes, upgrade prerequisites, manual upgrade steps, and post-upgrade expectations. * Highlighted major features, deprecations, security updates, bug fixes, operational notes, and component version updates. * Included configuration, CI, development, and related-project changelog updates. * Added contributor acknowledgements and a link to the full changelog. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
55195ded0c |
test(e2e): capture previous container logs and stop killing the snapshot (#3346)
## What this PR does Root-causing a recent E2E failure was nearly blocked because the diagnostics captured on failure had collected almost nothing. Two independent mechanisms were responsible, and both had been silently in place for a long time. **The failing container's previous logs were never captured.** The pod had crash-looped, so the decisive evidence lived in the first container instance, reachable only through `kubectl logs --previous`. Chainsaw's `podLogs` action has no `previous` field, so this could not be expressed per-suite; the conclusion had to be reconstructed from indirect evidence instead. A new capture runs from the global catch and from the BATS failure hook, selecting containers at runtime by `restartCount > 0` across both init and regular containers, so a healthy run stays quiet and future suites are covered without per-suite wiring. **The catch was structurally guaranteed to be killed.** A Chainsaw `script` op is bounded by `timeouts.exec`, which is 5m, while the catch's own inner budgets summed to roughly 960s. It reported `SCRIPT | ERROR | signal: killed` and produced nothing — and its second leg, the data-plane capture, had therefore never run at all. The kill signature was itself misleading: `timeout` exits 124 and cannot produce that message, which comes from Go's `exec.Cmd` killing the process group. Each catch op now carries an explicit timeout above the budgets it contains, so the inner one fires and leaves a partial, usable artifact instead of losing everything. **The snapshot was truncated at 60 seconds.** `crust-gather --duration` defaults to 60s and was never passed, so every snapshot ever taken was cut short, with nothing in the artifact saying so. The API discovery it performs first is not covered by that budget at all, which is where the outer wall-clock was going on a cluster with a degraded aggregated APIService — precisely the state a failed run is in. Both budgets are now set explicitly, and the collector's output goes to a log inside the snapshot with a completeness verdict, so "is this snapshot complete?" is answerable from the artifact. Failure paths are reported rather than swallowed: an unreachable API says so instead of implying nothing had restarted, a truncated capture is kept and marked, and every dropped item is logged, since silent truncation reads as full coverage. ### What is not verified here The mechanism claims are read from the pinned versions of Chainsaw and crust-gather. Everything cluster-facing is not verified by this PR: the go-template's output shape against a live API server, `--previous` returning the first instance for a real crash-looper, and whether the new collection budgets suffice on a degraded cluster. The bats suite covers the selection helpers only, and those cases are mutation-tested. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Diagnostics** * Improved end-to-end failure evidence by capturing “previous instance” logs for crash-looping pods (including restarted init containers) and prioritizing relevant namespaces. * Added time-bounded diagnostics collection with explicit outer timeouts; collector output is saved to per-snapshot log files. * Classifies collector outcomes as complete, timeout-truncated, or failed, while preserving the original test failure signal. * **Documentation** * Expanded E2E failure-diagnostics guidance with new timeout and evidence-capture conventions. * **Tests** * Added unit-style tests for previous-log selection, prioritization, row limiting, and deterministic log artifact naming. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d0c0f399f8 |
fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404)
## The bug Promotion rewrites the rc version substring to the stable version across the vendored image references. That rewrite lived inline in `promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus `packages/apps/kubernetes/images/*.tag` **only**, on the premise that the kubernetes app was the only one whose `.tag` files carry the cozystack version. Nine other `.tag` files do, and `system/multus` seds its reference straight into a vendored upstream daemonset manifest. #3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33 of 55 first-party references still reading `v1.6.0-rc.4`**. `hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind spot from the other direction — both scanned the depth-2 `values.yaml` alone, so those images were never retagged to the stable version (**30 references selected before, 42 after**) and never mirrored to the public registry for a nightly. A retag miss leaves an image short of a tag, since the digest still resolves inside one registry. A mirror miss is worse: the host rewrite walks the same file list, so a reference that is neither mirrored nor rewritten leaves the published tree pointing at the private build registry. `v1.6.0` is the first release to go through promotion — every earlier release was a full rebuild from the tag, which stamped every file uniformly. The glob has always been wrong; the old path just made it unobservable. ## Why it could not be caught before The rewrite was workflow-inline, so there was nothing to unit test. The only way to observe the miss was to cut a release. It is a script now — `hack/promote-rewrite-tags.sh` — and `hack/promote-rewrite-tags_test.bats` round-trips it against the real tree under `make unit-tests`, with no cluster, registry or release involved. ## The fix The three call sites each carried their own idea of where a reference can live, which is how they drifted apart. That knowledge moves into `hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()` for the files, `collect_image_refs()` for the references inside them — and all three source it, so a new storage location is declared once and reaches every consumer. The rewrite also asserts its own postcondition: it scans **wider** than it rewrites and fails the promotion when a reference survives in a location the enumeration does not know about, rather than shipping it quietly. The wide scan is shape-filtered to image-reference position, because `X.Y.Z-rc.N` is an ordinary version string that other things legitimately contain — `kubevirt-csi-driver/go.sum` pins `github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:` instead would have been wrong: `system/cilium/values.yaml` keeps `tag:` and `digest:` under separate keys. `docs/agents/image-refs.md` records the contract — three tag classes, three storage shapes, the invariants, and which consumer does what. ## Not a bug: kamaji `cluster-api-control-plane-provider-kamaji` stays at `v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild of an upstream component, versioned by that component rather than by the cozystack version line, so no rewrite is owed. A test pins this so a future fix here cannot over-reach. ## Documented known gaps, deliberately not fixed here Both are pre-existing and in the **host** dimension; the version dimension this PR owns is complete, because the version always lives in a `tag` key regardless of where the host sits. - **The nightly host rewrite cannot reach a split host.** It is a literal `<src-registry>/` substring replace, so `keycloak-operator` (host in a sibling `registry:` key) and `kubeovn` (`global.registry.address`, no trailing slash) are mirrored but keep pointing at the build registry in the published tree. Fixing it needs structure-aware rewriting. - **A reference inside a gzip is invisible.** `capi-providers-cpprovider` ships `files/components.gz`, and the promote postcondition cannot catch this class at all since `grep -rIl` skips binaries. Nothing is owed today — kamaji is component-versioned and the image is still mirrored and retagged via its `.tag` — so the residual is that a nightly's kamaji ConfigMap keeps the build-registry host. ## Testing - 17 tests in the new suite; `make bats-unit-tests` green (312 tests, exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow. - Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`, `1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no false negatives. - Every test is mutation-proven. Reverting the enumeration to the old glob, deleting any of the five YAML shapes, corrupting emitted digests, reverting extras to parse-only, dropping the unreadable-file guard, broadening shape 4 beyond `global.images`, or planting an unenumerated first-party reference in any storage shape each fail a named test. ## Reviewer note, out of scope `platform-migrations` is pinned at two different digests — `core/platform/values.yaml` at `v1.5.0` and `backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned pin; no Makefile writes that key, tracked in #3143). A `promote-retag.sh` dry-run emits **two copies to the same destination tag**, so the write-once guard will fail the promotion. This is independent of this PR and needs a decision about the backup-client image version, so it is left alone here — but it blocks the 1.6.0 promote and wants fixing before the re-cut. ## Release note ```release-note Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry. ``` Fixes the promote half of #3143. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Image promotion and nightly mirroring now detect digest-pinned image references across values files, standalone tag files, and supported templates. - Stable release preparation supports RC→stable tag rewriting via a dedicated promotion script, with broader verification to ensure no RC references remain. - **Bug Fixes** - Improved completeness of ref discovery and scanning, including previously missed template/tag locations. - **Documentation** - Added and linked guidance documenting image-reference invariants and how tooling enumerates supported storage shapes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
676c8b888a |
docs: add changelog for v1.6.0
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
1f302d0d78 |
docs(agents): correct issue ref, mirror claim and example in image-refs
The build-list violations bullet pointed at #3143, which tracks the adjacent gap (packages in the list going stale between releases) and never covers packages absent from the list; they are now tracked in the newly filed #3416. The nightly split-host paragraph claimed both affected images are mirrored while the published tree points at the build registry — today neither ref carries the source-registry host (keycloak-operator is never rebuilt by CI, kubeovn is built externally by cozystack/kubeovn-chart), so the ownership filter skips them and the gap is latent, not live; reworded to say exactly that. The opening example paired a v1.6.0 tag with the real v1.5.0 grafana digest — a ref that will never exist — and grafana is slated to stop being built; replaced with the literal committed cozystack-api ref. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
6a1e2d0d06 |
feat(backups): support PostgreSQL point-in-time recovery (PITR)
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested capability for the new backups.cozystack.io API (issue #2774). The recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing already existed on the barman-cloud-plugin branch; this adds the reliability and verification layer the issue asked for: - Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set and the recovery Cluster is not healthy, the driver reads the CNPG full-recovery pod logs and, once several distinct attempts have exited with PostgreSQL's "recovery ended before configured recovery target was reached" FATAL, fails the RestoreJob with conditions RecoveryConverged=False / Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the restore deadline. Health is checked first, so a converged recovery always wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a transient archive-lag miss on a near-now target and ignores transient recovery-pod crashes. Adds the pods/log RBAC grant the log read requires. - e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures the server timestamp, writes an after marker, restores to that timestamp and asserts the before row survived and the after row did not; step 46 submits a recoveryTime an hour ahead and asserts it fails fast with reason RecoveryTargetUnreachable. - Working, uncommented example (45-restorejob-pitr.yaml) and a docs section (docs/operations/backup-classes.md) covering the recoverable window, how to discover the earliest/latest restorable time from the backup catalog, the archive-lag tolerance, and idempotency under GitOps. Refs #2774 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
aef9e201c7 |
fix(ci): make workflow cancellation effective (#3401)
## What this PR does - allows superseded PR workflow runs to cancel `finalize` and `E2E Tests` instead of continuing because of job-level `always()` - makes `update-website-docs` cancellation-responsive while preserving its behavior when changelog generation fails or is skipped - retains step-level `always()` conditions for reports and sandbox cleanup ### Screenshots Not applicable. ### Release note ```release-note fix(ci): ensure superseded and manually canceled workflows stop long-running jobs ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Prevented release and validation-related jobs from running when a workflow is cancelled. * Updated pull request check, end-to-end testing, and stable documentation update conditions to avoid executing after cancellation. * Kept existing success-gating so follow-up tasks still require prerequisite steps to complete successfully. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |