Now that the required "E2E Tests" context is a commit status rather
than a job, a run that dies before posting leaves the head SHA at
"Expected" — which blocks the merge instead of passing it. Two paths
reached that state.
Same-repo: a `labeled` event shared the PR's concurrency group, so
labelling an open PR cancelled the live opened/synchronize run, and
the replacement run was then discarded by `plan`'s full-e2e guard --
so `e2e-report` skipped with it and nothing posted the status the
cancelled run was going to. Observed on live CI: run 30614258694
(cancelled) followed by 30614280603 (skipped) on the same SHA, 24s
apart, from a maintainer adding kind/* labels just after a push.
Label events now get their own concurrency group, so they can only
supersede each other. Release promotion is unaffected: `gh pr create
--label` attaches `release` at creation, so a promote PR gets a
single run that already sees the label (verified on #3397 and #3425,
one run each, build/finalize gated off).
Fork: `resolve` had no equivalent, so a cancelled triggering run fell
through to the `CONCLUSION != 'success'` branch and posted `failure`
— a false red on a SHA whose e2e never ran, which the replacement run
for that same SHA would not flip back (it takes the discarded-label
path and posts nothing). It now returns early on a cancelled or
wholly-skipped conclusion. Checked on the conclusion rather than the
job list because a cancelled run can report no jobs at all, which
would make the `Plan build` lookup miss.
Both stay fail-closed: declining to post can only preserve a verdict
a real run reached, never invent one.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Route the required "E2E Tests" merge gate through a GitHub commit status
posted with the default GITHUB_TOKEN (+ statuses:write), replacing the
COZYSTACK_CI app check-run. The app path needed the app granted checks:write
(it isn't — checks.create 403s "Resource not accessible by integration"), and
the earlier default-token check-run floated under a random check-suite
("PR size label"). A commit status has neither problem: branch protection
matches it by context name, it carries no check-suite grouping, and it is safe
with the default token because a fork's pull_request token is read-only and
cannot post one — only same-repo runs (e2e-report, guarded !fork) and the
privileged workflow_run in e2e-fork.yaml can set it.
The security property is unchanged: a never-posted required status stays
"Expected" and blocks the merge, so a skipped job can't smuggle a green
(#3257). Removes all COZYSTACK_CI app-token machinery from e2e-fork.yaml —
`resolve` posts the status directly, no `check_run_id` threading, no `open`
step.
Branch protection must require the "E2E Tests" status context (no GitHub App
and no app-id pin needed).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The fork e2e path in e2e-fork.yaml still ran the pre-#2826 BATS app loop
(`hack/e2e-apps/*.bats` + `test-apps-%`), which the merged Chainsaw
migration deleted — so the fork run was already broken. Rewrite it to the
same Chainsaw + Test-Impact-Analysis path as the in-tree job, and clear the
remaining review follow-ups on the publish job.
- Run E2E via a single `make test-chainsaw` invocation (no per-app retry —
e2e-testing.md §1). Suite selection and the full-e2e override come from
`resolve`'s TRUSTED base-repo listFiles/labels (new `changed_b64` /
`full_e2e` outputs), never a git diff of the fork-controlled merge ref, so
TIA cannot be shaped into skipping the suite (keeps the #3257 property).
Collect + upload the Chainsaw JUnit report like the in-tree job.
- Restrict fork image pushes to a trusted allowlist derived from the base
parent (HEAD^1) Makefiles, so a fork cannot mint an arbitrary repo path
under the CI registry by renaming an image. The charset guard stays as
defence in depth.
- `skopeo copy --all` so a multi-platform build's index digest (the one
baked into pr.patch) is preserved.
- Pin the flux CLI (FLUX_VERSION) in the privileged publish job instead of
install.sh's rolling latest.
Docs: e2e-testing.md §10 records that the fork path now runs Chainsaw + TIA
and that fork suite selection must stay sourced from trusted data.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
docs/agents/e2e-testing.md is the mandated reference for E2E CI changes but
described a single pull-requests.yaml pipeline. Document the split (same-repo
e2e in pull-requests.yaml; fork e2e in the privileged e2e-fork.yaml via
workflow_run), that the required "E2E Tests" context is a synthetic check-run
rather than a job (so renaming the e2e job must preserve it), that fork builds
export OCI archives instead of pushing, and the repo prerequisite Actions →
"Require approval for all outside collaborators".
Refs: #3257
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Moves release validation from promote time to rc time, making promotion
itself a pure retag operation. This is the follow-up to the v1.6.0
retrospective: the promote flow's checks ran on the promote PR (coupled
to promotion, skippable by never promoting), the changelog and website
docs were produced only after the release went public, and the finalize
tag push silently failed to trigger `tags.yaml` at all
(persisted-credential bug, root-caused below).
The release lifecycle after this PR:
1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes
the staging branch — and now runs a **mandatory full e2e** for every
`-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and
any published tag) get the same validation as a **manual button**
(`workflow_dispatch`).
2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**:
the release changelog is generated from the rc tag (reusable
`changelog-generate.yaml`) and committed to the rc staging branch, ready
for review long before promotion.
3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence:
the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a
successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input
for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages
the release, picks up the rc-time changelog (or generates one), and
**opens the website docs PR** on cozystack/website from the staging
branch — parked with an explicit DO-NOT-MERGE-until-finalize contract.
4. The promote PR **no longer gets `full-e2e` automatically** — its tree
is the rc's already-validated digests with only tag strings rewritten; a
maintainer can still add the label manually to force the suite.
5. Finalize performs the irreversible side effects (tag, publish, retag,
chart) — and with `persist-credentials: false` its tag push now actually
authenticates as the CI app, so the `tags.yaml` backstops
(`generate-changelog`, `update-website-docs`) fire for promoted tags.
This is the one-line fix for the v1.6.0 incident where the tag triggered
nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader`
silently defeats the app token injected via `git remote set-url`, and
`GITHUB_TOKEN` pushes create no workflow runs.
Everything is pinned by executable contract tests
(`hack/promote-gate-contract.bats`, extended
`hack/release-changelog-contract.bats` /
`release-changelog-behaviour.bats` — 48 tests total, including behaviour
tests that execute the new `hack/parse-rc-tag.sh` and
`hack/select-changelog-source.sh` for real), and `docs/release.md` is
rewritten to describe the new lifecycle end-to-end.
### Commit map (review commit-by-commit)
- `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing
changelog-before-publication rework (changelog generated at promote,
committed to the staging branch before the PR opens, finalize publishes
it as the release body; tags.yaml job becomes a verbatim-port backstop),
rebased onto main.
- `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane
(prerelease → staging branch, stable → tag; draft releases fail fast;
dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts).
- `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut
rc tags (also on `release_exists` re-runs); fixes the
`cut-prerelease.yaml` comment that falsely claimed e2e already existed.
- `2dfbeea77` — `changelog-generate.yaml` (reusable core) +
`changelog-rc.yaml` (rc-time button committing to the staging branch);
promote-rc gains a top-level `parse` job with a strict no-whitespace tag
policy (`hack/parse-rc-tag.sh`); source selection is
validate-then-fallthrough (`hack/select-changelog-source.sh`) so an
invalid higher-priority changelog can never silently win.
- `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override +
removal of the automatic `full-e2e` label (pull-requests.yaml
release-path e2e now requires an explicit label; a post-open label now
actually starts the run).
- `936190d4b` — `website-docs` job at promote time behind a
FETCH_REF-support guard (see dependency below), with the
DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before
`release-next`; `tags.yaml::update-website-docs` stays as the backstop.
- `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0
root cause).
- `717c8860f` — `docs/release.md` coherence pass over the whole new
lifecycle.
### Dependency
The promote-time website docs job needs cozystack/website#631
(`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard
detects the missing support and degrades to a ⚠️ line in the promote PR
body — promotion is unaffected and the tag-time backstop backfills, so
this PR is safe to merge first.
### Merge instructions
Please **merge-commit, do not squash**: the 13 commits are deliberately
atomic (the credential root-cause fix is separable from the feature
work) and squashing would collapse ~3k lines into one commit, hurting
bisectability.
### First live run
The e2e lane is verified by lint, contract tests and dry-traces, but has
not yet executed against a real tag; the intended first live run is a
manual `e2e-tag.yaml` dispatch against an existing published rc tag
before the gate is relied on for a real promotion.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added reusable changelog generation with validation, “present”
signaling, and safe artifact upload behavior (including RC-time
support).
* Added an E2E release-tag workflow with tag parsing, deterministic
sandboxing, and comprehensive report collection.
* Improved RC promotion with a stricter parse front-door, verified E2E
green gate (optional bypass), and earlier parallel changelog/asset
staging.
* **Documentation**
* Refreshed release-process docs to match the updated RC-to-stable,
changelog, and website-docs timing rules.
* **Bug Fixes**
* Improved PR finalization to source release bodies from verified
changelogs and warn/fallback on missing or empty content.
* Tightened label gating so E2E runs only under the intended conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The e2e conventions, the previous-logs capture script, and the global
chainsaw catch all claimed the v0.2.15 podLogs collector exposes only
container/tail/selector. It also accepts name, namespace, timeout and
the cluster fields; only the previous option is genuinely missing. The
wrong list steers catch authors toward imperative scripts for
cross-namespace log capture that podLogs handles declaratively.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three review nits, all verified against the tree, plus a tighter scope than the
first attempt at them.
The behaviour suite asserted "promote-rc does not trim the rc tag" against the
whole file. promote-rc.yaml has no .trim() today, so it passed, but any unrelated
.trim() added anywhere in the workflow would have failed a test about tag policy.
Scoping it to the parse JOB was not enough either: that job is ~180 lines, most
of them the e2e gate's github-script, whose string handling has nothing to do
with tag policy. Both greps are now scoped to the parse STEP, and run through a
code_lines filter copied from the contract suite so a YAML comment mentioning
.trim() cannot fail them.
Mutation-checked in both directions: a .trim() injected into the parse step fails
the test, one injected into the e2e-gate step of the same job does not.
- The Step 2 checklist entry still said "must be main" while Step 2 itself now
lists three valid starting points (main, detached at the release tag, detached
at the rc tag being promoted). The checklist is what an agent ticks off, so it
was the half that mattered; it now points at Step 2 rather than re-encoding the
count, so a fourth configuration cannot desync it. The heading said "Checking
current branch", which two of the three states do not have.
- hack/testdata/needs-semantics.yaml named release-changelog-contract.bats as its
consumer; the only consumer is release-changelog-behaviour.bats.
50/50 tests green across the three suites (11 + 24 + 15).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Final coherence pass over docs/release.md (and the releasing.md
pointer) after several incremental patches to the release-cycle
rework left the whole-document narrative uneven.
- Rewrite the regular and patch release sequences so a maintainer
reading top-to-bottom performs the actual flow: cut rc -> watch
mandatory rc E2E -> [optional Generate RC Changelog button] ->
promote dispatch (E2E gate; changelog generated or picked up;
website docs PR opened and parked) -> review the promote PR (no
E2E by default; E2E/changelog/website-docs lines) -> merge ->
finalize -> merge the parked website PR, then announce.
- Purge the last old-model sentence: "vX.Y.0 is tagged from the
last RC or a later commit in main" -> promoted from the rc by
digest, tag cut write-once at the promote PR merge commit.
- State the alpha/beta contract where prereleases are discussed:
built and published, button-only E2E (no mandatory rc-e2e), and
no promotion path (promote-rc accepts vX.Y.Z-rc.N alone).
- Fix the Phase 4 heading (was "chore(release): cut vX.Y.Z"; the
PR is "chore(release): promote ... -> ...") and add the missing
website-docs line to the reviewer checklist; add a
keep-parked-until-publish item to the pre-release checklist.
- Add cut-prerelease.yaml, changelog-rc.yaml, and
changelog-generate.yaml to "See also" so the workflow inventory
is complete and consistent with the "seven workflows + reusable
changelog-generate" count.
- releasing.md: replace the stale "typically involves ... creating
git tags / building artifacts" list (invalidated by the
promote-not-rebuild model) with the real high-level sequence.
Every workflow/job/step name cited was verified against the YAML on
this branch. hack/release-changelog-contract.bats stays green (24/24);
no test pins docs/release.md strings, so no test change was needed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two more review passes (Claude + Codex) on the previous commit. Codex
found the one that would have broken the very next promotion.
The agent reads its own instructions from the working tree, which is the
rc tag. Every rc cut before this change carries the old §2 of
docs/agents/changelog.md — two valid configurations, "stop" on anything
else — and a detached rc-tag checkout is exactly that "anything else". So
promoting v1.6.0-rc.4 would have presented a compliant agent with
instructions telling it to refuse, silently, producing no changelog on
every promotion of an existing rc. Verified against the tag. The same
applies to hack/validate-changelog.sh, absent from those trees entirely,
which would have failed the Verify step with "command not found" and, in
open-pr, taken the promote PR down with it. Both are now overlaid from the
dispatch ref, best-effort, in both jobs.
The structural validation added last round rejected every PATCH
changelog. It asserted a `# Cozystack vX.Y.Z` header and a 20-line floor;
patch releases use `# vX.Y.Z (date)` and v1.5.1 is a complete 19-line
release. Good AI output would have been discarded and the promote PR
mislabelled "generation failed". Measured across the tree: the header form
holds for 23 of 115 changelogs, so it was never an invariant.
The predicates now live in hack/validate-changelog.sh, tested directly
against real changelogs rather than through YAML greps — the reason the
defect shipped is that the suite pinned finalize's guard but nothing on
the promote-side validator, and stayed green when the validator was
mutated away. It accepts both header conventions, has no line floor, and
checks the version in the header, the release-link comment and the compare
link, so a changelog generated for the wrong version is caught. All 16
current-era changelogs pass; whitespace, empty, truncated and
wrong-version inputs are rejected.
Also fixed:
- A fatal artifact download in open-pr would abort the job before the PR
step — the same wedge as a skipped job, one step lower. Download is now
best-effort and the commit and PR body key off what is on disk, not an
upstream job output that says nothing about whether the transfer worked.
- A pre-existing changelog (hand-written, or preserved across the
force-push) reached the PR without ever being validated, was advertised
with ✅ and published verbatim. It now goes through the same validator,
as does the tags.yaml backstop.
- `gh pr view <branch>` matches closed PRs, so re-dispatching after
closing an abandoned promote PR edited the dead PR and opened nothing.
Branch on state explicitly.
- `gh pr edit --body` unconditionally overwrote the PR body, destroying
the upstream breaking-change notes docs/release.md instructs reviewers
to paste there. Refresh only while the body is still the generated one.
- open-pr consumed raw dispatch input while its siblings used the
validated parse; the changelog job's bash parse did not trim, so an
input with a stray space passed promote and failed there. Both fixed.
- The prompt and §2 said "trailing release-link comment"; it is the first
thing in every changelog.
Tests grow to 16, each mutation-checked: dropping the overlay, making the
download fatal, reverting the existing-changelog check to `-s`, and
removing the backstop validator each fail the suite.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Dual-model review (Claude + Codex) of the previous two commits found seven
substantive defects, two of them destructive. Both models independently
reached three of them; the split is noted per item.
Descope the website half (Claude, verified by reproduction). The docs job
was moved to promote time, but the website Makefile pins the fetch ref
with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line
BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that
cannot exist at promote time, since promote's own validate step hard-fails
if it does. update_apps.sh logs the failed fetch and exits 0 AFTER
truncating each destination file, so every app doc would become a stub,
`changed=true` would open the PR, and the new auto-merge would land it
unreviewed. Not fixable from this repo. update-website-docs returns to
tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize
auto-merge is dropped. Opening it earlier needs a cozystack/website change
first and is left as a follow-up. This also removes the unordered-sibling
race Codex found, where the release PR could merge before the docs job
opened its PR, leaving finalize nothing to merge and no retry.
Fix the non-blocking guarantee (both models). Only the AI step carried
continue-on-error; the changelog JOB did not, so a flaky `npm i -g
@github/copilot` failed it and `needs` skipped open-pr entirely — staging
branch pushed, release drafted, no promote PR. Gate open-pr on promote
alone.
Generate from a checkout the contract permits (both models). The job
checked out the rc staging branch, which docs/agents/changelog.md §2
rejects — a compliant agent stops and writes nothing, so the normal path
produced no changelog at all. Check out the rc tag detached and document
that configuration in §2. The checkout ref now comes from the validated
parse output rather than raw dispatch input.
Port, don't regenerate, in the backstop (both models). For a patch release
promoted onto release-X.Y the changelog never reaches main, so the
backstop fired and ran a second AI pass; once its PR merged,
update-releasenotes.yaml overwrote the published body with unreviewed
text. Detect the file at the tag and port it verbatim; the AI runs only
when it is absent from both main and the tag.
Survive re-dispatch (Codex). promote force-pushes the staging branch,
erasing a hand-written changelog — the documented recovery when generation
fails. Preserve an existing one across the rebuild, and stop open-pr
overwriting it with generated output.
Validate structurally, not by size (Codex). A truncated or whitespace-only
AI run passed `[ -s ]`, was committed, advertised as valid in the PR body,
and published verbatim as release notes. Assert the header, the compare
link and a plausible line count; finalize additionally refuses
whitespace-only content rather than blanking the notes.
Add hack/release-changelog-contract.bats — nine tests pinning each link in
the chain, mutation-checked: removing open-pr's if:, breaking finalize's
body read, and restoring paths-ignore each fail the suite.
Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which
described the old flow throughout.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Root-causing a recent E2E failure was nearly blocked because the
diagnostics captured on failure had collected almost nothing. Two
independent mechanisms were responsible, and both had been silently in
place for a long time.
**The failing container's previous logs were never captured.** The pod
had crash-looped, so the decisive evidence lived in the first container
instance, reachable only through `kubectl logs --previous`. Chainsaw's
`podLogs` action has no `previous` field, so this could not be expressed
per-suite; the conclusion had to be reconstructed from indirect evidence
instead. A new capture runs from the global catch and from the BATS
failure hook, selecting containers at runtime by `restartCount > 0`
across both init and regular containers, so a healthy run stays quiet
and future suites are covered without per-suite wiring.
**The catch was structurally guaranteed to be killed.** A Chainsaw
`script` op is bounded by `timeouts.exec`, which is 5m, while the
catch's own inner budgets summed to roughly 960s. It reported `SCRIPT |
ERROR | signal: killed` and produced nothing — and its second leg, the
data-plane capture, had therefore never run at all. The kill signature
was itself misleading: `timeout` exits 124 and cannot produce that
message, which comes from Go's `exec.Cmd` killing the process group.
Each catch op now carries an explicit timeout above the budgets it
contains, so the inner one fires and leaves a partial, usable artifact
instead of losing everything.
**The snapshot was truncated at 60 seconds.** `crust-gather --duration`
defaults to 60s and was never passed, so every snapshot ever taken was
cut short, with nothing in the artifact saying so. The API discovery it
performs first is not covered by that budget at all, which is where the
outer wall-clock was going on a cluster with a degraded aggregated
APIService — precisely the state a failed run is in. Both budgets are
now set explicitly, and the collector's output goes to a log inside the
snapshot with a completeness verdict, so "is this snapshot complete?" is
answerable from the artifact.
Failure paths are reported rather than swallowed: an unreachable API
says so instead of implying nothing had restarted, a truncated capture
is kept and marked, and every dropped item is logged, since silent
truncation reads as full coverage.
### What is not verified here
The mechanism claims are read from the pinned versions of Chainsaw and
crust-gather. Everything cluster-facing is not verified by this PR: the
go-template's output shape against a live API server, `--previous`
returning the first instance for a real crash-looper, and whether the
new collection budgets suffice on a degraded cluster. The bats suite
covers the selection helpers only, and those cases are mutation-tested.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Diagnostics**
* Improved end-to-end failure evidence by capturing “previous instance”
logs for crash-looping pods (including restarted init containers) and
prioritizing relevant namespaces.
* Added time-bounded diagnostics collection with explicit outer
timeouts; collector output is saved to per-snapshot log files.
* Classifies collector outcomes as complete, timeout-truncated, or
failed, while preserving the original test failure signal.
* **Documentation**
* Expanded E2E failure-diagnostics guidance with new timeout and
evidence-capture conventions.
* **Tests**
* Added unit-style tests for previous-log selection, prioritization, row
limiting, and deterministic log artifact naming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The build-list violations bullet pointed at #3143, which tracks the
adjacent gap (packages in the list going stale between releases) and
never covers packages absent from the list; they are now tracked in
the newly filed #3416. The nightly split-host paragraph claimed both
affected images are mirrored while the published tree points at the
build registry — today neither ref carries the source-registry host
(keycloak-operator is never rebuilt by CI, kubeovn is built externally
by cozystack/kubeovn-chart), so the ownership filter skips them and
the gap is latent, not live; reworded to say exactly that. The opening
example paired a v1.6.0 tag with the real v1.5.0 grafana digest — a
ref that will never exist — and grafana is slated to stop being built;
replaced with the literal committed cozystack-api ref.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubectl logs --previous returns the immediately preceding terminated
container instance, not the original first one — after multiple restarts
the first instance is already gone. Reword the helper, the BATS and
Chainsaw failure handlers, and the diagnostics doc to match the actual
--previous semantics.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Review of the previous commit found the postcondition itself could abort
a release. It grepped the bare "X.Y.Z-rc.N" substring across the tree, and
three live files carry such strings without being image refs:
kubevirt-csi-driver's go.sum pins github.com/golang/protobuf v1.4.0-rc.2
-- a version cozystack actually cut, so promoting v1.4.0 would have
failed on a Go checksum line -- metallb_test.yaml names v1.5.0-rc.2 in a
comment, and the console pnpm-lock.yaml carries rolldown@1.0.0-rc.15. The
remediation the error message gave would have made things worse: declaring
go.sum as a ref source would sed a checksum file.
Match the version only in image-reference position now: after an
image/repository/tag key with no intervening '#', or immediately before an
@sha256 digest. Narrowing it to "${RC}@sha256:" alone would have been
wrong -- cilium keeps `tag: v1.5.0` and `digest:` under separate keys, so
the version is not always adjacent to the digest. Verified against four
real rc versions that previously aborted.
The suite also only tested the fail-closed direction, which is why that
defect survived, and the collector had no completeness guard at all: a
first-party ref added at an unenumerated path on a `latest@` tag was
mirrored by nothing and passed all eighteen tests. Since the mirror's
failure mode is the worst of the three -- the published nightly points at
the private build registry -- that gap mattered most where coverage was
thinnest.
Added: a fail-open test using the three real rc versions above; a
collector completeness diff that discovers refs by content and fails on
any the enumeration misses; coverage for the OCI-artifact shape, whose
deletion previously left every suite green; and an unreadable-file test,
since a read error was being treated as "no match" and silently skipped.
The component-versioned test was vacuous -- its rc version shared no
substring with the kamaji tag it claimed to protect, so it passed by
construction. It now uses an rc version sharing kamaji's 0.19.0 prefix,
which a loosely-anchored implementation would corrupt.
Round-trip fixture discovery matched a combined 'ghcr.io/cozystack/
cozystack/' prefix, which skips keycloak-operator (host in a sibling
registry: key) and kubeovn (host in global.registry.address). It now
matches 'cozystack/cozystack' and stamps split `tag:` keys too.
Also documented two pre-existing gaps rather than half-fixing them: the
nightly host rewrite is a contiguous-substring replace and cannot reach a
split host, and a ref inside a gzip is invisible to every tool including
the postcondition, which skips binaries. Both are host-dimension defects;
the version dimension this change owns is unaffected, because the version
always lives in a tag key regardless of where the host sits.
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Root-causing a crash-loop failure was blocked twice over by the
diagnostics collected on failure.
`kubectl logs` shows only the current container instance, so for a
crash-looping pod the suites' `podLogs` collector captured the Nth
restart -- a fresh replay of the same startup path -- while the
evidence that explained the failure sat in the first, interrupted
instance. Chainsaw's podLogs collector has no `previous` option, so
add e2e-capture-previous-logs.sh and drive it from the global catch
and from the BATS failure hook. It discovers containers with
restartCount > 0 at runtime (init containers included, where an
interrupted bootstrap lives), orders the app suites' namespace first,
and dumps each to the job log and the snapshot tree. Containers that
never restarted are skipped, so a healthy run emits one line rather
than a page of "previous terminated container not found".
Every way that capture can come up short is named rather than
inferred, because kubectl exits 1 for a garbage-collected container,
an RBAC denial and an unreachable node alike: the container cap, a
pod list that never returned (which is not the same as nothing having
restarted), a read that timed out versus one kubectl refused and the
reason it gave, and a previous instance that produced no output. A
partial capture is kept and marked truncated rather than deleted --
it is the only copy of a log already gone from the cluster.
The crust-gather snapshot was killed by the Chainsaw op containing it.
A script op is bounded by timeouts.exec (5m), and the catch's own
wall-clock wrappers summed to 960s, so the script was SIGKILLed
partway through -- taking out the process group mid-write and skipping
the data-plane capture entirely. Give each catch op an explicit
timeout above the budgets it contains, so an inner timeout fires
instead and leaves a partial, still-servable snapshot behind. The
data-plane leg drops from 600s to 300s at that call site so both legs
fit one envelope; the BATS hook keeps the wider budget, having no
containing op.
Set crust-gather --duration explicitly as well. It is the collection
phase's own budget, it defaults to 60s, and on elapse it abandons the
collection and skips its finish step, leaving a truncated tree with no
indication anything is missing. The outer wall-clock has to exceed it
because the API discovery that runs first is not covered by that
budget at all -- with an unhealthy aggregated APIService, the state a
failed run is usually in, discovery alone can consume the whole
timeout before a single object is written. Send each collector's
output to a log beside its snapshot instead of /dev/null, with a
complete/truncated/failed verdict, so that question is answerable
from the artifact.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Promotion rewrote the rc version substring across the depth-2 package
values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the
premise that the kubernetes app was the only one whose .tag files carry
the cozystack version. Nine other .tag files do, and system/multus seds
its ref straight into a vendored upstream manifest. v1.6.0 was therefore
staged with 33 of 55 first-party refs still reading v1.6.0-rc.4.
promote-retag.sh and nightly-mirror.sh had the same blind spot from the
other direction: both scanned only the depth-2 values.yaml, so those
images were never retagged to the stable version (30 refs selected
before, 42 after) and never mirrored to the public registry for a
nightly. A retag miss merely leaves an image short of a tag, since the
digest still resolves inside one registry; a mirror miss is worse,
because the host rewrite walks the same file list and a ref that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.
The three call sites each carried their own idea of where a ref can
live, which is how they drifted apart. Move that knowledge into
hack/lib/image-refs.sh as the single enumeration -- image_ref_files()
for the files, collect_image_refs() for the refs inside them -- and have
all three source it, so a new storage location is declared once and
reaches every consumer.
The rewrite itself moves out of promote-rc.yaml into
hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only
observable by cutting a release: there was nothing to unit test. It now
also asserts its own postcondition, scanning wider than it rewrites and
failing the promotion when any ref survives in a location the
enumeration does not know about, rather than shipping it quietly.
Component-versioned images (kamaji's v0.19.0-cozystack.N) and
third-party pass-through images do not ride the cozystack version line
and are deliberately left alone; a test pins that so a future fix here
cannot over-reach. capi-providers-cpprovider is a documented known gap:
its ref is also embedded in a gzipped copy that the chart actually
ships, and rewriting only the readable copy would diverge the two.
docs/agents/image-refs.md records the contract -- three tag classes,
three storage shapes, the invariants, and which consumer does what.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The downstream-coordination checklist still directed contributors at an
"externalIPs to exposureClass" migration; exposureClass was removed in
3dbc7d366 and the native replacement key is publishing.loadBalancerClass.
Last leftover of the ExposureClass removal this PR sweeps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Cozystack is upstream for repositories that nothing keeps in sync with
it, and no CI job compares the two sides, so a change here breaks them
in silence. Only two narrow paths are automated, and both fire on a
release rather than on a PR.
It has already happened. The website docs generator works from an app
list hardcoded in its own Makefile, and packages/apps/opensearch was
never added to it, so an application we ship has no reference page at
all. The Terraform provider still offers a Kubernetes version this repo
removed. The plugin repo still sends operators to hack/ scripts that
were deleted, one of them rewritten as a Go command.
Add a Downstream Repositories trigger map to the contributing guide,
naming per repository which change here forces a change over there and
which file to touch, and a checklist in the PR template that links to
it. The template addresses AI agents directly: they author a large share
of PRs and will otherwise tick a box without opening the follow-up it
claims.
The coupling is wider than the obvious ones, and mostly invisible. talm
restates the node contract, overlapping ours but already diverged, so it
must be diffed rather than overwritten. cozyhr reads an annotation the
operator writes and silently merges the wrong values if it is renamed.
cozy-proxy is vendored in from its own repo, yet consumes labels
produced here and quietly stops handling external VM services if they
move. The telemetry server queries metric names emitted here, and a
rename just empties its dashboards. None of these fail loudly.
Guard the map with a unit test, because a map about silent drift must
not drift silently itself. It pins every in-repo path the map cites to a
file that exists, pins the map and the checklist to the same repository
list, and pins the workflow carve-out below to the map's location, to
the job that reads the flag, and to the output that carries it: cutting
any one of those three leaves the other two looking wired up. The header
states plainly what the suite cannot check, so a green tick is not
mistaken for a correct map: whether a coupling is real, and whether it
still holds over there, stays a human's job.
Editing the map alone is a docs-only PR, which skips the unit tests and
would exempt the map from its own guard. Rather than widen the docs-only
rule, which also gates the Talos image build, the plan step exports a
separate flag that only the unit-test job reads: a docs PR now runs the
tests and still builds nothing. It detects the map by its old path too,
since a rename is reported as the new path alone.
Document that gh pr create --body and --body-file replace the body
wholesale, which would otherwise drop the checklist from every
CLI-authored PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The v0.2.15 syntax/semantics traps were tribal knowledge scattered across PR
review notes (the number-literal caveat lived only in #2826's body, which
evaporates on merge). Consolidate them into a durable subsection so the next
author finds them before hitting each one fresh: filter-as-list conditions,
no number-literals/backticks in JMESPath, error: fails hard on an unknown
GVK, no conditional Test-level skip, step-level-only finally, bounded/blocking
cleanup surfacing latent teardown bugs, projection-style failure diffs, and
the catch-script cwd.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- scope create-github-app-token to the current repo with read-only
contents permission (was org-wide, inheriting the app's blanket
installation permissions); both token steps only read releases/assets
- fail-fast the Talos image upload when the disk is missing
(if-no-files-found: error) so drift fails in build-talos, not later
as a confusing e2e download error
- download the flux install script to a tempfile before executing it,
so a failed curl is not masked by the piped `sudo bash` exiting 0
- fix the regular-PR artifact-source comment (Talos disk comes from
build-talos, pr.patch from finalize) and the TIA scope wording in
docs/agents/e2e-testing.md
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch
diverged landed a competing single-VM build restructure (#2939) and
test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build
supersedes #2939:
- .github/workflows/pull-requests.yaml: keep #2983's
plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer
resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact
interface (pr-patch + talos-image) is identical on both sides, so e2e depends
on finalize/build-talos with no input rewiring. main's top-level permissions
and SHA-pinned action refs applied across all jobs.
- All other files merged cleanly (only pull-requests.yaml conflicted).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The "Pin the e2e management cluster Kubernetes version" item is no longer
in-flight — it is merged and live in hack/e2e-prepare-cluster.bats
(--kubernetes-version v1.33.12). Remove it from the "not yet the merged
standard" section; its rationale lives in the bats comment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The "What TIA does and does not do" section described the old single
Build job and the `detect-changes` paths-filter. Update it for the new
`plan` job (docs gate + build matrix), note that image builds are now
diff-scoped by hack/build-matrix.sh (distinct from TIA's app-loop trim),
and point the "widen the skip" advice at the plan docs-gate.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on
every release tag as a non-blocking alarm — its own header notes "a
failure here is an alarm, not a release blocker." It duplicates coverage
the release-cut PR already provides: the "Prepare release" commit bakes
image digests into `packages/core/` (and other packages), which TIA
escalates to the full suite, so the release PR exercises the whole suite
against the commit the tag will point at.
It has also never been green since it was added (2026-05-02): every tag
run failed at the build step. Even after #2973 fixed the registry login
today, the next run still failed (Build Talos image). Removing a
perpetually-red, non-gating duplicate removes noise without losing
coverage.
Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s
full-suite pattern, repoints its unit test to `pull-requests.yaml`, and
updates the e2e-testing doc.
Refs #2397
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Promotes universal project guidance that previously lived only in a
local
(gitignored) `CLAUDE.md` into the committed agent docs, so every
contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
- `overview.md`: Build & Development Commands (root + per-package make
targets,
build env vars), Values schema generation, a Testing section, Domain
references
- `contributing.md`: a "Verify Before Considering Done" matrix + Review
Expectations
- Dedup: `overview.md`'s Git Commits block now points to
`contributing.md`
> Note: `overview.md` links to `docs/agents/e2e-testing.md` added in
#2932 — merge
> that one first or together.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Expanded contribution guidance with a “Verify Before Considering Done”
pre-submission checklist and clearer expectations for including
behavioral regression tests in bug-fix PRs.
* Broadened build/development documentation with a “Build and
Development Commands” guide, including required environment variables,
values-schema generation, and test entry points.
* Updated contribution workflow notes by consolidating commit-convention
guidance and adding pointers to task-specific design references.
* Added a Claude entry-point document that directs to the main agents
guide and explains local override placement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from coderabbitai on docs/agents/overview.md:145:
internal/controller/dashboard/README.md does not exist (no such directory),
and there is no dashboard reference doc to point to. Remove the dead bullet;
the backups DESIGN.md reference is valid and kept.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:145:
reword the awkward "on the template" phrasing to "aligned with the template".
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/overview.md:103:
the Helm-only target is make helm-unit-tests; make unit-tests runs the full
unit suite. Point the bullet at the precise target and note the difference.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on docs/agents/overview.md:72:
make unit-tests runs the full suite (Helm, BATS, Go, presets, readiness),
not only Helm unit tests.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promote the universal project guidance that previously lived only in a
local CLAUDE.md into the committed agent docs, so every contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
overview.md gains Build and Development Commands (root + per-package
make targets, build env vars), Values schema generation, a Testing
section, and Domain references. contributing.md gains a "Verify Before
Considering Done" matrix and Review Expectations. Dedup: overview's Git
Commits block now points to contributing.md instead of restating it.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Capture the e2e stabilization conventions established over the last few
months as a routed agent doc: no retries on deterministic CI steps,
event-driven backstops over fixed sleeps, no EXIT/RETURN traps,
fail-fast HelmRelease-readiness gating, the status.history remediation
guard, first-class failure diagnostics, and a reviewer checklist. Also
documents Test-Impact Analysis (TIA) and its real scope — it trims only
the per-app loop, runs after install, and most PRs escalate to the full
suite. Wire it into AGENTS.md so every agent reads it before e2e/CI work.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- tags.yaml: the check_changelog step now looks at origin/main rather than
the working tree. After Round 2 we check out the release tag, which never
has the changelog file (the PR opened by this job merges to main after
the tag is cut), so the previous `[ -f ]` test always returned false and
the gate was effectively dead on reruns.
- changelog.md: Step 6's `RELEASE_END=$(git log -1 --format=%ai HEAD)`
(lines 179, 222) was missed in the previous "never HEAD" sweep — now
uses v<new_version>, matching Step 5.
- changelog.md: Step 2 ("on main branch") was incompatible with the CI flow
which now runs detached at the release tag. Relaxed it to list both
configurations (interactive on main; CI detached at v<version>), while
the "Scope and boundaries" section continues to forbid the agent from
switching branches.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- tags.yaml: drop "(or HEAD)" from the changelog prompt; the doc says
"never HEAD" and the workflow now matches.
- docs/agents/changelog.md: exclude the current release file from the
"previous contributors" set used for first-time-contributor detection.
The previous globbing on `docs/changelogs/v*.md` swept up the file
being generated, so `comm -23` would never report any newcomers.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- Drop the "orphan backport, drop entry" rule: a backport whose original is on
main but hasn't shipped in a stable tag is the first delivery and must stay
in the changelog.
- Add `v` prefix to `<previous_version>`/`<new_version>` placeholders in all
four `git log` invocations for consistency with the rest of the doc.
- Apply the bot/CI account filter to the new-contributors detection (and the
standalone previous-contributors extraction) so bot accounts can no longer
leak into the human Contributors list.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The generate-changelog job in .github/workflows/tags.yaml previously
checked out main and ran the AI agent there. The agent followed
docs/agents/changelog.md, which instructed it to compute the release
range with `git log <previous_version>..HEAD`. For minor releases
(vX.Y.0) cut from main this happens to give the right answer; for
patch releases (vX.Y.Z, Z>0) cut from a release-X.Y branch, HEAD-on-
main is a strict superset of the release tag and contains commits
that were merged to main both before and after the tag.
The v1.3.1 changelog generated by this workflow (#2480) demonstrated
the failure mode: it included 8 PRs merged to main but never shipped
in v1.3.1, 6 backport PRs that landed on release-1.3 *after* v1.3.1
was tagged, both originals and their backports as separate entries,
a hallucinated 2024 PR, and the cozystack-ci bot in the contributors
list. The corrected v1.3.1 changelog contains the single fix that
v1.3.1 actually ships.
This PR fixes the root cause and tightens the agent guardrails:
* tags.yaml: check out the release tag commit (`ref: ${{ steps.tag.
outputs.tag }}`) instead of `main`, so HEAD == the release commit
and `git log v<prev>..HEAD` corresponds to the actual release. The
Create-changelog-branch step still creates the PR branch from
origin/main, so PRs continue to merge cleanly into main.
* tags.yaml: update the AI prompt to state explicitly that HEAD is
the release commit and that the upper bound for the release range
is the new tag, not main.
* docs/agents/changelog.md: replace `..HEAD` with `..v<new_version>`
in every example so the instruction is unambiguous regardless of
which branch is checked out.
* docs/agents/changelog.md: add a hard rule that backport PRs MUST
be merged into the original entry (single line, two PR numbers),
never listed as a second entry. Document the edge case where the
original PR is outside the range — drop the entry.
* docs/agents/changelog.md: forbid using the brief description
verbatim as the detailed description, and forbid inventing entries
for PRs that are not in the release range.
* docs/agents/changelog.md: filter bot/CI accounts (`app/*`,
`*[bot]`, cozystack-ci, github-actions, dependabot, renovate) out
of the human Contributors list. Bot attribution on individual
entries is still valid.
The corresponding correction for the v1.3.1 changelog itself is in
PR #2480 on the changelog-v1.3.1 branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adopt CNCF/Kubernetes label conventions for issues and PRs and add
automated labeling.
**Canonical label file**: `.github/labels.yml`. Synced into the
repository by `.github/workflows/labels.yaml` (EndBug/label-sync@v2) on
push to `main`, weekly cron, and manual dispatch. UI-only label edits
are overwritten — propose changes via PR to this file.
### Label namespaces
Following the [Kubernetes label
scheme](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md):
- `kind/*` — issue or PR type (bug, feature, documentation, support,
cleanup, regression, flake, failing-test, api-change, breaking-change)
- `priority/*` — urgency (critical-urgent, important-soon,
important-longterm, backlog)
- `triage/*` — review state (needs-triage, accepted, needs-information,
not-reproducible, duplicate, unresolved)
- `lifecycle/*` — issue/PR lifecycle (active, frozen, stale, rotten)
- `area/*` — subsystem; 15 seeded plus `area/uncategorized` fallback.
Extensible — propose a new area when no existing one fits.
- `do-not-merge/*` — PR merge blockers (work-in-progress, hold)
Cozystack-specific labels preserved: `epic`, `community`, `help wanted`,
`good first issue`, `quality-of-life`, `upstream-issue`, `backport`,
`backport-previous`, `release`, `automated`, `debug`, `sponsored`,
`lgtm`, `ok-to-test`, `security/*`, `size:*`.
### `area/*` set
15 areas seeded by activity in open issues and PRs: `area/ai`,
`area/api`, `area/build`, `area/ci`, `area/dashboard`, `area/database`,
`area/extra`, `area/kubernetes`, `area/monitoring`, `area/networking`,
`area/platform`, `area/release`, `area/storage`, `area/testing`,
`area/virtualization`. Plus `area/uncategorized` as the auto-labeler
fallback.
### Migration safety
Existing labels are renamed via `aliases:` in `labels.yml`. GitHub
preserves the label ID, so all currently tagged issues and PRs keep
their tags under the new name without losing references:
| Old | New |
|---|---|
| `bug` | `kind/bug` |
| `enhancement` | `kind/feature` |
| `documentation` | `kind/documentation` |
| `question` | `kind/support` |
| `frozen` | `lifecycle/frozen` |
| `stale` | `lifecycle/stale` |
| `duplicate` | `triage/duplicate` |
| `do-not-merge` | `do-not-merge/work-in-progress` |
| `do not merge` | `do-not-merge/work-in-progress` |
`delete-other-labels: false` on the initial rollout. Generic
GitHub-default labels (`wontfix`, `invalid`) are preserved untouched and
will be removed in a follow-up cleanup PR. EndBug processes aliases
sequentially, so the second of the two `do-not-merge*` aliases hits a
name collision and logs a warning — the legacy label survives that one
sync and is cleaned up in the same follow-up.
### PR auto-labeling
`.github/workflows/pr-labeler.yaml` parses each PR title on `opened`,
`edited`, `reopened`, and `synchronize` and applies labels additively
(never removes):
- **type → `kind/*`**: feat, fix, docs, chore, refactor (others get no
kind)
- **scope → `area/*`**: scope mapping covers all current cozystack
components (full table in `docs/agents/contributing.md`)
- **`!` after type or `BREAKING CHANGE:` footer**: applies
`kind/breaking-change`
- **`[Backport release-1.x]` prefix**: stripped before parsing;
`area/release` and `backport` labels added
- **Composite scope** (`feat(platform, system, apps): …`): each part
mapped independently
- **Bracket fallback** (`[scope] description`): maps `area/*` but cannot
infer `kind/*`
- **Unmapped scope or non-conventional title**: applies
`area/uncategorized` for human review
### Schema validation
`.github/workflows/labels.yaml` runs a `validate` job on every PR
touching `labels.yml` or its workflow. Asserts:
- description ≤ 100 chars (GitHub REST API limit)
- color is 6-char hex without leading `#`
- unique top-level names
- aliases do not collide with top-level names
Sync runs only on push to main, weekly cron, and manual dispatch; PR
runs validate-only.
### Hardcoded label/title references updated
- `.github/ISSUE_TEMPLATE/bug_report.md`: `labels: 'bug'` → `labels:
'kind/bug'`
- `.github/workflows/tags.yaml`:
- changelog PR labels `['documentation', 'automated']` →
`['kind/documentation', 'automated']`
- release PR title `Release v${version}` → `chore(release): cut
v${version}` (so the auto-labeler applies `kind/cleanup` +
`area/release`)
- changelog PR title `docs: add changelog for v${version}` →
`docs(release): add changelog for v${version}` (so the auto-labeler
applies `kind/documentation` + `area/release`)
### Documentation
- `AGENTS.md`: Activation entry pointing agents to `labels.yml` and the
PR title auto-labeling rules. States explicitly that `area/*` accuracy
outweighs reuse — propose a new area when none fits, do not shoehorn
into a wrong one.
- `docs/agents/contributing.md`: PR Title Auto-Labeling section with
type→kind and scope→area tables.
### Out of scope (follow-up PRs)
- Removal of redundant labels (`wontfix`, `invalid`, plus the surviving
legacy `do not merge` if alias-rename collision keeps it)
- Org-wide sync from `cozystack/.github/labels.yml`
- Dosu bot configuration update for `lifecycle/stale` (requires
dashboard access)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automated PR labeling from Conventional Commits (type → kind/*, scope
→ area/*), plus a comprehensive namespaced label taxonomy.
* **Chores**
* Workflows to validate, sync, and auto-apply labels (including
scheduled/manual runs and validation checks).
* Added repository-wide label configuration and normalized bug label
metadata to namespaced form.
* Updated release PR titling/labeling conventions.
* **Documentation**
* Contributor and agent guidance on PR title conventions, label
mappings, and triage procedures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Scopes section was read as an exhaustive enumeration, which led to
review feedback flagging any scope outside the list as invalid. The
intent has always been that contributors pick the most specific scope
for the change and extend the list when a genuinely new area appears.
Reword the section accordingly and add operator as an example scope.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe when to set upgradeCRDs: CreateReplace (operators that evolve
their CRD set additively between versions) and the data-loss risk of
enabling it on operators that drop fields.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:73:
add space before closing pipe in the type to kind mapping table for
consistency with other rows.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:88:
… replaced with ... for compatibility across editors and tools.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add .github/labels.yml as the canonical label set, synced into the
repository by .github/workflows/labels.yaml using EndBug/label-sync.
Conventions follow the Kubernetes scheme:
https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md
Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/,
do-not-merge/. Cozystack-specific labels preserved (epic, community,
security/*, size:*).
Migration via aliases keeps references on existing issues and PRs:
- bug -> kind/bug
- enhancement -> kind/feature
- documentation -> kind/documentation
- question -> kind/support
- frozen -> lifecycle/frozen
- stale -> lifecycle/stale
- do-not-merge -> do-not-merge/work-in-progress
delete-other-labels is false on the initial rollout; redundant labels
("do not merge", duplicate, invalid, wontfix) stay until a follow-up
PR removes them after stabilisation.
The labels workflow has a validate job (python3 schema check) that
runs on PR. Sync runs only on push to main, weekly cron, and manual
dispatch. Schema invariants:
- description <= 100 chars (GitHub REST API limit)
- color is 6-char hex without leading #
- unique top-level names
- aliases do not collide with top-level names
PR auto-labeling (.github/workflows/pr-labeler.yaml):
- Parses PR title as Conventional Commits header (type, scope, !).
- type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs ->
kind/documentation, chore/refactor -> kind/cleanup; style, perf,
test, build, ci, revert -> no kind label).
- scope -> area/* via embedded mapping; composite scopes split on
comma. Bracket-style fallback ([scope] description) maps area/*
but cannot infer kind/*.
- '[Backport release-1.x]' prefix is stripped; area/release and
backport labels are added.
- '!' after type or 'BREAKING CHANGE:' footer in body adds
kind/breaking-change.
- Unmapped scope or non-conventional title adds area/uncategorized
to flag for human review.
- Additive only — never removes existing labels.
Hardcoded label references updated:
- .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug)
- .github/workflows/tags.yaml (documentation -> kind/documentation)
AGENTS.md gains an Activation entry pointing agents to labels.yml
as the source of truth and to contributing.md for the title
auto-labeling table.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:11:
Scope linters kept flagging valid scopes like 'agents' as unknown because
the list read as exhaustive. Annotate it as examples (not exhaustive) and
add 'agents' to the Other group so both humans and review bots stop
tripping on scopes that are already in regular use across the repo
history.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:30:
Reword "likely to need regenerated" (regional construction) to
"likely needs to be regenerated" for standard technical prose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai on docs/agents/contributing.md:26:
Replace the hard-coded packages/extra/<name> path in the example with
packages/<apps-or-extra>/<name> so the example matches the preceding
text that describes both apps and extra packages.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pre-commit CI runs make generate in every package and fails with exit
123 on any uncommitted generator output. Add explicit guidance so
agents stage regenerated README.md, values.schema.json and
packages/system/<name>-rd artifacts alongside the hand edits.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>