Commit Graph
539 Commits
Author SHA1 Message Date
Aleksei Sviridkin cc8b01d5cd docs(ci): say when each lane's status opener actually lands
The reference section claimed both lanes open the required context when
their run starts, and enumerated the triggers it protects against: a
re-run, a reopen, a `full-e2e` label. Every one of those starts a Pull
Request run, and on a fork that run cannot open the status at all. The
opener is guarded same-repo only, because a fork's `pull_request` token
is read-only, and the fork's opener lives in the privileged workflow,
which starts only once the unprivileged build has finished. So on the
lane the sentence was most likely to be read about, an already-green
head SHA keeps reading green for the whole build.

The gap is structural. A read-only token cannot post a status at any
point in the run, and that same read-only token is what makes publishing
the gate with the default token safe in the first place. It is bounded
by the build rather than open-ended. What was wrong was only the claim
that it does not exist.

Say so in all three places that carried the overstatement: the section
itself, the opener's own comment, and the contract test, which pins that
each lane has an opener and cannot express when it lands.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 02:10:35 +03:00
Aleksei Sviridkin c45179a9d0 docs(ci): correct the reason the build matrix keeps the rename-detected list
The comment justified the split by saying a path that no longer exists
has nothing to build. That is only half the picture: on a rename between
packages the source package is still there and still needs rebuilding,
and rename detection drops it from the list the matrix reads, so the
matrix under-builds. The behaviour predates this change and is out of
scope for a docs-only decision, but the comment presented it as
deliberate and correct rather than as inherited and left alone.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 01:26:28 +03:00
Aleksei Sviridkin 9c68b1f83e fix(ci): stop a cancelled fork run from publishing a verdict
The invariant that a cancelled run publishes nothing was written down in
three places and held in two of them.

The fork lane's terminal reporter ran under `always()`, which includes
cancellation. When a second privileged run supersedes the first, the
dying one still posted `failure` on a head SHA it no longer owned, for a
suite that never finished. It self-corrected once the superseding run
reported, but that can be two hours later, and until then a legitimate
pull request reads red. The same-repo lane was moved off `always()` for
exactly this reason. Move this one too: a failed publish or e2e still
reports, since neither is a cancellation, and a cancelled run now leaves
the `pending` it opened standing until its successor concludes.

The same-repo guard's rationale cited a scenario that no longer exists.
It described a label event cancelling the run that opened the pull
request, which the concurrency split made impossible for every label the
workflow discards. The guard is still needed, for a push, for a
`full-e2e` label superseding the head run, and for a manual cancel, so
the comment now names those instead of an extinct one.

The reference section listed the pieces this invariant rests on and
omitted the reporters, which is the half a reader is most likely to
touch, and counted them, which was already wrong and would go wrong
again on the next addition. It now enumerates without counting.

The contract pins both reporters, because both lanes have had this
defect at different times in the same shape.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 01:26:09 +03:00
Aleksei Sviridkin 8a21085324 fix(ci): decide docs-only from both sides of a rename
The two lanes answered "is this docs-only" from different sources and
could disagree. `plan` reads `git diff --name-only`, which applies rename
detection and reports a rename as its new path alone, so moving a code
file into docs/ read as docs-only. The fork lane reads the PR file list,
where a rename carries `previous_filename`, and reads the same change as
code.

Disagreement is not a near-miss here. The same-repo lane builds nothing
and uploads no patch artifact; the fork lane concludes e2e is required
and then fails closed because that artifact is missing, reporting that
the fork build did not complete. It completed — it correctly built
nothing. There is no push the contributor can make to clear it and no
label that overrides it, so a legitimate contribution is permanently
red.

Point the docs-only decision at the rename-blind list, which the job
already computes for the downstream-trigger-map flag and whose comment
names this exact trap. Both lanes now agree, and the same-repo side
stops shipping a removed code path untested — until now such a change
skipped every build and reported e2e as not required. The build matrix
keeps reading the rename-detected list: it picks packages to rebuild,
and a path that no longer exists has nothing to build.

The contract file gains the agreement as a pin, and grows a header to
match what it now covers. The fork half is pinned on the docs-only
expression rather than on `previous_filename`, which also appears in the
file list a few lines down and would have kept the assertion satisfied
with the check deleted.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 01:11:30 +03:00
Aleksei Sviridkin d3515509c8 fix(ci): let a status-API blip skip the opener instead of failing the run
The step that claims the required context is the first in `plan`, and
`checks`, `build`, `build-talos`, `finalize` and `e2e` all hang off that
job. A transient failure from the status API would therefore discard a
full build and suite for a call that is an optimisation, not a gate.

Mark it `continue-on-error`. Losing the opener degrades exactly to the
behaviour that preceded it — the head SHA keeps whatever status it had
until `e2e-report` concludes — while a run that dies before reporting
leaves the merge blocked either way. The terminal post keeps its own
error handling, because that one failing silently is what would let a
run finish with no verdict at all.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 00:54:29 +03:00
Aleksei Sviridkin ee844c98a5 fix(ci): open the "E2E Tests" status when the same-repo run starts
Moving the gate from a job to a commit status lost a property the job
had for free. A commit status is latest-wins per (sha, context) and
never expires, so a head SHA that already went green keeps reading green
from the moment a new run starts until `e2e-report` posts at the end.
Re-running all jobs, reopening, or adding full-e2e therefore leaves the
pull request mergeable for the whole of a suite that may be about to
fail. While the gate was a job, GitHub created a fresh non-success check
run each time and the window did not exist.

Claim the context at the start of `plan`, the way the fork lane already
claims it in `resolve`, and let `e2e-report` conclude it as before. The
call is same-repo only: a fork's `pull_request` token is read-only, so
posting from there would 403 and fail the job on every fork PR — and
that read-only token is what makes the whole commit-status design safe.
Fork PRs get their opening status from the privileged workflow instead.

The contract test grows a case pinning that each lane writes the context
from two points, one of them pending. Both assertions on the same-repo
opener are scoped to the `plan` job: the bare fork guard occurs on
several steps in other jobs, so a whole-file "at least one" passed with
the opener's guard deleted, pinning unrelated guards and reporting that
as coverage.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 00:40:49 +03:00
Aleksei Sviridkin 3722b97fb5 fix(ci): terminate the fork TIA file list so its last path is read
The privileged lane builds the changed-file list in JavaScript and hands
it to `hack/select-e2e.sh` through base64. `Array.join` puts a separator
between entries and none at the end, so the decoded file had no trailing
newline — and the selector reads it with `while IFS= read -r file`,
which POSIX defines as returning non-zero at EOF without a newline.
The loop body never runs for that last path, so it is silently dropped.

For a single-file pull request, the modal fork contribution, that is the
whole list: the selection comes back empty, the Chainsaw step is skipped
on it, the job concludes success and the required status is published
green with no suite executed. With more files the last one is dropped
instead, which under-selects — and it can also suppress an escalation,
since a path the selector does not recognise inside packages/ is exactly
what would have widened the run to the full suite.

Nothing in the log gave it away: the step prints the list with `sed`,
which renders an unterminated last line the same as a terminated one, so
the log showed a path the selector had never seen.

Terminate the list where it is built. The in-tree lane is unaffected and
always was — it feeds `git diff --name-only`, whose output ends with a
newline — which is why this is the first caller to meet the behaviour.
The read loop itself still drops an unterminated last line for any
future caller; fixing that belongs with the selector and its own test
surface, not here.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 00:23:01 +03:00
Aleksei Sviridkin fd06fb32e0 docs(ci): say what fork TIA hardening buys, and what it does not
Four comments and a documentation bullet claimed that routing the
changed-file list through the base-repo API stops a fork from making
test selection pick nothing. It does not. The `e2e` job runs from the
fork's own tree, so `hack/select-e2e.sh` is the fork's script: a PR that
edits it to print nothing gets `skip=true`, the suite step never runs,
the job succeeds and the required status goes green — the exact outcome
the paragraph presented as prevented. The same holds for the suites
under `hack/e2e-chainsaw/`, the install bats and the testing package's
recipes, all of which the fork authors and this job executes.

Hardening the inputs is still worth doing, since a merge-ref diff would
be a second bypass on top of the one that exists, so the code does not
change. What changes is the claim: the inputs are trusted, the decision
is not, and the fork lane is a correctness gate against an honest
contributor's regression rather than a boundary against a hostile fork.
State plainly what does contain a hostile fork — the maintainer
approval required before any fork run executes, the absence of any
credential in that job, and a diff editing the harness being visible in
review.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 00:04:15 +03:00
Aleksei Sviridkin 6309eff784 fix(ci): paginate the fork PR association lookup
The check that the head SHA carries exactly one open pull request is a
check on list completeness, and it was reading a single unpaginated
page. GitHub returns 30 associations by default, so the one shape the
check could not see was the shape it exists to catch: a second open PR
sitting past the page boundary. Route the call through `github.paginate`
like the three other lookups in the same script, and read the result as
the array paginate returns rather than as a response envelope.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 00:03:51 +03:00
Aleksei Sviridkin 9022d352c9 fix(ci): derive the fork image allowlist before loading the push token
Two comments described the privileged push job as running nothing but
fixed binaries over data, never `make`. The allowlist derivation added
later runs `make -n` once per image-building package — 32 on the current
tree — and it sat after the OCIR login, so the credential was live for
all of them. The property still held in substance, since those Makefiles
come from the base tree rather than the fork, but it was a narrower
property than the one written down, and the written one is what a future
reader would try to preserve.

Order the derivation ahead of the login. It needs nothing the login
provides, and above it the make expansion cannot see the push token at
all, which makes the stronger claim true rather than merely arguable.
Restate both comments to say what now holds: no fork-authored script in
the job at all, one base-tree `make -n` before the credential loads,
fixed binaries only once it has. Drop `yq` from the list of binaries the
push job runs — yq is used in the e2e job, not this one.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 23:29:01 +03:00
Aleksei Sviridkin 260ce359ac fix(ci): resolve the fork PR only when the head SHA has exactly one
A branch may have several open pull requests at once, against different
base branches, and they all sit on the same head commit. `resolve` took
the first open association and derived everything from it — the merge
ref it checks out, the docs-only verdict, the file list TIA is scoped
from, the full-e2e label — while writing the `E2E Tests` status to the
head SHA every one of those PRs shares. A verdict computed for one of
them therefore satisfies the others, and a PR whose own base sees code
can be greened by a sibling whose diff is entirely under docs/, with no
suite run. The `HEAD^2 == head_sha` assert further down cannot notice:
it holds for every PR on that commit by construction.

Require exactly one open association and fail closed otherwise, naming
the count in the status description so the state is diagnosable without
opening the run. The zero case, previously reported as "no open PR",
folds into the same check.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 23:28:01 +03:00
Aleksei Sviridkin 7c5e96947e fix(ci): stop two runs from publishing "E2E Tests" on one SHA
Giving every `labeled` event its own concurrency group stopped a
discarded label from cancelling a live run, but it also stopped a
`full-e2e` label from doing so — and that label is not discarded. `plan`
proceeds for it and `e2e-report` publishes with it, so the labelled run
and the head run it was meant to widen both post the `E2E Tests`
context, on the same head SHA, with no ordering between them.

That was a deliberate trade at the time: commit statuses are latest-wins
per context, so the reasoning went, the full-suite verdict lands last
whenever its suite finishes last, and the label is an explicit opt-in to
a longer run. This commit reverses that decision, because the premise
only holds while the full run succeeds. The full suite is the superset,
so a full-e2e run that fails early — a build flake, a suite failing at
minute twenty — finishes before the narrower head run it was meant to
widen, and that run's later green erases the failure. The required
status then reads success for a SHA whose suite did not pass. One
publisher per head SHA is worth more than the minutes of runner time a
cancelled narrow run costs.

Move aside only the label events `plan` discards, by making the
concurrency condition the exact complement of `plan`'s guard. A
`full-e2e` run stays in the main group and supersedes the head run the
way a push does, so exactly one publisher of the context is alive per
head SHA. Opting into the full suite now costs one suite instead of two,
and a second label landing afterwards can no longer cancel the full-e2e
run, because the two are no longer grouped together.

The contract test gains the complement — the group key must exclude the
publishing label rather than every label — and a check that the label
name is one and the same string in all three guards that read it, since
renaming it in one place splits the decision without touching a line
that reads load-bearing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 23:07:33 +03:00
Aleksei Sviridkin 3c50a09457 fix(ci): keep a discarded run out of the fork e2e gate's group
The privileged fork e2e run groups by fork repo + branch with
cancel-in-progress, so every completed Pull Request run for that branch
lands in the same group -- including a run that built nothing.
Labelling an open fork PR produces exactly that: pull-requests.yaml
triggers on `labeled` and `plan` discards every label other than
full-e2e, so all jobs skip. The privileged run for that event cancels
the one that is mid-suite on the same head SHA, then `resolve`
recognises the discarded trigger and posts no status at all. The
required "E2E Tests" context is a commit status, so the head SHA is
left at "Expected" with nothing left to flip it -- the run that
replaced the owner is the very one that decided to stay silent -- and
the PR stays unmergeable until the contributor pushes again.

Key the group by the triggering run's id when that run concluded
`cancelled` or `skipped`, so a run that will post nothing can cancel
only itself. Real runs keep superseding each other on fork repo +
branch.

The contract test ties the conclusions in the key to the ones `resolve`
stays silent for. Those two lists sit in different languages in
different halves of the file, and a third conclusion added to the guard
alone would restore the wedge without touching a line a reviewer reads
as load-bearing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 22:40:34 +03:00
Aleksei Sviridkin 353b8a1a6d ci(backport): skip a target whose backport already merged
A second label event on a PR whose first backport already merged
(for example backport-previous added shortly after backport merged)
re-enters the backport job for the target that already has the
change. The action's branch name for a backport is deterministic, so
that leg's cherry-pick is empty rather than conflicting, and git still
exits non-zero for an empty pick. Under this repository's
draft_commit_conflicts setting the action reads that exit code as a
conflict and tries to commit an empty working tree, which also fails
and surfaces as a false backport-failed comment on the original PR
for a branch the change has been on for a while.

Add a guard step that looks up whether a merged pull request already
exists on this target's deterministic backport branch, and skip the
checkout and the backport action for that leg when one does. The
guard changes nothing about the empty-cherry-pick behavior itself,
which lives in the action and needs a fix there; it only keeps a
known-stale re-run from reaching the action and misreporting.

Extend the existing backport.yaml contract tests with the new guard's
invariants: the merged (not merely closed) check on the deterministic
branch name, both downstream steps gated on the guard's output, and
step ordering.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 22:21:03 +03:00
Aleksei Sviridkin 954be15341 fix(e2e): bound every read in the report collector and the step that runs it (#3596)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

A single hung read in `hack/cozyreport.sh` did not truncate
`cozyreport.tgz`, it lost the artifact outright. The collector writes
its tarball on its last line, the `Collect report` step carried no
`timeout-minutes` of its own, and `Upload cozyreport.tgz` is the step
directly after it — so a read that never returned spent whatever was
left of the job budget, the job was cancelled, and the upload never ran.
The runs where the report is the only evidence left are the runs that
produced none. `|| true` guards against a non-zero exit, not against a
read that does not return.

Two halves, independent of each other.

**The step gets a ceiling of its own**, 30 minutes, in all three
workflows that run it. The number is measured against the step rather
than divided out of the job: pulling the `Collect report` timings out of
recent e2e runs gives 3 to 42 seconds, on failed jobs as much as on
green ones, and 30 minutes leaves five sixths of the job budget for the
steps that upload what this one produced. It carries `continue-on-error:
true` beside it, because the step's `|| true` is its standing statement
that collecting diagnostics is never fatal and a step killed at its
ceiling never reaches that `|| true` — without the tolerance, a passing
run whose collection ran long would be published as failed, and every
later `failure()` in the job would start meaning "the tests failed or
the report was slow". That tolerance also renders the overrun green, so
a following step turns the collect step's `outcome` into a `::warning::`
annotation: a bound that fires is recorded, and the outer bound is not
exempt from the rule the inner ones follow.

**Every read in the collector is bounded**, against four sections
before, and gates and host commands are reads too. An `if kubectl get
deploy …` that decides whether a module runs costs more unbounded than
anything it guards, because the module below it never starts; `df -h`
blocks indefinitely on a stale mount and takes the tarball with it by
exactly the mechanism above. A gate cut off short is recorded, since
skipping the module leaves the same empty tree as a cluster that never
ran the component, while a gate answering "not installed" is recorded
nowhere — that is the cluster answering correctly, and a marker firing
on every install without KubeVirt is one a reader learns to skip.

The six LINSTOR reads run a command inside a pod and go through a reader
of their own. They are the only reads whose duration a workload sets
rather than the apiserver, and `kubectl exec` returns the remote
command's exit status — a 124 there is either the collector's deadline
or a command that itself exited 124, so the existing classifier would
have named this script's own timeout as the cause of something that may
have happened inside a pod. The new note states what the status leaves
open, and sits beside the output rather than inside it, since two of the
six stream a gzipped tarball. A cut-off bundle is kept rather than
dropped, and the note says so: the emptiness check at those call sites
asks whether the archive lists a first member, which a truncated gzip
does.

What this does not do is bound the report. The pod, VM, VMI and services
walks carry a count cap and a time budget; every other per-object walk
carries neither, so their cost is the cluster's to set, and a failed
install with a hundred not-Ready HelmReleases spends four hundred
bounded reads in one of them. A per-read ceiling removes the read that
never returns, not the thousand reads that each return slowly. The file
header says so where it previously implied the opposite, and the step
ceiling is what covers the rest.

Two frozen lists in the tests are gone, replaced by one invariant
derived from the code: no read below the library guard runs without a
ceiling. Both lists were already wrong. The header named four bounded
sections while the file had ten, and the per-section counts of unbounded
reads only ever matched a redirect into `$VAR/`, which no gate, selector
or command substitution has — so they described the scan rather than the
file.

The guard that held the old exec assumption was satisfied by leaving
every exec unbounded, which is the state the file was in. It now also
requires each exec to be bounded, and requires that reader not to borrow
the classification written for reads whose 124 is unambiguous. Matching
on the verb rather than on the string `kubectl exec` is part of it: half
of these calls spell it `kubectl -n cozy-linstor exec`, which that
string never saw.

Two properties worth knowing before this lands. `kubectl get events -A
--sort-by=.lastTimestamp` sorts client-side, so kubectl buffers the
whole event list and writes nothing until it finishes — under a flat
per-read ceiling a cut-off there leaves a file holding only the marker,
not a short tail. That read had no ceiling at all before, where the same
slowness could take the whole tarball, so this is a narrower failure
rather than a new one. And `kubectl logs -n cozy-kamaji
deployment/kamaji` gains `--tail=2000`, matching every other controller
log in the file: it was the only unbounded-in-size log, which under a
wall-clock ceiling is the one most likely to be cut at its *end*, where
a control-plane bootstrap failure sits. Longer kamaji history than 2000
lines is no longer collected.

Two accepted imperfections in how the change is split across commits,
named rather than left to be found. `docs/agents/e2e-testing.md` keeps
this material in a single shared paragraph that all three commits touch,
so in the middle commit one sentence about the module gates lands a
commit ahead of the gates themselves. And two corrections to the exec
reader — it now keeps the remote command's whole message in the listing
rather than quoting the line `kubectl exec` always appends last, and its
cut-off note names both of the two things that can happen to a partial
bundle instead of asserting one — sit in the last commit rather than in
the one that introduced that reader. Both are true of the branch tip,
where every statement is correct and appears once; the cost of moving
them is another rebase across the one shared paragraph, which is not
worth what it buys.

Refs #3568.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(e2e): every read in the diagnostic report collector now carries a wall-clock ceiling — module gates, the reads that run a command inside a LINSTOR pod, and the host commands included — and a ceiling that fires is recorded in the artifact rather than only in the job log. The workflow step that runs the collector carries a ceiling of its own, so a read that does not return costs one file and a warning annotation instead of consuming the job budget and losing cozyreport.tgz entirely.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added bounded diagnostic collection across Kubernetes, Talos, LINSTOR,
host, controller, and command outputs.
- Reports preserve partial results and identify timeouts, failures,
missing output, and truncated artifacts.
- Added warnings when report collection fails or exceeds its time limit.

- **Bug Fixes**
- Corrected resource targets, selectors, deployment extraction, and
command matching.
- Improved handling for empty selections, unavailable resources, and
missing timeout tools.

- **Documentation**
- Expanded guidance for diagnostic markers, collection limits, timeouts,
and incomplete output.

- **Tests**
- Added coverage for bounded reads, failure reporting, markers,
selectors, logs, and subprocess mocking.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-07 17:47:51 +03:00
Aleksei Sviridkin 7e2db4da7e fix(e2e): bound every remaining read in the report collector
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and
the rest -- nodes, namespaces, events, HelmReleases, packages,
packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads
and every module gate -- read with none. The tarball is written on the
collector's last line, so a single read that never returns does not
truncate the report, it loses it, on the run where it is the only
evidence left.

Each read now goes through the shape that fits it: the existing bounded
reader for a read whose output is a file, the existing selector for one
that feeds a loop, and a new probe for the `if kubectl get deploy ...`
gates. A gate is a read like any other, and an unbounded one is worse
than the reads it guards, since the module below it never starts. Its
cut-off is recorded, because skipping the module leaves the same empty
tree as a cluster that never had the component; a gate that answers "not
installed" is recorded nowhere, since that is the cluster answering
correctly and a marker firing on every install without KubeVirt is one a
reader learns to skip. The reads that stay on the bare wrapper say so
where they sit; what they share is that the output is not the reader's
to write, going into a variable or through a pipeline that transforms
it, so there is no moment at which a marker could attach.

The header claimed a per-section list of what was bounded, and the tests
froze that list plus a per-section count of the unbounded reads. Both are
gone. The list had already gone stale, naming four sections while the
file had ten, and the counts were an undercount by construction -- they
matched a redirect to `$VAR/`, which no gate, selector or command
substitution has. One test replaces both: no read below the library guard
runs without a ceiling, checked by looking for a counterexample.

A ceiling per read is not a ceiling on the report, and the header now
says so where it used to imply the opposite. The pod, VM, VMI and
services walks carry a count cap and a time budget; the walks over nodes,
namespaces, HelmReleases, packages, PVCs and certificates carry neither,
so their cost is the cluster's to set. What this removes is the read that
never returns.

The bounded reader now names the tool it ran and describes what that tool
returned. It served kubectl alone until this change gave it talosctl and
the four host commands, so a `df` that hit a stale mount left a marker
reading "kubectl exited 1 ... not because the object ended" -- two false
statements in one line, in an artifact whose premise is that its notes
can be believed.

The exec reader keeps the remote command's whole message in the listing
it belongs to. `kubectl exec` appends its own `command terminated with
exit code N` after whatever the command said, so quoting the last line
there drops the cause and keeps the trailer, and sending the message
away from the file leaves a zero-byte listing -- the "returned nothing,
or never ran" question this collector exists to answer. Its cut-off note
also stops asserting that the archive is beside it: a stream that got
members across is kept and truncated, one killed before its first byte
leaves zero bytes and the call site's emptiness check removes it, and
the reader runs before that check, so it names both.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 14:38:29 +03:00
Aleksei Sviridkin 96e33fe656 ci(e2e): give the report collection step a ceiling of its own
The step that runs hack/cozyreport.sh carried no timeout-minutes in any
of the three workflows that run it, so it inherited only the job's 180.
The collector writes its tarball on its last line and the upload is the
next step, so a read that never returns spends what is left of the job
budget, the job is cancelled, and the upload never runs -- the artifact
is lost entirely rather than truncated, on exactly the run where it is
the only evidence left. `|| true` guards against a non-zero exit, not
against a read that does not return.

30 minutes is measured against the step rather than divided out of the
job: it takes 3-42s across sampled runs, on failed e2e jobs as well as
green ones, and 30 leaves five sixths of the job budget for the steps
that upload what this one produced.

A test pins the ceiling in all three workflows and requires it to stay
under the job's own, since a step ceiling at or above the job's cap is
the same as no ceiling at all.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 14:38:29 +03:00
Aleksei Sviridkin 8aa30472b6 docs(ci): correct the job-level permissions comment
The comment above the top-level `permissions` block claimed that jobs
request the minimum extra scopes on top of it. A job-level block does
not add to the workflow-level one: it replaces it, and every scope the
job does not name is set to none. A job needing one extra scope has to
restate `contents: read` alongside it or lose read access to the
repository.

The corrected wording states the replace semantics, so the next job
block written against this comment grants the whole set it needs
rather than only the delta.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 00:34:53 +03:00
Andrei Kvapil df157da618 ci(workflows): open the E2E breakpoint for maintainer-authored PRs (#3158)
## What this PR does

Decouples opening the E2E SSH breakpoint from the `debug` label.

Today, the only way to open a breakpoint on a failed feature PR is to
add the `debug` label. That same label also flips every job's `runs-on`
onto self-hosted runners, so you cannot get a breakpoint on the normal
ephemeral runner — debugging always drags the whole pipeline onto
self-hosted infrastructure.

This adds a maintainer-authorship path to the breakpoint's `if`
condition: a failed PR whose author is an `OWNER`, `MEMBER`, or
`COLLABORATOR` now opens the breakpoint automatically, with no label
required and without moving the jobs onto self-hosted runners. The
breakpoint simply holds the default `oracle-vm` ephemeral runner the job
already runs on.

The existing `debug` and `release` label paths are unchanged, and the
`authorized-users` list still controls who may actually SSH into an open
breakpoint (second defense layer). Workflow triggers are untouched —
this is purely an `if`-condition change.

### Release note

```release-note
ci(workflows): open the E2E SSH breakpoint automatically on failed maintainer-authored PRs, without requiring the `debug` label or moving the jobs onto self-hosted runners
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Updated E2E failure debugging so breakpoints can now open for pull
requests authored by repository maintainers, not just when specific
labels are applied.
* Clarified the conditions and runner behavior for breakpoint access in
the workflow documentation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-05 14:29:45 +02:00
Myasnikov Daniil dfb1a41f28 ci(release): refuse a build-release dispatch from a non-line ref
The workflow_dispatch trigger added for the freeze takes any ref, and the
"Run workflow" ref selector preselects the default branch — which is
exactly where this workflow's own recovery advice, and cut-prerelease's
warning, send an operator. IMAGE_TAG is github.ref_name, so a dispatch
from main spends a two-hour `make build` republishing
cozystack-packages:main and every main image tag while build-main.yaml
may be writing the same ones: the 409 tag-collision class #2711 fixed.

Nothing is ever mislabeled, since the content really is that ref, so the
cost is a wasted runner rather than a wrong artifact. Guard it anyway: it
is five lines, and the mis-click is easy.

Anchored at both ends so the per-release staging branches (release-1.6.1,
release-1.6.0-rc.4) do not qualify — those are built by tags.yaml — and
gated on ref_type so a tag named release-1.6 cannot pass the pattern on
its own. Runs before the checkout and the OCIR login, so a refused
dispatch never touches the credentials. Fails rather than skips: a
skipped job reads in the run list like one that produced the artifact.

The push trigger was already filtered to release-X.Y; this closes the
dispatch path to the same set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-04 14:23:51 +05:00
Myasnikov Daniil e8675dd321 docs(release): reword the changelog-backstop comments for the freeze model
Three comments still described the changelog reaching main by default,
with a release-X.Y-targeted promote PR as a patch-release corner case.
After the freeze that is inverted: promote-rc.yaml bases the promote PR
on release-X.Y whenever the branch exists, and cutting vX.Y.0-rc.1
creates it, so every release merges its changelog onto the maintenance
line and none reach main on the way. The tags.yaml backstop is now the
normal route, and its self-skip on `exists` is the exception.

These comments are the first thing anyone debugging a missing changelog
reads, so pointing at the wrong model of which path is normal costs real
time. Reworded in place: the generate-changelog job header, the
three-paths comment above check_changelog, its inline at_tag note,
finalize's explanation of why update-releasenotes.yaml is not trusted
with the release body, and the contract test that pins the porting
behaviour. Prose only, no behaviour change.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-04 14:23:32 +05:00
Myasnikov Daniil 5ed6b2204a ci(release): build the packages artifact when a freeze creates a line
build-release.yaml publishes `cozystack-packages:<line>` on push to a
release line, and pull-requests.yaml's overlay reads it so a release-line
PR tests its own line's binaries instead of main's (#3471, #3437).

It cannot fire for the push that creates the line. The freeze points
release-X.Y at a commit that is already on main, so the push carries no
new commits, and GitHub does not run a workflow whose paths/paths-ignore
filter finds no changed files ("If there are no files changed, the
workflow will not run"). The line would therefore have no artifact until
its first cherry-pick merged, and in that window the overlay finds
nothing to pull and leaves every package on its committed ref — at
freeze time the previous release's, which is exactly the
cross-generation mix #3437 fixed. The old flow had no such window: it
branched at the promote merge commit, which carried real commits and its
own release's refs.

Add workflow_dispatch to build-release.yaml and have the freeze step
dispatch it for the branch it just created, so the artifact exists from
the moment the line does. The dispatch is non-fatal: without it the
overlay no-ops and early cherry-pick PRs test their committed refs,
which is where they were before #3471. The tag is pushed and the line is
frozen by that point, so failing there would misreport both.

The trigger also gives a line build a re-run button, which previously
needed an empty commit pushed to the line.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:38:26 +05:00
Myasnikov Daniil 66ed32c6f9 ci(release): print the recovery command if the freeze push fails
Address review feedback from lexfrei on .github/workflows/cut-prerelease.yaml:293:
the tag push and this branch push are independent, so a transient failure
here leaves the line tagged but not frozen, and the write-once tag guard
refuses a same-tag re-dispatch — recovery is manual.

Fail with the exact push command rather than leaving the operator to
reconstruct it. Deliberately not an atomic two-ref push: the tag push
relies on being anchored to the branch tip for GitHub to populate the
push event's base_ref, which tags.yaml requires, and whether a multi-ref
atomic push preserves that is only observable on a live release cut.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:32:00 +05:00
Myasnikov Daniil a59ad88eb5 ci(release): distinguish absent from broken in the freeze branch probe
Address review feedback from lexfrei on .github/workflows/cut-prerelease.yaml:286:
the release-X.Y existence probe piped ls-remote straight into cut, so a
transport or auth failure produced empty output and read as "branch not
found", steering into the create path. The write-once tag guard, the
stale-tip guard, and the refuse step above all make this distinction;
match them here.

Behaviour is unchanged on the happy path — pipefail already failed the
step on a real error, and the non-forced push below fails closed on a
conflict. This makes the intent explicit rather than incidental.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil 6ff5b98d56 ci(backport): target the newest existing release line
Determine the backport targets by enumerating real release-X.Y branches
instead of deriving them from getLatestRelease.

getLatestRelease returns the newest published stable. Now that cutting
an rc creates release-X.Y before vX.Y.0 exists, that name lagged by a
full line for the whole freeze window: a `backport` on a fix for the
release being stabilised would have landed on the previous line and
missed the release it was written for. The freeze window is when
backports matter most, since the frozen branch is the only way in.

Also drops the contiguous-minor assumption — `previous` was computed as
min-1, naming a branch that does not exist whenever a minor is skipped.
Sorting is numeric, so release-1.10 correctly outranks release-1.9.

Outside a freeze window the result is unchanged: with release-1.0..1.5
present it still resolves to release-1.5 / release-1.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil 53a7fc4dcd feat(release): cutting an rc freezes the line into release-X.Y
Cutting the first vX.Y.0-rc.N now creates release-X.Y at the tagged
commit, and every later cut for that line must be dispatched from the
branch — a dispatch from main is refused.

Before this, only the stable promote-PR merge created release-X.Y, so
rc.2 and rc.3 were cut from main's tip and silently picked up every
feature merged since rc.1. The release therefore shipped content no rc
had ever validated. Freezing at the rc closes that: main stays open for
the next minor, and fixes reach the release by cherry-pick onto the
branch.

promote-rc.yaml needs no change — it already prefers release-X.Y as the
promote PR base when that branch exists, so the stable release is now
cut from the frozen tree.

Gated to -rc (alpha/beta are pre-freeze builds off an open main) and to
patch 0 (a patch-line rc is cut from a release-X.Y that already exists).
The branch is create-only and never force-moved: moving it would re-open
the freeze and drag in whatever the new tip holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
myasnikovdaniil f1f383626e fix(ci): overlay images from the PR base branch, and publish per-line artifacts (#3471)
## What this PR does

The PR finalize job overlays image refs for every package a PR did not
rebuild, so e2e and the installer do not test last-release images for
everything outside the PR's build matrix. It could only ever read
`cozystack-packages:main`, which meant a **release-line PR was handed
main's binaries to run against its own line's charts**.

cozystack/cozystack#3437 is the demonstration: a one-line change on
`release-1.6` deactivating an app failed install deterministically,
twice, with

```
helmrelease/backupstrategy-controller: Helm install failed …
error validating data: SchemaError(github.com/cozystack/cozystack/pkg/apis/core/v1alpha1.Option.spec):
unknown model in reference: "github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec"
```

main's `cozystack-controller` served an aggregated OpenAPI that
`release-1.6`'s charts could not validate against. Nothing in that PR
was broken; the lane was. Both branches carry `option_types.go` and key
`OptionSpec` identically in the committed generated OpenAPI, so this is
a generation mismatch at runtime, not a codegen drift.

This PR fixes it by giving every base branch its own artifact to overlay
from, rather than by turning the overlay off.

### The change

1. **`pull-requests.yaml`** reads `cozystack-packages:${BASE_REF}`
(`github.base_ref`) instead of a hardcoded `:main`.
2. **`build-release.yaml`** (new) publishes that artifact for maintained
`release-<major>.<minor>` branches exactly as `build-main.yaml` does for
main: every image tagged with the branch, and the whole packages tree
pushed with each reference digest-pinned to what the run just built.

An earlier revision of this branch simply skipped the overlay for
non-main bases. That fixed the wrong-images problem but left
release-line PRs testing their line's *last release*: for any package
the PR did not rebuild, the committed ref is the released digest, so a
component changed by an earlier backport was still exercised as its
pre-backport binary until the next rc. Per-line artifacts remove that
gap too, which is why the guard was replaced rather than kept.

### Three deliberate choices

**The trigger matches line branches only** (`release-[0-9]+.[0-9]+`).
The per-release and rc staging branches `promote-rc.yaml` and
`tags.yaml` create — `release-1.6.1`, `release-1.6.0-rc.4` — must not
trigger a full rebuild; their images come from the tag build and
rebuilding them is waste.

**`WRITE_CACHE` stays `0`.** `CACHE_REGISTRY/<img>:buildcache` is a
single ref per image and `build-main.yaml` is deliberately its only,
serialized writer so concurrent builds cannot race on the cache manifest
— the 409 class #2711 fixed for image tags. A line build can overlap a
main build, so writing here would reintroduce that race. Line builds
read the cache.

**A missing artifact still degrades to committed refs, but says so.** On
a release line it emits a `::warning::` naming the branch. Silent
degradation is indistinguishable from a working overlay, which is how a
mis-specified branch filter could hide for a whole release cycle.

### Cost

One `make build` per push to a maintained line — in practice per merged
backport. That is the price of release-line PRs testing their line's tip
instead of its last release.

### Verification

`hack/overlay-main-images_test.bats` pins the artifact tag to the base
branch, rejects a hardcoded `:main` in either overlay step, and pins
`build-release.yaml`'s branch filter, image tag and `WRITE_CACHE: '0'`.
Mutation-checked: restoring `:main`, setting `WRITE_CACHE: '1'`, and
broadening the filter to `release-*` each fail a test. 13/13 green;
`actionlint` and `zizmor` clean.

Worth an explicit ack in review: the branch-filter pattern is the one
thing no local test can prove, since only GitHub evaluates it. If it
does not match, `build-release` never runs and the new `::warning::` is
what surfaces it on the next release-line PR.

### Backport

`release-1.6` needs this too — for `pull_request` events GitHub builds
the workflow from the merge ref, so a release-line PR only stops
receiving main's images once the change is on its base branch.
cozystack/cozystack#3472 carried the interim guard and is closed in
favour of backporting this instead.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a “Build release line” workflow that builds and publishes images
and the packages artifact for maintained `release-<major>.<minor>`
branches.

* **Bug Fixes**
* Updated PR workflow finalization to use base-branch–specific package
overlays, avoiding incorrect main-branch image references when targeting
release branches.
* Improved fallback behavior when the base-branch packages artifact is
unavailable.

* **Tests**
* Added workflow wiring tests to verify base-branch artifact usage and
that each maintained release line publishes its own packages artifact.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-03 18:22:47 +05:00
Myasnikov Daniil 6a73a6caa9 docs(agents): call the E2E gate a commit status, not a check-run
The §10 heading, four workflow comments and both TOCTOU comments
still called the required "E2E Tests" context a check-run, while the
body of that same section — and the implementation — insist it is a
commit status. The distinction is the whole reason the section exists:
a check-run made with the default token floats into an arbitrary
check-suite, which is why the gate is a status. Leaving the wrong
primitive in the heading steers a maintainer wiring branch protection
at exactly the wrong thing.

The paragraph explaining why it is NOT a check-run keeps that wording,
since there the term is the rejected alternative. Also adds the blank
line missing before the "Chainsaw v0.2.15 gotchas" heading.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:57 +05:00
Myasnikov Daniil 1582eff3f7 docs(ci): correct the stale shared-runner claims in the fork e2e lane
Two comments in e2e-fork.yaml justified themselves with a threat that
no longer exists: the allowlist derivation guarded a `git worktree
add` against "a crashed/concurrent run on the shared self-hosted
runner", and the sandbox name was said to disambiguate "concurrent
fork runs on a shared host". The dedicated runner was decommissioned
in #2937 — these labels are ephemeral Oracle ARC VMs, one per job,
as .github/actionlint.yaml states. A comment that asserts a shared
host next to a step handling a push token misleads exactly the
reader who is auditing that boundary.

Both guards stay; only their stated reason changes to the one that
actually holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:47 +05:00
Myasnikov Daniil 1d668e1fcb fix(ci): scope the fork publish OCIR login to a run-local auth store
The `publish` job ran `skopeo login` and `docker login` with no
DOCKER_CONFIG override and no logout, so the OCIR push credential
landed in the runner's default stores (~/.docker/config.json and
$XDG_RUNTIME_DIR/containers/auth.json). Every other OCIR login in
this repo — pull-requests.yaml's build, build-talos and finalize —
already scopes to $RUNNER_TEMP/.docker, and `Set up Docker config`
in those jobs copies ~/.docker verbatim into the build context, so
a token left in the default store is a token handed to whatever
runs next.

Scope both logins to $RUNNER_TEMP/.docker (skopeo via --authfile,
flux via the docker keychain's DOCKER_CONFIG), carry the same env
into the push step that reads the auth, and drop the credential in
an always() logout step so a failed push does not leave it behind.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:32 +05:00
Myasnikov Daniil c46037ce7b fix(ci): stop a cancelled run from wedging the "E2E Tests" gate
Now that the required "E2E Tests" context is a commit status rather
than a job, a run that dies before posting leaves the head SHA at
"Expected" — which blocks the merge instead of passing it. Two paths
reached that state.

Same-repo: a `labeled` event shared the PR's concurrency group, so
labelling an open PR cancelled the live opened/synchronize run, and
the replacement run was then discarded by `plan`'s full-e2e guard --
so `e2e-report` skipped with it and nothing posted the status the
cancelled run was going to. Observed on live CI: run 30614258694
(cancelled) followed by 30614280603 (skipped) on the same SHA, 24s
apart, from a maintainer adding kind/* labels just after a push.
Label events now get their own concurrency group, so they can only
supersede each other. Release promotion is unaffected: `gh pr create
--label` attaches `release` at creation, so a promote PR gets a
single run that already sees the label (verified on #3397 and #3425,
one run each, build/finalize gated off).

Fork: `resolve` had no equivalent, so a cancelled triggering run fell
through to the `CONCLUSION != 'success'` branch and posted `failure`
— a false red on a SHA whose e2e never ran, which the replacement run
for that same SHA would not flip back (it takes the discarded-label
path and posts nothing). It now returns early on a cancelled or
wholly-skipped conclusion. Checked on the conclusion rather than the
job list because a cancelled run can report no jobs at all, which
would make the `Plan build` lookup miss.

Both stay fail-closed: declining to post can only preserve a verdict
a real run reached, never invent one.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:07 +05:00
Timofei Larkin e5d4940338 chore(codeowners): don't gate per-app generated files on architect review
api/apps/v1alpha1/*/types.go and packages/system/*/cozyrds/*.yaml are
entirely produced by `make generate` from each app's own values.yaml,
but the /api/ codeowner rule pulled every app's regenerated types.go
into an architect-only review, regardless of which area actually
changed. Correctness there is already enforced by the pre-commit
make-generate CI hooks, not by human review, so list no owner for
these generated paths and let that check carry the weight instead.

Also fixes a stale api/.gitattributes pattern that never matched any
real file (zz_generated_deepcopy.go vs the actual
zz_generated.deepcopy.go), and adds a matching linguist-generated
attribute for the cozyrds ApplicationDefinition manifests, so both
collapse in the PR diff view like the Go files already do.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-08-03 14:08:36 +03:00
Andrei Kvapil e16df56019 chore(codeowners): scope CODEOWNERS to per-area reviewers (#3228)
## What this PR does

Replaces the single catch-all owner line in `.github/CODEOWNERS` with
area-scoped reviewer ownership, so each part of the tree routes to the
people who actually own it (per the Contributor Ladder), instead of
requiring the whole roster on every change.

Highlights:
- Architects (`@kvaps @lllamnyp`) are the default owner and own the API
surface (`/api/`) and the VPC app.
- CI / automation → `@myasnikovdaniil @sircthulhu`; backups → `@androndo
@lllamnyp`; virtualization (kubevirt, vm-instance/disk) → `@scooby87
@lllamnyp @kvaps @myasnikovdaniil`.
- `cozystack-api` / controller / lineage webhook → `@IvanHunters @kvaps
@lllamnyp`.
- General packages / apps / CLI ownership → `@lexfrei @IvanHunters
@myasnikovdaniil` (+ `@scooby87` for apps), with `kubernetes` and
`extra` scoped separately.
- Flux → `@kingdonb`; GPU (hami, gpu-operator, GPU dashboards) →
`@lexfrei`; metrics-server and seaweedfs → `@nbykov0`.
- Governance / licensing files → `@tym83 @kvaps`.

Ordering follows "last match wins" (broad → specific). The API review
gate block is kept last so it cannot be overridden. This is
review-routing only — no runtime/functional change — and only binds once
"Require review from Code Owners" is enabled in branch protection.

This file is what the CNCF Incubation review reads as "code ownership",
so it is paired with #3462, which documents the same people in
`MAINTAINERS.md` and introduces the Reviewers table the ladder already
defined. Merging one without the other leaves the two describing
different projects, which is the gap both PRs exist to close. Three of
the people named here — `@kingdonb`, `@matthieu-robin`,
`@mattia-eleuteri` — hold only read access today, and a CODEOWNERS entry
for someone without write is silently ignored by GitHub, so the org-side
permission changes have to land alongside.

### Screenshots

N/A — governance/config change, no user-facing or UI impact.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walked the trigger map against the diff: no row matches. The change
touches only `.github/CODEOWNERS` — no package, no `values.schema.json`,
no `ApplicationDefinition`, no platform variant, no release asset, no
node prerequisite. Every path named here was checked against current
`main`; all 65 exist.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated CODEOWNERS from a single catch-all to an ordered, path-scoped
reviewer mapping.
* Clarified “last matching pattern wins” behavior and ensured the API
review gate rules remain last to avoid overrides.
* Expanded explicit ownership coverage across dashboards/examples,
packages and Go components, apps, platform/infrastructure modules,
CI/automation, and governance/licensing documentation (including
SECURITY).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-30 11:04:03 +02:00
Myasnikov Daniil 507336dba0 fix(ci): report "E2E Tests" green on a release promotion PR
Under main's rc-gate model a promote PR carries `release` and deliberately not
`full-e2e`, so the `e2e` job skips: the immutable rc whose digests the tree pins
was already verified, which is the cost the rc gate exists to remove. While the
required gate was a job, that skip satisfied branch protection. Now that it is a
commit status, e2e-report saw PLAN_RESULT=success, CODE=true (a promotion
rewrites ~38 non-docs image refs) and E2E_RESULT=skipped, fell to the last
branch and posted failure — so every release would have needed a full suite
re-run or an admin override of the very gate this PR strengthens.

Give e2e-report the carve-out the `e2e` job already has: release without
full-e2e reports success and says why. Adding full-e2e still forces a real run
and a real verdict.

Close the second route to the same red as well. `gh pr create --label` attaches
the label just after opening, so the `labeled` event cancels the `opened` run
through cancel-in-progress; under always() that dying run still posted failure
with PLAN_RESULT=cancelled, and the `labeled` run replacing it is skipped by the
allow-list guard, so nothing flipped the status back. !cancelled() makes a
cancelled run report nothing, which is the same fail-closed posture used
elsewhere here: not posting leaves the required status "Expected", which blocks.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-30 11:54:52 +05:00
Myasnikov Daniil 6728bd4245 fix(ci): don't clobber a fork PR's green "E2E Tests" on a label event
Same defect as the e2e-report fix, on the fork path. main's `labeled`
pull_request trigger produces a "Pull Request" run whose `plan` guard
discards every label but full-e2e, so every job skips and nothing is built.
workflow_run still fires for it, and `resolve` then reddened the required
status down either branch: a non-success conclusion fails closed, and a
`success` conclusion falls through to the missing-pr-patch check, which also
fails closed. Labelling an already-green open fork PR therefore turned its
E2E Tests red until the next push.

Detect that shape directly — the triggering run's `Plan build` job concluded
`skipped` — and return without posting anything. Statuses are per-SHA and an
unposted required status stays "Expected", which blocks the merge, so
declining to post can only preserve a verdict some real run reached, never
invent one. A fork that edits the workflow to force this path leaves its head
SHA with no status at all, i.e. not mergeable. The job already holds the
actions:read scope the job lookup needs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 18:35:14 +05:00
Myasnikov Daniil 8b79a0f20c fix(ci): don't clobber a green "E2E Tests" status on a discarded label event
main gained a `labeled` pull_request trigger whose `plan` guard discards
every label event except full-e2e. e2e-report carried no such guard, so
labelling an already-green open PR started a run in which `plan` was
skipped, e2e-report still ran (always()), read PLAN_RESULT != 'success' and
posted "E2E Tests" = failure — overwriting the passing required status and
making a merge-ready PR unmergeable until the next push.

Give e2e-report the same `labeled` guard as `plan`, so it runs if and only if
`plan` runs. Skipping posts nothing, which is fail-closed: an unposted
required status stays "Expected" and blocks the merge, so the guard can only
ever preserve a verdict, never invent one.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:57:17 +05:00
Myasnikov Daniil 2b53ac32a1 ci(e2e): run the fork e2e job in the workspace, not /tmp
e2e-fork.yaml's e2e job was written against the pre-78fc3f917 in-tree job,
which copied the checkout to /tmp/$SANDBOX_NAME, ran every step from there
and removed it on teardown. main has since dropped that workspace: e2e is a
single job on an ephemeral runner, so there is no cross-job rendezvous and
the checkout is never reused.

Keeping it here would also reintroduce the unguarded
`rm -rf /tmp/$SANDBOX_NAME` that commit deliberately deleted rather than
guarded — when an upstream step fails the unconditional sandbox-id write is
skipped, SANDBOX_NAME stays empty, and the `if: always()` removal expands
to `rm -rf /tmp/`.

Drop the Prepare/Remove workspace steps and the seven `cd
/tmp/$SANDBOX_NAME` prefixes, and point the chainsaw-report and cozyreport
upload paths at _out/. /tmp/changed.txt stays — it is a scratch file, not a
workspace copy, and the in-tree lane writes it the same way.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:48:43 +05:00
Myasnikov Daniil 60e910b863 docs(ci): correct the select-e2e.sh escalation claim in e2e-fork (#3262)
Both comments stated that select-e2e.sh escalates to the full suite on
any unrecognised path, so fork TIA "can only narrow". That is not what
it does: it escalates for an unrecognised path INSIDE packages/, but a
path it does not recognise at all (hack/*.mk, for one) selects nothing,
the Chainsaw step is skipped, and the required status still concludes
success.

Found by coderabbitai in review. The behaviour is pre-existing and
identical for same-repo PRs through pull-requests.yaml, so it is tracked
separately in #3392 rather than fixed on the fork lane alone — which
would leave forks stricter than same-repo for the same diff. Correct the
comments here so the guarantee is not overstated in the meantime.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov Daniil 705e76f5db fix(ci): fail closed on renamed and truncated fork PR file lists (#3262)
Address review feedback from coderabbitai on
.github/workflows/e2e-fork.yaml: the docs-only decision read only
`filename`, so a rename reported the NEW path alone. A PR renaming
packages/<app>/values.yaml -> docs/whatever.md therefore looked
docs-only, concluded "E2E Tests" green, and never ran the suite —
while in fact removing a code path. Both sides of a rename must be
under docs/ for a PR to count as docs-only.

`pulls.listFiles` is also capped at 3000 files, and a truncated list
could omit the only non-docs path. Fail closed at the cap instead of
ruling docs-only on a list known to be incomplete.

Feed both sides of a rename to TIA as well: the old path is a removal
the selector must see. Extra paths can only widen the selection, so
this cannot cause a suite to be skipped.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov DaniilandClaude Opus 4.8 1a1f3696ee fix(ci): fix the nullglob-defeated OCI-archive guard in the fork publish
`count=$(ls _out/oci/*.oci.tar | wc -l)` runs under `shopt -s nullglob`, so
when nothing matches the glob expands to zero words and `ls` lists the current
directory instead — count is always >=1 and the "at least one archive" guard
never fires. A zero-archive build then falls through to `flux push` and defers
the real failure to an obscure image-pull error deep in the ~180-minute e2e
run. Glob into an array (empty under nullglob) and test its length; reuse it
for the push loop.

Reported by coderabbitai on #3262.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 3b70fda870 fix(ci): make fork OCI_EXPORT_DIR absolute so archives land where the upload reads
The fork build sets OCI_EXPORT_DIR to a relative `_out/oci`, but each image is
built with `make -C <pkg> image`, so buildx resolves the relative `dest=`
against the package directory — the archive landed in <pkg>/_out/oci/ while the
`if-no-files-found: error` upload reads repo-root `_out/oci/*.oci.tar`. Every
non-docs fork build would have failed at that upload (never caught, since
same-repo PRs leave OCI_EXPORT_DIR empty). Root it at github.workspace so the
dest resolves to repo-root _out/oci regardless of the sub-make cwd.

Reported by gemini-code-assist on #3262 (its abspath suggestion resolves against
the package dir under `make -C`, so this fixes it at the workflow level instead).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 0f6e99eff8 fix(ci): publish the "E2E Tests" gate as a commit status, not a check-run
Route the required "E2E Tests" merge gate through a GitHub commit status
posted with the default GITHUB_TOKEN (+ statuses:write), replacing the
COZYSTACK_CI app check-run. The app path needed the app granted checks:write
(it isn't — checks.create 403s "Resource not accessible by integration"), and
the earlier default-token check-run floated under a random check-suite
("PR size label"). A commit status has neither problem: branch protection
matches it by context name, it carries no check-suite grouping, and it is safe
with the default token because a fork's pull_request token is read-only and
cannot post one — only same-repo runs (e2e-report, guarded !fork) and the
privileged workflow_run in e2e-fork.yaml can set it.

The security property is unchanged: a never-posted required status stays
"Expected" and blocks the merge, so a skipped job can't smuggle a green
(#3257). Removes all COZYSTACK_CI app-token machinery from e2e-fork.yaml —
`resolve` posts the status directly, no `check_run_id` threading, no `open`
step.

Branch protection must require the "E2E Tests" status context (no GitHub App
and no app-id pin needed).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 03b519dc75 fix(ci): fork e2e — Chainsaw+TIA run and image-push hardening
The fork e2e path in e2e-fork.yaml still ran the pre-#2826 BATS app loop
(`hack/e2e-apps/*.bats` + `test-apps-%`), which the merged Chainsaw
migration deleted — so the fork run was already broken. Rewrite it to the
same Chainsaw + Test-Impact-Analysis path as the in-tree job, and clear the
remaining review follow-ups on the publish job.

- Run E2E via a single `make test-chainsaw` invocation (no per-app retry —
  e2e-testing.md §1). Suite selection and the full-e2e override come from
  `resolve`'s TRUSTED base-repo listFiles/labels (new `changed_b64` /
  `full_e2e` outputs), never a git diff of the fork-controlled merge ref, so
  TIA cannot be shaped into skipping the suite (keeps the #3257 property).
  Collect + upload the Chainsaw JUnit report like the in-tree job.
- Restrict fork image pushes to a trusted allowlist derived from the base
  parent (HEAD^1) Makefiles, so a fork cannot mint an arbitrary repo path
  under the CI registry by renaming an image. The charset guard stays as
  defence in depth.
- `skopeo copy --all` so a multi-platform build's index digest (the one
  baked into pr.patch) is preserved.
- Pin the flux CLI (FLUX_VERSION) in the privileged publish job instead of
  install.sh's rolling latest.

Docs: e2e-testing.md §10 records that the fork path now runs Chainsaw + TIA
and that fork suite selection must stay sourced from trusted data.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 9cddb532bf fix(ci): publish "E2E Tests" check-run as the COZYSTACK_CI app
The synthetic "E2E Tests" required check-run was created with the default
GITHUB_TOKEN (the "GitHub Actions" app). A token-created check-run has no
owning job, so GitHub files it under an arbitrary GitHub Actions check-suite
for the head SHA — in practice the earliest one, the PR-size labeler — so
"E2E Tests" rendered under "PR size label".

Publish it as the COZYSTACK_CI GitHub App instead (create-github-app-token,
scoped to permission-checks: write) so the check lands in the app own
dedicated suite. This is also stricter: only the CI app can satisfy the gate.

- pull-requests.yaml (e2e-report): mint the app token, pass it to checks.create.
- e2e-fork.yaml: split resolve into a verdict decision on the default
  workflow_run token (keeps the actions:read / pull-requests:read lookups) and
  an "open" step that publishes the check as the app; report re-mints the token
  so checks.update runs as the same app that opened the pending run.
- docs/agents/e2e-testing.md: document the change and the branch-protection
  requirement.

Requires main branch protection to require "E2E Tests" from the COZYSTACK_CI
app id (not app_id 15368).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 eba55976a0 fix(ci): decide fork docs-only from the PR file list and harden publish (#3257)
resolve inferred "docs-only" from the absence of a pr-patch artifact, but
artifact presence is decided by pull-requests.yaml running from the fork's
merge commit — a fork could suppress it (code=false, or if:false on finalize)
and conclude the required E2E Tests check green with e2e never run, re-opening
the very bypass this feature closes.

Derive docs-only from trusted base-repo data (pulls.listFiles): green only
when every changed file is under docs/; any non-docs file makes e2e required,
and a missing pr-patch then fails the check closed. Resolve the PR open-only
(no closed/merged fallback).

Also harden the privileged jobs:
- drop `packages: write` from publish (it authenticates to OCIR, not GHCR);
- assert at least one OCI archive before pushing, and validate each archive
  basename against a strict charset before using it as a registry repo;
- install flux from a downloaded script rather than piping curl to sudo bash;
- assert `git rev-parse HEAD^2` equals the resolved head SHA in publish and
  e2e, so a moved merge ref cannot operate on a different tree than the check.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 076658e70f fix(ci): avoid zizmor github-env finding on fork e2e sandbox name
The `Set sandbox ID` step wrote a computed value to $GITHUB_ENV, which the
zizmor github-env audit flags as high severity (write to GITHUB_ENV may allow
code execution). Set SANDBOX_NAME as a job-level env var instead — no
$GITHUB_ENV write, so the audit does not apply. Uses github.run_id, which is
unique per run; in workflow_run context GITHUB_REF is the default branch and
cannot disambiguate concurrent fork runs sharing a self-hosted host.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 76c8f890fa fix(ci): export OCI archives for buildx calls that bypass image-tags (#3257)
Fork PRs build images to per-image OCI archives instead of pushing,
gated on OCI_EXPORT_DIR. The `--output type=oci` was appended only inside
the `image-tags` macro, so `packages/system/capi-providers-cpprovider`,
whose `image:` recipe uses a hardcoded `--tag` and never calls
`image-tags`, ran buildx with no exporter under export mode (PUSH/LOAD
forced to 0). No archive was written, and the fork build's
`if-no-files-found: error` upload then failed the whole build. Its
`--cache-to type=inline` also needs an image exporter.

- hack/common-envs.mk: extract the OCI-output snippet from `image-tags`
  into a reusable `oci-output <image-name>` macro so any buildx call can
  opt in; `image-tags` now ends with `$(call oci-output,$(1))` (no-op for
  the ~33 conforming packages). Move `comma :=` above its first user.
- packages/system/capi-providers-cpprovider/Makefile: add
  `$(call oci-output,cluster-api-control-plane-provider-kamaji)` (archive
  base name equals the OCIR repo the privileged push keys on) and drop the
  inline cache under export (`--cache-to type=inline` gated off).
- .github/workflows/pull-requests.yaml: wrap the `E2E Tests` check-run
  creation in try/catch. On error, `core.setFailed` so the required merge
  gate fails loudly and fail-closed instead of silently succeeding.

Repo-wide audit of every `docker buildx build` reachable from the root
`build:` matrix confirms capi was the only recipe bypassing both
`image-tags` and `oci-output`.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:25 +05:00
Myasnikov DaniilandClaude Opus 4.8 e0ac2f9e12 ci(e2e): run fork-PR e2e via privileged workflow_run (#3257)
Fork PRs cannot push images (pull_request runs from a fork receive no
secrets), so `make image` pushed anonymously and died with `denied`.
That failed every build, collapsed the build->finalize->e2e chain, and
left the required "E2E Tests" job skipped — which branch protection
treats as satisfied, so an approved fork PR could merge with e2e never
having run.

pull-requests.yaml (fork PRs):
- build / build-talos / finalize export images to per-image OCI archives
  (OCI_EXPORT_DIR) and upload them instead of pushing, so the red
  "denied" build wall disappears and the chain no longer collapses.
- the in-tree e2e job is renamed off "E2E Tests" and guarded to
  same-repo PRs; a new e2e-report job concludes the required "E2E Tests"
  context as an explicit check-run (success on a docs-only PR or when the
  in-tree e2e passed, failure otherwise). A skipped job can no longer
  satisfy the required check.

e2e-fork.yaml (new, privileged workflow_run from the default branch):
- resolves the fork PR, opens a pending "E2E Tests" check-run, and
  fails-closed if the fork build failed;
- pushes the pre-built image archives to OCIR by digest and flux-pushes
  the digest-pinned packages tree — the OCIR token is live only for
  trusted binaries (git/skopeo/flux/yq) over data, never for `make` or
  any fork-authored script;
- runs the e2e suite against the pushed closure with no push token and
  concludes the "E2E Tests" check-run on the PR head SHA.

Fork build code never receives registry credentials; the fork's images
and test scripts run only in the credential-less, ephemeral e2e job.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:25 +05:00
Myasnikov Daniil 2330d6f3ca fix(ci): overlay images from the PR base branch, and publish per-line artifacts
Supersedes the interim guard in this branch`s first commit, which skipped the
overlay entirely for release-line PRs. Skipping fixed the wrong images but left
those PRs testing their line`s last release: for any package the PR did not
rebuild, the committed ref is the released digest, so a component changed by an
earlier backport was exercised as its pre-backport binary until the next rc.

The overlay now reads `cozystack-packages:<base branch>` instead of always
`:main`, and build-release.yaml publishes that artifact for every maintained
`release-<major>.<minor>` branch the way build-main.yaml does for main: images
tagged with the branch, and the whole packages tree pushed with each reference
digest-pinned to what the run just built. Each base branch therefore has its own
generation to overlay from, which is what the original bug was really about —
#3437 failed install because main`s cozystack-controller served an aggregated
OpenAPI release-1.6`s charts could not validate against.

Three deliberate choices:

  * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The
    per-release and rc staging branches promote-rc.yaml and tags.yaml create
    (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their
    images come from the tag build, and rebuilding them would be waste.
  * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per
    image and build-main.yaml is deliberately its only, serialized writer so
    concurrent builds cannot race on the cache manifest (the 409 class #2711
    fixed for image tags). A line build can overlap a main build, so writing here
    would reintroduce that race. Line builds read the cache.
  * A missing artifact still degrades to committed refs, but on a release line it
    now emits a ::warning:: naming the branch. Silent degradation is
    indistinguishable from a working overlay, which is how a mis-specified branch
    filter would hide for a whole release cycle.

Cost: one `make build` per push to a maintained line, i.e. per merged backport.

hack/overlay-main-images_test.bats pins the artifact tag to the base branch,
rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s
branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main,
setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test.
13/13 green; actionlint and zizmor clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:29:43 +05:00
myasnikovdaniil cc44f6c981 ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449)
## What this PR does

Moves release validation from promote time to rc time, making promotion
itself a pure retag operation. This is the follow-up to the v1.6.0
retrospective: the promote flow's checks ran on the promote PR (coupled
to promotion, skippable by never promoting), the changelog and website
docs were produced only after the release went public, and the finalize
tag push silently failed to trigger `tags.yaml` at all
(persisted-credential bug, root-caused below).

The release lifecycle after this PR:

1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes
the staging branch — and now runs a **mandatory full e2e** for every
`-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and
any published tag) get the same validation as a **manual button**
(`workflow_dispatch`).
2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**:
the release changelog is generated from the rc tag (reusable
`changelog-generate.yaml`) and committed to the rc staging branch, ready
for review long before promotion.
3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence:
the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a
successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input
for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages
the release, picks up the rc-time changelog (or generates one), and
**opens the website docs PR** on cozystack/website from the staging
branch — parked with an explicit DO-NOT-MERGE-until-finalize contract.
4. The promote PR **no longer gets `full-e2e` automatically** — its tree
is the rc's already-validated digests with only tag strings rewritten; a
maintainer can still add the label manually to force the suite.
5. Finalize performs the irreversible side effects (tag, publish, retag,
chart) — and with `persist-credentials: false` its tag push now actually
authenticates as the CI app, so the `tags.yaml` backstops
(`generate-changelog`, `update-website-docs`) fire for promoted tags.
This is the one-line fix for the v1.6.0 incident where the tag triggered
nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader`
silently defeats the app token injected via `git remote set-url`, and
`GITHUB_TOKEN` pushes create no workflow runs.

Everything is pinned by executable contract tests
(`hack/promote-gate-contract.bats`, extended
`hack/release-changelog-contract.bats` /
`release-changelog-behaviour.bats` — 48 tests total, including behaviour
tests that execute the new `hack/parse-rc-tag.sh` and
`hack/select-changelog-source.sh` for real), and `docs/release.md` is
rewritten to describe the new lifecycle end-to-end.

### Commit map (review commit-by-commit)

- `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing
changelog-before-publication rework (changelog generated at promote,
committed to the staging branch before the PR opens, finalize publishes
it as the release body; tags.yaml job becomes a verbatim-port backstop),
rebased onto main.
- `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane
(prerelease → staging branch, stable → tag; draft releases fail fast;
dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts).
- `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut
rc tags (also on `release_exists` re-runs); fixes the
`cut-prerelease.yaml` comment that falsely claimed e2e already existed.
- `2dfbeea77` — `changelog-generate.yaml` (reusable core) +
`changelog-rc.yaml` (rc-time button committing to the staging branch);
promote-rc gains a top-level `parse` job with a strict no-whitespace tag
policy (`hack/parse-rc-tag.sh`); source selection is
validate-then-fallthrough (`hack/select-changelog-source.sh`) so an
invalid higher-priority changelog can never silently win.
- `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override +
removal of the automatic `full-e2e` label (pull-requests.yaml
release-path e2e now requires an explicit label; a post-open label now
actually starts the run).
- `936190d4b` — `website-docs` job at promote time behind a
FETCH_REF-support guard (see dependency below), with the
DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before
`release-next`; `tags.yaml::update-website-docs` stays as the backstop.
- `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0
root cause).
- `717c8860f` — `docs/release.md` coherence pass over the whole new
lifecycle.

### Dependency

The promote-time website docs job needs cozystack/website#631
(`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard
detects the missing support and degrades to a ⚠️ line in the promote PR
body — promotion is unaffected and the tag-time backstop backfills, so
this PR is safe to merge first.

### Merge instructions

Please **merge-commit, do not squash**: the 13 commits are deliberately
atomic (the credential root-cause fix is separable from the feature
work) and squashing would collapse ~3k lines into one commit, hurting
bisectability.

### First live run

The e2e lane is verified by lint, contract tests and dry-traces, but has
not yet executed against a real tag; the intended first live run is a
manual `e2e-tag.yaml` dispatch against an existing published rc tag
before the gate is relied on for a real promotion.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added reusable changelog generation with validation, “present”
signaling, and safe artifact upload behavior (including RC-time
support).
* Added an E2E release-tag workflow with tag parsing, deterministic
sandboxing, and comprehensive report collection.
* Improved RC promotion with a stricter parse front-door, verified E2E
green gate (optional bypass), and earlier parallel changelog/asset
staging.
* **Documentation**
* Refreshed release-process docs to match the updated RC-to-stable,
changelog, and website-docs timing rules.
* **Bug Fixes**
* Improved PR finalization to source release bodies from verified
changelogs and warn/fallback on missing or empty content.
* Tightened label gating so E2E runs only under the intended conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-29 17:14:29 +05:00