Commit Graph
474 Commits
Author SHA1 Message Date
Myasnikov Daniil 5ed6b2204a ci(release): build the packages artifact when a freeze creates a line
build-release.yaml publishes `cozystack-packages:<line>` on push to a
release line, and pull-requests.yaml's overlay reads it so a release-line
PR tests its own line's binaries instead of main's (#3471, #3437).

It cannot fire for the push that creates the line. The freeze points
release-X.Y at a commit that is already on main, so the push carries no
new commits, and GitHub does not run a workflow whose paths/paths-ignore
filter finds no changed files ("If there are no files changed, the
workflow will not run"). The line would therefore have no artifact until
its first cherry-pick merged, and in that window the overlay finds
nothing to pull and leaves every package on its committed ref — at
freeze time the previous release's, which is exactly the
cross-generation mix #3437 fixed. The old flow had no such window: it
branched at the promote merge commit, which carried real commits and its
own release's refs.

Add workflow_dispatch to build-release.yaml and have the freeze step
dispatch it for the branch it just created, so the artifact exists from
the moment the line does. The dispatch is non-fatal: without it the
overlay no-ops and early cherry-pick PRs test their committed refs,
which is where they were before #3471. The tag is pushed and the line is
frozen by that point, so failing there would misreport both.

The trigger also gives a line build a re-run button, which previously
needed an empty commit pushed to the line.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:38:26 +05:00
Myasnikov Daniil 66ed32c6f9 ci(release): print the recovery command if the freeze push fails
Address review feedback from lexfrei on .github/workflows/cut-prerelease.yaml:293:
the tag push and this branch push are independent, so a transient failure
here leaves the line tagged but not frozen, and the write-once tag guard
refuses a same-tag re-dispatch — recovery is manual.

Fail with the exact push command rather than leaving the operator to
reconstruct it. Deliberately not an atomic two-ref push: the tag push
relies on being anchored to the branch tip for GitHub to populate the
push event's base_ref, which tags.yaml requires, and whether a multi-ref
atomic push preserves that is only observable on a live release cut.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:32:00 +05:00
Myasnikov Daniil a59ad88eb5 ci(release): distinguish absent from broken in the freeze branch probe
Address review feedback from lexfrei on .github/workflows/cut-prerelease.yaml:286:
the release-X.Y existence probe piped ls-remote straight into cut, so a
transport or auth failure produced empty output and read as "branch not
found", steering into the create path. The write-once tag guard, the
stale-tip guard, and the refuse step above all make this distinction;
match them here.

Behaviour is unchanged on the happy path — pipefail already failed the
step on a real error, and the non-forced push below fails closed on a
conflict. This makes the intent explicit rather than incidental.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil 6ff5b98d56 ci(backport): target the newest existing release line
Determine the backport targets by enumerating real release-X.Y branches
instead of deriving them from getLatestRelease.

getLatestRelease returns the newest published stable. Now that cutting
an rc creates release-X.Y before vX.Y.0 exists, that name lagged by a
full line for the whole freeze window: a `backport` on a fix for the
release being stabilised would have landed on the previous line and
missed the release it was written for. The freeze window is when
backports matter most, since the frozen branch is the only way in.

Also drops the contiguous-minor assumption — `previous` was computed as
min-1, naming a branch that does not exist whenever a minor is skipped.
Sorting is numeric, so release-1.10 correctly outranks release-1.9.

Outside a freeze window the result is unchanged: with release-1.0..1.5
present it still resolves to release-1.5 / release-1.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil 53a7fc4dcd feat(release): cutting an rc freezes the line into release-X.Y
Cutting the first vX.Y.0-rc.N now creates release-X.Y at the tagged
commit, and every later cut for that line must be dispatched from the
branch — a dispatch from main is refused.

Before this, only the stable promote-PR merge created release-X.Y, so
rc.2 and rc.3 were cut from main's tip and silently picked up every
feature merged since rc.1. The release therefore shipped content no rc
had ever validated. Freezing at the rc closes that: main stays open for
the next minor, and fixes reach the release by cherry-pick onto the
branch.

promote-rc.yaml needs no change — it already prefers release-X.Y as the
promote PR base when that branch exists, so the stable release is now
cut from the frozen tree.

Gated to -rc (alpha/beta are pre-freeze builds off an open main) and to
patch 0 (a patch-line rc is cut from a release-X.Y that already exists).
The branch is create-only and never force-moved: moving it would re-open
the freeze and drag in whatever the new tip holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
myasnikovdaniil f1f383626e fix(ci): overlay images from the PR base branch, and publish per-line artifacts (#3471)
## What this PR does

The PR finalize job overlays image refs for every package a PR did not
rebuild, so e2e and the installer do not test last-release images for
everything outside the PR's build matrix. It could only ever read
`cozystack-packages:main`, which meant a **release-line PR was handed
main's binaries to run against its own line's charts**.

cozystack/cozystack#3437 is the demonstration: a one-line change on
`release-1.6` deactivating an app failed install deterministically,
twice, with

```
helmrelease/backupstrategy-controller: Helm install failed …
error validating data: SchemaError(github.com/cozystack/cozystack/pkg/apis/core/v1alpha1.Option.spec):
unknown model in reference: "github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec"
```

main's `cozystack-controller` served an aggregated OpenAPI that
`release-1.6`'s charts could not validate against. Nothing in that PR
was broken; the lane was. Both branches carry `option_types.go` and key
`OptionSpec` identically in the committed generated OpenAPI, so this is
a generation mismatch at runtime, not a codegen drift.

This PR fixes it by giving every base branch its own artifact to overlay
from, rather than by turning the overlay off.

### The change

1. **`pull-requests.yaml`** reads `cozystack-packages:${BASE_REF}`
(`github.base_ref`) instead of a hardcoded `:main`.
2. **`build-release.yaml`** (new) publishes that artifact for maintained
`release-<major>.<minor>` branches exactly as `build-main.yaml` does for
main: every image tagged with the branch, and the whole packages tree
pushed with each reference digest-pinned to what the run just built.

An earlier revision of this branch simply skipped the overlay for
non-main bases. That fixed the wrong-images problem but left
release-line PRs testing their line's *last release*: for any package
the PR did not rebuild, the committed ref is the released digest, so a
component changed by an earlier backport was still exercised as its
pre-backport binary until the next rc. Per-line artifacts remove that
gap too, which is why the guard was replaced rather than kept.

### Three deliberate choices

**The trigger matches line branches only** (`release-[0-9]+.[0-9]+`).
The per-release and rc staging branches `promote-rc.yaml` and
`tags.yaml` create — `release-1.6.1`, `release-1.6.0-rc.4` — must not
trigger a full rebuild; their images come from the tag build and
rebuilding them is waste.

**`WRITE_CACHE` stays `0`.** `CACHE_REGISTRY/<img>:buildcache` is a
single ref per image and `build-main.yaml` is deliberately its only,
serialized writer so concurrent builds cannot race on the cache manifest
— the 409 class #2711 fixed for image tags. A line build can overlap a
main build, so writing here would reintroduce that race. Line builds
read the cache.

**A missing artifact still degrades to committed refs, but says so.** On
a release line it emits a `::warning::` naming the branch. Silent
degradation is indistinguishable from a working overlay, which is how a
mis-specified branch filter could hide for a whole release cycle.

### Cost

One `make build` per push to a maintained line — in practice per merged
backport. That is the price of release-line PRs testing their line's tip
instead of its last release.

### Verification

`hack/overlay-main-images_test.bats` pins the artifact tag to the base
branch, rejects a hardcoded `:main` in either overlay step, and pins
`build-release.yaml`'s branch filter, image tag and `WRITE_CACHE: '0'`.
Mutation-checked: restoring `:main`, setting `WRITE_CACHE: '1'`, and
broadening the filter to `release-*` each fail a test. 13/13 green;
`actionlint` and `zizmor` clean.

Worth an explicit ack in review: the branch-filter pattern is the one
thing no local test can prove, since only GitHub evaluates it. If it
does not match, `build-release` never runs and the new `::warning::` is
what surfaces it on the next release-line PR.

### Backport

`release-1.6` needs this too — for `pull_request` events GitHub builds
the workflow from the merge ref, so a release-line PR only stops
receiving main's images once the change is on its base branch.
cozystack/cozystack#3472 carried the interim guard and is closed in
favour of backporting this instead.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a “Build release line” workflow that builds and publishes images
and the packages artifact for maintained `release-<major>.<minor>`
branches.

* **Bug Fixes**
* Updated PR workflow finalization to use base-branch–specific package
overlays, avoiding incorrect main-branch image references when targeting
release branches.
* Improved fallback behavior when the base-branch packages artifact is
unavailable.

* **Tests**
* Added workflow wiring tests to verify base-branch artifact usage and
that each maintained release line publishes its own packages artifact.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-03 18:22:47 +05:00
Myasnikov Daniil 6a73a6caa9 docs(agents): call the E2E gate a commit status, not a check-run
The §10 heading, four workflow comments and both TOCTOU comments
still called the required "E2E Tests" context a check-run, while the
body of that same section — and the implementation — insist it is a
commit status. The distinction is the whole reason the section exists:
a check-run made with the default token floats into an arbitrary
check-suite, which is why the gate is a status. Leaving the wrong
primitive in the heading steers a maintainer wiring branch protection
at exactly the wrong thing.

The paragraph explaining why it is NOT a check-run keeps that wording,
since there the term is the rejected alternative. Also adds the blank
line missing before the "Chainsaw v0.2.15 gotchas" heading.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:57 +05:00
Myasnikov Daniil 1582eff3f7 docs(ci): correct the stale shared-runner claims in the fork e2e lane
Two comments in e2e-fork.yaml justified themselves with a threat that
no longer exists: the allowlist derivation guarded a `git worktree
add` against "a crashed/concurrent run on the shared self-hosted
runner", and the sandbox name was said to disambiguate "concurrent
fork runs on a shared host". The dedicated runner was decommissioned
in #2937 — these labels are ephemeral Oracle ARC VMs, one per job,
as .github/actionlint.yaml states. A comment that asserts a shared
host next to a step handling a push token misleads exactly the
reader who is auditing that boundary.

Both guards stay; only their stated reason changes to the one that
actually holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:47 +05:00
Myasnikov Daniil 1d668e1fcb fix(ci): scope the fork publish OCIR login to a run-local auth store
The `publish` job ran `skopeo login` and `docker login` with no
DOCKER_CONFIG override and no logout, so the OCIR push credential
landed in the runner's default stores (~/.docker/config.json and
$XDG_RUNTIME_DIR/containers/auth.json). Every other OCIR login in
this repo — pull-requests.yaml's build, build-talos and finalize —
already scopes to $RUNNER_TEMP/.docker, and `Set up Docker config`
in those jobs copies ~/.docker verbatim into the build context, so
a token left in the default store is a token handed to whatever
runs next.

Scope both logins to $RUNNER_TEMP/.docker (skopeo via --authfile,
flux via the docker keychain's DOCKER_CONFIG), carry the same env
into the push step that reads the auth, and drop the credential in
an always() logout step so a failed push does not leave it behind.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:32 +05:00
Myasnikov Daniil c46037ce7b fix(ci): stop a cancelled run from wedging the "E2E Tests" gate
Now that the required "E2E Tests" context is a commit status rather
than a job, a run that dies before posting leaves the head SHA at
"Expected" — which blocks the merge instead of passing it. Two paths
reached that state.

Same-repo: a `labeled` event shared the PR's concurrency group, so
labelling an open PR cancelled the live opened/synchronize run, and
the replacement run was then discarded by `plan`'s full-e2e guard --
so `e2e-report` skipped with it and nothing posted the status the
cancelled run was going to. Observed on live CI: run 30614258694
(cancelled) followed by 30614280603 (skipped) on the same SHA, 24s
apart, from a maintainer adding kind/* labels just after a push.
Label events now get their own concurrency group, so they can only
supersede each other. Release promotion is unaffected: `gh pr create
--label` attaches `release` at creation, so a promote PR gets a
single run that already sees the label (verified on #3397 and #3425,
one run each, build/finalize gated off).

Fork: `resolve` had no equivalent, so a cancelled triggering run fell
through to the `CONCLUSION != 'success'` branch and posted `failure`
— a false red on a SHA whose e2e never ran, which the replacement run
for that same SHA would not flip back (it takes the discarded-label
path and posts nothing). It now returns early on a cancelled or
wholly-skipped conclusion. Checked on the conclusion rather than the
job list because a cancelled run can report no jobs at all, which
would make the `Plan build` lookup miss.

Both stay fail-closed: declining to post can only preserve a verdict
a real run reached, never invent one.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:07 +05:00
Myasnikov Daniil 507336dba0 fix(ci): report "E2E Tests" green on a release promotion PR
Under main's rc-gate model a promote PR carries `release` and deliberately not
`full-e2e`, so the `e2e` job skips: the immutable rc whose digests the tree pins
was already verified, which is the cost the rc gate exists to remove. While the
required gate was a job, that skip satisfied branch protection. Now that it is a
commit status, e2e-report saw PLAN_RESULT=success, CODE=true (a promotion
rewrites ~38 non-docs image refs) and E2E_RESULT=skipped, fell to the last
branch and posted failure — so every release would have needed a full suite
re-run or an admin override of the very gate this PR strengthens.

Give e2e-report the carve-out the `e2e` job already has: release without
full-e2e reports success and says why. Adding full-e2e still forces a real run
and a real verdict.

Close the second route to the same red as well. `gh pr create --label` attaches
the label just after opening, so the `labeled` event cancels the `opened` run
through cancel-in-progress; under always() that dying run still posted failure
with PLAN_RESULT=cancelled, and the `labeled` run replacing it is skipped by the
allow-list guard, so nothing flipped the status back. !cancelled() makes a
cancelled run report nothing, which is the same fail-closed posture used
elsewhere here: not posting leaves the required status "Expected", which blocks.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-30 11:54:52 +05:00
Myasnikov Daniil 6728bd4245 fix(ci): don't clobber a fork PR's green "E2E Tests" on a label event
Same defect as the e2e-report fix, on the fork path. main's `labeled`
pull_request trigger produces a "Pull Request" run whose `plan` guard
discards every label but full-e2e, so every job skips and nothing is built.
workflow_run still fires for it, and `resolve` then reddened the required
status down either branch: a non-success conclusion fails closed, and a
`success` conclusion falls through to the missing-pr-patch check, which also
fails closed. Labelling an already-green open fork PR therefore turned its
E2E Tests red until the next push.

Detect that shape directly — the triggering run's `Plan build` job concluded
`skipped` — and return without posting anything. Statuses are per-SHA and an
unposted required status stays "Expected", which blocks the merge, so
declining to post can only preserve a verdict some real run reached, never
invent one. A fork that edits the workflow to force this path leaves its head
SHA with no status at all, i.e. not mergeable. The job already holds the
actions:read scope the job lookup needs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 18:35:14 +05:00
Myasnikov Daniil 8b79a0f20c fix(ci): don't clobber a green "E2E Tests" status on a discarded label event
main gained a `labeled` pull_request trigger whose `plan` guard discards
every label event except full-e2e. e2e-report carried no such guard, so
labelling an already-green open PR started a run in which `plan` was
skipped, e2e-report still ran (always()), read PLAN_RESULT != 'success' and
posted "E2E Tests" = failure — overwriting the passing required status and
making a merge-ready PR unmergeable until the next push.

Give e2e-report the same `labeled` guard as `plan`, so it runs if and only if
`plan` runs. Skipping posts nothing, which is fail-closed: an unposted
required status stays "Expected" and blocks the merge, so the guard can only
ever preserve a verdict, never invent one.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:57:17 +05:00
Myasnikov Daniil 2b53ac32a1 ci(e2e): run the fork e2e job in the workspace, not /tmp
e2e-fork.yaml's e2e job was written against the pre-78fc3f917 in-tree job,
which copied the checkout to /tmp/$SANDBOX_NAME, ran every step from there
and removed it on teardown. main has since dropped that workspace: e2e is a
single job on an ephemeral runner, so there is no cross-job rendezvous and
the checkout is never reused.

Keeping it here would also reintroduce the unguarded
`rm -rf /tmp/$SANDBOX_NAME` that commit deliberately deleted rather than
guarded — when an upstream step fails the unconditional sandbox-id write is
skipped, SANDBOX_NAME stays empty, and the `if: always()` removal expands
to `rm -rf /tmp/`.

Drop the Prepare/Remove workspace steps and the seven `cd
/tmp/$SANDBOX_NAME` prefixes, and point the chainsaw-report and cozyreport
upload paths at _out/. /tmp/changed.txt stays — it is a scratch file, not a
workspace copy, and the in-tree lane writes it the same way.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:48:43 +05:00
Myasnikov Daniil 60e910b863 docs(ci): correct the select-e2e.sh escalation claim in e2e-fork (#3262)
Both comments stated that select-e2e.sh escalates to the full suite on
any unrecognised path, so fork TIA "can only narrow". That is not what
it does: it escalates for an unrecognised path INSIDE packages/, but a
path it does not recognise at all (hack/*.mk, for one) selects nothing,
the Chainsaw step is skipped, and the required status still concludes
success.

Found by coderabbitai in review. The behaviour is pre-existing and
identical for same-repo PRs through pull-requests.yaml, so it is tracked
separately in #3392 rather than fixed on the fork lane alone — which
would leave forks stricter than same-repo for the same diff. Correct the
comments here so the guarantee is not overstated in the meantime.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov Daniil 705e76f5db fix(ci): fail closed on renamed and truncated fork PR file lists (#3262)
Address review feedback from coderabbitai on
.github/workflows/e2e-fork.yaml: the docs-only decision read only
`filename`, so a rename reported the NEW path alone. A PR renaming
packages/<app>/values.yaml -> docs/whatever.md therefore looked
docs-only, concluded "E2E Tests" green, and never ran the suite —
while in fact removing a code path. Both sides of a rename must be
under docs/ for a PR to count as docs-only.

`pulls.listFiles` is also capped at 3000 files, and a truncated list
could omit the only non-docs path. Fail closed at the cap instead of
ruling docs-only on a list known to be incomplete.

Feed both sides of a rename to TIA as well: the old path is a removal
the selector must see. Extra paths can only widen the selection, so
this cannot cause a suite to be skipped.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov DaniilandClaude Opus 4.8 1a1f3696ee fix(ci): fix the nullglob-defeated OCI-archive guard in the fork publish
`count=$(ls _out/oci/*.oci.tar | wc -l)` runs under `shopt -s nullglob`, so
when nothing matches the glob expands to zero words and `ls` lists the current
directory instead — count is always >=1 and the "at least one archive" guard
never fires. A zero-archive build then falls through to `flux push` and defers
the real failure to an obscure image-pull error deep in the ~180-minute e2e
run. Glob into an array (empty under nullglob) and test its length; reuse it
for the push loop.

Reported by coderabbitai on #3262.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 3b70fda870 fix(ci): make fork OCI_EXPORT_DIR absolute so archives land where the upload reads
The fork build sets OCI_EXPORT_DIR to a relative `_out/oci`, but each image is
built with `make -C <pkg> image`, so buildx resolves the relative `dest=`
against the package directory — the archive landed in <pkg>/_out/oci/ while the
`if-no-files-found: error` upload reads repo-root `_out/oci/*.oci.tar`. Every
non-docs fork build would have failed at that upload (never caught, since
same-repo PRs leave OCI_EXPORT_DIR empty). Root it at github.workspace so the
dest resolves to repo-root _out/oci regardless of the sub-make cwd.

Reported by gemini-code-assist on #3262 (its abspath suggestion resolves against
the package dir under `make -C`, so this fixes it at the workflow level instead).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 0f6e99eff8 fix(ci): publish the "E2E Tests" gate as a commit status, not a check-run
Route the required "E2E Tests" merge gate through a GitHub commit status
posted with the default GITHUB_TOKEN (+ statuses:write), replacing the
COZYSTACK_CI app check-run. The app path needed the app granted checks:write
(it isn't — checks.create 403s "Resource not accessible by integration"), and
the earlier default-token check-run floated under a random check-suite
("PR size label"). A commit status has neither problem: branch protection
matches it by context name, it carries no check-suite grouping, and it is safe
with the default token because a fork's pull_request token is read-only and
cannot post one — only same-repo runs (e2e-report, guarded !fork) and the
privileged workflow_run in e2e-fork.yaml can set it.

The security property is unchanged: a never-posted required status stays
"Expected" and blocks the merge, so a skipped job can't smuggle a green
(#3257). Removes all COZYSTACK_CI app-token machinery from e2e-fork.yaml —
`resolve` posts the status directly, no `check_run_id` threading, no `open`
step.

Branch protection must require the "E2E Tests" status context (no GitHub App
and no app-id pin needed).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 03b519dc75 fix(ci): fork e2e — Chainsaw+TIA run and image-push hardening
The fork e2e path in e2e-fork.yaml still ran the pre-#2826 BATS app loop
(`hack/e2e-apps/*.bats` + `test-apps-%`), which the merged Chainsaw
migration deleted — so the fork run was already broken. Rewrite it to the
same Chainsaw + Test-Impact-Analysis path as the in-tree job, and clear the
remaining review follow-ups on the publish job.

- Run E2E via a single `make test-chainsaw` invocation (no per-app retry —
  e2e-testing.md §1). Suite selection and the full-e2e override come from
  `resolve`'s TRUSTED base-repo listFiles/labels (new `changed_b64` /
  `full_e2e` outputs), never a git diff of the fork-controlled merge ref, so
  TIA cannot be shaped into skipping the suite (keeps the #3257 property).
  Collect + upload the Chainsaw JUnit report like the in-tree job.
- Restrict fork image pushes to a trusted allowlist derived from the base
  parent (HEAD^1) Makefiles, so a fork cannot mint an arbitrary repo path
  under the CI registry by renaming an image. The charset guard stays as
  defence in depth.
- `skopeo copy --all` so a multi-platform build's index digest (the one
  baked into pr.patch) is preserved.
- Pin the flux CLI (FLUX_VERSION) in the privileged publish job instead of
  install.sh's rolling latest.

Docs: e2e-testing.md §10 records that the fork path now runs Chainsaw + TIA
and that fork suite selection must stay sourced from trusted data.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 9cddb532bf fix(ci): publish "E2E Tests" check-run as the COZYSTACK_CI app
The synthetic "E2E Tests" required check-run was created with the default
GITHUB_TOKEN (the "GitHub Actions" app). A token-created check-run has no
owning job, so GitHub files it under an arbitrary GitHub Actions check-suite
for the head SHA — in practice the earliest one, the PR-size labeler — so
"E2E Tests" rendered under "PR size label".

Publish it as the COZYSTACK_CI GitHub App instead (create-github-app-token,
scoped to permission-checks: write) so the check lands in the app own
dedicated suite. This is also stricter: only the CI app can satisfy the gate.

- pull-requests.yaml (e2e-report): mint the app token, pass it to checks.create.
- e2e-fork.yaml: split resolve into a verdict decision on the default
  workflow_run token (keeps the actions:read / pull-requests:read lookups) and
  an "open" step that publishes the check as the app; report re-mints the token
  so checks.update runs as the same app that opened the pending run.
- docs/agents/e2e-testing.md: document the change and the branch-protection
  requirement.

Requires main branch protection to require "E2E Tests" from the COZYSTACK_CI
app id (not app_id 15368).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 eba55976a0 fix(ci): decide fork docs-only from the PR file list and harden publish (#3257)
resolve inferred "docs-only" from the absence of a pr-patch artifact, but
artifact presence is decided by pull-requests.yaml running from the fork's
merge commit — a fork could suppress it (code=false, or if:false on finalize)
and conclude the required E2E Tests check green with e2e never run, re-opening
the very bypass this feature closes.

Derive docs-only from trusted base-repo data (pulls.listFiles): green only
when every changed file is under docs/; any non-docs file makes e2e required,
and a missing pr-patch then fails the check closed. Resolve the PR open-only
(no closed/merged fallback).

Also harden the privileged jobs:
- drop `packages: write` from publish (it authenticates to OCIR, not GHCR);
- assert at least one OCI archive before pushing, and validate each archive
  basename against a strict charset before using it as a registry repo;
- install flux from a downloaded script rather than piping curl to sudo bash;
- assert `git rev-parse HEAD^2` equals the resolved head SHA in publish and
  e2e, so a moved merge ref cannot operate on a different tree than the check.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 076658e70f fix(ci): avoid zizmor github-env finding on fork e2e sandbox name
The `Set sandbox ID` step wrote a computed value to $GITHUB_ENV, which the
zizmor github-env audit flags as high severity (write to GITHUB_ENV may allow
code execution). Set SANDBOX_NAME as a job-level env var instead — no
$GITHUB_ENV write, so the audit does not apply. Uses github.run_id, which is
unique per run; in workflow_run context GITHUB_REF is the default branch and
cannot disambiguate concurrent fork runs sharing a self-hosted host.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 76c8f890fa fix(ci): export OCI archives for buildx calls that bypass image-tags (#3257)
Fork PRs build images to per-image OCI archives instead of pushing,
gated on OCI_EXPORT_DIR. The `--output type=oci` was appended only inside
the `image-tags` macro, so `packages/system/capi-providers-cpprovider`,
whose `image:` recipe uses a hardcoded `--tag` and never calls
`image-tags`, ran buildx with no exporter under export mode (PUSH/LOAD
forced to 0). No archive was written, and the fork build's
`if-no-files-found: error` upload then failed the whole build. Its
`--cache-to type=inline` also needs an image exporter.

- hack/common-envs.mk: extract the OCI-output snippet from `image-tags`
  into a reusable `oci-output <image-name>` macro so any buildx call can
  opt in; `image-tags` now ends with `$(call oci-output,$(1))` (no-op for
  the ~33 conforming packages). Move `comma :=` above its first user.
- packages/system/capi-providers-cpprovider/Makefile: add
  `$(call oci-output,cluster-api-control-plane-provider-kamaji)` (archive
  base name equals the OCIR repo the privileged push keys on) and drop the
  inline cache under export (`--cache-to type=inline` gated off).
- .github/workflows/pull-requests.yaml: wrap the `E2E Tests` check-run
  creation in try/catch. On error, `core.setFailed` so the required merge
  gate fails loudly and fail-closed instead of silently succeeding.

Repo-wide audit of every `docker buildx build` reachable from the root
`build:` matrix confirms capi was the only recipe bypassing both
`image-tags` and `oci-output`.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:25 +05:00
Myasnikov DaniilandClaude Opus 4.8 e0ac2f9e12 ci(e2e): run fork-PR e2e via privileged workflow_run (#3257)
Fork PRs cannot push images (pull_request runs from a fork receive no
secrets), so `make image` pushed anonymously and died with `denied`.
That failed every build, collapsed the build->finalize->e2e chain, and
left the required "E2E Tests" job skipped — which branch protection
treats as satisfied, so an approved fork PR could merge with e2e never
having run.

pull-requests.yaml (fork PRs):
- build / build-talos / finalize export images to per-image OCI archives
  (OCI_EXPORT_DIR) and upload them instead of pushing, so the red
  "denied" build wall disappears and the chain no longer collapses.
- the in-tree e2e job is renamed off "E2E Tests" and guarded to
  same-repo PRs; a new e2e-report job concludes the required "E2E Tests"
  context as an explicit check-run (success on a docs-only PR or when the
  in-tree e2e passed, failure otherwise). A skipped job can no longer
  satisfy the required check.

e2e-fork.yaml (new, privileged workflow_run from the default branch):
- resolves the fork PR, opens a pending "E2E Tests" check-run, and
  fails-closed if the fork build failed;
- pushes the pre-built image archives to OCIR by digest and flux-pushes
  the digest-pinned packages tree — the OCIR token is live only for
  trusted binaries (git/skopeo/flux/yq) over data, never for `make` or
  any fork-authored script;
- runs the e2e suite against the pushed closure with no push token and
  concludes the "E2E Tests" check-run on the PR head SHA.

Fork build code never receives registry credentials; the fork's images
and test scripts run only in the credential-less, ephemeral e2e job.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:25 +05:00
Myasnikov Daniil 2330d6f3ca fix(ci): overlay images from the PR base branch, and publish per-line artifacts
Supersedes the interim guard in this branch`s first commit, which skipped the
overlay entirely for release-line PRs. Skipping fixed the wrong images but left
those PRs testing their line`s last release: for any package the PR did not
rebuild, the committed ref is the released digest, so a component changed by an
earlier backport was exercised as its pre-backport binary until the next rc.

The overlay now reads `cozystack-packages:<base branch>` instead of always
`:main`, and build-release.yaml publishes that artifact for every maintained
`release-<major>.<minor>` branch the way build-main.yaml does for main: images
tagged with the branch, and the whole packages tree pushed with each reference
digest-pinned to what the run just built. Each base branch therefore has its own
generation to overlay from, which is what the original bug was really about —
#3437 failed install because main`s cozystack-controller served an aggregated
OpenAPI release-1.6`s charts could not validate against.

Three deliberate choices:

  * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The
    per-release and rc staging branches promote-rc.yaml and tags.yaml create
    (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their
    images come from the tag build, and rebuilding them would be waste.
  * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per
    image and build-main.yaml is deliberately its only, serialized writer so
    concurrent builds cannot race on the cache manifest (the 409 class #2711
    fixed for image tags). A line build can overlap a main build, so writing here
    would reintroduce that race. Line builds read the cache.
  * A missing artifact still degrades to committed refs, but on a release line it
    now emits a ::warning:: naming the branch. Silent degradation is
    indistinguishable from a working overlay, which is how a mis-specified branch
    filter would hide for a whole release cycle.

Cost: one `make build` per push to a maintained line, i.e. per merged backport.

hack/overlay-main-images_test.bats pins the artifact tag to the base branch,
rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s
branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main,
setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test.
13/13 green; actionlint and zizmor clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:29:43 +05:00
myasnikovdaniil cc44f6c981 ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449)
## What this PR does

Moves release validation from promote time to rc time, making promotion
itself a pure retag operation. This is the follow-up to the v1.6.0
retrospective: the promote flow's checks ran on the promote PR (coupled
to promotion, skippable by never promoting), the changelog and website
docs were produced only after the release went public, and the finalize
tag push silently failed to trigger `tags.yaml` at all
(persisted-credential bug, root-caused below).

The release lifecycle after this PR:

1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes
the staging branch — and now runs a **mandatory full e2e** for every
`-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and
any published tag) get the same validation as a **manual button**
(`workflow_dispatch`).
2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**:
the release changelog is generated from the rc tag (reusable
`changelog-generate.yaml`) and committed to the rc staging branch, ready
for review long before promotion.
3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence:
the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a
successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input
for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages
the release, picks up the rc-time changelog (or generates one), and
**opens the website docs PR** on cozystack/website from the staging
branch — parked with an explicit DO-NOT-MERGE-until-finalize contract.
4. The promote PR **no longer gets `full-e2e` automatically** — its tree
is the rc's already-validated digests with only tag strings rewritten; a
maintainer can still add the label manually to force the suite.
5. Finalize performs the irreversible side effects (tag, publish, retag,
chart) — and with `persist-credentials: false` its tag push now actually
authenticates as the CI app, so the `tags.yaml` backstops
(`generate-changelog`, `update-website-docs`) fire for promoted tags.
This is the one-line fix for the v1.6.0 incident where the tag triggered
nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader`
silently defeats the app token injected via `git remote set-url`, and
`GITHUB_TOKEN` pushes create no workflow runs.

Everything is pinned by executable contract tests
(`hack/promote-gate-contract.bats`, extended
`hack/release-changelog-contract.bats` /
`release-changelog-behaviour.bats` — 48 tests total, including behaviour
tests that execute the new `hack/parse-rc-tag.sh` and
`hack/select-changelog-source.sh` for real), and `docs/release.md` is
rewritten to describe the new lifecycle end-to-end.

### Commit map (review commit-by-commit)

- `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing
changelog-before-publication rework (changelog generated at promote,
committed to the staging branch before the PR opens, finalize publishes
it as the release body; tags.yaml job becomes a verbatim-port backstop),
rebased onto main.
- `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane
(prerelease → staging branch, stable → tag; draft releases fail fast;
dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts).
- `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut
rc tags (also on `release_exists` re-runs); fixes the
`cut-prerelease.yaml` comment that falsely claimed e2e already existed.
- `2dfbeea77` — `changelog-generate.yaml` (reusable core) +
`changelog-rc.yaml` (rc-time button committing to the staging branch);
promote-rc gains a top-level `parse` job with a strict no-whitespace tag
policy (`hack/parse-rc-tag.sh`); source selection is
validate-then-fallthrough (`hack/select-changelog-source.sh`) so an
invalid higher-priority changelog can never silently win.
- `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override +
removal of the automatic `full-e2e` label (pull-requests.yaml
release-path e2e now requires an explicit label; a post-open label now
actually starts the run).
- `936190d4b` — `website-docs` job at promote time behind a
FETCH_REF-support guard (see dependency below), with the
DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before
`release-next`; `tags.yaml::update-website-docs` stays as the backstop.
- `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0
root cause).
- `717c8860f` — `docs/release.md` coherence pass over the whole new
lifecycle.

### Dependency

The promote-time website docs job needs cozystack/website#631
(`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard
detects the missing support and degrades to a ⚠️ line in the promote PR
body — promotion is unaffected and the tag-time backstop backfills, so
this PR is safe to merge first.

### Merge instructions

Please **merge-commit, do not squash**: the 13 commits are deliberately
atomic (the credential root-cause fix is separable from the feature
work) and squashing would collapse ~3k lines into one commit, hurting
bisectability.

### First live run

The e2e lane is verified by lint, contract tests and dry-traces, but has
not yet executed against a real tag; the intended first live run is a
manual `e2e-tag.yaml` dispatch against an existing published rc tag
before the gate is relied on for a real promotion.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added reusable changelog generation with validation, “present”
signaling, and safe artifact upload behavior (including RC-time
support).
* Added an E2E release-tag workflow with tag parsing, deterministic
sandboxing, and comprehensive report collection.
* Improved RC promotion with a stricter parse front-door, verified E2E
green gate (optional bypass), and earlier parallel changelog/asset
staging.
* **Documentation**
* Refreshed release-process docs to match the updated RC-to-stable,
changelog, and website-docs timing rules.
* **Bug Fixes**
* Improved PR finalization to source release bodies from verified
changelogs and warn/fallback on missing or empty content.
* Tightened label gating so E2E runs only under the intended conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-29 17:14:29 +05:00
Andrei Kvapil 78fc3f9178 refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows)
The E2E job once ran as several state-sharing self-hosted-runner jobs
that rendezvoused through a sha256(REPO:WORKFLOW:REF) key and a per-run
/tmp/$SANDBOX_NAME copy of the checkout. Since the move to the ephemeral
pool it is a single job: there is no cross-job rendezvous and the
checkout is never reused, so copying it into /tmp and removing it
afterwards is dead weight.

Run every step in $GITHUB_WORKSPACE instead: drop the Prepare/Remove
workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes, and point the
upload paths at _out/. This deletes the unguarded `rm -rf
/tmp/$SANDBOX_NAME` rather than guarding it: when an upstream step fails
or the job is cancelled, the unconditional `Set sandbox ID` is skipped,
SANDBOX_NAME stays empty, and the `if: always()` steps interpolate it —
the removal expanding to `rm -rf /tmp/` and failing the cleanup step.
`Set sandbox ID` stays, since the container still needs a name.

Apply the same treatment to nightly.yaml, and drop the Makefile dead
code the single-runner retirement left behind: the world-writable
mkdir/chmod of /tmp/$SANDBOX_NAME (backed a bind mount removed long ago)
and the `-e SANDBOX_NAME` container env (nothing in the e2e-sandbox
image reads it).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 17:49:02 +02:00
Myasnikov Daniil fea40d6314 fix(ci): only overlay current-main images on main-based PRs
The PR finalize job pulls cozystack-packages:main and repoints every package
the PR did not rebuild at current-main images. On a main-based PR that is the
whole point: it stops e2e from testing last-release images for everything
outside the build matrix. On a release-line PR it is wrong — the committed refs
there are not stale, they are that line`s released digests, and the charts are
written against exactly those.

So a release-line PR was installing main`s binaries onto its own charts, and the
mismatch grows with every commit main gains. #3437 is the demonstration: a
one-line change on release-1.6 that deactivates an app failed install
deterministically, twice, with

  SchemaError(...core/v1alpha1.Option.spec): unknown model in reference:
  "...core~1v1alpha1.OptionSpec"

from main`s cozystack-controller serving an aggregated OpenAPI that branch`s
charts cannot validate against. Nothing in the PR was broken; the lane was.
Left alone this makes every 1.6 backport look red, which is when release-branch
PRs are busiest.

Both overlay steps are now gated on `github.base_ref == main`, so a release-line
PR keeps its committed digests — the behaviour that predates the overlay.
Retargeting the overlay at a per-line artifact would be better but is not
possible today: build-main.yaml publishes only cozystack-packages:main, and the
registry carries no release-* equivalent (verified against the packages repo`s
tag list: `main` plus per-PR tags, nothing else).

hack/overlay-main-images_test.bats pins the wiring per step, so adding a third
overlay step without the guard fails the suite. Mutation-checked by removing one
guard. 12/12 green; actionlint clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-27 21:41:57 +05:00
Myasnikov Daniil a2667c58ca fix(ci): grant rc-e2e the checks:write ceiling e2e-tag.yaml declares
A caller job's permissions are the ceiling for every job in the workflow it
calls, and GitHub validates that ceiling when it CREATES the run — before the
calling job's `if:` is evaluated, let alone before a step runs. e2e-tag.yaml's
e2e job declares `checks: write` for its workflow_dispatch-only breakpoint, so
granting only `contents: read` here made the run fail validation with
"requesting 'checks: write', but is only allowed 'checks: none'".

A startup failure takes the whole run with it, and an `if:` cannot gate it, so
this was not limited to rc cuts: EVERY tag push — rc and stable, including the
one finalize pushes to publish a release — would have produced no build, no
draft release and no staging branch.

The comment claimed the opposite: that because workflow_call can never reach the
breakpoint branch, the grant was unnecessary. Runtime reachability has no bearing
on a static check. e2e-tag.yaml now states the contract for its callers, next to
the permission block that was being read as "contents: read is all this needs".

hack/promote-gate-contract.bats pins both halves of the pair, and separately pins
the `E2E <tag> (full suite)` job name the gate correlates evidence by. Neither is
reachable from any PR lane — tags.yaml runs only on tag pushes — so a regression
here would otherwise surface at the next release. Mutation-checked: dropping the
grant fails the first pin, renaming the e2e job fails the second.

changelog-rc.yaml's reusable-workflow call now pins `contents: read` explicitly,
matching promote-rc.yaml's identical call. The outcome is the same either way,
but the docs describe an omitted block as the repository default rather than the
workflow's own top-level block, and this is the file class where that ambiguity
just cost a release lane.

Reported by CodeRabbit on #3449; blast radius and the missing pins found by an
adversarial re-check.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-27 14:03:49 +05:00
Myasnikov Daniil 01e1e7188e ci(release): stop finalize persisting credentials so the tag push triggers tags.yaml
pull-requests-release.yaml's finalize job checked out the repo with the default
persist-credentials, leaving GITHUB_TOKEN embedded as http.extraheader. Every
later tag/branch push re-injects the CI app token via `git remote set-url`, but a
persisted extraheader silently wins over the URL credential — so those pushes
authenticated as GITHUB_TOKEN. A GITHUB_TOKEN-authenticated push creates no
workflow run (GitHub anti-recursion), which is exactly why v1.6.0's stable tag
never triggered tags.yaml: the generate-changelog and update-website-docs
backstops never ran.

Set persist-credentials: false on the finalize checkout, so the set-url app token
actually authenticates the pushes — the same handling promote-rc.yaml already
uses. Audited every git/gh step in the job: both tag steps set-url the app token
before pushing, and Ensure-maintenance-branch / Publish-draft-release / the
retag+chart steps use the app token via github-script or a separate registry
login — none depends on the persisted GITHUB_TOKEN extraheader.

Pin it in hack/promote-gate-contract.bats.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 18:16:26 +05:00
Myasnikov Daniil 936190d4b6 ci(release): open the website docs PR at promote time
The cozystack/website 'update managed apps reference' PR used to be opened only
at tag time by tags.yaml::update-website-docs — after finalize pushed the stable
tag, i.e. after the release was already published (too late to review) — and it
fetched README content from that stable tag (BRANCH=RELEASE_TAG), which finalize
only creates post-merge.

Add a promote-time website-docs job to promote-rc.yaml (needs [parse, promote])
that opens the same update-docs-vX.Y.Z PR from the release-X.Y.Z staging branch
via the website Makefile's FETCH_REF variable, so the docs are reviewable
alongside the promote PR while the stable tag is still unborn. It refreshes the
trunk pins from the staging ref before release-next so a new minor inherits fresh
Talos/cozystack pins (the v1.6 talos-pin incident class), then runs update-all
with FETCH_REF pinned to the staging branch.

The job is non-blocking: a FETCH_REF-support guard fails loudly (warn + exit 1)
when cozystack/website main predates the FETCH_REF Makefile variable — otherwise
make would silently ignore the unknown FETCH_REF override, fall back to the
not-yet-created stable tag, and update_apps.sh would warn-and-continue into STUB
docs. Any failure surfaces as a website-docs warning line in the promote PR body
(plumbed through open-pr via needs.website-docs.result) and never blocks
promotion; the tag-time job stays as the backstop and no-ops when promote already
did the work.

MERGE-TIMING CONTRACT: the website PR must NOT be merged before finalize
publishes the release — doing so flips the site's latest-version pointer to a
version whose docs and openapi are not on GitHub yet, 404ing every version-pin.
The website PR body and the promote PR body both say so in bold.

Pin the contract in hack/promote-gate-contract.bats (needs shape, FETCH_REF
invocation, the guard, non-persisted checkout credentials, the DO-NOT-MERGE body
wording, the promote-PR status line, and the tags.yaml backstop comment) and
relax the release-changelog-contract needs pin to tolerate the appended
website-docs dependency.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 18:16:26 +05:00
Myasnikov Daniil ff7ddd3c9a ci(release): gate promotion on a green rc e2e and drop the automatic full-e2e label
Promotion now assumes checks passed at rc time instead of re-running
them. At dispatch, before any staging happens, promote-rc verifies the
rc tag has a green full e2e via either evidence path: the rc's
tags.yaml push run containing a successful 'rc-e2e / E2E <tag> (full
suite)' workflow_call job, or a successful manual e2e-tag.yaml dispatch
correlated by its exact tag-bearing job name (immutable rc digests make
past evidence valid). No evidence fails the dispatch with both paths
and the remediation named. A skip_e2e_gate boolean input bypasses the
gate loudly for emergencies; the promote PR body always carries the
verification state (verified or bypassed), and 'verified' can only
originate in the gate's green branch.

The promote PR keeps the release label but no longer gets full-e2e
automatically: its tree is the rc's already-validated digests with only
tag strings rewritten, so pull-requests.yaml now runs e2e on the
release path only when a maintainer opts in by adding full-e2e. The
regular-PR path and the ordinary full-e2e forcing are untouched, and
the new labeled trigger makes a post-open label actually start the run.

A new promote-gate contract suite pins the gate's DAG position, the
override default, the label set, the body lines, and the label-gate
condition with comment-proof executable-line counts.

Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 18:16:26 +05:00
Myasnikov Daniil 2dfbeea776 ci(release): dispatch changelog generation at rc time
Extract the promote-time changelog generation into a reusable workflow
(changelog-generate.yaml, workflow_call) and add a dispatchable wrapper
(changelog-rc.yaml) so a maintainer can generate the release changelog
the moment an rc goes green, while there is still time to review it.

The reusable core keeps promote-rc's observable behaviour: detached
rc-tag checkout, read-only app token for the AI step, the
validate-changelog.sh gate, non-blocking AI failure, and the same
changelog-vX.Y.Z artifact. .release-tooling is overlaid from
github.ref_name, which under workflow_call is the caller's dispatch ref
(and coincides with the reusable workflow's own ref for a local uses:),
so "current tooling against an old rc tag" is preserved. Secrets are
declared explicitly rather than inherited, so a caller that forgets one
fails at parse time.

The rc tag is parsed by a shared hack/parse-rc-tag.sh that REJECTS
whitespace outright (a stripped 'v1. 6.0-rc.2' would silently become a
different tag). promote-rc parses it ONCE in a cheap `parse` job that
both `promote` and `changelog` depend on — so they still run in parallel
with each other while consuming one validated tag, and a whitespace tag
fails the whole promotion rather than letting promote succeed while the
changelog job rejects the same input and the PR opens empty. The
reusable core mirrors the reject-whitespace policy inline because its
parse must run before any checkout.

changelog-rc.yaml verifies the rc staging branch release-X.Y.Z-rc.N
exists UP FRONT (in parse, which generate depends on), so a missing
branch fails in seconds rather than after a 30-minute AI run, then
commits the result there as docs/changelogs/vX.Y.Z.md idempotently
(compare-before-push).

promote-rc's changelog job is a thin uses: call. Source selection —
validate-then-fall-through across the rc tag tree, the rc staging branch
(the rc-time pickup), then AI generation, discarding any invalid source
with a loud warning so it never suppresses a valid lower-priority one —
lives in hack/select-changelog-source.sh, executed after the tooling
overlay. open-pr carries the picked-up file onto release-X.Y.Z and
finalize publishes it.

Contract and behaviour bats are repointed and extended: the shared
parser, the source-selection script, and promote's parse policy are all
exercised by execution on adversarial inputs; the remaining grep pins
match executable lines with exact counts so a comment or dead block
cannot satisfy them. docs/release.md lists the new workflow and the
rc-time entry point. actionlint and zizmor (default and pedantic) are
clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 18:16:16 +05:00
Myasnikov Daniil 025564d961 ci(release): make full e2e mandatory for freshly cut rc tags
Wire the published-tag e2e lane into tags.yaml as a leaf rc-e2e job:
needs prepare-release, fires only for -rc. tags (alpha/beta stay
button-only via e2e-tag.yaml dispatch), and calls the reusable lane
with the pushed tag. The staging branch and published rc release both
exist before the job starts because needs waits for the whole
prepare-release job (Publish rc release + Create release branch).

rc-e2e deliberately also runs when prepare-release short-circuits via
release_exists: that flag means skip-rebuild, not skip-validation, and
revalidating a published rc is idempotent. The caller grants only
contents: read — the lane's checks: write is needed solely by its
dispatch-only breakpoint, unreachable under workflow_call.

Also correct the cut-prerelease handoff comment, which claimed
tags.yaml already ran e2e (it never did).

Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 18:16:08 +05:00
Myasnikov Daniil 5aec70b3ae ci(release): add published-tag e2e validation lane
The release bits currently get their only e2e on the promote PR, which
couples validation to promotion and leaves alpha/beta tags with no e2e
path at all. This lane decouples them: a workflow_call + dispatch
workflow that validates any PUBLISHED release tag — the digest-pinned
tree (staging branch for prereleases, the tag itself for stable), the
release's nocloud disk asset, and the full Chainsaw suite, mirroring
the proven release-path e2e from pull-requests.yaml (no TIA, no PR
context). Draft releases fail fast with a clear message: checks in the
new release model happen at rc time, not at promote time.

A dispatch-only debug input exposes the same SSH breakpoint as the PR
lane; workflow_call runs can never trip it. Artifacts are tag-suffixed.

Wiring it as the mandatory post-rc-cut validation lands separately in
tags.yaml; for alpha/beta/stable this workflow is the manual button.

Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 16:24:01 +05:00
Myasnikov Daniil dd308644c8 ci(release): execute the release-changelog logic in tests, not just grep it
The existing suite asserts on workflow YAML. That catches a rename or a
deleted `if:` and nothing else: it cannot tell whether the shell inside a
`run:` block does what the step name claims, and it stayed green while the
logic underneath was wrong — twice, in the two preceding rounds.

Add hack/release-changelog-behaviour.bats, which runs things for real.

The GitHub Actions semantics the whole non-blocking design rests on are
now verified by executing them. hack/testdata/needs-semantics.yaml mirrors
the promote / changelog / open-pr shapes with a fatal (non-AI) step in the
changelog job, and `act` runs it. Result: open-pr DOES run when a job it
needs has failed, `needs.changelog.result` is `failure`, and — the part
that could not be read off the YAML — a failed job's outputs come back
EMPTY, while a successful job's propagate. That empty-output behaviour is
precisely why open-pr re-establishes changelog presence from disk instead
of trusting needs.changelog.outputs. Deleting the `if:` from the fixture
makes open-pr get skipped, so the gate is demonstrably load-bearing.

The git logic moves into hack/changelog-preserve.sh and is exercised
against real bare remotes: a branch that exists, one that does not, one
with no changelog, one with an empty changelog, and the full
preserve/checkout -B/restore/force-push cycle promote actually performs.
The last of these directly reproduces the regression the preserve step
exists to prevent. An empty changelog is deliberately NOT preserved — it
would suppress generation and then publish as blank notes.

finalize's release-body selection runs under node across all three inputs
(good, whitespace-only, absent) and the resulting decision table is
compared as a whole.

Replace the shell overlay with a sparse second actions/checkout of the
dispatch ref at .release-tooling. The overlay had a bootstrap problem I
introduced: the script that fetches missing tooling was itself missing
from the rc tree, so it could never have worked for exactly the trees that
need it. actions/checkout comes from the runner, not the repo, so it has
no such circularity. .release-tooling is gitignored because promote runs
`git add -A`, and the contract suite pins that.

All 21 tests mutation-checked. Breaking changelog-preserve.sh, making it
return an empty file, removing the fixture's `if:`, dropping the tooling
checkout, and renaming its path each fail the suite and pass on restore.

Not covered, and stated in the suite header rather than implied: the AI
generation step, the real GitHub API calls, and whether the sparse
checkout resolves on GitHub's runners.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 11:34:17 +05:00
Myasnikov Daniil 34e45295fe ci(release): fix second-round review findings
Two more review passes (Claude + Codex) on the previous commit. Codex
found the one that would have broken the very next promotion.

The agent reads its own instructions from the working tree, which is the
rc tag. Every rc cut before this change carries the old §2 of
docs/agents/changelog.md — two valid configurations, "stop" on anything
else — and a detached rc-tag checkout is exactly that "anything else". So
promoting v1.6.0-rc.4 would have presented a compliant agent with
instructions telling it to refuse, silently, producing no changelog on
every promotion of an existing rc. Verified against the tag. The same
applies to hack/validate-changelog.sh, absent from those trees entirely,
which would have failed the Verify step with "command not found" and, in
open-pr, taken the promote PR down with it. Both are now overlaid from the
dispatch ref, best-effort, in both jobs.

The structural validation added last round rejected every PATCH
changelog. It asserted a `# Cozystack vX.Y.Z` header and a 20-line floor;
patch releases use `# vX.Y.Z (date)` and v1.5.1 is a complete 19-line
release. Good AI output would have been discarded and the promote PR
mislabelled "generation failed". Measured across the tree: the header form
holds for 23 of 115 changelogs, so it was never an invariant.

The predicates now live in hack/validate-changelog.sh, tested directly
against real changelogs rather than through YAML greps — the reason the
defect shipped is that the suite pinned finalize's guard but nothing on
the promote-side validator, and stayed green when the validator was
mutated away. It accepts both header conventions, has no line floor, and
checks the version in the header, the release-link comment and the compare
link, so a changelog generated for the wrong version is caught. All 16
current-era changelogs pass; whitespace, empty, truncated and
wrong-version inputs are rejected.

Also fixed:

- A fatal artifact download in open-pr would abort the job before the PR
  step — the same wedge as a skipped job, one step lower. Download is now
  best-effort and the commit and PR body key off what is on disk, not an
  upstream job output that says nothing about whether the transfer worked.
- A pre-existing changelog (hand-written, or preserved across the
  force-push) reached the PR without ever being validated, was advertised
  with ✅ and published verbatim. It now goes through the same validator,
  as does the tags.yaml backstop.
- `gh pr view <branch>` matches closed PRs, so re-dispatching after
  closing an abandoned promote PR edited the dead PR and opened nothing.
  Branch on state explicitly.
- `gh pr edit --body` unconditionally overwrote the PR body, destroying
  the upstream breaking-change notes docs/release.md instructs reviewers
  to paste there. Refresh only while the body is still the generated one.
- open-pr consumed raw dispatch input while its siblings used the
  validated parse; the changelog job's bash parse did not trim, so an
  input with a stray space passed promote and failed there. Both fixed.
- The prompt and §2 said "trailing release-link comment"; it is the first
  thing in every changelog.

Tests grow to 16, each mutation-checked: dropping the overlay, making the
download fatal, reverting the existing-changelog check to `-s`, and
removing the backstop validator each fail the suite.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 11:34:17 +05:00
Myasnikov Daniil f49d54a689 ci(release): fix review findings; keep website docs at tag time
Dual-model review (Claude + Codex) of the previous two commits found seven
substantive defects, two of them destructive. Both models independently
reached three of them; the split is noted per item.

Descope the website half (Claude, verified by reproduction). The docs job
was moved to promote time, but the website Makefile pins the fetch ref
with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line
BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that
cannot exist at promote time, since promote's own validate step hard-fails
if it does. update_apps.sh logs the failed fetch and exits 0 AFTER
truncating each destination file, so every app doc would become a stub,
`changed=true` would open the PR, and the new auto-merge would land it
unreviewed. Not fixable from this repo. update-website-docs returns to
tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize
auto-merge is dropped. Opening it earlier needs a cozystack/website change
first and is left as a follow-up. This also removes the unordered-sibling
race Codex found, where the release PR could merge before the docs job
opened its PR, leaving finalize nothing to merge and no retry.

Fix the non-blocking guarantee (both models). Only the AI step carried
continue-on-error; the changelog JOB did not, so a flaky `npm i -g
@github/copilot` failed it and `needs` skipped open-pr entirely — staging
branch pushed, release drafted, no promote PR. Gate open-pr on promote
alone.

Generate from a checkout the contract permits (both models). The job
checked out the rc staging branch, which docs/agents/changelog.md §2
rejects — a compliant agent stops and writes nothing, so the normal path
produced no changelog at all. Check out the rc tag detached and document
that configuration in §2. The checkout ref now comes from the validated
parse output rather than raw dispatch input.

Port, don't regenerate, in the backstop (both models). For a patch release
promoted onto release-X.Y the changelog never reaches main, so the
backstop fired and ran a second AI pass; once its PR merged,
update-releasenotes.yaml overwrote the published body with unreviewed
text. Detect the file at the tag and port it verbatim; the AI runs only
when it is absent from both main and the tag.

Survive re-dispatch (Codex). promote force-pushes the staging branch,
erasing a hand-written changelog — the documented recovery when generation
fails. Preserve an existing one across the rebuild, and stop open-pr
overwriting it with generated output.

Validate structurally, not by size (Codex). A truncated or whitespace-only
AI run passed `[ -s ]`, was committed, advertised as valid in the PR body,
and published verbatim as release notes. Assert the header, the compare
link and a plausible line count; finalize additionally refuses
whitespace-only content rather than blanking the notes.

Add hack/release-changelog-contract.bats — nine tests pinning each link in
the chain, mutation-checked: removing open-pr's if:, breaking finalize's
body read, and restoring paths-ignore each fail the suite.

Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which
described the old flow throughout.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 11:34:16 +05:00
Myasnikov Daniil ba67fea7df ci(release): drop paths-ignore from the release finalize trigger
The promote PR now carries docs/changelogs/vX.Y.Z.md, and Prepare stable
branch explicitly tolerates producing no tag-string changes when the
digests are already stable. Together those can yield a docs-only promote
PR, which `paths-ignore: docs/**/*` would filter out — finalize never
runs, so the stable tag is never cut and the release is never published,
with no error raised anywhere.

The `if:` gate (merged + `release` label + release-bot author) is what
actually decides whether this workflow does anything, so let it do the
filtering. The cost is a skipped job on unrelated closed PRs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 11:33:10 +05:00
Myasnikov Daniil 75b5192f1b ci(release): produce changelog and docs before the release goes public
Changelog generation and the website-docs PR both lived in tags.yaml,
gated on the stable tag push. Under the promote flow that tag is created
by pull-requests-release.yaml finalize, which also publishes the release
— so both jobs could only ever run once the release was already public.
A reader hitting the release in that window saw an empty body, and the
site lagged until someone merged the docs PR.

Move both to promote time:

- promote-rc.yaml gains a `changelog` job that runs in parallel with
  `promote` and hands the file to a new `open-pr` join job, which commits
  it onto the release-X.Y.Z branch so it rides the promote PR. Generation
  is AI-driven and slow while `promote` spends its wall clock on the Talos
  assets; neither needs the other, so both start at once.
- The website-docs job moves to promote-rc.yaml too, depending only on
  `promote`. Different repo, so it cannot ride the PR — it is opened early
  and finalize merges it after the release publishes.
- finalize reads docs/changelogs/vX.Y.Z.md from the merge commit and uses
  it as the published release body. update-releasenotes.yaml is no longer
  on the critical path: it triggers on the same push and would race this
  job, and if it won it would find no published release, skip, and never
  fire again. It also only watches main, so a patch release promoted onto
  release-X.Y was never synced at all.
- tags.yaml keeps generate-changelog as a backstop only. It self-skips
  when the changelog is already on main, and covers the two cases where
  promotion does not put it there: generation failed (non-blocking), or
  the promote PR targeted a maintenance line.

Generation stays non-blocking, so a flaky AI step cannot wedge a
promotion. When it fails the promote PR body says so explicitly, and the
escape hatch is to commit docs/changelogs/vX.Y.Z.md to the branch — which
also makes a hand-written changelog first-class: if the file is already
in the rc tree, generation is skipped entirely.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-24 11:33:10 +05:00
Myasnikov Daniil baf0158d81 fix(release): reuse only an OPEN promote PR, not one in any state
The promote step guarded PR creation with `gh pr view <branch>`, which
resolves a pull request by head branch in any state. A closed one
satisfied the guard, so creation was skipped while the step still exited
0 — a promotion that reports success and leaves nothing to merge, the
same silent-skip shape as the enumeration bug fixed in #3404.

STABLE_BRANCH is derived from the stable version, so every rc promoted to
that version shares it. Both re-dispatch paths this workflow documents as
supported are therefore wedged once any promote PR for that version has
been closed: re-dispatching the same rc, and promoting a newer rc to the
same version. Neither recovers by reopening the old PR, because the
step's own force-push makes its head unreachable and GitHub refuses with
"state cannot be changed. The <branch> branch was force-pushed or
recreated."

Hit live while promoting v1.6.0-rc.4 after #3397 had been closed: the run
went green and opened no PR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-22 17:18:00 +05:00
myasnikovdaniil d0c0f399f8 fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404)
## The bug

Promotion rewrites the rc version substring to the stable version across
the vendored image references. That rewrite lived inline in
`promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus
`packages/apps/kubernetes/images/*.tag` **only**, on the premise that
the kubernetes app was the only one whose `.tag` files carry the
cozystack version. Nine other `.tag` files do, and `system/multus` seds
its reference straight into a vendored upstream daemonset manifest.

#3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33
of 55 first-party references still reading `v1.6.0-rc.4`**.

`hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind
spot from the other direction — both scanned the depth-2 `values.yaml`
alone, so those images were never retagged to the stable version (**30
references selected before, 42 after**) and never mirrored to the public
registry for a nightly. A retag miss leaves an image short of a tag,
since the digest still resolves inside one registry. A mirror miss is
worse: the host rewrite walks the same file list, so a reference that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

`v1.6.0` is the first release to go through promotion — every earlier
release was a full rebuild from the tag, which stamped every file
uniformly. The glob has always been wrong; the old path just made it
unobservable.

## Why it could not be caught before

The rewrite was workflow-inline, so there was nothing to unit test. The
only way to observe the miss was to cut a release.

It is a script now — `hack/promote-rewrite-tags.sh` — and
`hack/promote-rewrite-tags_test.bats` round-trips it against the real
tree under `make unit-tests`, with no cluster, registry or release
involved.

## The fix

The three call sites each carried their own idea of where a reference
can live, which is how they drifted apart. That knowledge moves into
`hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()`
for the files, `collect_image_refs()` for the references inside them —
and all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite also asserts its own postcondition: it scans **wider** than
it rewrites and fails the promotion when a reference survives in a
location the enumeration does not know about, rather than shipping it
quietly. The wide scan is shape-filtered to image-reference position,
because `X.Y.Z-rc.N` is an ordinary version string that other things
legitimately contain — `kubevirt-csi-driver/go.sum` pins
`github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a
cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:`
instead would have been wrong: `system/cilium/values.yaml` keeps `tag:`
and `digest:` under separate keys.

`docs/agents/image-refs.md` records the contract — three tag classes,
three storage shapes, the invariants, and which consumer does what.

## Not a bug: kamaji

`cluster-api-control-plane-provider-kamaji` stays at
`v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild
of an upstream component, versioned by that component rather than by the
cozystack version line, so no rewrite is owed. A test pins this so a
future fix here cannot over-reach.

## Documented known gaps, deliberately not fixed here

Both are pre-existing and in the **host** dimension; the version
dimension this PR owns is complete, because the version always lives in
a `tag` key regardless of where the host sits.

- **The nightly host rewrite cannot reach a split host.** It is a
literal `<src-registry>/` substring replace, so `keycloak-operator`
(host in a sibling `registry:` key) and `kubeovn`
(`global.registry.address`, no trailing slash) are mirrored but keep
pointing at the build registry in the published tree. Fixing it needs
structure-aware rewriting.
- **A reference inside a gzip is invisible.**
`capi-providers-cpprovider` ships `files/components.gz`, and the promote
postcondition cannot catch this class at all since `grep -rIl` skips
binaries. Nothing is owed today — kamaji is component-versioned and the
image is still mirrored and retagged via its `.tag` — so the residual is
that a nightly's kamaji ConfigMap keeps the build-registry host.

## Testing

- 17 tests in the new suite; `make bats-unit-tests` green (312 tests,
exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow.
- Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`,
`1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no
false negatives.
- Every test is mutation-proven. Reverting the enumeration to the old
glob, deleting any of the five YAML shapes, corrupting emitted digests,
reverting extras to parse-only, dropping the unreadable-file guard,
broadening shape 4 beyond `global.images`, or planting an unenumerated
first-party reference in any storage shape each fail a named test.

## Reviewer note, out of scope

`platform-migrations` is pinned at two different digests —
`core/platform/values.yaml` at `v1.5.0` and
`backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned
pin; no Makefile writes that key, tracked in #3143). A
`promote-retag.sh` dry-run emits **two copies to the same destination
tag**, so the write-once guard will fail the promotion. This is
independent of this PR and needs a decision about the backup-client
image version, so it is left alone here — but it blocks the 1.6.0
promote and wants fixing before the re-cut.

## Release note

```release-note
Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry.
```

Fixes the promote half of #3143.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Image promotion and nightly mirroring now detect digest-pinned image
references across values files, standalone tag files, and supported
templates.
- Stable release preparation supports RC→stable tag rewriting via a
dedicated promotion script, with broader verification to ensure no RC
references remain.

- **Bug Fixes**
- Improved completeness of ref discovery and scanning, including
previously missed template/tag locations.

- **Documentation**
- Added and linked guidance documenting image-reference invariants and
how tooling enumerates supported storage shapes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-22 16:41:17 +05:00
Myasnikov Daniil 09323fff13 fix(ci): cover every image-ref storage shape in promote, retag and mirror
Promotion rewrote the rc version substring across the depth-2 package
values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the
premise that the kubernetes app was the only one whose .tag files carry
the cozystack version. Nine other .tag files do, and system/multus seds
its ref straight into a vendored upstream manifest. v1.6.0 was therefore
staged with 33 of 55 first-party refs still reading v1.6.0-rc.4.

promote-retag.sh and nightly-mirror.sh had the same blind spot from the
other direction: both scanned only the depth-2 values.yaml, so those
images were never retagged to the stable version (30 refs selected
before, 42 after) and never mirrored to the public registry for a
nightly. A retag miss merely leaves an image short of a tag, since the
digest still resolves inside one registry; a mirror miss is worse,
because the host rewrite walks the same file list and a ref that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

The three call sites each carried their own idea of where a ref can
live, which is how they drifted apart. Move that knowledge into
hack/lib/image-refs.sh as the single enumeration -- image_ref_files()
for the files, collect_image_refs() for the refs inside them -- and have
all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite itself moves out of promote-rc.yaml into
hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only
observable by cutting a release: there was nothing to unit test. It now
also asserts its own postcondition, scanning wider than it rewrites and
failing the promotion when any ref survives in a location the
enumeration does not know about, rather than shipping it quietly.

Component-versioned images (kamaji's v0.19.0-cozystack.N) and
third-party pass-through images do not ride the cozystack version line
and are deliberately left alone; a test pins that so a future fix here
cannot over-reach. capi-providers-cpprovider is a documented known gap:
its ref is also embedded in a gzipped copy that the chart actually
ships, and rewriting only the readable copy would diverge the two.

docs/agents/image-refs.md records the contract -- three tag classes,
three storage shapes, the invariants, and which consumer does what.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 20:14:24 +05:00
Myasnikov Daniil 792951b9c6 fix(ci): make workflow cancellation effective
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-21 17:57:48 +05:00
myasnikovdaniil 8020656725 fix(ci): resolve nightly packages digest without image-specific skopeo ops (#3332)
## What this PR does

The Nightly workflow has **never completed a run**. Every scheduled run
since `NIGHTLY_ENABLED` was set has failed in `mirror` at "Resolve
source commit and version", ~80s in, with:

```
level=fatal msg="unsupported image-specific operation on artifact with type \"application/vnd.cncf.flux.config.v1+json\""
```

Nothing has ever been published — `cozy-installer:nightly` and
`cozystack-packages:nightly` both 404 on GHCR, and `cozystack-nocloud`
does not exist.

This fixes that failure and the next two behind it.

### 1. The digest lookup could never have worked

`cozystack-packages` is a **Flux OCI artifact** (config mediaType
`application/vnd.cncf.flux.config.v1+json`), not a container image. A
plain `skopeo inspect` parses the config blob as an image config and
fatals on it, so `skopeo inspect --format '{{.Digest}}'` was never
viable against this ref — the `--raw` call on the very next line already
acknowledged this.

Replaced with `skopeo inspect --raw` + `skopeo manifest-digest`, which
hashes the manifest bytes — exactly what the registry addresses the
artifact by, with no image-specific parsing.

This also collapses two registry fetches into one: the digest and the
`org.opencontainers.image.revision` annotation now come from the **same
bytes**, which strengthens the race the original comment describes —
there is no longer any window between pinning the tag and reading it.

### 2. `flux pull artifact` rejects a non-existent output path (×2)

`flux pull artifact --output _baked` fails with `invalid output path
"_baked": no such file or directory`. `_baked` is neither tracked nor
created by anything. `pull-requests.yaml` does `rm -rf` + `mkdir -p`
first; `nightly.yaml` omitted it. Since the workflow always died at step
1, this was never reached.

Same bug for `_rw` in the e2e staging step. Both now create the
directory first.

## Verification

- Reproduced the original fatal against the live artifact with skopeo
1.22.2.
- Ran the fixed resolve step end-to-end against the real registry:
returns the digest the registry reports, and a `sha` that matches the
current `main` HEAD.
- Reproduced the `_baked` failure and confirmed the fix extracts the
tree.
- `actionlint` clean; `zizmor --offline` clean; `shellcheck` clean on
changed lines.

Also checked and **not** changed, each disproven rather than assumed:

- `skopeo copy --multi-arch all` on a Flux artifact — works; `copy` is
mediaType-agnostic.
- The `awk -F@ '/artifact successfully pushed/'` digest capture —
matches current flux CLI output, and is the same idiom
`packages/core/installer/Makefile` already uses in production.
- `nightly-mirror.sh`'s `inspect --format Digest` — all mirrored refs
there are genuine image manifests, so it is correct as written.
- `make -C packages/core/talos talos-nocloud` — exists and is CI-proven
via `tags.yaml`.

## Honest scope

This fixes the `mirror` job's blockers. It does **not** promise a green
nightly on the first run: every step after `mirror` is still unproven,
having never executed. The most likely next wall is `build-disk` —
`cozystack-nocloud` is a brand-new package, so the first `oras push`
creates it, and the follow-up `oras tag` can hit `403 permission_denied`
until the package is linked (the risk the workflow already documents
in-line). The first-ever full e2e run may well surface its own issues.

Reviewing that follow-on separately seems better than growing this PR.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved nightly artifact mirroring and staging reliability.
* Prevented artifact pull failures by ensuring required output
directories are created before use.
  * Enhanced retrieval of artifact version and source revision metadata.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-20 16:49:38 +05:00
Aleksei Sviridkin 0f4a2715ba ci(labeler): map cert-manager, external-secrets and metrics-server scopes
PRs scoped to these three components fell through to
area/uncategorized because the scope map had no entry for them.
metrics-server serves the monitoring pipeline; cert-manager and
external-secrets are platform services.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-20 05:29:41 +03:00
Myasnikov Daniil 90254efa6d fix(ci): resolve nightly packages digest without image-specific skopeo ops
The Nightly workflow has never completed a run. Every scheduled run since
NIGHTLY_ENABLED was set has failed in `mirror` at "Resolve source commit
and version", ~80s in:

  level=fatal msg="unsupported image-specific operation on artifact with
  type \"application/vnd.cncf.flux.config.v1+json\""

cozystack-packages is a Flux OCI artifact, not a container image: its
config mediaType is application/vnd.cncf.flux.config.v1+json. Plain
`skopeo inspect` parses the config blob as an image config and fatals on
any non-image artifact, so resolving the digest with
`inspect --format '{{.Digest}}'` could never work.

Fetch the :main manifest once with `inspect --raw` and derive both the
digest (`skopeo manifest-digest`, which just hashes the manifest bytes —
no image-specific parsing) and the revision annotation from those exact
bytes. This also tightens the race the original comment describes: the
digest and the annotation now come from a single fetch, so there is no
window at all between pinning the tag and reading its revision, and one
registry round-trip is saved.

Because the workflow never ran past this step, the rest was unproven.
Also fix both `flux pull artifact` calls, which passed an --output
directory that does not exist and would have failed with "invalid output
path": create it first, matching the established idiom in
pull-requests.yaml ("Pull current-main packages tree").

Verified against the live OCIR artifact: the fixed step resolves
packages_ref to the same digest the registry reports and yields main's
current HEAD sha; the fixed pull extracts the tree. skopeo copy on Flux
artifacts, the flux push digest awk parse, the cozy-installer chart name
and nightly-mirror.sh's per-image digest verification were each checked
and are correct as-is.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-16 22:40:05 +05:00
Aleksei Sviridkin 90342063c0 chore(github): add area/keycloak, area/tenant, area/kamaji labels
Three scopes recur across PR history with no fitting area/* label
(keycloak, tenant, kamaji). Add the labels and map their scopes in the
auto-labeler, and fill two gaps in existing mappings (mongodb ->
area/database, csi -> area/storage) so these stop landing in
area/uncategorized.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-16 20:10:20 +03:00
myasnikovdaniil a269eb3965 chore(deps): update actions/checkout action to v6.0.3 (#3133)
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [actions/checkout](https://redirect.github.com/actions/checkout) |
action | patch | `v6.0.2` → `v6.0.3` |

---

### Release Notes

<details>
<summary>actions/checkout (actions/checkout)</summary>

###
[`v6.0.3`](https://redirect.github.com/actions/checkout/blob/HEAD/CHANGELOG.md#v603)

[Compare
Source](https://redirect.github.com/actions/checkout/compare/v6.0.2...v6.0.3)

- Fix checkout init for SHA-256 repositories by
[@&#8203;yaananth](https://redirect.github.com/yaananth) in
[#&#8203;2439](https://redirect.github.com/actions/checkout/pull/2439)
- fix: expand merge commit SHA regex and add SHA-256 test cases by
[@&#8203;yaananth](https://redirect.github.com/yaananth) in
[#&#8203;2414](https://redirect.github.com/actions/checkout/pull/2414)

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/cozystack/cozystack).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0ZWQiLCJkby1ub3QtbWVyZ2UvaG9sZCJdfQ==-->
2026-07-15 14:38:36 +05:00