Files
cozystack/docs/release/ci-pipeline.md
T
Myasnikov Daniil 35a4f7c5d9 docs(release): split the release document into one page per topic
docs/release.md had grown into one page that mixed the release model,
the CI pipeline, changelogs, signatures, nightlies and the fires seen
at release time. A reader looking for one of them read all of it, and
an edit to one part reflowed the rest. It now indexes one page per
topic under docs/release/.

The pages already describe how fixes reach a release line through the
backport batch, which the following commits add. Workflows, scripts
and tests that named docs/release.md now point at the page that holds
the text they mean.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 15:03:06 +00:00

27 KiB

What CI does during the release process

The numbered process of a regular release is implemented by seven workflows, including the reusable/manual e2e-tag.yaml, plus the reusable changelog-generate.yaml that two of them share to produce the changelog. Knowing which job does what makes the failure modes much easier to diagnose.

  1. cut-prerelease.yaml — manual dispatch (from main or release-X.Y): the sole entry point for creating a pre-release tag. Validates the -alpha/-beta/-rc tag, then pushes it (write-once, at the branch tip) as the CI app so tags.yaml fires. Cutting the first vX.Y.0-rc.N also creates release-X.Y, freezing the line; afterwards a vX.Y.* dispatch from main is refused. Stable tags are never created here.
  2. tags.yaml — fires on any pre-release tag push: build-amd64 and build-arm64 build the pre-release side by side, then prepare-release publishes it and pushes its digest-pinned staging branch, then — only for -rc.N tags — the mandatory rc-e2e job calls e2e-tag.yaml in the same run (-alpha.N / -beta.N skip it and are validated on demand with e2e-tag.yaml's manual button). generate-changelog and update-website-docs also live here but are backstops only — both are normally produced at promotion time (see 4); on a stable tag push each self-skips or no-ops when the promote flow already did the work. These stable-tag backstops only fire because finalize pushes the stable tag with the CI app token (a GITHUB_TOKEN push triggers no run — see 5).
  3. e2e-tag.yaml — reusable from tags.yaml::rc-e2e and manually dispatchable for a published tag. It checks that the tag's release carries its nocloud disk, resolves the digest-pinned tree and runs the full install, OpenAPI, and Chainsaw suite on Talos containers. A tag cut before the container lane existed has to be dispatched from its own ref. Its dispatch run title is the constant workflow name, so the promote gate correlates a manual run through the successful tag-bearing E2E job name rather than the title.
  4. promote-rc.yaml — manual dispatch, from main only (the signature over a stable release's assets names that ref, see Release signatures and SBOM). Five jobs: parse validates the rc tag and requires green full-E2E evidence before any staging; promote rewrites and commits the rc tags, publishes that packages tree as a run-specific temporary reproducible candidate, commits its digest pin in the release-X.Y.Z tree, and drafts the stable release; changelog calls the reusable changelog-generate.yaml in parallel to produce docs/changelogs/vX.Y.Z.md (regenerating it, or copying an rc-time changelog already committed to the release-X.Y.Z-rc.N staging branch by changelog-rc.yaml — see 7); website-docs (needs parse, promote) opens the cozystack/website docs PR from the staging branch via FETCH_REF (see Phase 3); open-pr joins them, commits the changelog onto the staging branch, and opens the release-X.Y.Z promote PR with the release label but no automatic full-e2e. So the changelog and website docs are reviewed as part of the promotion, the already-checked rc suite is not repeated by default, and the PR body carries ✅ verified or ⚠️ bypassed plus ✅/⚠️ changelog and website-docs lines. Both changelog and website-docs are non-blocking. No stable registry name is created at dispatch.
  5. .github/workflows/pull-requests.yaml verifies a promotion candidate against the prospective merge tree and original rc artifact before the required E2E Tests status can turn green. pull-requests-release.yaml fires after merge and repeats that verification before any stable tag, then cuts the write-once stable + Go-module tags, publishes the release with the merged changelog as its body, retags the candidate and rc images to stable by digest (:latest gated on newest-stable), and publishes the stable chart.
  6. update-releasenotes.yaml — fires on pushes to main that touch docs/changelogs/v*.md; syncs that content into the corresponding GitHub Release body. No longer the primary path (finalize sets the body directly, which is race-free and works for maintenance lines too) — this now covers later edits to a published changelog and manual re-syncs via workflow_dispatch.
  7. changelog-rc.yaml — manual dispatch, optional: generate the changelog at rc time instead of waiting for promotion, while there is still time to review it. It calls the same reusable changelog-generate.yaml and commits the result to the release-X.Y.Z-rc.N staging branch as docs/changelogs/vX.Y.Z.md; the promote-time changelog job then copies that file instead of running the AI again. Skipping this workflow changes nothing — promote generates the changelog itself, exactly as before.

Phase 1 — build-amd64, build-arm64 and prepare-release (hard gate)

On an rc tag push: builds images for both architectures, commits digest pins, creates the draft release, and pushes the release-X.Y.Z-rc.N staging branch. build-amd64 holds the release-exists no-op and the rejection of a hand-pushed stable tag, and builds the amd64 images and the release assets (its make build also pushes the packages artifact and the installer chart, pinned on amd64 digests); prepare-release stitches, gates, republishes those two from the repinned tree, and publishes the release. (The stable promote PR is opened later by promote-rc.yaml, not here. When the stable tag itself is pushed by finalize, a draft already exists, so this job is a no-op — see Phase 5.)

The commit (Prepare release vX.Y.Z, authored by cozystack-ci[bot]) is digest pins and image tags only:

  • packages/core/{installer,platform,testing}/values.yaml
  • packages/system/*/values.yaml (cozystack-api, dashboard, kamaji, linstor, kubevirt-csi-node, etc.)
  • packages/apps/kubernetes/images/{kubevirt-csi-driver,ubuntu-container-disk-*}.tag
  • packages/system/dashboard/templates/configmap.yaml (the $tenantText value)

Things that look surprising in the diff but are normal:

  • Zero-diff packages: when buildx fully caches and upstream base images haven't moved, the digest is byte-identical to the previous release and won't appear. Verify the build step actually ran — don't assume "no diff = nothing built."
  • Sudden churn on ubuntu-container-disk-* tags: cloud-images.ubuntu.com/noble/current/ is a moving target; these often rebuild even without code changes.
  • Switched registry: if a self-built image moved from docker.io to ghcr.io, the registry portion of the digest changes — that's a deliberate move, not a regression.
  • targetVersion is NOT touched by the release PR. Platform migration targetVersion is bumped earlier in a feature commit (e.g. [platform] Bump migration targetVersion to 39 for migration 38). The release PR only re-pins the platform-migrations image digest.

Multi-arch images

Every pre-release is published for amd64 and arm64. tags.yaml::build-arm64 builds the images on a native arm64 runner and pushes each as <tag>-arm64, while build-amd64 builds amd64, the Talos installer and matchbox for both arches, and the assets on x86 at the same time. build-amd64 hands its stamped tree, the log of the tags it pushed and the assets to prepare-release as workflow artifacts; a missing one fails the upload or the download rather than publishing a partial release. prepare-release waits for both builds and runs only if both succeeded: when either fails, the rc is not prepared at all, so a release is never published amd64 only. It applies the stamped tree and runs hack/stitch-multiarch.sh, which joins each image with its arm64 twin into one index, moves onto it every tag the build pushed for that image (the rc tag and, under PUBLISH_VERSIONED=1, its component version, which the build records in PUSHED_TAGS_LOG), and repins the tree on the index digests, then hack/verify-multiarch.sh, then republishes the packages artifact and the installer chart from the repinned tree. A stable tag skips build-arm64 and builds nothing: promotion copies the rc's indexes with every platform, so the stable release is exactly as multi-arch as its rc. Because the two builds run in parallel, an rc that is not prepared (its arm64 build failed, or prepare-release failed before the stitch) still leaves behind what the amd64 build pushed: amd64-only images under :<rc> and under the component-version tags, and the cozystack-packages:<rc> artifact and the installer chart, both pinned on those amd64 digests. A component tag an earlier rc had moved to an index then points at an amd64-only image again, until a run that succeeds stitches it, a re-run of the same run included.

verify-multiarch.sh is a hard gate. It resolves every ref the tree pins, third-party images included, and fails naming each one that is not an index carrying both linux/amd64 and linux/arm64, or that it cannot resolve at all. The fix is to make the image multi-arch (build it for both, or pin a multi-arch digest of the same version); an image that cannot exist for arm64 goes in hack/multiarch-allowlist by repository, with the reason on the same line.

That gate sees only what the tree pins by digest. An image a vendored chart renders from its own defaults, or one an operator launches at runtime, is invisible to it. rc-e2e closes most of that gap: before tearing the sandbox down, e2e-tag.yaml reads every image the three Talos nodes pulled from their image stores (hack/collect-images.sh), each by the digest the pull resolved to, and runs verify-multiarch.sh --refs-file on that list with the same allowlist. A pre-release fails on any single-arch image, on any ref it cannot read or that is still rate-limited after the retries, and on a list the collection could not produce; a stable tag's run only warns, since it ships its rc's images. A tag whose tree predates the audit (it has no hack/verify-multiarch.sh) only warns and passes, so an rc cut before the audit existed can still be re-validated. The list and the audit output are in the run's image-list-<tag> artifact. Neither check covers a package that e2e never installs and whose images the tree does not pin by digest: an arm64 break there shows up only on an arm64 cluster.

build-arm64-nightly.yaml builds main for arm64 every day, so an arm64 build break shows up there before the next rc. Its first step prints verify-multiarch.sh --report for main's committed tree: main pins amd64 images, so its first-party refs are all listed and are expected to be, since the rc's stitch turns them into indexes. A third-party ref that an in-tree rebuild stamps over at build time, as the Harbor components and shadowbox are, is listed too and goes away on the rc. The lines that matter are the third-party refs no build replaces and refs that cannot be resolved; each of those fails the next rc until it is fixed or allowlisted. matchbox is listed too and does not matter: build-amd64 rebuilds it as a two-arch index. The stitch names the images it skips in its own summary: an image with no <tag>-arm64 twin, one whose digest is also pinned in a file the rewrite does not reach, and a pair that is not one amd64 and one arm64 image. matchbox (already an index) and the e2e sandbox (never built for arm64, and allowlisted) are skipped on every rc and pass the gate; any other skipped image fails it.

RC E2E — mandatory check before promotion

For an rc, prepare-release must finish before tags.yaml::rc-e2e calls the published-tag lane, so the full suite tests the digest-pinned staging tree that promotion will reuse, and the release it runs against already carries its disk. promote-rc.yaml::parse then checks the latest tags.yaml push run whose head_branch is the rc tag and accepts it only when the reusable E2E job succeeded. GitHub exposes called-workflow job names as E2E Release Candidate / E2E vX.Y.Z-rc.N (full suite), which the gate matches by the exact tag-bearing suffix.

The alternate evidence path scans retained successful workflow_dispatch runs of e2e-tag.yaml and accepts one whose direct E2E job is named E2E vX.Y.Z-rc.N (full suite). The workflow's run title is always E2E Release Tag and therefore cannot correlate the input tag; artifact names also carry the tag but expire independently, so the job name is the strongest available correlator. If Actions retention or manual deletion removes the run/job metadata, or that job is renamed without updating the matcher, the gate fails closed and tells the maintainer to run the E2E Release Tag button again. skip_e2e_gate=true is an emergency override, not alternate evidence: it writes a loud warning and a ⚠️ PR-body line.

Phase 2 — the changelog (generated at rc time or promotion, not here)

The changelog for vX.Y.Z is produced by promote-rc.yaml's changelog job (a thin call to the reusable changelog-generate.yaml) and committed onto the release-X.Y.Z staging branch by open-pr, so it rides the promote PR and is on the base branch the moment the release publishes. Finalize then uses that file verbatim as the GitHub Release body. Generating it here — on the stable tag push — would be too late by construction: that tag is created by finalize, which has already published the release. It can also be produced earlier, at rc time, via the optional changelog-rc.yaml button — which runs the same reusable core and commits the file to the release-X.Y.Z-rc.N staging branch, from which the promote-time changelog job copies it (validated) instead of regenerating.

The changelog job runs in parallel with promote, detached at the rc tag (the "promotion" configuration in §2 of agents/changelog.md), using a separate read-only GitHub App token for the AI step so the model cannot mutate the repo even with --allow-all-tools. Every write — the commit onto the staging branch and the PR — happens in open-pr under a write-scoped token.

Generation is deliberately non-blocking: a Copilot outage or exhausted token quota must never block a release. Only the AI step carries continue-on-error, and open-pr is gated on promote alone (if: !cancelled() && needs.promote.result == 'success') so the promote PR opens regardless. When the changelog is missing the PR body says so explicitly — that ⚠️ line is the signal to act on.

Recovery, in order of preference:

  1. Commit docs/changelogs/vX.Y.Z.md to the release-X.Y.Z branch by hand (follow agents/changelog.md). promote preserves it across a re-dispatch, and open-pr will not overwrite it with generated output.
  2. Re-dispatch promote-rc.yaml — generation runs again; anything already on the branch wins.
  3. Merge without one and let tags.yaml::generate-changelog (the backstop) open a changelog PR against main after the fact. The release is published with the draft's body until that merges.

Known failure modes:

  • Copilot quota exhausted (COPILOT_GITHUB_TOKEN 402). Refill or rotate, then re-dispatch.
  • AI step timeout (hard 30-min cap).
  • Output truncated or whitespace-only — caught by the Verify changelog step (hack/validate-changelog.sh), which accepts either header convention (# Cozystack vX.Y.Z for a minor or # vX.Y.Z (<date>) for a patch), and requires the leading release-link comment pointing at releases/tag/vX.Y.Z, a compare/...vX.Y.Z link ending in this version, and at least one ## section — deliberately with no line-count floor, since a complete short patch changelog (v1.5.1 ships in 19 lines) must not be rejected as a fragment. It downgrades to the missing-changelog path rather than shipping a fragment as release notes.

tags.yaml::generate-changelog is the backstop. It self-skips when the changelog is already on main. It earns its keep in two cases: generation failed during promotion, or the promote PR targeted release-X.Y so the file never reached main. Since the rc freeze the second case is every release, minor and patch alike — the promote PR always targets the release line (see Regular Releases), which makes this backstop the only route by which a published changelog reaches main. In that case it ports the reviewed file from the tag commit verbatim rather than regenerating — a second AI pass there would spend quota and then overwrite already-published, already-reviewed release notes.

Phase 3 — update-website-docs (now the backstop)

The cozystack/website "update managed apps reference" PR is normally opened earlier, at promote time, by promote-rc.yaml's website-docs job — not here. That job fetches the README content from the release-X.Y.Z staging branch promote just pushed (via the website Makefile's FETCH_REF variable), so the docs are ready to review alongside the promote PR while the stable tag is still unborn. It refreshes the trunk version pins from the staging ref before promoting next/ → vX.Y/, so a new minor's docs inherit fresh Talos/cozystack pins instead of a stale next.yaml. It is non-blocking: if it fails (or trips its FETCH_REF-support guard against a website checkout that predates that Makefile variable), the promote PR opens with a ⚠️ website-docs line and this tag-time job backfills the same branch.

Merge-timing contract: the website PR must NOT be merged before the release is published (finalize). Merging it early flips the site's latest-version pointer to a version whose versioned docs and openapi.json are not on GitHub yet, so every version-pin and API link for the new version 404s. Both the website PR body and the promote PR body say so; park the website PR until the GitHub release for vX.Y.Z is live, then merge.

This tag-time update-website-docs job stays as the backstop. It runs with if: !cancelled() && needs.prepare-release.result == 'success' && !contains(github.ref_name, '-') (surviving a failed/skipped changelog phase, stopping on cancel, and skipping every pre-release tag), decides whether to promote next/ → vX.Y/, runs make update-all, stages content hugo.yaml data/versions, and force-pushes the same update-docs-vX.Y.Z branch. On the normal path — where promote-rc.yaml::website-docs already did the work — it no-ops into "No changes to commit". At tag time the stable tag exists, so its BRANCH=RELEASE_TAG fetch resolves without needing FETCH_REF. For it to fire at all, finalize must push the stable tag with the CI app token (see Phase 5) — a tag pushed with the default GITHUB_TOKEN triggers no workflow run.

If anyone changes the website Makefile to write somewhere else (e.g. static/, i18n/), the git add list must grow in both jobs or the PR silently drops files.

Phase 4 — review and merge the promote PR

Reviewer checklist for the chore(release): promote vX.Y.Z-rc.N -> vX.Y.Z PR:

  • The PR body says ✅ RC full e2e was verified; if it says the gate was bypassed, obtain explicit maintainer acceptance or add full-e2e and wait for that opt-in run.
  • Verify release packages candidate is green. It pulls the candidate and original rc artifact by digest, compares the candidate to GitHub's prospective merge tree, and feeds the required E2E Tests status even when full E2E is not repeated. Do not edit packages/ on this PR after generation; if the guard reports drift, re-dispatch promotion to create and pin a new run-specific candidate. Note what a green here does not cover: GitHub fires no pull_request event when the base branch advances, so a packages/ change merged to the base afterwards is never re-verified on the PR. Finalize repeats the check against the real merge commit and catches that case before any stable name exists — but it catches it after the promote PR has already merged, so a packages/ change landing on the base while a promote PR is open is worth holding back.
  • The PR body's website-docs line is present: if ✅, the cozystack/website docs PR is open and parked — do NOT merge it until vX.Y.Z is published (see Phase 3); if ⚠️, the tag-time backstop will backfill it after finalize.
  • Diff is digest pins + image tags only — nothing else.
  • No accidental config drift (a value file you don't recognize).
  • If any chart bumped, check the new digest pulls — crane manifest ghcr.io/cozystack/cozystack/<name>@<digest> should return.
  • Build artifacts on the draft release page exist and match the expected count.
  • The PR includes docs/changelogs/vX.Y.Z.md, it has content, and the entries match the commit range (see Changelog pre-publish verification). If the PR body carries the ⚠️ "No changelog in this PR" line, resolve that before merging — finalize publishes this file as the release body, unless the PR also carries docs/releasenotes/vX.Y.Z.md (see Phase 6).

Phase 5 — pull-requests-release.yaml (finalize)

Fires on merge of a PR that is merged, carries the release label, and is authored by cozystack-ci[bot] (the last guard closes the "name a branch release-X.Y.Z, get it labelled, merge it" hole). Head branch must match release-X.Y.Z[-suffix]. This is where every consumer-visible or write-once side effect of a promotion happens — after the rc's mandatory E2E was checked at dispatch and the PR merged, making promotion transactional at stable names. Steps:

  1. Verify the packages candidate before creating any stable tag. hack/verify-promoted-packages.sh pulls the candidate by the digest committed in installer values, proves it matches the merged packages/ tree after normalizing only the impossible self-reference, rejects surviving rc image tags, then follows the candidate's embedded pre-push platformSourceRef back to the original rc artifact and compares normalized container repo@digest sets. Any package edit after candidate publication or any changed container digest aborts finalize while all stable names are still absent.
  2. Create the tag at the merge commit (write-once). The merge commit of Prepare release vX.Y.Z did not exist before the PR opened, so there is nothing to move — the tag is created here for the first time. A pre-existing tag at a different commit fails the step loudly rather than being force-moved (see Tag immutability). The checkout uses persist-credentials: false so this push authenticates as the CI app (via git remote set-url), not the default GITHUB_TOKEN — a GITHUB_TOKEN push creates no workflow run, so a persisted-credential push would silently skip tags.yaml entirely and its stable-tag backstops (generate-changelog, update-website-docs) would never fire.
  3. Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag (write-once, stable only) at the same commit, so Go consumers of api/apps/v1alpha1 get the release. Moved here from tags.yaml, whose prepare-release body is skipped for a promoted stable (the draft already exists).
  4. Ensure the maintenance branch release-X.Y exists at the tag commit. Since the rc freeze creates it, this is normally a no-op fast-forward — the merge commit is already the branch tip. Still created if missing; updated fast-forward-only — a non-fast-forward update warns and is left for a maintainer rather than being force-updated.
  5. Publish the draft release, with docs/releasenotes/vX.Y.Z.md, or without it docs/changelogs/vX.Y.Z.md, from the merge commit as its body (absent or whitespace-only files warn and leave the draft's body in place, rather than publishing blank notes, and a body over 100000 bytes is published as a link, see Phase 6). make_latest is computed against published-non-prerelease tags: prereleases stay false; tags older than the current max stay false (and the current max is force-restored to latest if necessary, so an older patch tag cut after a newer minor won't downgrade latest).
  6. Retag the verified packages candidate and rc container images to stable by digest (hack/promote-retag.sh, no container rebuild) and publish the stable cozy-installer chart. :latest (on both the images and the chart) moves only when this release's make_latest was true — the same decision as step 5, so the release's latest and the images' :latest never disagree. The chart is packaged with platformVersion stamped into its default values so the documented helm --install --version X.Y.Z path reports the stable version.

Phase 6 — update-releasenotes.yaml (sync GitHub Release body)

Fires on pushes to main that touch docs/changelogs/v*.md. Reads each docs/changelogs/vX.Y.Z.md and PATCHes the matching GitHub Release's body if it differs. Since Phase 5 now sets the body directly from the merge commit, this is no longer on the critical path — it covers later edits to a published changelog, the backstop PR landing a changelog that reached main after the fact, and manual re-syncs via workflow_dispatch. It only watches main, which is why finalize (not this job) is what populates a release promoted onto release-X.Y — since the freeze, that is every release rather than only a patch.

The release body is docs/releasenotes/vX.Y.Z.md when it exists and is not blank, otherwise the changelog, which stays the full record either way, and a push to main touching either file fires this sync. Finalize sets the body after the stable tags exist, where a body GitHub refuses would strand the release, so at publish time a body over 100000 bytes is replaced by a one-line link to docs/changelogs/vX.Y.Z.md on the tag, with a warning in the run's log. This sync never writes that link: it warns and leaves such a release's body as it is. The tags.yaml backstop ports only the changelog to main, so release notes have to be added to main as well, otherwise the sync falls back to the changelog and replaces the release notes with it whenever the changelog fits.