Finalize sets the release body after the stable tag, the
api/apps/v1alpha1 tag and the release-X.Y fast-forward already exist,
so a body GitHub refuses fails the step with the release still a
draft, and the image retag and chart publish behind it never run.
GitHub's web UI cut the v1.7.0-rc.2 changelog at about 125000
characters, and the v1.7.0 changelog is 259,026.
A release can now carry a short hand-written docs/releasenotes/<tag>.md
that becomes its body, while the changelog stays the full record. Both
finalize and update-releasenotes take the release notes when they have
content and the changelog otherwise, and hold the body to 100000
bytes. Bytes are never fewer than characters, and the margin makes
GitHub's exact limit, and its answer to an oversized body, irrelevant.
Over the limit, finalize publishes a one-line link to the changelog on
the tag instead, and the sync leaves the published body alone: the
tags.yaml backstop brings a changelog to main without its release
notes, and the sync would otherwise replace them.
A release without release notes whose changelog is under the limit
still gets its changelog byte for byte.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every flux install fetched fluxcd.io/install.sh and ran it as root, so
whatever that host served at the moment executed on runners that later
hold registry credentials. Use the upstream setup action instead, pinned
by commit and given an explicit version: it checks the release tarball
against that release's checksums file and runs no script fetched at run
time.
nightly and the fork e2e lane still took yq from the floating latest
release. They now use the pinned, checksum-verified install the
promotion steps already have.
The helm-unittest plugin embeds its own Helm and was installed at
whatever version was newest, so a plugin release could change chart
test results on a PR that touched no chart. Pin it, and replace a
different version a persistent runner may carry.
A contract test scans every workflow for these three tools, so an
unpinned flux, yq or helm-unittest install copied from an older step
fails the unit lane.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three claims that were stronger than the code, plus one legibility fix.
The PR gate's comment said a `packages/` edit "on the PR or its base"
fails the status before merge. The base half is not true: GitHub fires
no `pull_request` event when the base branch advances, it only
recomputes refs/pull/N/merge, so a change landing on the base after this
went green is never re-verified and the PR merges on the stale result.
Checking the merge tree does make base drift visible WHEN the job runs,
which is worth keeping — but finalize is what actually catches the drift
case, after the merge and before any stable name. Say both, in the
workflow and in the reviewer checklist, so nobody reads the green as
covering more than it does.
"Transactional at every consumer-visible name" is a judgement call
dressed as an absolute, inherited from the "this dispatch performs NO
registry mutation" it replaced. The dispatch now does write to the
registry: the `promotion-*` candidate is a real, publicly listable tag.
What it creates is no *stable* name — no vX.Y.Z, no :latest, no stable
installer. Stated that way the sentence stops needing the reader to
already know which writes were meant.
The retention paragraph listed two of the conditions the workflow
enforces. It now lists all of them, including the ones the workflow
gained here: same-repository, bot-authored, `release-X.Y.Z` branch, the
`release` label, exactly one valid digest pin, and protection from a
base-branch tip as well as an open PR.
Also: `flux pull artifact` reads its credentials from
~/.docker/config.json, which is why finalize now runs `docker login` —
but its runner class is not one of the docker-using build ones and the
setup step installs everything except docker. Check for it there, where
the message can name what is missing, instead of failing with "command
not found" halfway down the release path. And the verifier's exclusion
list is a restatement of flux's `excludeOCI`, not a derivation — the CLI
does not expose it — so the coupling and the pinned version are now
written down next to the list.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three review findings on the new promotion steps.
`yq` was fetched from `releases/latest/download` in all three steps this
pipeline installs it in — the promote dispatch, the promote PR gate and
finalize. That makes an unpinned moving dependency part of what decides
whether a stable release may be created, and part of what rewrites the
installer pin: two promotions of the same rc can be served different
binaries. Pin it the way FLUX_VERSION already is, version-gated so a
matching preinstalled yq is reused, on the version the e2e sandbox image
already tracks.
A version authenticates nothing about the bytes it names, though, and
these are the workflows that mint write-once stable names. yq is
therefore also pinned BY CONTENT: its SHA-256 is committed beside its
version and verified before install.
Scoped to yq deliberately. It is the one downloaded binary with write
authority over what a release contains — it stamps the candidate digest
into installer values at promote time and platformVersion into the
stable chart at finalize. skopeo, helm and oras only move bytes that are
already digest-addressed, so a substituted one is caught downstream; a
substituted yq writes the wrong pin and every later check verifies that
pin faithfully.
The Flux installer is left as it is, on purpose. It fetches
flux_${FLUX_VERSION}_checksums.txt from the pinned release and verifies
the binary against it, so the binary is not the exposure — the ~6 KB
bootstrap script from fluxcd.io is, and it is mutable by design and
fetched identically in ten other workflows here. Pinning its bytes in
these three files would break on the next regeneration of a script the
project does not control while leaving the same fetch unpinned
everywhere else. The comment names the residual risk instead.
No precedent existed either way: nothing under .github/workflows/ or
hack/ verified a downloaded tool before this, and every version-pinned
download in the tree (oras 1.2.0, cozyvalues-gen 1.6.0) is unverified.
This is a new convention rather than an application of one, confined to
the release path and justified by write authority, and the comment says
so where the next reader will be standing. The three unpinned fetches
outside this pipeline (nightly.yaml, e2e-fork.yaml) are left alone.
The checksum is trust-on-first-use, which is the honest description: it
fixes the bytes to what was checked when the version was chosen and
makes any later substitution loud. Provenance is the release's own
published checksums list cross-checked against the downloaded asset;
both agree on
fa52a4e758c63d38299163fbdd1edfb4c4963247918bf9c1c5d31d84789eded4, and
that binary self-reports v4.53.3.
Verified locally against the real asset: the version gate skips a
matching preinstalled yq and re-installs a mismatched one, the checksum
line reports OK for the genuine binary and exits 1 for a tampered copy,
which aborts the step under the runner's `bash -e`.
`verify-release-candidate` had no `timeout-minutes`, so it inherited the
6-hour default while `e2e-report` waits on it through `needs` — a
stalled `flux pull artifact` would hold the required `E2E Tests` status
pending for the whole of it. 30 minutes, matching `checks` and
`finalize`.
The contract pins that all three install steps carry both versions and
the checksum, run the checksum check, name no `latest/download`, and
agree on one yq version and one checksum. So the unpinned one-line idiom
cannot come back into a release step by copy-paste from the workflows
that still use it, and bumping the version in one file without its
checksum cannot pass. Both mutations are caught.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Serialize the tag-rewritten packages tree during RC promotion and pin its immutable digest in the installer. Verify the candidate against the prospective merge and the original RC artifact before creating stable names.
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The comment above the top-level `permissions` block claimed that jobs
request the minimum extra scopes on top of it. A job-level block does
not add to the workflow-level one: it replaces it, and every scope the
job does not name is set to none. A job needing one extra scope has to
restate `contents: read` alongside it or lose read access to the
repository.
The corrected wording states the replace semantics, so the next job
block written against this comment grants the whole set it needs
rather than only the delta.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three comments still described the changelog reaching main by default,
with a release-X.Y-targeted promote PR as a patch-release corner case.
After the freeze that is inverted: promote-rc.yaml bases the promote PR
on release-X.Y whenever the branch exists, and cutting vX.Y.0-rc.1
creates it, so every release merges its changelog onto the maintenance
line and none reach main on the way. The tags.yaml backstop is now the
normal route, and its self-skip on `exists` is the exception.
These comments are the first thing anyone debugging a missing changelog
reads, so pointing at the wrong model of which path is normal costs real
time. Reworded in place: the generate-changelog job header, the
three-paths comment above check_changelog, its inline at_tag note,
finalize's explanation of why update-releasenotes.yaml is not trusted
with the release body, and the contract test that pins the porting
behaviour. Prose only, no behaviour change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
pull-requests-release.yaml's finalize job checked out the repo with the default
persist-credentials, leaving GITHUB_TOKEN embedded as http.extraheader. Every
later tag/branch push re-injects the CI app token via `git remote set-url`, but a
persisted extraheader silently wins over the URL credential — so those pushes
authenticated as GITHUB_TOKEN. A GITHUB_TOKEN-authenticated push creates no
workflow run (GitHub anti-recursion), which is exactly why v1.6.0's stable tag
never triggered tags.yaml: the generate-changelog and update-website-docs
backstops never ran.
Set persist-credentials: false on the finalize checkout, so the set-url app token
actually authenticates the pushes — the same handling promote-rc.yaml already
uses. Audited every git/gh step in the job: both tag steps set-url the app token
before pushing, and Ensure-maintenance-branch / Publish-draft-release / the
retag+chart steps use the app token via github-script or a separate registry
login — none depends on the persisted GITHUB_TOKEN extraheader.
Pin it in hack/promote-gate-contract.bats.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Dual-model review (Claude + Codex) of the previous two commits found seven
substantive defects, two of them destructive. Both models independently
reached three of them; the split is noted per item.
Descope the website half (Claude, verified by reproduction). The docs job
was moved to promote time, but the website Makefile pins the fetch ref
with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line
BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that
cannot exist at promote time, since promote's own validate step hard-fails
if it does. update_apps.sh logs the failed fetch and exits 0 AFTER
truncating each destination file, so every app doc would become a stub,
`changed=true` would open the PR, and the new auto-merge would land it
unreviewed. Not fixable from this repo. update-website-docs returns to
tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize
auto-merge is dropped. Opening it earlier needs a cozystack/website change
first and is left as a follow-up. This also removes the unordered-sibling
race Codex found, where the release PR could merge before the docs job
opened its PR, leaving finalize nothing to merge and no retry.
Fix the non-blocking guarantee (both models). Only the AI step carried
continue-on-error; the changelog JOB did not, so a flaky `npm i -g
@github/copilot` failed it and `needs` skipped open-pr entirely — staging
branch pushed, release drafted, no promote PR. Gate open-pr on promote
alone.
Generate from a checkout the contract permits (both models). The job
checked out the rc staging branch, which docs/agents/changelog.md §2
rejects — a compliant agent stops and writes nothing, so the normal path
produced no changelog at all. Check out the rc tag detached and document
that configuration in §2. The checkout ref now comes from the validated
parse output rather than raw dispatch input.
Port, don't regenerate, in the backstop (both models). For a patch release
promoted onto release-X.Y the changelog never reaches main, so the
backstop fired and ran a second AI pass; once its PR merged,
update-releasenotes.yaml overwrote the published body with unreviewed
text. Detect the file at the tag and port it verbatim; the AI runs only
when it is absent from both main and the tag.
Survive re-dispatch (Codex). promote force-pushes the staging branch,
erasing a hand-written changelog — the documented recovery when generation
fails. Preserve an existing one across the rebuild, and stop open-pr
overwriting it with generated output.
Validate structurally, not by size (Codex). A truncated or whitespace-only
AI run passed `[ -s ]`, was committed, advertised as valid in the PR body,
and published verbatim as release notes. Assert the header, the compare
link and a plausible line count; finalize additionally refuses
whitespace-only content rather than blanking the notes.
Add hack/release-changelog-contract.bats — nine tests pinning each link in
the chain, mutation-checked: removing open-pr's if:, breaking finalize's
body read, and restoring paths-ignore each fail the suite.
Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which
described the old flow throughout.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The promote PR now carries docs/changelogs/vX.Y.Z.md, and Prepare stable
branch explicitly tolerates producing no tag-string changes when the
digests are already stable. Together those can yield a docs-only promote
PR, which `paths-ignore: docs/**/*` would filter out — finalize never
runs, so the stable tag is never cut and the release is never published,
with no error raised anywhere.
The `if:` gate (merged + `release` label + release-bot author) is what
actually decides whether this workflow does anything, so let it do the
filtering. The cost is a skipped job on unrelated closed PRs.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Changelog generation and the website-docs PR both lived in tags.yaml,
gated on the stable tag push. Under the promote flow that tag is created
by pull-requests-release.yaml finalize, which also publishes the release
— so both jobs could only ever run once the release was already public.
A reader hitting the release in that window saw an empty body, and the
site lagged until someone merged the docs PR.
Move both to promote time:
- promote-rc.yaml gains a `changelog` job that runs in parallel with
`promote` and hands the file to a new `open-pr` join job, which commits
it onto the release-X.Y.Z branch so it rides the promote PR. Generation
is AI-driven and slow while `promote` spends its wall clock on the Talos
assets; neither needs the other, so both start at once.
- The website-docs job moves to promote-rc.yaml too, depending only on
`promote`. Different repo, so it cannot ride the PR — it is opened early
and finalize merges it after the release publishes.
- finalize reads docs/changelogs/vX.Y.Z.md from the merge commit and uses
it as the published release body. update-releasenotes.yaml is no longer
on the critical path: it triggers on the same push and would race this
job, and if it won it would find no published release, skip, and never
fire again. It also only watches main, so a patch release promoted onto
release-X.Y was never synced at all.
- tags.yaml keeps generate-changelog as a backstop only. It self-skips
when the changelog is already on main, and covers the two cases where
promotion does not put it there: generation failed (non-blocking), or
the promote PR targeted a maintenance line.
Generation stays non-blocking, so a flaky AI step cannot wedge a
promotion. When it fails the promote PR body says so explicitly, and the
escape hatch is to commit docs/changelogs/vX.Y.Z.md to the branch — which
also makes a hand-written changelog first-class: if the file is already
in the rc tree, generation is skipped entirely.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promotion mutated the registry at dispatch time — retagging ~37 images to
vX.Y.Z (+:latest), publishing the stable cozy-installer chart — before the
promote PR's e2e ran and before any merge. An abandoned promotion then left
stable-named bytes for a version that never shipped, and the (correct) image
write-once check wedged any later re-promotion to the same version.
Move every irreversible registry side effect into pull-requests-release.yaml
finalize, which runs only after the promote PR merged (= after its e2e passed),
making promotion atomic. promote-rc.yaml now only stages the release-X.Y.Z
branch, drafts the stable release, and opens the PR; an abandoned promotion
leaves nothing but an overwritable draft + branch.
Folded into the same restructure:
- Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag in finalize, at the stable
tag's commit. It was gated in tags.yaml on `release_exists==false &&
is_stable`, unsatisfiable under promotion (the draft always pre-exists), so
promoted stables never got their submodule tag — a regression for Go
consumers. Finalize is where the stable tag is born, so it cuts the matching
submodule tag write-once (stable only).
- Gate :latest by max-semver. promote-retag.sh now moves :latest only when
MOVE_LATEST=1 (default off); finalize passes it the same make_latest decision
the release gets, so a patch on an older line moves neither the release's
`latest` nor the images' :latest. Added MOVE_LATEST bats coverage.
- Stamp platformVersion into the published stable chart's default values
(uncommitted, at package time) so `helm --install --version X.Y.Z` reports the
stable version in telemetry instead of the rc version baked into the operator
image. Not committed to the branch — that would leak platformVersion=vX.Y.Z
onto main and mis-stamp the next rc.
- promote-rc: serialize all promotions (static concurrency group, not per-rc);
label the PR full-e2e so the full suite runs; scope the rc->stable sed to the
depth-2 values.yaml glob plus the kubernetes .tag files (the build's own write
set); tolerate a leftover draft so re-promotion is not wedged.
- finalize: assert the PR author is cozystack-ci[bot], and run on an ephemeral
oracle-vm runner (needs skopeo/helm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on
.github/workflows/pull-requests-release.yaml: the fast-forward catch block
treated every updateRef failure as a recoverable warning, so an auth, API, or
malformed-ref error would silently leave the maintenance branch stale. Warn
only on the 422 non-fast-forward rejection and fail the job otherwise.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replace the force-retag operations in the release flow with write-once /
compare-before-force guards, so published tags are never moved (the cache
poisoning on api/apps/v1alpha1 via the Go module proxy, plus SBOM/scanner
drift, all stem from moved tags).
- tags.yaml: the api/apps/v1alpha1 submodule tag is now write-once (create
if absent, no-op if unchanged, fail loudly if it would move). The
release-X.Y.Z staging branch uses a compare-before-force guard (skip when
unchanged; force+log only when the staging branch genuinely moves — it is
a mutable staging ref, not an artifact). The build is now skipped when a
release (draft OR published) already exists, so a stable tag prepared by
promotion is never rebuilt here.
- pull-requests-release.yaml: the stable tag is created at the PR merge
commit with a write-once guard instead of `git tag -f`/`git push -f` —
the merge commit did not exist before the PR opened, so force is
impossible by construction. The maintenance branch is fast-forward-only
(no `updateRef force:true`); a non-FF update warns and is left for a
maintainer.
- auto-release.yaml: removed. Stable patch tags were auto-bumped nightly via
delete-recreate; stable now appears only through an explicit rc promotion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Raise the OpenSSF Scorecard Token-Permissions and Pinned-Dependencies
checks (both currently 0/10) ahead of CNCF Incubation due diligence:
- Declare a top-level read-only default (permissions: contents: read) in
every workflow, plus an explicit per-job permissions: block that grants
only the writes that job actually needs. Workflows that previously held
write scopes at the workflow level (backport, pr-labeler, pr-size,
stale) now hold them at job level instead.
- Pin every GitHub-owned and third-party action to a full commit SHA with
a version comment. scorecard.yml was already pinned in #2721.
No triggers, job graph, or step logic change; only token scopes and
action refs. Self-hosted release jobs keep the exact write scopes they
already declared.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
When the git tag was deleted and re-created between draft creation
and merge, GitHub orphaned the draft release: tag_name became
"untagged-<hash>" while `name` was preserved. The finalize step
looked up the draft by tag_name and threw `Draft release for <tag>
not found`, blocking the release.
Fall back to matching by `name` only when the candidate is the
GitHub-orphaned form (tag_name matches /^untagged-/), repair the
draft's tag_name via updateRelease, then publish. Hard-fail is
preserved when neither lookup matches.
Same failure recurred on the v1.4.1 finalize run; this commit
prevents recurrence but does not auto-recover the stuck v1.4.2
draft.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Publish draft release step used the workflow's default GITHUB_TOKEN,
which does not return draft releases from listReleases — so the script
failed with "Draft release for vX.Y.Z not found" and every release tag
had to be undrafted manually.
This same bug was fixed once before in 4db55ac5eb ([ci] Add Github
token to fetch draft releases) by adding github-token: GH_PAT to the
listing step. It was lost during the refactor in 66a756b6 that combined
the separate listing/flags/publish steps into a single Publish step,
and not restored when f96b2da1 migrated remaining github-token usages
from GH_PAT to the cozystack-ci App token.
Add github-token: app-token.outputs.token to the merged step so it
shares the same auth as the rest of the workflow.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozystack-ci[bot] noreply email with brackets is rejected by DCO
probot as an invalid email address, causing release PRs to fail DCO
checks.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace the cozystack-bot personal access token (GH_PAT) with a
GitHub App (cozystack-ci) installation token across all CI/CD
workflows. This improves security by using short-lived, scoped
tokens instead of a long-lived PAT tied to a user account.
Changes:
- tags.yaml: use app token in all 3 jobs (prepare-release,
generate-changelog, update-website-docs)
- auto-release.yaml: use app token for daily patch releases
- pull-requests-release.yaml: use app token for release finalization
The cozystack-ci GitHub App (ID: 3297617) is installed org-wide
with contents:write and pull-requests:write permissions. Secrets
COZYSTACK_CI_APP_ID and COZYSTACK_CI_PRIVATE_KEY are set at org
level.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace unreliable getLatestRelease() API with semver-based max version
detection. After publishing a backport release, explicitly restore the
latest flag on the highest semver release to handle cases where GitHub
API ignores make_latest: 'false'.
Also remove dead code (unused steps) from tags.yaml workflow.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Skip long workflows on PRs that only change files inside `./docs` directory.
- Not applicable to other docs in this repository, such as `packages/apps/**/*.md`,
as they're part of the build.
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Expanded support for prerelease tags to include "alpha" and "beta"
suffixes (e.g., `-alpha.1`, `-beta.2`) in addition to "rc".
- **Style**
- Improved formatting and consistency in comments and log messages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This change also fixes `finalizing release` workflow
https://github.com/cozystack/cozystack/pull/890#issuecomment-2830525103
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated release tag validation to require a dot between "rc" and the
number (e.g., `v0.31.5-rc.1` instead of `v0.31.5-rc1`).
- Adjusted error messages to reflect the new release tag format.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Improved reliability of GitHub Actions workflows by ensuring only one
job per pull request or branch runs at a time. If a new workflow run is
triggered, any previous in-progress runs for the same group will be
automatically canceled, preventing overlapping executions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Myasnikov Daniil