admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-10-27 07:14:04 +09:00
Code Issues Packages Projects Releases Wiki Activity
7,643 Commits 529 Branches 291 Tags
feat/cozystack-controller-gpu-only
Commit Graph
38 Commits
Author SHA1 Message Date
myasnikovdaniil d1d4e92acb feat(release): sign the release assets and attach a source SBOM (#4861) 
## What this PR does

Implements the release-asset part of #4759: a release cut from a tree
that has this change gets an SBOM of its source and one keyless
signature that covers all of its assets.

These are the two CLOMonitor checks still failing for this repository,
`sbom` and `signed_releases`, after #4757 and #4899 brought the project
score to 96.69.

`hack/upload-assets.sh` now collects its asset list into an array and
hands it to the new `hack/sign-release-assets.sh` before the first
upload. The script writes three files into `_out/assets`, and they are
uploaded with the rest:

| File | What it is |
|---|---|
| `cozystack-source-sbom.spdx.json` | SPDX 2.3 SBOM made with syft, over
`git archive HEAD` (the committed release tree, without `_out/`,
`.release-tooling` or runner leftovers) and the unpacked
`cozypkg-*.tar.gz` binaries. It does **not** describe the container
images that `packages/` pins by digest (their SBOMs are #2937, A2 in
#3278) or the Talos ISO, disk, kernel and initramfs images. It is named
by what it covers, so `cozystack-sbom` stays free for an SBOM of the
release as a whole. CLOMonitor's `sbom` check matches `sbom` in an asset
name. |
| `cozystack-checksums.txt` | SHA-256 of every uploaded asset and of the
SBOM. |
| `cozystack-checksums.txt.sigstore.json` | Sigstore bundle of `cosign
sign-blob` over the checksums, keyless, signed with the job's GitHub
OIDC identity. |

Before it returns, the script runs the same `cosign verify-blob` that
`docs/release.md` gives users, with `--certificate-identity-regexp
'^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)'`:
`tags.yaml` on a `v*` tag push, `promote-rc.yaml` dispatched from
`main`. A signature that check would reject therefore fails the job
before anything is uploaded. It also refuses its own three outputs as
inputs: `promote-rc.yaml` downloads every rc asset, and a stable release
has to get freshly computed checksums and a fresh signature, since
promotion restamps several assets.

Workflows: `tags.yaml::prepare-release` and `promote-rc.yaml::promote`,
the two jobs that call `make upload_assets`, get `id-token: write` and
two setup steps:
- syft v1.54.1 from its release tarball, checked against a committed
SHA-256 (`c069905b…`, the release's checksums entry; the checksums file
verifies against anchore/syft's keyless release signature). Not
`anchore/sbom-action/download-syft`: pinned by commit, it still runs
syft's `install.sh` fetched from a tag at run time, in a job that holds
the app token, the GHCR login and `id-token: write`. v1.54.1 is past
GHSA-mw2c-m758-9v5q (fixed in v1.52.0).
`hack/ci-tool-installs-contract.bats` now fails on
`anchore/sbom-action`, on a syft install script, on a syft download
without `sha256sum -c` and on version or checksum drift between the two
workflows.
- `sigstore/cosign-installer` v4.1.2, pinned by SHA, with
`cosign-release: v3.1.3`: the version `docs/release.md` asks users for,
so the job checks its own signature the way they will. The action
verifies that binary against cosign's keyless and KMS bundles with its
digest-pinned bootstrap cosign. `hack/ci-tool-installs-contract.bats`
holds the action commit and the release equal in both workflows and
fails on a step without `cosign-release`.

`docs/release.md` gets a "Release signatures and SBOM" section with the
user-side verification: cosign v3.1.3 or later (v2.6.5 on the v2 line),
the two commands joined with `&&`, what the SBOM covers and what the
certificate identity proves. It now also says, at each place
`promote-rc.yaml` is run, that it is dispatched from `main` only.

### Decisions for the release owners

- **Fail closed.** If Fulcio or Rekor is down, the release job fails and
nothing is uploaded. I chose that so that no release goes out unsigned.
If you would rather publish unsigned than block, the change is a `||
echo ::warning::` around the call in `upload-assets.sh`.
- **Promotion runs the rc tree's script.** `promote-rc.yaml` calls `make
upload_assets` from the rc checkout, not from `.release-tooling`. The
first signed stable release is therefore the promotion of the first rc
tagged after this merges. `release-1.6` gets signatures only if this is
backported. CLOMonitor reads the latest release, so the score moves only
when that release carries the files.
- **Certificate identity pins the refs** (decided in review):
`tags.yaml@refs/tags/v…` and `promote-rc.yaml@refs/heads/main`. A
`promote-rc.yaml` dispatched from another branch fails the job's own
check at the upload step, after the staging branch, the `promotion-*`
candidate and the empty draft release exist; nothing in `parse` refuses
a non-`main` dispatch up front. Until the `v*` tag rulesets exist, a
hand-pushed `v*` tag whose tree has an edited `tags.yaml` still signs,
which `docs/release.md` says.
- **Follow-ups after the first signed release** (#4759 stays open for
them):
- `project.documentation.signature-verification` in
`security-insights.yml` (#4757);
- the SBOM row in `docs/security/self-assessment.md`, which says "Not
published at this time"; the image caveat above stays in it.

I left both alone so that they do not describe releases that don't exist
yet.

### How it was checked

- Bats 1.14.0 (the version CI installs), with gh, syft and cosign
stubbed:
- `hack/sign-release-assets.bats`, 8 tests: what the checksums cover;
that syft sees the committed tree and the cozypkg binaries but not
untracked files; that the verify identity accepts `tags.yaml` on a `v*`
tag and `promote-rc.yaml` on `main`, and rejects a fork, another
workflow, `tags.yaml.evil`, `tags.yaml` on a branch, and
`promote-rc.yaml` on `release-1.6`, on `main-evil` and on a tag; that
`docs/release.md` shows the same regexp as the script; that a failed
verification fails the script; that it refuses a missing asset, assets
split over two directories (with a same-named file in both, so the test
fails if the directory check is removed) and its own outputs as input.
- `hack/upload-assets.bats`, 4 tests, new: the exact upload list and
order (the assets, then the SBOM, the checksums and the bundle); that
the checksums name every upload but the last two; zero `gh` calls when
verification fails or an asset is missing. The old `upload-assets.sh`
fails all four, a version that signs after uploading fails the last two.
- `hack/ci-tool-installs-contract.bats`, 8 tests. Putting
`download-syft` back, dropping the `sha256sum -c` line, letting
`SYFT_VERSION` drift between the workflows, dropping `cosign-release`
from one step, letting it drift, unpinning the installer or giving one
step another installer commit each fails it.
- Putting the old any-ref regexp back in the script fails the identity
test and the docs tie; putting it back in the docs alone fails the tie.
- The meta-suites `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage` and `bats-runner-contract`
pass: 90 tests across these eight files.
- The pinned regexp gives the same accept/reject results under `grep -E`
and Go's `regexp` (what cosign uses). cosign-installer's install script
at the pinned commit, run as written with `cosign-release: v3.1.3`,
verifies the binary's keyless and KMS bundles and installs v3.1.3 (its
SHA-256 matches `cosign_checksums.txt`). `--yes`, `--bundle`,
`--certificate-identity-regexp` and `--certificate-oidc-issuer` are
listed, not deprecated, in cosign v3.1.3's help.
- The `Set up syft` step's script, run as written with
`RUNNER_TEMP`/`GITHUB_PATH` set: the checksum passes and `syft version`
reports 1.54.1. A real syft v1.54.1 run through
`hack/sign-release-assets.sh` (cosign stubbed) over this tree and the
v1.6.5 `cozypkg-linux-amd64.tar.gz` took 10 s and wrote an SPDX 2.3
document with 920 packages: 440 golang, 373 npm, 103 github, 4 without a
purl; 60 come from the cozypkg binary.
- cosign behaviour behind the docs, with a key-signed legacy bundle
whose `cert` holds the bare public key, against the documented identity
regexp (tlog ignored, since an attacker can upload their own entry):
v2.6.4 and v3.1.2 print "Verified OK", v2.6.5 and v3.1.3 reject it.
`--new-bundle-format` makes v2.6.4 reject it but not v3.1.2, so the docs
give the minimum version and not the flag. A real keyless protobuf
bundle (syft's) verifies on v2.6.5 and v3.1.3.
- zizmor 1.26.1 `--offline` reports no findings. actionlint 1.7.12 (with
`.github/actionlint.yaml`) reports the same 13 pre-existing info-level
shellcheck notes in `tags.yaml`/`promote-rc.yaml` as before, nothing
new. shellcheck passes on both scripts. `hack/check-commit-trailers.sh`
passes.
- Not checked: a real keyless signature. It needs the OIDC token of a
job in this repository, so the first rc after the merge is the
end-to-end test. Cutting an `-alpha.N` first would exercise it without
touching an rc.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I checked the trigger map file by file. The only entry
`hack/upload-assets.sh` can trigger is renaming a release asset, because
the website install pages hardcode asset names. This PR adds three
assets and renames none of the released ones, `cozypkg-checksums.txt`
included (the SBOM was renamed to `cozystack-source-sbom.spdx.json` in
review, before any release carried it). The workflow and docs changes
match no entry.

### Release note

```release-note
feat(release): releases carry cozystack-source-sbom.spdx.json, an SPDX SBOM of the source tree and the cozypkg binaries (not of the container or Talos images), cozystack-checksums.txt with the SHA-256 of each asset, and cozystack-checksums.txt.sigstore.json, a keyless Sigstore signature over the checksums made by the release workflow; docs/release.md shows how to verify a download with cosign v3.1.3 or later.
```

Refs #4759



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Releases now include an SPDX software bill of materials, SHA-256
checksums, and a keyless Sigstore signature bundle.
* Release assets are verified against the expected workflow identity
before upload; uploads stop if verification fails.
* **Documentation**
* Added guidance on release signatures, SBOM coverage, asset
verification, and which branch dispatches release workflows.
* **Tests**
* Added coverage for generated release files, upload behavior, and
signing and verification failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-20 15:20:41 +05:00
Marian Koreniuk bc24b588b2 feat(release): sign the release assets and attach a source SBOM 
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed.

hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order.

The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only.

Refs #4759

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
 2026-10-19 21:18:18 +02:00
myasnikovdaniil ebceef4105 ci(retention): sweep packages named by the published nightly tree (#4826) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The Artifact Retention workflow has failed every day since mid-August,
and it never pruned anything. It lists the org's container packages with
`GET /orgs/cozystack/packages?package_type=container`, and that endpoint
answers `400 Invalid argument` to the app installation token the job
uses. Others hit the same in actions/github-script#449 and cli/cli#9606.
An old `|| true` used to hide it. This is item A19 in #3278.

The per-package endpoints accept the same token: a probe run confirmed
it, and the dry-run below read every package with it. So the job no
longer lists the org. The new `hack/retention-packages.sh` takes the
package names from the published `cozystack-packages:nightly` tree,
reading refs through the shared `hack/lib/image-refs.sh`. It also adds
`cozy-installer` and `cozystack-nocloud`, which the nightly publishes
outside the tree. The committed tree can't be the source: it misses
images whose digest only the build pins, and those still get dated
nightly tags every day.

Failures now turn the run red instead of passing silently:

- if the name list is empty, the nightly sweep is skipped and the run
fails, but abandoned promotion candidates are still handled, because
their protection doesn't depend on that list
- if a package's versions can't be read, that package is skipped and the
run fails at the end
- if a DELETE is refused, `hack/retention-delete.sh` logs an error with
the HTTP status, goes on with the remaining versions and fails the run
(before, this was only a warning)
- if names resolve but none of the packages is linked to this
repository, nothing gets swept, so the run fails too, after the
promotion sweep has run (that sweep itself is skipped only when
candidate protection can't be resolved)

Promotion-candidate protection itself is unchanged. The contract test
that pinned the old org listing now pins the new behaviour, and both new
scripts have bats tests.

Known gap: a component that disappears from the nightly tree drops off
the list, so its old dated nightlies are never pruned. Today none of the
packages outside the list has more dated nightlies than the keep window.

Dry-run on this branch is green:
https://github.com/cozystack/cozystack/actions/runs/38075159900. It
resolves 59 packages, skips `cozystack/cozystack-ui` with the warning,
and would delete 418 old nightly versions plus one abandoned promotion
candidate. Deleting needs a different token. The app token lists
versions, but a one-version apply run
(https://github.com/cozystack/cozystack/actions/runs/37939905343) got
`404 Package not found` on DELETE for a version it had just listed. The
same DELETE with this repository's `GITHUB_TOKEN` and the job's
`packages: write` worked in a second run
(https://github.com/cozystack/cozystack/actions/runs/37941084456), with
no change to package settings. So `hack/retention-delete.sh` now deletes
with `GITHUB_TOKEN`, and listing stays on the app token.

That delete was shown on a package linked to this repository. One of the
swept packages, `cozystack/cozystack-ui`, is linked to another
repository, and I didn't try deleting there. Such a package is skipped
before its versions are read and named in a warning, so it can't turn
every apply run red. The follow-up for it is in #3278.

`RETENTION_APPLY` stays off, so the scheduled run is a green dry-run
until someone turns it on.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

N/A

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(retention): the GHCR retention workflow takes package names from the published nightly tree instead of the org package listing, which rejects the app token, and fails the run whenever names, versions or deletions cannot be resolved.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Nightly package retention now targets packages referenced by the
published nightly tree and keeps the 14 newest eligible versions per
package.
* Promotion candidates older than 30 days are eligible for pruning only
when their digests are not protected by qualifying release pull requests
or branch tips.
* **Bug Fixes**
* Packages linked to another repository are skipped, and sweep failures
are reported without preventing other eligible packages from being
processed.
* **Documentation**
* Clarified which nightly versions and promotion candidates qualify for
retention.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 12:02:35 +05:00
Aleksei Sviridkin da1f1cc387 ci(retention): delete with the workflow token 
The app installation token lists package versions but its DELETE of a
listed version is answered with 404, while the repository's
GITHUB_TOKEN with packages: write deleted the same version without any
change to package settings. Deletes therefore go through GITHUB_TOKEN;
listing keeps the app token.

That deletion was shown on a package linked to this repository. A
package linked elsewhere is not attempted: it is skipped before its
versions are read and named in a warning, so it cannot turn every
apply run red. A failed read of the link still fails the run, and so
does a sweep in which no resolved package is linked, since that sweep
prunes nothing.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-10 21:15:48 +03:00
Aleksei Sviridkin f1c0f6f3f6 ci(retention): sweep packages named by the published tree 
The org container-package listing answers 400 to the app installation
token, so the job has never reached its sweep. Take the names from the
published nightly tree via hack/retention-packages.sh and keep calling
only the per-package versions endpoints, which accept that token.

An unresolved name list now fails the run: the nightly publishes into
these packages daily, so an empty list cannot mean there is nothing to
prune. It skips only the nightly sweep. The promotion sweep reads one
fixed package and is kept safe by the protected set, which the name
list does not feed, so it still runs.

A versions read that fails, in either sweep, is reported with the API
error and fails the run once the rest is swept, instead of aborting
the step through set -e. A failed candidate read leaves no ids behind.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-10 21:10:33 +03:00
Aleksei Sviridkin da28ea7ee8 ci(e2e): run nightly and release-tag e2e on the container lane 
Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.

Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.

DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-09 16:57:43 +03:00
Myasnikov Daniil d3df698c43 docs(release): gate maintenance cuts on backport delivery 
Listing labelled pull requests cannot establish whether their changes
landed on a maintenance branch.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
 2026-10-07 05:19:16 +03:00
Aleksei Sviridkin 51a2d859ac docs(release): describe what the release job builds and what blocks an rc 
The amd64 release job builds the Talos installer and matchbox for both
arches, so the pages that said CI never builds both in one job, and
that build-amd64 builds amd64 only, were wrong. Every rc stitch also
skips matchbox and the e2e sandbox, and the nightly report lists
matchbox and the refs an in-tree rebuild stamps over; the page said
every such skip and line fails the next rc. Name the ones that do not.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 5924b3fd34 docs: describe how a pre-release is built and gated for arm64 
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-28 01:32:37 +03:00
IvanHunters 0fa6cf72a5 chore(ci): complete the backport label rename across all consumers 
The initial rename missed several literal consumers of the old label
names beyond labels.yml and backport.yaml:

- stale.yaml exempt-pr-labels still listed backport,backport-previous.
  actions/stale matches label names exactly and does not resolve
  label-sync aliases, so after the rename a kind/backport PR would lose
  its stale exemption and get auto-closed after inactivity. This is the
  one behavioural regression; the rest are documentation drift.
- PULL_REQUEST_TEMPLATE.md told contributors to apply the backport label.
- release.md label-target table and the gh pr list --search commands
  referenced the old names (the searches would silently return nothing).
- contributing.md claimed a backport label is auto-applied on the
  [Backport] prefix; pr-labeler only adds area/release, so the line was
  already stale and is corrected while renaming.

Historical narrative in release.md and the backport scope-to-area doc
rows (a Conventional-Commit scope, not the label) are left as-is.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-27 13:08:15 +05:00
Myasnikov Daniil 04fa8471fe docs(release): say what the gates actually guarantee 
Three claims that were stronger than the code, plus one legibility fix.

The PR gate's comment said a `packages/` edit "on the PR or its base"
fails the status before merge. The base half is not true: GitHub fires
no `pull_request` event when the base branch advances, it only
recomputes refs/pull/N/merge, so a change landing on the base after this
went green is never re-verified and the PR merges on the stale result.
Checking the merge tree does make base drift visible WHEN the job runs,
which is worth keeping — but finalize is what actually catches the drift
case, after the merge and before any stable name. Say both, in the
workflow and in the reviewer checklist, so nobody reads the green as
covering more than it does.

"Transactional at every consumer-visible name" is a judgement call
dressed as an absolute, inherited from the "this dispatch performs NO
registry mutation" it replaced. The dispatch now does write to the
registry: the `promotion-*` candidate is a real, publicly listable tag.
What it creates is no *stable* name — no vX.Y.Z, no :latest, no stable
installer. Stated that way the sentence stops needing the reader to
already know which writes were meant.

The retention paragraph listed two of the conditions the workflow
enforces. It now lists all of them, including the ones the workflow
gained here: same-repository, bot-authored, `release-X.Y.Z` branch, the
`release` label, exactly one valid digest pin, and protection from a
base-branch tip as well as an open PR.

Also: `flux pull artifact` reads its credentials from
~/.docker/config.json, which is why finalize now runs `docker login` —
but its runner class is not one of the docker-using build ones and the
setup step installs everything except docker. Check for it there, where
the message can name what is missing, instead of failing with "command
not found" halfway down the release path. And the verifier's exclusion
list is a restatement of flux's `excludeOCI`, not a derivation — the CLI
does not expose it — so the coupling and the pinned version are now
written down next to the list.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-14 18:37:58 +05:00
Myasnikov Daniil f21d277eba fix(release): retain live promotion candidates 
Protect package candidates referenced by open release PRs while pruning abandoned run-specific promotion artifacts. Document the stable-candidate lifecycle and maintenance-branch rollout requirement.

Assisted-By: Claude <noreply@anthropic.com>

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-14 18:37:57 +05:00
Aleksei Sviridkin e0d6c6be92 fix(ci): stop labeled events from cancelling and re-running backports 
A run joins its concurrency group before any job-level `if` is
evaluated, so `prepare`'s guard cannot stop a label event from
disturbing the run already backporting a PR. Labels set with the
default GITHUB_TOKEN start no run, but labels set by third-party
GitHub Apps do, and they arrive in bursts, so each one cancelled the
backport the merge had just started.

The delivery survived anyway, which is why this went unnoticed: the
old guard admitted a run on the PR's cumulative labels, so the last
label of a burst requalified and redid the work. In the bursts on
record the kill lands within seconds, well before the cherry-pick,
so the cost was a restart rather than a lost backport.

Move a label event that requests no backport into a group of its own,
and let one that does request a backport queue rather than cancel.
Both halves are needed. `backport` and `backport-previous` are
separate requests that both have to complete, so they share a group
and must not cancel each other, while every other label
does no work here and only has to stay out of the way.

Scope `prepare` to the labels each trigger is entitled to read at the
same time. Reading the cumulative label set on a `labeled` event let
any later unrelated label on a merged PR still carrying `backport`
re-enter the job to redo a backport already delivered. That redo is
also what redelivered a killed backport, so narrowing the guard alone
would turn a noisy delivery into a missing one: neither change is
safe to land without the other.

Bound both jobs with `timeout-minutes` for the same reason, since a
queued request now waits on the run ahead of it rather than replacing
it. The ceiling bounds execution and not the wait for a runner, which
no job-level key reaches, so 30 is generous rather than calculated:
every execution measured on this workflow has been seconds, against
minutes of queue that nothing here can bound.

Document the recovery path the narrowing removes: with no accidental
re-run left, re-applying whichever backport label the PR already
carries is how an operator retries a merge-triggered run that died.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-11 03:14:09 +03:00
Myasnikov Daniil 24e9c5396c docs(release): resync the release doc with the freeze model 
The freeze changed three things the prose still described the old way.

The backport table claimed `backport-previous` resolves to
`release-X.(Y-1)` and that resolution runs through `getLatestRelease`.
Neither is true any more: the job enumerates the real `release-X.Y`
branches, sorts them numerically descending, and takes the newest and
second-newest existing lines. The `Y-1` derivation is the min-1 bug this
branch dropped. Line 155 already carried the correct description, so the
file contradicted itself.

The freeze window's effect on backport targets lived only in a code
comment. During it `backport` aims at the line being stabilised and
`backport-previous` at the last published stable, so the line one step
further back has no label for the duration. Document the trade-off where
the labels are described.

The changelog backstop and Phase 6 both framed a promote PR targeting
`release-X.Y` as "a patch release". After the freeze every promote PR
targets the release line, minor and patch alike, which makes the
backstop the only path a `.0` changelog takes to `main`.

Also note that re-cutting a line after an unusable rc.1 means deleting
`release-X.Y` by hand — the refuse step blocks every other route, by
design.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-03 21:35:01 +05:00
Myasnikov Daniil b55e822eaf docs(release): align the patch-release diagrams with the freeze model 
Address review feedback from coderabbitai on docs/release.md:150: the
patch-release gitGraphs still branched release-1.2 off main after the
v1.2.0 commit, depicting the branch as created at the stable release.
Under the rc freeze the line forks at vX.Y.0-rc.1 and the release commit
never lands on main, as the surrounding prose already states.

Fork release-1.2 at the rc.1 commit and place 'Release v1.2.0' on the
release line in all three patch-section diagrams.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-03 21:31:59 +05:00
Myasnikov Daniil 3f15229795 docs(release): fix the Backports link fragment 
Address review feedback from lexfrei and coderabbitai on docs/release.md:39:
the anchor pointed at #backporting, but the heading is '## Backports'
(slug #backports), so the link resolved nowhere and tripped markdownlint
MD051.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-03 21:31:59 +05:00
Myasnikov Daniil e5428caaa6 docs(release): document the rc freeze 
The Release Candidates section said the opposite of current behaviour —
"new features and changes can still be added before the regular release"
— so it is rewritten around the freeze, along with the Regular Release
step list, both gitGraph diagrams, and the workflow reference sections.

The promote PR now targets release-X.Y rather than main, so the diagrams
show main forking away at the freeze and never receiving the release
commit. Patch Releases collapses to "same as above, the branch already
exists", since after the freeze both flows are identical.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-03 21:31:59 +05:00
Myasnikov Daniil 717c8860f6 docs(release): make the release playbook coherent with the promote flow 
Final coherence pass over docs/release.md (and the releasing.md
pointer) after several incremental patches to the release-cycle
rework left the whole-document narrative uneven.

- Rewrite the regular and patch release sequences so a maintainer
  reading top-to-bottom performs the actual flow: cut rc -> watch
  mandatory rc E2E -> [optional Generate RC Changelog button] ->
  promote dispatch (E2E gate; changelog generated or picked up;
  website docs PR opened and parked) -> review the promote PR (no
  E2E by default; E2E/changelog/website-docs lines) -> merge ->
  finalize -> merge the parked website PR, then announce.
- Purge the last old-model sentence: "vX.Y.0 is tagged from the
  last RC or a later commit in main" -> promoted from the rc by
  digest, tag cut write-once at the promote PR merge commit.
- State the alpha/beta contract where prereleases are discussed:
  built and published, button-only E2E (no mandatory rc-e2e), and
  no promotion path (promote-rc accepts vX.Y.Z-rc.N alone).
- Fix the Phase 4 heading (was "chore(release): cut vX.Y.Z"; the
  PR is "chore(release): promote ... -> ...") and add the missing
  website-docs line to the reviewer checklist; add a
  keep-parked-until-publish item to the pre-release checklist.
- Add cut-prerelease.yaml, changelog-rc.yaml, and
  changelog-generate.yaml to "See also" so the workflow inventory
  is complete and consistent with the "seven workflows + reusable
  changelog-generate" count.
- releasing.md: replace the stale "typically involves ... creating
  git tags / building artifacts" list (invalidated by the
  promote-not-rebuild model) with the real high-level sequence.

Every workflow/job/step name cited was verified against the YAML on
this branch. hack/release-changelog-contract.bats stays green (24/24);
no test pins docs/release.md strings, so no test change was needed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:26 +05:00
Myasnikov Daniil 01e1e7188e ci(release): stop finalize persisting credentials so the tag push triggers tags.yaml 
pull-requests-release.yaml's finalize job checked out the repo with the default
persist-credentials, leaving GITHUB_TOKEN embedded as http.extraheader. Every
later tag/branch push re-injects the CI app token via `git remote set-url`, but a
persisted extraheader silently wins over the URL credential — so those pushes
authenticated as GITHUB_TOKEN. A GITHUB_TOKEN-authenticated push creates no
workflow run (GitHub anti-recursion), which is exactly why v1.6.0's stable tag
never triggered tags.yaml: the generate-changelog and update-website-docs
backstops never ran.

Set persist-credentials: false on the finalize checkout, so the set-url app token
actually authenticates the pushes — the same handling promote-rc.yaml already
uses. Audited every git/gh step in the job: both tag steps set-url the app token
before pushing, and Ensure-maintenance-branch / Publish-draft-release / the
retag+chart steps use the app token via github-script or a separate registry
login — none depends on the persisted GITHUB_TOKEN extraheader.

Pin it in hack/promote-gate-contract.bats.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:26 +05:00
Myasnikov Daniil 936190d4b6 ci(release): open the website docs PR at promote time 
The cozystack/website 'update managed apps reference' PR used to be opened only
at tag time by tags.yaml::update-website-docs — after finalize pushed the stable
tag, i.e. after the release was already published (too late to review) — and it
fetched README content from that stable tag (BRANCH=RELEASE_TAG), which finalize
only creates post-merge.

Add a promote-time website-docs job to promote-rc.yaml (needs [parse, promote])
that opens the same update-docs-vX.Y.Z PR from the release-X.Y.Z staging branch
via the website Makefile's FETCH_REF variable, so the docs are reviewable
alongside the promote PR while the stable tag is still unborn. It refreshes the
trunk pins from the staging ref before release-next so a new minor inherits fresh
Talos/cozystack pins (the v1.6 talos-pin incident class), then runs update-all
with FETCH_REF pinned to the staging branch.

The job is non-blocking: a FETCH_REF-support guard fails loudly (warn + exit 1)
when cozystack/website main predates the FETCH_REF Makefile variable — otherwise
make would silently ignore the unknown FETCH_REF override, fall back to the
not-yet-created stable tag, and update_apps.sh would warn-and-continue into STUB
docs. Any failure surfaces as a website-docs warning line in the promote PR body
(plumbed through open-pr via needs.website-docs.result) and never blocks
promotion; the tag-time job stays as the backstop and no-ops when promote already
did the work.

MERGE-TIMING CONTRACT: the website PR must NOT be merged before finalize
publishes the release — doing so flips the site's latest-version pointer to a
version whose docs and openapi are not on GitHub yet, 404ing every version-pin.
The website PR body and the promote PR body both say so in bold.

Pin the contract in hack/promote-gate-contract.bats (needs shape, FETCH_REF
invocation, the guard, non-persisted checkout credentials, the DO-NOT-MERGE body
wording, the promote-PR status line, and the tags.yaml backstop comment) and
relax the release-changelog-contract needs pin to tolerate the appended
website-docs dependency.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:26 +05:00
Myasnikov Daniil ff7ddd3c9a ci(release): gate promotion on a green rc e2e and drop the automatic full-e2e label 
Promotion now assumes checks passed at rc time instead of re-running
them. At dispatch, before any staging happens, promote-rc verifies the
rc tag has a green full e2e via either evidence path: the rc's
tags.yaml push run containing a successful 'rc-e2e / E2E <tag> (full
suite)' workflow_call job, or a successful manual e2e-tag.yaml dispatch
correlated by its exact tag-bearing job name (immutable rc digests make
past evidence valid). No evidence fails the dispatch with both paths
and the remediation named. A skip_e2e_gate boolean input bypasses the
gate loudly for emergencies; the promote PR body always carries the
verification state (verified or bypassed), and 'verified' can only
originate in the gate's green branch.

The promote PR keeps the release label but no longer gets full-e2e
automatically: its tree is the rc's already-validated digests with only
tag strings rewritten, so pull-requests.yaml now runs e2e on the
release path only when a maintainer opts in by adding full-e2e. The
regular-PR path and the ordinary full-e2e forcing are untouched, and
the new labeled trigger makes a post-open label actually start the run.

A new promote-gate contract suite pins the gate's DAG position, the
override default, the label set, the body lines, and the label-gate
condition with comment-proof executable-line counts.

Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:26 +05:00
Myasnikov Daniil 2dfbeea776 ci(release): dispatch changelog generation at rc time 
Extract the promote-time changelog generation into a reusable workflow
(changelog-generate.yaml, workflow_call) and add a dispatchable wrapper
(changelog-rc.yaml) so a maintainer can generate the release changelog
the moment an rc goes green, while there is still time to review it.

The reusable core keeps promote-rc's observable behaviour: detached
rc-tag checkout, read-only app token for the AI step, the
validate-changelog.sh gate, non-blocking AI failure, and the same
changelog-vX.Y.Z artifact. .release-tooling is overlaid from
github.ref_name, which under workflow_call is the caller's dispatch ref
(and coincides with the reusable workflow's own ref for a local uses:),
so "current tooling against an old rc tag" is preserved. Secrets are
declared explicitly rather than inherited, so a caller that forgets one
fails at parse time.

The rc tag is parsed by a shared hack/parse-rc-tag.sh that REJECTS
whitespace outright (a stripped 'v1. 6.0-rc.2' would silently become a
different tag). promote-rc parses it ONCE in a cheap `parse` job that
both `promote` and `changelog` depend on — so they still run in parallel
with each other while consuming one validated tag, and a whitespace tag
fails the whole promotion rather than letting promote succeed while the
changelog job rejects the same input and the PR opens empty. The
reusable core mirrors the reject-whitespace policy inline because its
parse must run before any checkout.

changelog-rc.yaml verifies the rc staging branch release-X.Y.Z-rc.N
exists UP FRONT (in parse, which generate depends on), so a missing
branch fails in seconds rather than after a 30-minute AI run, then
commits the result there as docs/changelogs/vX.Y.Z.md idempotently
(compare-before-push).

promote-rc's changelog job is a thin uses: call. Source selection —
validate-then-fall-through across the rc tag tree, the rc staging branch
(the rc-time pickup), then AI generation, discarding any invalid source
with a loud warning so it never suppresses a valid lower-priority one —
lives in hack/select-changelog-source.sh, executed after the tooling
overlay. open-pr carries the picked-up file onto release-X.Y.Z and
finalize publishes it.

Contract and behaviour bats are repointed and extended: the shared
parser, the source-selection script, and promote's parse policy are all
exercised by execution on adversarial inputs; the remaining grep pins
match executable lines with exact counts so a comment or dead block
cannot satisfy them. docs/release.md lists the new workflow and the
rc-time entry point. actionlint and zizmor (default and pedantic) are
clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:16 +05:00
Myasnikov Daniil 6fc9708514 docs(release): align validator description with validate-changelog.sh checks 
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 11:38:59 +05:00
Myasnikov Daniil f49d54a689 ci(release): fix review findings; keep website docs at tag time 
Dual-model review (Claude + Codex) of the previous two commits found seven
substantive defects, two of them destructive. Both models independently
reached three of them; the split is noted per item.

Descope the website half (Claude, verified by reproduction). The docs job
was moved to promote time, but the website Makefile pins the fetch ref
with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line
BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that
cannot exist at promote time, since promote's own validate step hard-fails
if it does. update_apps.sh logs the failed fetch and exits 0 AFTER
truncating each destination file, so every app doc would become a stub,
`changed=true` would open the PR, and the new auto-merge would land it
unreviewed. Not fixable from this repo. update-website-docs returns to
tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize
auto-merge is dropped. Opening it earlier needs a cozystack/website change
first and is left as a follow-up. This also removes the unordered-sibling
race Codex found, where the release PR could merge before the docs job
opened its PR, leaving finalize nothing to merge and no retry.

Fix the non-blocking guarantee (both models). Only the AI step carried
continue-on-error; the changelog JOB did not, so a flaky `npm i -g
@github/copilot` failed it and `needs` skipped open-pr entirely — staging
branch pushed, release drafted, no promote PR. Gate open-pr on promote
alone.

Generate from a checkout the contract permits (both models). The job
checked out the rc staging branch, which docs/agents/changelog.md §2
rejects — a compliant agent stops and writes nothing, so the normal path
produced no changelog at all. Check out the rc tag detached and document
that configuration in §2. The checkout ref now comes from the validated
parse output rather than raw dispatch input.

Port, don't regenerate, in the backstop (both models). For a patch release
promoted onto release-X.Y the changelog never reaches main, so the
backstop fired and ran a second AI pass; once its PR merged,
update-releasenotes.yaml overwrote the published body with unreviewed
text. Detect the file at the tag and port it verbatim; the AI runs only
when it is absent from both main and the tag.

Survive re-dispatch (Codex). promote force-pushes the staging branch,
erasing a hand-written changelog — the documented recovery when generation
fails. Preserve an existing one across the rebuild, and stop open-pr
overwriting it with generated output.

Validate structurally, not by size (Codex). A truncated or whitespace-only
AI run passed `[ -s ]`, was committed, advertised as valid in the PR body,
and published verbatim as release notes. Assert the header, the compare
link and a plausible line count; finalize additionally refuses
whitespace-only content rather than blanking the notes.

Add hack/release-changelog-contract.bats — nine tests pinning each link in
the chain, mutation-checked: removing open-pr's if:, breaking finalize's
body read, and restoring paths-ignore each fail the suite.

Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which
described the old flow throughout.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 11:34:16 +05:00
myasnikovdaniil d0c0f399f8 fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404) 
## The bug

Promotion rewrites the rc version substring to the stable version across
the vendored image references. That rewrite lived inline in
`promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus
`packages/apps/kubernetes/images/*.tag` **only**, on the premise that
the kubernetes app was the only one whose `.tag` files carry the
cozystack version. Nine other `.tag` files do, and `system/multus` seds
its reference straight into a vendored upstream daemonset manifest.

#3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33
of 55 first-party references still reading `v1.6.0-rc.4`**.

`hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind
spot from the other direction — both scanned the depth-2 `values.yaml`
alone, so those images were never retagged to the stable version (**30
references selected before, 42 after**) and never mirrored to the public
registry for a nightly. A retag miss leaves an image short of a tag,
since the digest still resolves inside one registry. A mirror miss is
worse: the host rewrite walks the same file list, so a reference that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

`v1.6.0` is the first release to go through promotion — every earlier
release was a full rebuild from the tag, which stamped every file
uniformly. The glob has always been wrong; the old path just made it
unobservable.

## Why it could not be caught before

The rewrite was workflow-inline, so there was nothing to unit test. The
only way to observe the miss was to cut a release.

It is a script now — `hack/promote-rewrite-tags.sh` — and
`hack/promote-rewrite-tags_test.bats` round-trips it against the real
tree under `make unit-tests`, with no cluster, registry or release
involved.

## The fix

The three call sites each carried their own idea of where a reference
can live, which is how they drifted apart. That knowledge moves into
`hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()`
for the files, `collect_image_refs()` for the references inside them —
and all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite also asserts its own postcondition: it scans **wider** than
it rewrites and fails the promotion when a reference survives in a
location the enumeration does not know about, rather than shipping it
quietly. The wide scan is shape-filtered to image-reference position,
because `X.Y.Z-rc.N` is an ordinary version string that other things
legitimately contain — `kubevirt-csi-driver/go.sum` pins
`github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a
cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:`
instead would have been wrong: `system/cilium/values.yaml` keeps `tag:`
and `digest:` under separate keys.

`docs/agents/image-refs.md` records the contract — three tag classes,
three storage shapes, the invariants, and which consumer does what.

## Not a bug: kamaji

`cluster-api-control-plane-provider-kamaji` stays at
`v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild
of an upstream component, versioned by that component rather than by the
cozystack version line, so no rewrite is owed. A test pins this so a
future fix here cannot over-reach.

## Documented known gaps, deliberately not fixed here

Both are pre-existing and in the **host** dimension; the version
dimension this PR owns is complete, because the version always lives in
a `tag` key regardless of where the host sits.

- **The nightly host rewrite cannot reach a split host.** It is a
literal `<src-registry>/` substring replace, so `keycloak-operator`
(host in a sibling `registry:` key) and `kubeovn`
(`global.registry.address`, no trailing slash) are mirrored but keep
pointing at the build registry in the published tree. Fixing it needs
structure-aware rewriting.
- **A reference inside a gzip is invisible.**
`capi-providers-cpprovider` ships `files/components.gz`, and the promote
postcondition cannot catch this class at all since `grep -rIl` skips
binaries. Nothing is owed today — kamaji is component-versioned and the
image is still mirrored and retagged via its `.tag` — so the residual is
that a nightly's kamaji ConfigMap keeps the build-registry host.

## Testing

- 17 tests in the new suite; `make bats-unit-tests` green (312 tests,
exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow.
- Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`,
`1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no
false negatives.
- Every test is mutation-proven. Reverting the enumeration to the old
glob, deleting any of the five YAML shapes, corrupting emitted digests,
reverting extras to parse-only, dropping the unreadable-file guard,
broadening shape 4 beyond `global.images`, or planting an unenumerated
first-party reference in any storage shape each fail a named test.

## Reviewer note, out of scope

`platform-migrations` is pinned at two different digests —
`core/platform/values.yaml` at `v1.5.0` and
`backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned
pin; no Makefile writes that key, tracked in #3143). A
`promote-retag.sh` dry-run emits **two copies to the same destination
tag**, so the write-once guard will fail the promotion. This is
independent of this PR and needs a decision about the backup-client
image version, so it is left alone here — but it blocks the 1.6.0
promote and wants fixing before the re-cut.

## Release note

```release-note
Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry.
```

Fixes the promote half of #3143.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Image promotion and nightly mirroring now detect digest-pinned image
references across values files, standalone tag files, and supported
templates.
- Stable release preparation supports RC→stable tag rewriting via a
dedicated promotion script, with broader verification to ensure no RC
references remain.

- **Bug Fixes**
- Improved completeness of ref discovery and scanning, including
previously missed template/tag locations.

- **Documentation**
- Added and linked guidance documenting image-reference invariants and
how tooling enumerates supported storage shapes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 16:41:17 +05:00
Myasnikov Daniil 09323fff13 fix(ci): cover every image-ref storage shape in promote, retag and mirror 
Promotion rewrote the rc version substring across the depth-2 package
values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the
premise that the kubernetes app was the only one whose .tag files carry
the cozystack version. Nine other .tag files do, and system/multus seds
its ref straight into a vendored upstream manifest. v1.6.0 was therefore
staged with 33 of 55 first-party refs still reading v1.6.0-rc.4.

promote-retag.sh and nightly-mirror.sh had the same blind spot from the
other direction: both scanned only the depth-2 values.yaml, so those
images were never retagged to the stable version (30 refs selected
before, 42 after) and never mirrored to the public registry for a
nightly. A retag miss merely leaves an image short of a tag, since the
digest still resolves inside one registry; a mirror miss is worse,
because the host rewrite walks the same file list and a ref that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

The three call sites each carried their own idea of where a ref can
live, which is how they drifted apart. Move that knowledge into
hack/lib/image-refs.sh as the single enumeration -- image_ref_files()
for the files, collect_image_refs() for the refs inside them -- and have
all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite itself moves out of promote-rc.yaml into
hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only
observable by cutting a release: there was nothing to unit test. It now
also asserts its own postcondition, scanning wider than it rewrites and
failing the promotion when any ref survives in a location the
enumeration does not know about, rather than shipping it quietly.

Component-versioned images (kamaji's v0.19.0-cozystack.N) and
third-party pass-through images do not ride the cozystack version line
and are deliberately left alone; a test pins that so a future fix here
cannot over-reach. capi-providers-cpprovider is a documented known gap:
its ref is also embedded in a gzipped copy that the chart actually
ships, and rewriting only the readable copy would diverge the two.

docs/agents/image-refs.md records the contract -- three tag classes,
three storage shapes, the invariants, and which consumer does what.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 20:14:24 +05:00
Myasnikov Daniil a4b5b9d57b docs(release): update workflow cancellation condition 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 18:41:45 +05:00
Myasnikov DaniilandClaude Opus 4.8 47eec0157f feat(release): add Cut Pre-release Tag workflow (#2677) 
Pre-release tags (vX.Y.Z-alpha.N / -beta.N / -rc.N) were cut by a manual
`git push origin HEAD:refs/tags/<tag>`. That leaves the immutable-tag work in
#2677 only partly enforceable: a repo ruleset cannot restrict `v*` tag
*creation* to CI while still letting maintainers hand-push pre-releases (fnmatch
has no way to match "v* except v*-rc.*"), so "do not hand-cut stable tags" stays
a matter of discipline backed only by an in-workflow guard.

Add cut-prerelease.yaml — a workflow_dispatch that becomes the sole entry point
for creating a pre-release tag:

- Generates the CozyStack CI app token (same app as promote-rc.yaml) and pushes
  the tag as the app, so tags.yaml fires (a tag pushed with the default
  GITHUB_TOKEN would not trigger it) and the push-event base_ref is populated
  (tags.yaml's Get-base-branch step needs it).
- Validates ^v\d+\.\d+\.\d+-(alpha|beta|rc)\.\d+$ and rejects stable vX.Y.Z —
  stable tags are cut write-once by the promote flow at a PR merge commit.
- Enforces dispatch from main or the matching release-X.Y branch, and refuses to
  move an existing tag (write-once).
- Pushes HEAD:refs/tags/<tag> from the branch tip — the exact form
  docs/release.md prescribes — on ubuntu-latest (no build runner needed).

With every pre-release cut by CI, repo admins can then enable a "Restrict
creations" tag ruleset on `v*` with the CI app as the sole bypass actor, turning
tag-creation discipline into hard enforcement.

Also update tags.yaml's two human-facing error messages and docs/release.md (the
RC and patch flows, the workflow list, and a new repo-side tag-ruleset
enforcement note) to point at the workflow instead of a manual push.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-09 12:21:58 +05:00
Myasnikov DaniilandClaude Fable 5 d252fc93a0 ci(release): make rc promotion transactional; defer registry writes to finalize 
Promotion mutated the registry at dispatch time — retagging ~37 images to
vX.Y.Z (+:latest), publishing the stable cozy-installer chart — before the
promote PR's e2e ran and before any merge. An abandoned promotion then left
stable-named bytes for a version that never shipped, and the (correct) image
write-once check wedged any later re-promotion to the same version.

Move every irreversible registry side effect into pull-requests-release.yaml
finalize, which runs only after the promote PR merged (= after its e2e passed),
making promotion atomic. promote-rc.yaml now only stages the release-X.Y.Z
branch, drafts the stable release, and opens the PR; an abandoned promotion
leaves nothing but an overwritable draft + branch.

Folded into the same restructure:

- Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag in finalize, at the stable
  tag's commit. It was gated in tags.yaml on `release_exists==false &&
  is_stable`, unsatisfiable under promotion (the draft always pre-exists), so
  promoted stables never got their submodule tag — a regression for Go
  consumers. Finalize is where the stable tag is born, so it cuts the matching
  submodule tag write-once (stable only).
- Gate :latest by max-semver. promote-retag.sh now moves :latest only when
  MOVE_LATEST=1 (default off); finalize passes it the same make_latest decision
  the release gets, so a patch on an older line moves neither the release's
  `latest` nor the images' :latest. Added MOVE_LATEST bats coverage.
- Stamp platformVersion into the published stable chart's default values
  (uncommitted, at package time) so `helm --install --version X.Y.Z` reports the
  stable version in telemetry instead of the rc version baked into the operator
  image. Not committed to the branch — that would leak platformVersion=vX.Y.Z
  onto main and mis-stamp the next rc.
- promote-rc: serialize all promotions (static concurrency group, not per-rc);
  label the PR full-e2e so the full suite runs; scope the rc->stable sed to the
  depth-2 values.yaml glob plus the kubernetes .tag files (the build's own write
  set); tolerate a leftover draft so re-promotion is not wedged.
- finalize: assert the PR author is cozystack-ci[bot], and run on an ephemeral
  oracle-vm runner (needs skopeo/helm).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 17:44:53 +05:00
Myasnikov DaniilandClaude Opus 4.8 f9ac1bf9cf docs(release): pin nightly examples to the full immutable tag 
nightly.yaml publishes each build under 0.0.0-nightly.<date>.<run-id>.<attempt>
(the run-id + attempt suffix makes every nightly tag immutable), but the docs
still showed the old date-only 0.0.0-nightly.<YYYYMMDD> form in the mirror-stage
description and both the helm-install and oras-pull pin examples. Update them to
the full form; the floating `nightly` example is unchanged.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 10:24:00 +05:00
Myasnikov Daniil 0e14b34164 docs(release): broaden section heading to cover all release workflows 
"What CI does on a tag push" only described one of the four triggers;
promote-rc.yaml uses manual dispatch and pull-requests-release.yaml fires
on PR merge. Rename to "What CI does during the release process" so
maintainers find the right entry point regardless of which workflow they
are debugging.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 10:03:58 +05:00
Myasnikov DaniilandClaude Opus 4.8 ba6cfbf300 docs(release): reconcile the stable-tag flow to promotion-only 
Address review feedback from coderabbitai on docs/release.md: the Regular and
Patch sections still told maintainers to push vX.Y.Z to start the build, while
the rest of the doc said the stable tag is created write-once at the merge
commit and stable comes only from rc promotion (with a 'can also push directly'
escape hatch). Rewrite both sections to the single promotion model — push an rc,
promote it via promote-rc.yaml, merge the promote PR, finalize creates the
write-once tag — drop the direct-push escape hatch, and add promote-rc.yaml to
the workflow list.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 10:03:58 +05:00
Myasnikov DaniilandClaude Opus 4.8 f30f31526a docs(release): document the GHCR nightly mirror + install path 
Replace the nightly-rc-tag description with the new model: a nightly is
an installable copy of main on GHCR (mirror OCIR->GHCR by digest +
re-published packages artifact, cozy-installer chart and nocloud disk),
validated by the full e2e suite, with no GitHub release and no Go-module
tag. Document the helm install path and the dated/floating tag scheme,
and update the Tag-immutability section and workflow link list.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 10:03:50 +05:00
Myasnikov Daniil 6a2f392b41 docs(release): rewrite for the write-once + rc-promotion model 
The release doc still described the removed force-retag flow and auto-release.yaml
cron. Update it to the model this PR ships: stable tags are created write-once at
the PR merge commit (never force-moved), the maintenance branch is
fast-forward-only, stable patches come only from rc promotion (nightly-rc.yaml +
promote-rc.yaml + retention.yaml), and the api/apps/v1alpha1 Go-module tag is cut
only for stable. Replaces the "Automated patch tag" and "Force-retagging"
sections (the latter becomes "Tag immutability") and refreshes the See-also list.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 10:03:49 +05:00
Myasnikov Daniil 8c4cfe80d8 docs(release): document automated patch-release cron and support window 
Address review feedback from lexfrei on docs/release.md: the Patch Releases section described only the manual process, so add an "Automated Patch Releases" subsection covering the nightly cron, the new-commits trigger, the 2-newest-lines support window, and the manual tag-push escape hatch for EOL lines.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-21 14:26:15 +03:00
Myasnikov Daniil 6d8634e3ef docs(release): expand release.md into release-engineer playbook 
Adds operational sections under the existing process-design walkthrough:
CI phase-by-phase behavior, backport bot mechanics + cherry-pick triage,
monotonic-counter migration trap, changelog failure modes + verification,
three release-time war stories, pre-release verification checklist,
force-retagging discussion + RFC link, bundle-PR splitting strategy.

Mermaid diagrams and numbered RC/regular/patch walkthroughs are kept
verbatim. Example versions normalized to simple semver-style (v1.2.0,
v1.2.1, release-1.2) consistent across the doc.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-26 00:29:24 +05:00
Nick Volynkin 12c6df83f5 [docs] Update release policy: Release Candidate versions 
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
 2025-05-05 11:52:06 +03:00
Nick VolynkinandAndrei Kvapil ffe6109dfb [docs] Describe the Cozystack release workflow 
Resolves #127

Co-authored-by: Andrei Kvapil <kvapss@gmail.com>

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
 2025-04-16 19:31:58 +03:00