Promotion rewrote the rc version substring across the depth-2 package
values.yaml plus packages/apps/kubernetes/images/*.tag alone, on the
premise that the kubernetes app was the only one whose .tag files carry
the cozystack version. Nine other .tag files do, and system/multus seds
its ref straight into a vendored upstream manifest. v1.6.0 was therefore
staged with 33 of 55 first-party refs still reading v1.6.0-rc.4.
promote-retag.sh and nightly-mirror.sh had the same blind spot from the
other direction: both scanned only the depth-2 values.yaml, so those
images were never retagged to the stable version (30 refs selected
before, 42 after) and never mirrored to the public registry for a
nightly. A retag miss merely leaves an image short of a tag, since the
digest still resolves inside one registry; a mirror miss is worse,
because the host rewrite walks the same file list and a ref that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.
The three call sites each carried their own idea of where a ref can
live, which is how they drifted apart. Move that knowledge into
hack/lib/image-refs.sh as the single enumeration -- image_ref_files()
for the files, collect_image_refs() for the refs inside them -- and have
all three source it, so a new storage location is declared once and
reaches every consumer.
The rewrite itself moves out of promote-rc.yaml into
hack/promote-rewrite-tags.sh. Being workflow-inline is why this was only
observable by cutting a release: there was nothing to unit test. It now
also asserts its own postcondition, scanning wider than it rewrites and
failing the promotion when any ref survives in a location the
enumeration does not know about, rather than shipping it quietly.
Component-versioned images (kamaji's v0.19.0-cozystack.N) and
third-party pass-through images do not ride the cozystack version line
and are deliberately left alone; a test pins that so a future fix here
cannot over-reach. capi-providers-cpprovider is a documented known gap:
its ref is also embedded in a gzipped copy that the chart actually
ships, and rewriting only the readable copy would diverge the two.
docs/agents/image-refs.md records the contract -- three tag classes,
three storage shapes, the invariants, and which consumer does what.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The Nightly workflow has **never completed a run**. Every scheduled run
since `NIGHTLY_ENABLED` was set has failed in `mirror` at "Resolve
source commit and version", ~80s in, with:
```
level=fatal msg="unsupported image-specific operation on artifact with type \"application/vnd.cncf.flux.config.v1+json\""
```
Nothing has ever been published — `cozy-installer:nightly` and
`cozystack-packages:nightly` both 404 on GHCR, and `cozystack-nocloud`
does not exist.
This fixes that failure and the next two behind it.
### 1. The digest lookup could never have worked
`cozystack-packages` is a **Flux OCI artifact** (config mediaType
`application/vnd.cncf.flux.config.v1+json`), not a container image. A
plain `skopeo inspect` parses the config blob as an image config and
fatals on it, so `skopeo inspect --format '{{.Digest}}'` was never
viable against this ref — the `--raw` call on the very next line already
acknowledged this.
Replaced with `skopeo inspect --raw` + `skopeo manifest-digest`, which
hashes the manifest bytes — exactly what the registry addresses the
artifact by, with no image-specific parsing.
This also collapses two registry fetches into one: the digest and the
`org.opencontainers.image.revision` annotation now come from the **same
bytes**, which strengthens the race the original comment describes —
there is no longer any window between pinning the tag and reading it.
### 2. `flux pull artifact` rejects a non-existent output path (×2)
`flux pull artifact --output _baked` fails with `invalid output path
"_baked": no such file or directory`. `_baked` is neither tracked nor
created by anything. `pull-requests.yaml` does `rm -rf` + `mkdir -p`
first; `nightly.yaml` omitted it. Since the workflow always died at step
1, this was never reached.
Same bug for `_rw` in the e2e staging step. Both now create the
directory first.
## Verification
- Reproduced the original fatal against the live artifact with skopeo
1.22.2.
- Ran the fixed resolve step end-to-end against the real registry:
returns the digest the registry reports, and a `sha` that matches the
current `main` HEAD.
- Reproduced the `_baked` failure and confirmed the fix extracts the
tree.
- `actionlint` clean; `zizmor --offline` clean; `shellcheck` clean on
changed lines.
Also checked and **not** changed, each disproven rather than assumed:
- `skopeo copy --multi-arch all` on a Flux artifact — works; `copy` is
mediaType-agnostic.
- The `awk -F@ '/artifact successfully pushed/'` digest capture —
matches current flux CLI output, and is the same idiom
`packages/core/installer/Makefile` already uses in production.
- `nightly-mirror.sh`'s `inspect --format Digest` — all mirrored refs
there are genuine image manifests, so it is correct as written.
- `make -C packages/core/talos talos-nocloud` — exists and is CI-proven
via `tags.yaml`.
## Honest scope
This fixes the `mirror` job's blockers. It does **not** promise a green
nightly on the first run: every step after `mirror` is still unproven,
having never executed. The most likely next wall is `build-disk` —
`cozystack-nocloud` is a brand-new package, so the first `oras push`
creates it, and the follow-up `oras tag` can hit `403 permission_denied`
until the package is linked (the risk the workflow already documents
in-line). The first-ever full e2e run may well surface its own issues.
Reviewing that follow-on separately seems better than growing this PR.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved nightly artifact mirroring and staging reliability.
* Prevented artifact pull failures by ensuring required output
directories are created before use.
* Enhanced retrieval of artifact version and source revision metadata.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
PRs scoped to these three components fell through to
area/uncategorized because the scope map had no entry for them.
metrics-server serves the monitoring pipeline; cert-manager and
external-secrets are platform services.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Nightly workflow has never completed a run. Every scheduled run since
NIGHTLY_ENABLED was set has failed in `mirror` at "Resolve source commit
and version", ~80s in:
level=fatal msg="unsupported image-specific operation on artifact with
type \"application/vnd.cncf.flux.config.v1+json\""
cozystack-packages is a Flux OCI artifact, not a container image: its
config mediaType is application/vnd.cncf.flux.config.v1+json. Plain
`skopeo inspect` parses the config blob as an image config and fatals on
any non-image artifact, so resolving the digest with
`inspect --format '{{.Digest}}'` could never work.
Fetch the :main manifest once with `inspect --raw` and derive both the
digest (`skopeo manifest-digest`, which just hashes the manifest bytes —
no image-specific parsing) and the revision annotation from those exact
bytes. This also tightens the race the original comment describes: the
digest and the annotation now come from a single fetch, so there is no
window at all between pinning the tag and reading its revision, and one
registry round-trip is saved.
Because the workflow never ran past this step, the rest was unproven.
Also fix both `flux pull artifact` calls, which passed an --output
directory that does not exist and would have failed with "invalid output
path": create it first, matching the established idiom in
pull-requests.yaml ("Pull current-main packages tree").
Verified against the live OCIR artifact: the fixed step resolves
packages_ref to the same digest the registry reports and yields main's
current HEAD sha; the fixed pull extracts the tree. skopeo copy on Flux
artifacts, the flux push digest awk parse, the cozy-installer chart name
and nightly-mirror.sh's per-image digest verification were each checked
and are correct as-is.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three scopes recur across PR history with no fitting area/* label
(keycloak, tenant, kamaji). Add the labels and map their scopes in the
auto-labeler, and fill two gaps in existing mappings (mongodb ->
area/database, csi -> area/storage) so these stop landing in
area/uncategorized.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cozystack is upstream for repositories that nothing keeps in sync with
it, and no CI job compares the two sides, so a change here breaks them
in silence. Only two narrow paths are automated, and both fire on a
release rather than on a PR.
It has already happened. The website docs generator works from an app
list hardcoded in its own Makefile, and packages/apps/opensearch was
never added to it, so an application we ship has no reference page at
all. The Terraform provider still offers a Kubernetes version this repo
removed. The plugin repo still sends operators to hack/ scripts that
were deleted, one of them rewritten as a Go command.
Add a Downstream Repositories trigger map to the contributing guide,
naming per repository which change here forces a change over there and
which file to touch, and a checklist in the PR template that links to
it. The template addresses AI agents directly: they author a large share
of PRs and will otherwise tick a box without opening the follow-up it
claims.
The coupling is wider than the obvious ones, and mostly invisible. talm
restates the node contract, overlapping ours but already diverged, so it
must be diffed rather than overwritten. cozyhr reads an annotation the
operator writes and silently merges the wrong values if it is renamed.
cozy-proxy is vendored in from its own repo, yet consumes labels
produced here and quietly stops handling external VM services if they
move. The telemetry server queries metric names emitted here, and a
rename just empties its dashboards. None of these fail loudly.
Guard the map with a unit test, because a map about silent drift must
not drift silently itself. It pins every in-repo path the map cites to a
file that exists, pins the map and the checklist to the same repository
list, and pins the workflow carve-out below to the map's location, to
the job that reads the flag, and to the output that carries it: cutting
any one of those three leaves the other two looking wired up. The header
states plainly what the suite cannot check, so a green tick is not
mistaken for a correct map: whether a coupling is real, and whether it
still holds over there, stays a human's job.
Editing the map alone is a docs-only PR, which skips the unit tests and
would exempt the map from its own guard. Rather than widen the docs-only
rule, which also gates the Talos image build, the plan step exports a
separate flag that only the unit-test job reads: a docs PR now runs the
tests and still builds nothing. It detects the map by its old path too,
since a rename is reported as the new path alone.
Document that gh pr create --body and --body-file replace the body
wholesale, which would otherwise drop the checklist from every
CLI-authored PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Migrates the E2E **app** test suite from BATS to [Kyverno
Chainsaw](https://github.com/kyverno/chainsaw) — a declarative,
Kubernetes-native E2E framework (CNCF, part of the Kyverno project;
successor to KUTTL) — and wires it into CI as the replacement for the
per-app BATS loop. This grows the original pilot (postgres + bucket)
into the full suite.
All 22 app suites are ported under `hack/e2e-chainsaw/` and the
`hack/e2e-apps/*.bats` files are removed:
`postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`,
`clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`,
`external-dns`, `kuberture`, `vminstance`, `gateway`,
`kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`,
`kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`.
**Approach**
- **Declarative** suites assert on `status.conditions` and concrete
fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait`
pair that appeared ~100 times collapses into a single `assert` that
polls existence and state together, with a structured diff on failure
and automatic `events`/`describe`/`podLogs` capture via `catch`
(previously only `harbor.bats` did this, by hand).
- **Imperative** suites keep their logic in `script` steps: `openbao`
init/unseal, `kuberture` external-dns split-horizon probes,
`vminstance`, the `gateway` admission/impersonation cases, and
`kubernetes-latest`/`previous`, which wrap the relocated
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up,
LB/NFS/ouroboros checks).
- `gateway` tests derive the tenant apex from the namespace
`namespace.cozystack.io/host` label at runtime, so they are
host-independent.
**CI**
- The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new
`test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`.
- The `chainsaw` binary is added to the e2e-sandbox image.
- `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap +
OpenAPI checks).
- The per-app 3-retry loop is dropped — assertion polling replaces the
fixed-timeout flakiness it papered over.
**Validation**
Ran against a development cluster: the DB/app, storage, and VM suites
pass; the three suites that depend on platform features not present on
that cluster (`gateway`, `kuberture`, `external-dns`) and the two
heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a
freshly installed platform.
Note for reviewers: service-port asserts use the `(ports[*].port)`
projection form rather than a number-literal filter (`` ports[?port ==
`N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal
comparisons.
**2026-07-09: reconciled with main (~520 commits of drift)**
- etcd suite re-ported to the v1alpha2 operator contracts (#2859):
`readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag
contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip
driving `examples/backups/etcd`
- 12 new `ingress-hostname-policy` gateway cases ported (apex derived at
runtime from the tenant-root namespace label)
- post-branch bats drift folded in: `mariadb-single` webhook guard,
kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket
readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant
PVC-reclaim guard (#3059), kuberture fail-loud negations,
SC-fallback-default test (#2872 B1)
- 4 suites that only exist on main since the branch was cut are ported:
`kubernetes-oidc-system`, `kubernetes-oidc-customconfig`,
`securitygroup`, `serviceexposure`
- CI seams: `nightly.yaml` converted to the chainsaw invocation,
`run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR
pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh`
replicated in the Chainsaw global catch
### Release note
```release-note
fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* End-to-end testing is now driven by a unified Chainsaw suite (platform
apps, Kubernetes, networking, security groups, service exposure).
* Test Impact Analysis selects affected suites, with an option to run
the full suite.
* CI produces a JUnit report artifact and collects suite-scoped
diagnostics on failure.
* **Bug Fixes**
* CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and
sandbox bucket state diagnostics.
* Improved teardown reliability with bounded wait + finalizer handling
to avoid hangs.
* **Documentation**
* Updated E2E testing docs and local guidance to reflect the new
Chainsaw-based workflow and suite paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Reconciles the talos-image-cache conflicts created when #3254 merged to
main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh,
a superset of #3254's logic + the node-join diagnose helper), drop the
BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path
versions of the mirror manifest, install bats, and unit test. Also picks
up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and
Talos v1.13.6 (#3240).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Finishes the runner decommission tracked in #2937. PR builds, the main
cache warmer, nightly, promote-rc and pull-requests-release already run
on the ephemeral pool; this moves the stragglers so no workflow depends
on the persistent runner anymore:
- tags.yaml prepare-release: large ephemeral shape (same as the cache
warmer), flux CLI bootstrap, and read-only OCIR access so release
builds warm-start from the shared mode=max build cache while still
pushing images to GHCR. Without the cache wiring every tag build on
an ephemeral runner would be 100% cold.
- tags.yaml generate-changelog / update-website-docs, backport.yaml,
update-releasenotes.yaml: GitHub-hosted runners — these jobs only
need git and the GitHub API, no docker or repo toolchain.
- pull-requests.yaml: the debug label no longer reroutes jobs to the
self-hosted runner (that path would queue forever once the runner is
gone). The label keeps gating the SSH breakpoint on e2e failure,
which works from ephemeral runners too.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Cosmetic follow-ups from the re-review (all fail-closed, no behavior change):
- Write-access gate: drop the inert 'maintain' from the allowed list.
getCollaboratorPermissionLevel's `permission` field returns only the legacy
roles admin/write/read/none (maintain collapses to write), so admin+write is
the full "can push" set; 'maintain' never matched.
- Stale-tip guard: resolve the remote tip without a pipe masking ls-remote's
exit code, mirroring the write-once query — a transport failure now reports a
distinct error instead of "branch not found".
- tags.yaml: the hand-pushed-stable rejection message now says stable tags are
finalized by the promote flow at the merge commit (pull-requests-release.yaml),
not "come from promote-rc.yaml" — promote-rc opens the PR; finalize cuts the tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Non-blocking review notes from #3253:
- Require write access to dispatch. workflow_dispatch already needs write
access to trigger, but add an explicit getCollaboratorPermissionLevel gate
(admin/maintain/write) as auditable defense-in-depth that fails closed.
- Constrain a `main` dispatch to a new minor's pre-release (patch Z == 0). A
patch-line pre-release built from main content would produce a wrong-tree
release-X.Y.Z-rc.N staging branch; it must be cut from its release-X.Y line.
- Guard against a stale checkout tip. workflow_dispatch pins the dispatch-time
SHA, not the push-time branch tip (the gap includes the concurrency-queue
wait); if the branch advanced, the push would tag a stale commit and yield an
empty base_ref, failing tags.yaml after burning the write-once name. Compare
local HEAD to the remote branch tip and fail with a re-run hint instead.
- Distinguish ls-remote "tag absent" (exit 2) from a transport/auth failure
(any other non-zero), so an infra blip cannot read as "free to create".
- Shallow checkout (fetch-depth: 1, no fetch-tags) instead of a full-history
clone to push one ref; drop the redundant local tag pre-check.
- Run pure-input validation (tag + dispatch branch) before minting the token
or checking out, so a bad dispatch fails in seconds.
actionlint + zizmor clean; run blocks bash -n checked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The zizmor "Audit workflows" gate (added in #3223) is red on main: five
actions/* references in the release workflows from #3017 are pinned to a
floating tag instead of a commit SHA (blanket unpinned-uses policy, High
severity):
- nightly.yaml (x3) actions/checkout@v4
- promote-rc.yaml actions/checkout@v4
- retention.yaml actions/create-github-app-token@v1
Pin each to the SHA the rest of the repo already standardizes on:
- actions/checkout -> 34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- actions/create-github-app-token -> d72941d797fd3113feb6b93fd0dec494b13a2547 # v1
`zizmor --offline .github/workflows/` now reports no findings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Pre-release tags (vX.Y.Z-alpha.N / -beta.N / -rc.N) were cut by a manual
`git push origin HEAD:refs/tags/<tag>`. That leaves the immutable-tag work in
#2677 only partly enforceable: a repo ruleset cannot restrict `v*` tag
*creation* to CI while still letting maintainers hand-push pre-releases (fnmatch
has no way to match "v* except v*-rc.*"), so "do not hand-cut stable tags" stays
a matter of discipline backed only by an in-workflow guard.
Add cut-prerelease.yaml — a workflow_dispatch that becomes the sole entry point
for creating a pre-release tag:
- Generates the CozyStack CI app token (same app as promote-rc.yaml) and pushes
the tag as the app, so tags.yaml fires (a tag pushed with the default
GITHUB_TOKEN would not trigger it) and the push-event base_ref is populated
(tags.yaml's Get-base-branch step needs it).
- Validates ^v\d+\.\d+\.\d+-(alpha|beta|rc)\.\d+$ and rejects stable vX.Y.Z —
stable tags are cut write-once by the promote flow at a PR merge commit.
- Enforces dispatch from main or the matching release-X.Y branch, and refuses to
move an existing tag (write-once).
- Pushes HEAD:refs/tags/<tag> from the branch tip — the exact form
docs/release.md prescribes — on ubuntu-latest (no build runner needed).
With every pre-release cut by CI, repo admins can then enable a "Restrict
creations" tag ruleset on `v*` with the CI app as the sole bypass actor, turning
tag-creation discipline into hard enforcement.
Also update tags.yaml's two human-facing error messages and docs/release.md (the
RC and patch flows, the workflow list, and a new repo-side tag-ruleset
enforcement note) to point at the workflow instead of a manual push.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three sandbox QEMU guests each run with `-smp 8`, demanding 24 vCPU
in total. On the 24-vCPU runner that is a 1:1 CPU commit with no host
headroom for QEMU's emulation/IO threads, vhost-net, the host NAT every
guest image pull traverses, containerd and the kernel. Under the peak
concurrency of "Install Cozystack" the guests lose cycles to CPU steal,
so in-guest readiness/liveness probes miss their deadlines and a
HelmRelease intermittently flips InProgress -> Failed with dependents
cascading to DependencyNotReady — on a different component each run, the
signature of resource thrash rather than a product bug.
Move the job to the 32-vCPU / 128 GiB runner class. Guest sizing is
unchanged (24 vCPU / 72 GiB), so the install does not slow down, while
the host keeps 8 vCPU and 56 GiB of headroom that absorbs the emulation
and network overhead the probes were starving against.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed
test-apps-% target; switch it to the single chainsaw invocation
(empty CHAINSAW_SUITES = full suite) and upload the JUnit report
- pin the chainsaw-report upload-artifact step to the same SHA the
rest of the workflows use (scorecard/zizmor gate)
- repoint hack/run-kubernetes-drain_test.bats at the relocated
hack/e2e-chainsaw/_lib/run-kubernetes.sh
- replicate cozytest.sh's NotReady-pod data-plane capture
(hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so
chainsaw-suite failures keep diagnostic parity with the BATS lanes
- refresh stale bats path comments
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Restores the two OpenSSF Scorecard checks that dropped the aggregate
score below our target, and adds a durable gate so they cannot silently
regress again.
**Root cause.** 15 of 17 workflows already declare a read-only top-level
token and pin their actions by SHA. Two did not — `build-main.yaml` and
`ui-test.yaml` (the latter added by the in-tree console move) — and that
was enough to zero out two checks:
- **Token-Permissions** is scored on the weakest workflow, so a single
workflow with an undeclared (default-write) top-level token drops the
whole check to `0`.
- **Pinned-Dependencies** counted the six tag-pinned action refs as
unpinned.
**Fixes**
- `build-main.yaml`, `ui-test.yaml`: add a top-level read-only
`permissions` block (job-level `packages: write` kept where the build
cache needs it) and pin every action to the SHA already used elsewhere
in the tree.
- `.github/zizmor.yml` + `.github/workflows/zizmor.yml` +
`.pre-commit-config.yaml`: add [zizmor](https://docs.zizmor.sh/) as a
gate. It runs on the PR diff (pre-commit) and tree-wide on every
workflow-touching PR, failing on any reintroduced unpinned action or
missing/overly-broad permissions block. Scope is intentionally limited
to the two Scorecard-relevant audits (`unpinned-uses`,
`excessive-permissions`), which are clean tree-wide as of this PR; the
remaining audits carry pre-existing debt and are disabled with a
documented TODO to burn them down and re-enable one at a time.
- `scorecard.yml`: wire an optional `SCORECARD_TOKEN` secret. The
default `GITHUB_TOKEN` cannot read branch-protection settings, so the
Branch-Protection check is currently scored on partial data. The token
is a no-op until the secret is added by a maintainer.
- `internal/fluxshardoperator/config_fuzz_test.go`: native Go fuzz
targets for the shard config parsers (round-trip and range invariants,
not just no-panic), giving the Scorecard Fuzzing check a real signal on
a hot-path parser.
**Verification**
- `zizmor --offline .github/workflows/` → *No findings to report* (gate
green).
- `go test ./internal/fluxshardoperator/ -run '^Fuzz'` passes; a 15s
live fuzz of `FuzzParsePinnedTenants` ran 10.5k execs with no crashers.
Follow-ups (tracked separately, not in this PR): merge Renovate's
digest-pinning PRs to lift Pinned-Dependencies further, tighten branch
protection on `main` and release branches, upgrade the OpenSSF Best
Practices badge, and sign releases.
### Release note
```release-note
ci: pin GitHub Actions by SHA and scope workflow token permissions to restore the OpenSSF Scorecard Token-Permissions and Pinned-Dependencies checks, and add a zizmor gate (pre-commit + PR workflow) that blocks future regressions
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added automated security scanning for workflow changes via a dedicated
CI workflow, along with a matching pre-commit hook.
* **Bug Fixes**
* Hardened CI workflows with workflow-level read-only token permissions
and pinned GitHub Action versions for more reliable execution.
* Improved OpenSSF Scorecard configuration to better handle
branch-protection checks when a token secret is available.
* **Tests**
* Added Go fuzz tests to ensure shard parsing/formatting remains
non-panicking and preserves round-trip/contract behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Implements the immutable-tag + rc-promotion flow from #2677. A **stable
release becomes a renamed release-candidate**: the bytes shipped as
`vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as
`vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt —
it is *promoted* by retagging the rc's existing images.
Layered onto the build matrix from #2937/#2983 (this PR is **stacked on
`refactor/build-matrix-2937`** and must merge after it).
## The five force-retag sites, removed
| Site | Before | After |
|---|---|---|
| `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` |
write-once (create-if-absent, fail if it would move) |
| `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` |
compare-before-force (no-op if unchanged; staging branch only) |
| `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` |
write-once at the PR merge commit (force impossible by construction) |
| `pull-requests-release.yaml` maintenance branch | `updateRef
force:true` | fast-forward-only |
| `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow
deleted** — stable only via explicit promote |
## Version decoupling (the enabler)
The operator baked its version into the image at build time, so an rc
image self-reported the rc string — blocking retag-promotion. It now
reads `COZYSTACK_VERSION` from the environment (threaded via
`cozystackOperator.platformVersion` → Deployment env, stamped by `make
manifests`), falling back to the build-time value when unset. The same
image bits can report any release name. The only runtime reader is the
telemetry metric `cozy_cluster_info{cozystack_version=...}`.
## Promotion flow
`promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`):
1. Validate the rc release exists and the stable tag does not.
2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from
`packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to
`:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`.
3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable
version (the `@sha256` wins regardless), restamp the version-stamped
assets (operator manifests, cozypkg, openapi; the heavy Talos assets are
copied verbatim from the rc draft), open a `release-X.Y.Z` PR.
4. Merging the PR reuses the existing release-PR e2e and
`pull-requests-release.yaml` finalize to cut the **write-once** stable
tag at the merge commit and publish the release. Squash is disallowed
(the tag needs a real merge commit).
`nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by
`NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line
(dry-run by default).
## Validation
**Local end-to-end — the core claim is proven, not asserted.** The whole
design rests on "copy-by-digest to a new tag preserves the digest, so
stable == the e2e-tested rc, bit-for-bit." This was exercised against a
real registry (`ttl.sh`) with real `skopeo`:
- Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp
tree exercising all three `values.yaml` digest shapes (single `image:`
string, split `repository`/`tag`/`digest` map, and the
`platformSourceRef` OCI artifact), then ran the actual
`hack/promote-retag.sh v1.4.0`.
- **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the
identical digest** (`sha256:fd8d9aa6…`), across all 17 platform
manifests. The script's own `skopeo inspect` post-check passed, and an
independent re-inspection confirmed it.
Local validation caught (and this PR fixes) two bugs that only surface
against a real registry — never in static lint:
1. `skopeo copy --multi-arch all --all` — mutually exclusive flags,
fatal error; every retag would have failed on CI. Now `--multi-arch
all`.
2. The same digest was retagged twice when an image appeared in two
value shapes; the ref set is now deduped on the canonical `repo@digest`.
**Other local checks:** `go test ./pkg/version` + `go build`/`vet`;
`helm template` renders `COZYSTACK_VERSION` in all 3 operator variants
(omitted when unset); `make manifests` stamps the version into the
install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint`
clean and `act -l` resolves the job graph on all workflows; the rc-tag
parser and nightly version-math unit-tested (accept/reject +
minor-vs-patch bumps).
**Still CI-only (cannot be exercised offline):** the full workflow
runtime — github-script API calls, OCIR auth, the rc-release /
staging-branch preconditions, the nightly `base_ref` push mechanics, and
self-hosted runners. The *logic* inside the steps is unit-tested; the
*orchestration* is not. Treat it as unproven until the workflows run.
## Out of scope / required follow-up (needs repo admin — not doable from
a PR)
- Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by
the CI app, no delete/update; "limit branches/tags updated in a single
push" = 1.
- Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo
variables.
- Registry-side pruning of `*-nightly.*` image tags (no OCIR delete
parity yet — tracked TODO in `retention.yaml`).
## Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added nightly publishing (mirror-by-digest, disk build, e2e
validation) and nightly retention pruning.
* Added RC-to-stable promotion via digest retagging (no rebuild).
* Operator/installer and console now expose version/platform metadata,
including runtime override via `COZYSTACK_VERSION`.
* **Bug Fixes**
* Enforced write-once tag behavior and fast-forward-only maintenance
updates across release workflows.
* Made nightly mirroring/selection and retention pruning more selective
and safer.
* **Tests**
* Added Bats coverage for nightly mirroring and RC retagging, plus new
Helm/unit checks.
* **Documentation**
* Updated the release model to center RC promotion and tag immutability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pin the remaining floating version tags in the build-main and ui-test
workflows to immutable commit SHAs, preserving the original tag in a
trailing comment so Renovate can keep the pins up to date.
Supply-chain hardening against tag hijacking/mutation; no functional
change — each SHA is the commit the tag currently resolves to.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Review feedback (non-blocking):
- .pre-commit-config.yaml pinned zizmor at v1.22.0 while the CI gate installs
1.26.1; the zizmorcore/zizmor-pre-commit rev is the zizmor version 1:1, so the
gap meant local and CI runs could differ. Align the hook to v1.26.1.
- ui-test.yaml's checkout did not set persist-credentials: false, unlike the
other two workflows hardened here (build-main.yaml, zizmor.yml). Set it so the
token is not left in .git/config for the later pnpm install step to read.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build-main.yaml and ui-test.yaml (the latter added by the in-tree console
move) landed without following the repo's workflow conventions: no top-level
permissions block and actions referenced by moving tags. Together they zero out
two OpenSSF Scorecard checks the rest of the tree already satisfies:
- Token-Permissions: scored on the weakest workflow, so one workflow with an
undeclared (default-write) top-level token drops the whole check to 0.
- Pinned-Dependencies: the six tag-pinned action refs count as unpinned.
Fix both files (top-level read-only token; job-level packages:write kept where
needed; all action refs pinned to the SHA already used elsewhere in the tree)
and add a durable gate so it cannot recur:
- .github/zizmor.yml enables the two Scorecard-relevant audits (unpinned-uses,
excessive-permissions), clean tree-wide as of this commit. The remaining
audits carry pre-existing debt and are disabled with a TODO to burn down and
re-enable one at a time.
- .pre-commit-config.yaml runs zizmor (offline) on changed workflows.
- .github/workflows/zizmor.yml runs the same audit tree-wide on every PR that
touches a workflow, failing on any reintroduced regression.
Also wire scorecard.yml to an optional SCORECARD_TOKEN secret: the default
GITHUB_TOKEN cannot read branch-protection settings, so the Branch-Protection
check is scored on partial data. The token is a no-op until the secret is added.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address review feedback from lexfrei on .github/workflows/promote-rc.yaml:200.
The "Prepare stable branch" step built the release-X.Y.Z branch with
`git checkout -B` + `git commit -s` (a fresh, timestamp-dependent SHA every run)
and pushed it with a plain non-force push. On a re-dispatch — retrying after a
later step failed, or promoting a newer rc to the same version — the new commit
is a sibling of the already-pushed one (same parent, different SHA), so the push
is rejected non-fast-forward and promotion wedges until the branch is deleted by
hand. That contradicts both the validate step (which tolerates a leftover draft
precisely so a newer rc can be re-promoted) and the workflow header's
"overwritable branch, nothing to wedge a later re-promotion".
Reuse the compare-before-force pattern from tags.yaml's Create release branch
step: skip when the remote already matches, force-push + log when the branch
genuinely moves. It is a mutable staging branch, so overwriting it is correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- nightly.yaml: pass the registry tokens to `skopeo login` over stdin
(--password-stdin) instead of `-p` on argv, matching promote-rc/finalize and
keeping the token out of the process list.
- dashboard configmap: require digits-then-dot (^v?[0-9]+\.) when deriving the
displayed platform version from the console image tag. The old ^v?[0-9] let a
tagless, port-carrying ref (ghcr.io:443/.../console, whose last ":"-segment is
"443/.../console") pass as a version. Verified: v1.5.0 / 1.5.0 still surface;
the port ref and :latest no longer do.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The tag workflow assumed the old hand-pushed-stable flow in several places that
break (or misbehave) under rc-promotion:
- Drop PUBLISH_FLOATING from the Build step. This job now only ever builds a
prerelease (a stable tag skips via its pre-existing draft, or is rejected —
below), so moving :latest here would track the newest rc, and an rc on an old
line would drag :latest backwards. :latest belongs to promotion (finalize).
- Publish rc releases. tags.yaml created the rc release as a draft and nothing
ever published it, so the community could not see the rc page or download its
nocloud-amd64.raw.xz — defeating the rc's stated purpose. Publish it
(prerelease:true, make_latest:false) once assets are uploaded; rc tags are
write-once, nothing to protect with draft status.
- Reject hand-pushed stable tags. A stable vX.Y.Z with no pre-existing draft is
a hand-push down the legacy full-rebuild path (full rebuild, moved :latest,
rogue release PR). Fail fast with a message pointing at promote-rc.yaml. This
replaces two now-unreachable steps that shared the same gate: the submodule
Go-module tag (moved to finalize in the previous commit) and "Create pull
request if not exists" (promote-rc owns PR creation).
- Gate generate-changelog and update-website-docs on stable (ref_name has no
'-'), so each rc no longer opens a changelog-vX.Y.Z-rc.N PR that never merges
and an "update managed apps for vX.Y.Z-rc.N" website PR.
- Paginate the release lookups (check_release, draft-reuse): a promote-created
draft can sit past the default first 30 and be missed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promotion mutated the registry at dispatch time — retagging ~37 images to
vX.Y.Z (+:latest), publishing the stable cozy-installer chart — before the
promote PR's e2e ran and before any merge. An abandoned promotion then left
stable-named bytes for a version that never shipped, and the (correct) image
write-once check wedged any later re-promotion to the same version.
Move every irreversible registry side effect into pull-requests-release.yaml
finalize, which runs only after the promote PR merged (= after its e2e passed),
making promotion atomic. promote-rc.yaml now only stages the release-X.Y.Z
branch, drafts the stable release, and opens the PR; an abandoned promotion
leaves nothing but an overwritable draft + branch.
Folded into the same restructure:
- Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag in finalize, at the stable
tag's commit. It was gated in tags.yaml on `release_exists==false &&
is_stable`, unsatisfiable under promotion (the draft always pre-exists), so
promoted stables never got their submodule tag — a regression for Go
consumers. Finalize is where the stable tag is born, so it cuts the matching
submodule tag write-once (stable only).
- Gate :latest by max-semver. promote-retag.sh now moves :latest only when
MOVE_LATEST=1 (default off); finalize passes it the same make_latest decision
the release gets, so a patch on an older line moves neither the release's
`latest` nor the images' :latest. Added MOVE_LATEST bats coverage.
- Stamp platformVersion into the published stable chart's default values
(uncommitted, at package time) so `helm --install --version X.Y.Z` reports the
stable version in telemetry instead of the rc version baked into the operator
image. Not committed to the branch — that would leak platformVersion=vX.Y.Z
onto main and mis-stamp the next rc.
- promote-rc: serialize all promotions (static concurrency group, not per-rc);
label the PR full-e2e so the full suite runs; scope the rc->stable sed to the
depth-2 values.yaml glob plus the kubernetes .tag files (the build's own write
set); tolerate a leftover draft so re-promotion is not wedged.
- finalize: assert the PR author is cozystack-ci[bot], and run on an ephemeral
oracle-vm runner (needs skopeo/helm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The first-ever `oras push` of cozystack-nocloud creates the GHCR package,
but without org.opencontainers.image.source it is created unlinked to the
repo, and the follow-up `oras tag … nightly` can 403 permission_denied:
write_package until someone links it by hand. Set the annotation on push so
the package is linked from the start (matches the --label the image builds
already carry).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Addresses two blockers that stop a stable release from being cut by the
promote flow (both reproduced against a real registry in review).
B1 — wrong registry: the workflow forced env.REGISTRY to the OCIR CI
registry and logged in only to OCIR, but release-tag images (rc and
stable) are built and pushed to GHCR by tags.yaml (hack/common-envs.mk
default). promote-retag.sh filters refs by the $REGISTRY/ prefix, so with
REGISTRY=OCIR every GHCR ref was dropped, the selection went empty, and
the retag step exited 1 before doing anything. Point the job at
ghcr.io/cozystack/cozystack (matching the script default) and log in to
GHCR via --password-stdin (also resolves the token-on-argv nit).
B2 — missing chart: promote-retag.sh only retags image refs found in
values.yaml, and the cozy-installer OCI chart is a separate artifact in no
values.yaml. make build (which pushes the chart) is gated on a
not-yet-existing release, so a promoted stable never published
cozy-installer:X.Y.Z — the documented `helm --install … --version X.Y.Z`
404'd with manifest-unknown. Repackage the chart from the restamped stable
tree (no rebuild — it references the retagged images + packages artifact by
digest) and helm push it, mirroring the nightly flow. Adds helm to the job.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The PR Auto-Label workflow triggered on pull_request_target 'edited',
which fires on every PR title *and body* edit. Long-lived bot PRs such
as Renovate repeatedly edit their description on schedule, causing the
label job to re-run dozens of times against an unchanged commit (e.g.
~48 runs on #2998, all on the same head SHA).
Drop 'edited' so the job only runs on opened/reopened/synchronize.
Labels derived from the PR title are re-applied on the next push or on
reopen. Title-only edits between pushes no longer trigger a relabel,
which is an acceptable trade for eliminating the churn.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two issues flagged in review on promote-rc.yaml:
- The "Validate rc is promotable" github-script called core.setFailed()
but kept running: a missing rc release then hit `rcRel.assets` and a
missing disk asset hit `disk.id`, throwing a TypeError that masked the
intended failure message. Return immediately after each setFailed so the
clean diagnostic is what surfaces.
- "Create draft stable release and upload assets" ran `make upload_assets`
without COZYSTACK_VERSION. upload_assets depends on `manifests`, which
re-renders the operator manifests; with no version it blanks the
platformVersion, silently undoing the stable stamp the Restamp step just
applied. Pass COZYSTACK_VERSION=${STABLE_VERSION} so the re-render
re-stamps the stable version.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/tags.yaml: the
single `skip` output meant 'a release already exists' but was reused to gate
update-website-docs, so a promoted stable tag (whose draft promote-rc already
created) skipped the website-docs job, not just the rebuild. Rename the output
to release_exists to reflect its real meaning and drop it from the
update-website-docs gate so a promotion still publishes docs (the rebuild steps
inside prepare-release stay gated on it).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on
.github/workflows/pull-requests-release.yaml: the fast-forward catch block
treated every updateRef failure as a recoverable warning, so an auth, API, or
malformed-ref error would silently leave the maintenance branch stale. Warn
only on the 422 non-fast-forward rejection and fail the job otherwise.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/promote-rc.yaml:
the rc staging checkout left the app token in .git/config even though the
Prepare stable branch step re-injects it via 'git remote set-url'. Set
persist-credentials: false to drop the redundant exposure.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/promote-rc.yaml:
the validate step only checked for an existing stable release, so a stray
refs/tags/vX.Y.Z (partial run or manual push) would slip through and fail later
in finalize — after promote-retag had already retagged images. Also fail early
when the stable git tag already exists.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/promote-rc.yaml:
actions/create-github-app-token was used at the floating @v1 and without a
repositories restriction, so the minted token carried owner-wide access. Pin it
to the SHA the rest of the repo uses and scope it to this repository.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/nightly.yaml:
actions/checkout writes the token into .git/config by default, and these jobs
copy the checkout into /tmp/$SANDBOX_NAME before running repo targets. Set
persist-credentials: false on all three source checkouts so the token is not
carried into the sandbox copies.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/nightly.yaml:
the version was 0.0.0-nightly.<date>, so a same-day re-dispatch or re-run
republished every image, chart and artifact under the same tag — the dated
nightly was not write-once. Append the run id and attempt so each build gets a
distinct, immutable tag, and widen retention.yaml's selector regex to match the
extra numeric components (still excluding the floating :nightly and any release
tag).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/retention.yaml:
gh api --paginate with --jq runs the filter once per page and emits one JSON
array per page, so total/drop_ids miscount once a package's versions span more
than one page. Pipe the raw pages through 'jq -s add' to merge them into a
single array before counting and slicing.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on .github/workflows/nightly.yaml:
the resolve step read cozystack-packages:main and the pull step re-fetched
the mutable :main tag, so a concurrent build-main push could check out commit
A while mirroring tree B. Resolve the :main digest once, derive the revision
sha from it, and pull the baked tree by that same digest.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Match the repo-wide SHA pin main applies to every github-script use, so
the new promote-rc workflow doesn't regress the Scorecard pinned-dependencies
check.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Nightlies are now GHCR OCI tags (0.0.0-nightly.<date>), not git tags or
GitHub releases, so:
- retention.yaml now prunes old GHCR container-package versions via the
Packages API instead of git tags/releases. Scoped to versions whose
tags are ALL 0.0.0-nightly.<YYYYMMDD>, so the floating :nightly tag,
release tags (vX.Y.Z, latest, rc, main) and untagged versions are
never touched. Keeps newest 14 per package; dry-run by default.
- nightly-rc.yaml is removed: it cut nightly *git tags* that drove a
release/e2e path we no longer use. The nightly.yaml workflow replaces
it with the mirror + full-e2e flow.
actionlint clean.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Scheduled nightly that publishes an installable nightly of main without
rebuilding it, then runs the full e2e suite against the published bits:
- mirror: resolves the exact commit cozystack-packages:main was built
from, runs nightly-mirror.sh to copy every cozystack-owned component
image OCIR->GHCR by digest, re-publishes the rewritten packages
artifact and the cozy-installer chart to GHCR (0.0.0-nightly.<date>
+ floating :nightly).
- build-disk: assembles the Talos nocloud disk from the upstream imager
(no cozystack-image dependency, no rebuild) and publishes it as a GHCR
OCI artifact (cozystack-nocloud) via oras, so a nightly is installable
on real hardware, not just e2e.
- e2e: stages the GHCR install closure (rewritten tree + chart pinned to
the GHCR packages artifact + the disk) and runs the FULL app suite (no
Test Impact Analysis - a nightly has no diff to scope against).
No GitHub release and no api/apps/v1alpha1 Go-module tag are created.
Gated by the NIGHTLY_ENABLED repo variable. actionlint clean.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The rc->stable substring rewrite ran `sed s/${RC_VERSION}/.../` with the dots
unescaped, so the rc version matched as a regex. Escape regex metacharacters so
the match is literal. Use `git checkout -B` for the stable staging branch so a
re-run resets the local branch instead of failing on "already exists".
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
nightly-rc.yaml's `release-[0-9]*.[0-9]*` glob also matched the per-tag staging
branches tags.yaml creates (release-1.4.0, release-1.4.0-rc.2,
release-1.5.0-nightly.<date>), cutting spurious nightly tags. Anchor the target
selection with a `^release-[0-9]+\.[0-9]+$` regex.
tags.yaml now exposes an is_stable output and gates the write-once
api/apps/v1alpha1 Go-module tag and the auto-opened release PR on it, so an
rc/beta/alpha (is_stable=false) or a nightly (is_stable unset) never pollutes the
module proxy or opens a release PR (#2677 goal 3).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- nightly-rc.yaml: daily cron that cuts a write-once
vX.Y.Z-nightly.<date> tag for each tracked target (main + active
release-X.Y branches) when its HEAD moved since the last nightly. The tag
drives the normal rc build/e2e path. Gated by the NIGHTLY_RC_ENABLED repo
variable so it can merge dark and be switched on deliberately.
- retention.yaml: daily cron that keeps the newest 14 nightlies per line and
deletes older nightly tags + their releases. Delete-only, scoped strictly
to the *-nightly.* namespace (rc/stable untouched). Dry-run by default;
flip the `apply` input or the RETENTION_APPLY repo variable to delete.
Registry image pruning is left as a tracked TODO (no OCIR delete parity).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add the rc->stable promotion path. A stable release is a renamed
release-candidate: the bytes shipped as vX.Y.Z are bit-for-bit the bytes
built and e2e-tested as vX.Y.Z-rc.N.
- hack/promote-retag.sh: reads the digest-pinned image refs the rc baked
into packages/*/*/values.yaml (single-string, split repository/tag/digest,
and the cozystack-packages OCI artifact) and `skopeo copy`s each — by
digest — to the stable tag and :latest. Copying by digest guarantees the
stable image is identical to the rc; the result is verified with
`skopeo inspect`. Validated offline with `sh -n` and the yq extraction
against the current tree; the skopeo path is CI-validated only.
- promote-rc.yaml: workflow_dispatch(rc_tag). Validates the rc exists and
the stable tag does not, retags via the script, rewrites the cosmetic
-rc.N substring in the vendored tags to the stable version (the @sha256
wins regardless), restamps the version-stamped assets (operator manifests,
cozypkg, openapi; the heavy Talos assets are copied from the rc draft),
and opens a release-X.Y.Z PR. Merging it reuses the existing release-PR
e2e + pull-requests-release.yaml finalize to cut the write-once stable tag.
Squash is disallowed on the PR (the stable tag needs a real merge commit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
