mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
11bac54f931682dd94de4081bb9e8a7eef60bef1
129
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov Daniil
|
975ea6acb7 |
ci(workflows): gate Talos builds on their inputs
Classify the rename-blind changed-path list against direct Talos sources and shared build inputs. This lets same-repository pull requests skip the heavy build while removals, renames, and build configuration changes still select it. Fork builds remain unconditional. Preserve the matchbox digest produced by a successful Talos build when finalize overlays base-branch image references. When the job is skipped, bootbox continues to receive the current base reference. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
62f2234aea |
fix(e2e): stop a failed teardown posting a failed test verdict
The teardown ends both merge-gating lanes under `if: always()`, and this branch made it `exit "$cleanup_rc"`. So a `docker compose down` hiccup, a busy zpool or a failed rmdir after a fully passing suite posts the required "E2E Tests" status as a failure, and nothing downstream can tell that apart from a test that actually broke. The runner is discarded when the job ends, so there is no next run for the strictness to protect; a reused host is a developer machine, and an annotation serves it. Loud, not fatal. Two comments described mechanisms that are gone. The breakpoint step still explained reading a soft-red output that this branch removed, so a maintainer would go looking for it; the node-join deadline now fails its suite like any other, which is what makes `failure()` sufficient. The build-talos header still described building the nocloud disk and e2e needing it, when the job builds neither and the container lane downloads no disk -- the profile is compiled by nightly and by the tag build's `make assets` instead, and what a PR still compiles is installer.yaml, which shares every input with it bar the platform and output format. And the platform-package override now states its consequence for both packages it replaces, not only for CDI: the merge-gating lane exercises neither the shipped CDI default nor `drbd.enabled: true`. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
Myasnikov Daniil
|
ad00f35f2d |
test(e2e): correct two comments that describe the opposite of the code
pull-requests.yaml said the QEMU substrate "is not wired into any workflow now" and that prepare-cluster and hack/e2e-prepare-cluster.bats stay in the tree for a nightly lane that will carry the coverage. Both halves are wrong in the direction that costs the most: nightly.yaml and e2e-tag.yaml already run `make prepare-env`, which calls prepare-cluster, so anyone trusting that comment deletes live code and takes nightly and every release-candidate run with it. docs/agents/e2e-testing.md states it correctly, so the tree disagreed with itself. hack/e2e-platform-packages.sh said the lane replaces "only LINSTOR with an otherwise-identical Package". It replaces two, and the kubevirt-cdi one is not identical: it raises the importer memory limit from CDI's 600M to 4Gi and the request from 60M to 256Mi. The consequence belongs next to the override -- the merge-gating lane does not exercise the shipped CDI default, and whether a tenant worker disk imports at 600M is answered by the QEMU lanes, which take no override. Also drops the dead half of `resolve_assets`. It resolved nocloud-amd64.raw.xz and exported disk_id; the container lane boots no disk and nothing in this workflow reads that output any more, so all the lookup could still do was fail a release-labelled PR over an asset it does not use. The job stays -- e2e's release arm gates on its result -- and now proves what is still worth proving, that the draft release exists. e2e-tag.yaml keeps resolving the asset, because it consumes it. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
Myasnikov Daniil
|
186547d835 |
test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos containers instead of QEMU guests, so a tenant worker sits at L2 rather than L3. Measured in CI on commits where both substrates ran the same tree: whole-job wall clock down 24-49 minutes, essentially all of it in Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to 466-809s and stop depending on the runner's vgif flag. The node-join soft-red gate goes with the substrate, which is what it was for. It existed because nested virtualisation on the shared runners degraded far enough that a worker registering in minutes elsewhere could miss any deadline this test could afford; #3513 established that the discriminator was the host's kvm_amd vgif flag, and removing a nesting level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from all four workflows, and a missed deadline is an ordinary red again. The ::warning annotation survives, because the reason a run went red should be legible without opening the log, and so does the 124-versus-everything -else test, because "the workers were slow" and "the wait never ran" send a reader to different places. Done by swapping the substrate inside the existing e2e jobs rather than by promoting a separate container job. Those jobs carry more than a substrate -- the GitHub App token, the report-overrun guard, the images list, the SSH breakpoint -- and promoting a job that never had them would have dropped four features without saying so. Three properties of container mode drive the design, and each fails quietly rather than loudly. machine.kernel.modules is a silent no-op -- kernel_module_spec.go returns early on ModeContainer with no error and no event -- so the host loads openvswitch and zfs before the nodes start and the workflow step asserts both rather than trusting them; a missing module surfaces an hour later looking like a CNI or a storage regression. Docker gives a privileged container its own tmpfs /dev seeded once at creation, so /dev/kvm works but ZFS zvols created later never appear and linstor-csi fails ControllerPublishVolume; the compose file binds the host devtmpfs, which is also why talosctl cluster create docker can never carry LINSTOR. And there is no ARP VIP, so 192.168.123.11 stands in for the QEMU lane's .10. The QEMU machinery stays in the tree and keeps running: nightly.yaml and e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated StorageClass, its Immediate binding mode, the tenant StorageClass propagation that only applies to remotely-accessible classes, and the Cozystack Talos node image with its extensions all keep nightly and release-candidate coverage. None of them is exercised per pull request any more. That is the trade. Nothing on the PR path builds or downloads a nocloud disk now, so build-talos drops the talos-nocloud step and the talos-image artifact. build-talos itself stays: finalize consumes its digest fragment. One property this does not fix, recorded beside the runner class because it will be read off a red run eventually: a container node reports the HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters that with kubelet systemReserved. That corrects the scheduler and not what a pod reads from /proc, so a workload sizing itself from /proc/cpuinfo or /proc/meminfo is configured for the whole runner. Also carried here because the same files carry them: the ghcr.io pull-through mirror removal (see #4007 -- whichever lands first makes the other a no-op), the authoritative HelmRelease readiness gate, the backup-credential preflight, the prepull guard against an image-less container fragment, and folding the two standalone OIDC suites into the latest-version tenant cluster. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e49633208c |
ci(tests): parallelize unit test targets
The BATS half of `make unit-tests` ran as one shell loop over ~60 files, which is a nine-minute serial tail no amount of runner CPU could touch. Each file becomes its own Make prerequisite instead, so `-j4` schedules them, and the workflow asks for `make unit-tests test-controllers -j4 --output-sync=target` in a single step: one make process owns all four runner CPUs, the Helm and Go targets share the same bounded pool rather than waiting for each other, and output stays grouped by target instead of four logs interleaving. The existence check moves into its own prerequisite rather than disappearing. It guards against the wildcard matching nothing, which would otherwise make the whole target succeed by having no work. COZYTEST_TRACE rides per target. It arrived with the serial loop it now replaces, and a target that dropped it would go back to streaming full xtrace for every suite -- so the parallelism guard requires the prefix rather than tolerating it. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
ad62a8d62b |
test(e2e): stop blocking merges on the tenant node-join deadline
The tenant node-join wait no longer discriminates. Nested virtualisation on the shared runners degrades far enough that a guest which registers in minutes on real hardware can miss any deadline this test can afford, so a red on that one wait says as much about the machine the run landed on as about the product, and it teaches a reader to re-run the job. The suite still fails on it. That is what keeps Chainsaw's catch running: the previous-instance container logs, the host snapshot, the data-plane capture and the event dump are all keyed on the test failing, and the report stays honest. What changes is one layer up. The wait writes a marker beside its diagnostics, and a new lane-side gate reads that marker to decide whether the job blocks. It opens only when every failed test in the run carries one, and blocks on anything it cannot attribute: a report it could not read, a failed test the report does not name, a failure with no marker. Only the lanes that block a merge ask it. The two pull-request lanes do; nightly and the tag lane run the same suites and stay hard. A soft pass on the tag lane would ship a release candidate whose tenant clusters brought up no worker nodes, and nightly is the only place an e2e failure reaches anyone who did not open a pull request, since no lane in this tree notifies anywhere. A softened red finishes the job green, so the job's result stops answering whether the suite passed, and both kinds of consumer are handed the decision instead of inferring it. A step gated on the job failing would silently stop running: the SSH breakpoint is that case, and it still opens for a pull request marked for debugging. A job that reports the result would silently start reporting a pass: the required commit status now says the deadline was missed and that the merge is not blocked on it, rather than that the suite passed. The deadline goes to 29m. That is the test's own budget and nothing more: its clock starts after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, while the chart's autoscaler and health-check timers have been running since the Machine was created, so no ordering against them is claimed. The chart's 20m and 30m give a scale to pick against, and a budget over the first leaves room for a remediation cycle without promising that one fits. The operation ceiling both suites give the script rises with the deadline, and the diagnostics phase budget rises inside it, because the collectors gated ahead of the serial console cost more between them than the old budget covered. Both budget guards read their terms from the sources that set them. The e2e job cap rises with them, and its guard holds what a guard can hold: the pair's operation and catch ceilings, and one figure across every lane that runs the suite. It also records that the suites' full summed ceilings have never fit any cap in that file, so the margin above the bound is taken from a measured run instead. The prose that restates the cap elsewhere moves with it, and the sweep that holds the deadline's restatements now reads the figure spelled in words as well as in minutes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
61f7939ab7 |
ci(workflows): make E2E breakpoint opt-in via labels only
Drop the author-association branch from the "Breakpoint on E2E failure" step condition, so a red end-to-end run only opens an SSH breakpoint when the pull request carries the debug or release label. The dropped branch held an ephemeral CI runner open for up to twenty minutes on every red maintainer-authored pull request, whether or not anyone intended to attach. Trim the step's explanatory comment to match: drop the paragraph about the removed authorship path, including its stale claim that only the debug label affects runner selection, since this workflow has no conditional runs-on to begin with. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Myasnikov Daniil
|
04fa8471fe |
docs(release): say what the gates actually guarantee
Three claims that were stronger than the code, plus one legibility fix. The PR gate's comment said a `packages/` edit "on the PR or its base" fails the status before merge. The base half is not true: GitHub fires no `pull_request` event when the base branch advances, it only recomputes refs/pull/N/merge, so a change landing on the base after this went green is never re-verified and the PR merges on the stale result. Checking the merge tree does make base drift visible WHEN the job runs, which is worth keeping — but finalize is what actually catches the drift case, after the merge and before any stable name. Say both, in the workflow and in the reviewer checklist, so nobody reads the green as covering more than it does. "Transactional at every consumer-visible name" is a judgement call dressed as an absolute, inherited from the "this dispatch performs NO registry mutation" it replaced. The dispatch now does write to the registry: the `promotion-*` candidate is a real, publicly listable tag. What it creates is no *stable* name — no vX.Y.Z, no :latest, no stable installer. Stated that way the sentence stops needing the reader to already know which writes were meant. The retention paragraph listed two of the conditions the workflow enforces. It now lists all of them, including the ones the workflow gained here: same-repository, bot-authored, `release-X.Y.Z` branch, the `release` label, exactly one valid digest pin, and protection from a base-branch tip as well as an open PR. Also: `flux pull artifact` reads its credentials from ~/.docker/config.json, which is why finalize now runs `docker login` — but its runner class is not one of the docker-using build ones and the setup step installs everything except docker. Check for it there, where the message can name what is missing, instead of failing with "command not found" halfway down the release path. And the verifier's exclusion list is a restatement of flux's `excludeOCI`, not a derivation — the CLI does not expose it — so the coupling and the pinned version are now written down next to the list. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
9fa8eeeda2 |
ci(release): pin the promotion toolchain and cap the candidate gate
Three review findings on the new promotion steps.
`yq` was fetched from `releases/latest/download` in all three steps this
pipeline installs it in — the promote dispatch, the promote PR gate and
finalize. That makes an unpinned moving dependency part of what decides
whether a stable release may be created, and part of what rewrites the
installer pin: two promotions of the same rc can be served different
binaries. Pin it the way FLUX_VERSION already is, version-gated so a
matching preinstalled yq is reused, on the version the e2e sandbox image
already tracks.
A version authenticates nothing about the bytes it names, though, and
these are the workflows that mint write-once stable names. yq is
therefore also pinned BY CONTENT: its SHA-256 is committed beside its
version and verified before install.
Scoped to yq deliberately. It is the one downloaded binary with write
authority over what a release contains — it stamps the candidate digest
into installer values at promote time and platformVersion into the
stable chart at finalize. skopeo, helm and oras only move bytes that are
already digest-addressed, so a substituted one is caught downstream; a
substituted yq writes the wrong pin and every later check verifies that
pin faithfully.
The Flux installer is left as it is, on purpose. It fetches
flux_${FLUX_VERSION}_checksums.txt from the pinned release and verifies
the binary against it, so the binary is not the exposure — the ~6 KB
bootstrap script from fluxcd.io is, and it is mutable by design and
fetched identically in ten other workflows here. Pinning its bytes in
these three files would break on the next regeneration of a script the
project does not control while leaving the same fetch unpinned
everywhere else. The comment names the residual risk instead.
No precedent existed either way: nothing under .github/workflows/ or
hack/ verified a downloaded tool before this, and every version-pinned
download in the tree (oras 1.2.0, cozyvalues-gen 1.6.0) is unverified.
This is a new convention rather than an application of one, confined to
the release path and justified by write authority, and the comment says
so where the next reader will be standing. The three unpinned fetches
outside this pipeline (nightly.yaml, e2e-fork.yaml) are left alone.
The checksum is trust-on-first-use, which is the honest description: it
fixes the bytes to what was checked when the version was chosen and
makes any later substitution loud. Provenance is the release's own
published checksums list cross-checked against the downloaded asset;
both agree on
fa52a4e758c63d38299163fbdd1edfb4c4963247918bf9c1c5d31d84789eded4, and
that binary self-reports v4.53.3.
Verified locally against the real asset: the version gate skips a
matching preinstalled yq and re-installs a mismatched one, the checksum
line reports OK for the genuine binary and exits 1 for a tampered copy,
which aborts the step under the runner's `bash -e`.
`verify-release-candidate` had no `timeout-minutes`, so it inherited the
6-hour default while `e2e-report` waits on it through `needs` — a
stalled `flux pull artifact` would hold the required `E2E Tests` status
pending for the whole of it. 30 minutes, matching `checks` and
`finalize`.
The contract pins that all three install steps carry both versions and
the checksum, run the checksum check, name no `latest/download`, and
agree on one yq version and one checksum. So the unpinned one-line idiom
cannot come back into a release step by copy-paste from the workflows
that still use it, and bumping the version in one file without its
checksum cannot pass. Both mutations are caught.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
|
||
|
Myasnikov Daniil
|
5d56681f8c |
fix(ci): put every label guard in lockstep, forks included
Promotion needs the `release` label to start work, so this PR added it to the set of labels whose `labeled` event is not discarded. That set is stated in four places and was updated in three. `gate-concurrency-contract.bats` caught the shape of the problem and then became the first casualty of it. It asserted that the concurrency key excludes exactly ONE label from the `-label` group, by counting occurrences of `github.event.label.name != '`. A second publishing label made the count 2, turning the suite red — and with it `checks`, then `finalize`, then `e2e`, then the required `E2E Tests` status. Raising the constant to 2 would re-hardcode the same brittleness one label later and throw away the invariant the test exists for: the set of labels excluded from the `-label` suffix is exactly the set `plan` runs for. Both sets are now extracted from the workflow and compared, so the assertion holds at any number of labels. `resolve_assets` was the guard that got missed. It sits on `e2e`'s release arm, so a label event `plan` runs for and it skips for leaves `e2e` with both arms false — `finalize` is force-skipped for release PRs — while `e2e-report` still runs. On a promote PR already carrying `full-e2e`, re-applying `release` therefore superseded the green run and posted `E2E Tests` = failure, because `full-e2e` bypasses the "e2e not repeated" branch and `E2E_RESULT` is `skipped`. `unlabeled` is not a trigger, so removing the label could not clear it; only a push could. `release` must also not publish on the fork lane. e2e-fork.yaml's `resolve` decides "this run built nothing, so it is not a verdict" by asking whether `Plan build` skipped. Admitting `release` at `plan` made that false: a fork run reaches `plan`, has `build`/`finalize` gated off by its own label, succeeds having produced no pr-patch, and the fork lane fails closed on the missing artifact — red on a suite that was never going to run, on a PR whose only fault is a mislabel. `plan` and `resolve_assets` now admit `release` for same-repo PRs only, and the concurrency key carries the complementary term, so NOT(A OR B OR (C AND NOT F)) stays exactly (NOT A) AND (NOT B) AND (NOT C OR F). For a same-repo PR F is false and every one of them reduces to what it was, so none of this changes non-fork behaviour. The same shape still reaches the fork lane from a PUSH to a fork PR somebody mislabelled `release` — `build` is gated on the label, not on the event, so this predates promotion using the label. Left alone; the comment in e2e-fork.yaml now records it instead of asserting an invariant this PR would have made false. The contract enumerated `plan` and `e2e-report` by name, which is why it did not catch `resolve_assets`: a named-job loop cannot notice a job nobody thought to name. It now derives the job list from the file and requires every job that decides from a label name to decide from the same set, and pins the fork term on both halves of the complement. The file's existing properties are kept — extraction runs over executable-only lines, so a commented-out key cannot satisfy the contract, and every comparison asserts a non-empty input first, so an empty extraction fails closed under both bats and cozytest.sh. Verified by mutation, each caught: excluding a label `plan` skips for, admitting one the key does not exclude, reverting `resolve_assets`, deleting `e2e-report`'s guard, narrowing `verify-release-candidate`'s, renaming the label in one guard only, commenting out the group key, and dropping the fork term from either side alone. Deleting `plan`'s step-level fork guard is caught too, by an assertion that had to stop matching on the bare identifier once `plan`'s own `if:` named it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
c2e145709b |
fix(release): publish stable packages candidate
Serialize the tag-rewritten packages tree during RC promotion and pin its immutable digest in the installer. Verify the candidate against the prospective merge and the original RC artifact before creating stable names. Assisted-By: Claude <noreply@anthropic.com> Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
1abc653cfb |
fix(ci): treat a disabled chainsaw suite as inert in test selection
Editing hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled selects the whole Chainsaw suite. The per-suite rule reads the suite name off the directory and ignores the suffix, so the change selects "backup"; the final intersection against the suites that actually exist finds no such suite and empties; and the safety net for an empty selection escalates to the full run. A disabled suite is registered nowhere and executed by nothing, which makes its edits the one file class that provably cannot regress a test. The mapping had it backwards: they bought the most expensive outcome the selector has. Rare in practice, and paid on every commit by anyone grooming or re-enabling such a suite. Classify *.disabled as inert ahead of the per-suite rule, and record it alongside the other rules that select nothing so the header stays an accurate enumeration. The safety net keeps its job for paths no rule has looked at, which is what it was added for. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a6c94c4528 |
fix(ci): correct an inverted caveat and clean up after the assertion
Two defects introduced by the previous two commits. The in-tree lane's comment said `skip=true` cannot mean "a workflow nobody classified", and then explained, after a colon, exactly how it can: `.github/` is inert as a directory and the e2e workflows are exempted from it by name, so a new one reads as inert. The clause meant to warn a reader instead told them the one unsafe case is safe, in the copy consulted when deciding whether to trust a green gate. The other three copies of the caveat state it correctly. The scratch-directory cleanup landed before the last assertion in every test added here. That passes the EXIT-trap guard and inverts the reason the guard exists: both runners set `-e`, so cleanup placed last is unreachable on failure and the fixture survives for inspection, which `docs/agents/e2e-testing.md` §3 states outright and the already-converted `hack/select-install_test.bats` follows. Placed early, a failing test deletes the evidence of its own failure, and one test had the call between two assertions, so which of them failed decided whether the directory survived. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
55f3c79e9f |
docs(ci): stop pointing at the inert list for paths it does not hold
Four places said an empty selection means the inert pattern matched, and sent the reader to `inert_config_pattern` to see which paths those are. Documentation, dashboards and `*.md` are dropped before that pattern is ever consulted, and a backups example whose app has no suite is dropped where backup examples are matched. A reader checking the pattern for any of them finds nothing and has to reconstruct the order by reading the script. Name the rules that produce an empty selection instead, and keep the claim to the one the lanes rely on: empty is a decision some rule reached, not a path nothing looked at. The same bullet claimed every path is classified. One is not. The inert pattern makes `.github/` inert as a whole directory and the workflows that run the suite are exempted from it by name, so a workflow added under `.github/workflows/` reads as inert and the fall-through never sees it. Today's enumeration is correct and I checked it against `rg -l test-chainsaw`, so this is not a live hole, but the sentence saying it cannot exist is what makes the reminder next to that list look decorative. Record it as the guard it is. Separately, the fork lane's rationale for appending a newline described a mechanism this branch removes: the reader now guards its own loop, so the terminator is no longer what keeps the last path alive. Say that it is now the text-file contract and a second line of defence across two files, which is a reason to keep it and a different reason than the one written there. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
ee4a48278f |
refactor(ci): drop the unreachable empty-selection branch in both lanes
Both lanes ran `if [ "$FULL_E2E" = "true" ] || [ -z "$SELECTED_APPS" ]` under a step gated on `steps.select.outputs.skip != 'true'`. When `select` runs, an empty result sets `skip=true` and the step never starts, so the only way to arrive with no selection is the `full-e2e` label — which sets `FULL_E2E=true`. The second test could never decide anything the first had not, while the comment above it advertised "full suite when the selection is empty" in the file whose whole point is that a green status means tests ran. Testing `FULL_E2E` alone keeps the real invariant visible. The comments now say what an empty selection means after the preceding commit: every changed path was explicitly classified inert, not that the selector had no idea. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
7dca642894 |
fix(ci): restore e2e for fork PRs and close the merge-gate bypass (#3262)
## What this PR does Fixes #3257 — fork PRs never ran e2e, and the skipped required check let them merge anyway. Fork PRs get no secrets on `pull_request`, so `make image` pushed anonymously and died with `denied`. That failed every build, collapsed the `build → finalize → e2e` chain, and left the required **E2E Tests** job *skipped* — which branch protection treats as satisfied, so an approved fork PR could merge with e2e never having run. This implements **Option A** from the issue (privileged `workflow_run` split), pushing fork e2e images to the existing CI registry (OCIR): **Build primitive (`hack/common-envs.mk`)** - New `OCI_EXPORT_DIR` mode: build each image to a per-image OCI archive instead of pushing, and force `PUSH`/`LOAD` off. The digest is captured via `--metadata-file` regardless of output type, so the refs baked into `pr.patch` match what the privileged run later pushes. Threaded through the `image-tags` macro every package already uses — no per-package Makefile change. **`pull-requests.yaml` (unprivileged, fork branch)** - `build` / `build-talos` / `finalize` export images to OCI archives and upload them as artifacts instead of pushing → the red `denied` build wall disappears and the chain no longer collapses. - The in-tree e2e job is renamed off `"E2E Tests"` and guarded to same-repo PRs. A new `e2e-report` job concludes the required `"E2E Tests"` context as an **explicit check-run** (success on a docs-only PR or when e2e passed, failure otherwise). A skipped job can no longer satisfy the required check — this also closes the same hole for same-repo PRs whose build fails. **`e2e-fork.yaml` (new, privileged `workflow_run` from the default branch)** - Resolves the fork PR, opens a pending `"E2E Tests"` check-run, and fails-closed if the fork build failed. - Pushes the pre-built image archives to OCIR **by digest** and flux-pushes the digest-pinned packages tree. The OCIR token is live only for trusted binaries (`git`/`skopeo`/`flux`/`yq`) over data — never for `make` or any fork-authored script. - Runs the e2e suite against the pushed closure with **no push token** and concludes the check-run on the PR head SHA. Fork build code never receives registry credentials; the fork's images and test scripts run only in the credential-less, ephemeral e2e job — a stronger posture than a `pull_request_target` run that would hand fork code the secrets directly. ### Validation Local: buildx OCI-archive output + digest capture, macro expansion for single- and multi-image packages, non-fork push path unchanged, `actionlint` clean. Needs live validation (why this is a **draft**): `workflow_run` artifact download across runs; that the `"E2E Tests"` commit status satisfies the required context and keeps a fork PR non-mergeable until the privileged run concludes it; `refs/pull/<N>/merge` fetch + `git apply --3way`; `skopeo --preserve-digests` end-to-end. `workflow_run` only runs from the default branch, so the privileged half is exercised by mirroring the workflow onto a personal fork's default branch and opening a throwaway fork PR into it — not by merging first. _Since review_: the fork `e2e` job ran the pre-#2826 BATS app loop that the merged Chainsaw migration deleted (already broken); it now runs the same Chainsaw + TIA path as the in-tree job, with suite selection and the `full-e2e` override sourced from `resolve`'s trusted base-repo file list / labels (not a merge-ref diff). Publish job hardened per the non-blocking follow-ups: base-tree image allowlist, `skopeo copy --all`, pinned flux CLI. The required `E2E Tests` gate is now published as a commit **status** (default token + `statuses:write`), not a COZYSTACK_CI app check-run — the app lacked `checks:write` (`checks.create` 403'd) and a default-token check-run floated under a labeler suite; a commit status has neither problem, and forks can't post one (their `pull_request` token is read-only). Two review-bot findings fixed too: fork `OCI_EXPORT_DIR` is now absolute (a relative path misplaced archives under `make -C <pkg>` and every fork build would fail at upload), and the "≥1 archive" guard no longer no-ops under `nullglob`. ### Repo-admin prerequisites One repository setting lives outside this diff: 1. **Branch protection: keep `E2E Tests` required.** The gate is now a commit **status** named `E2E Tests` (posted with the default `GITHUB_TOKEN` — see `docs/agents/e2e-testing.md` §10), not a job or an app check-run. Branch protection already requires `E2E Tests` (currently the e2e job's name, renamed here to `E2E (in-tree)`), so the required context now resolves to the status — **no GitHub App, no `checks:write` grant, no app-id pin needed**. Just don't pin the context to a specific app. 2. **Actions → "Require approval for all outside collaborators".** Each fork run must be a deliberate maintainer click before it executes, since the fork's unprivileged run produces the artifacts the privileged `e2e-fork.yaml` consumes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (7 files: two CI workflows, `docs/agents/e2e-testing.md`, `hack/common-envs.mk` + its bats suite, and two package `Makefile`s). The one trigger worth naming is "change developer tooling (`cozyvalues-gen`, `cozypkg`, the package `Makefile`s) → `development.md`" on the website. It does not fire here: `OCI_EXPORT_DIR` is a CI-only build mode that defaults off, and the local developer workflow is unchanged — `make image` still builds and pushes exactly as before, which the new default-path case in `hack/common-envs_test.bats` now pins. No chart values, no API surface, no app list, no platform variant, no Talos version bump and no release asset rename are touched. - [x] No downstream repository is affected by this change ### Release note ```release-note fix(ci): fork pull requests now run e2e via a privileged workflow, and a fork PR can no longer merge on a skipped (never-run) required E2E Tests check ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a dedicated fork E2E workflow that runs after the base PR and reports results via the required **E2E Tests** commit status. - Fork PRs now export build images as OCI archive artifacts for later publishing. - Added a same-repo E2E reporting job and renamed the in-tree E2E flow to **E2E (in-tree)**. - **Bug Fixes** - Added tighter, fail-closed gating to ensure merge blocking when **E2E Tests** isn’t correctly finalized. - **Documentation** - Documented fork vs same-repo E2E split, OCI export behavior, and required fork security settings. - **Tests** - Added Bats coverage validating `OCI_EXPORT_DIR` and preventing unintended registry pushes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
cc8b01d5cd |
docs(ci): say when each lane's status opener actually lands
The reference section claimed both lanes open the required context when their run starts, and enumerated the triggers it protects against: a re-run, a reopen, a `full-e2e` label. Every one of those starts a Pull Request run, and on a fork that run cannot open the status at all. The opener is guarded same-repo only, because a fork's `pull_request` token is read-only, and the fork's opener lives in the privileged workflow, which starts only once the unprivileged build has finished. So on the lane the sentence was most likely to be read about, an already-green head SHA keeps reading green for the whole build. The gap is structural. A read-only token cannot post a status at any point in the run, and that same read-only token is what makes publishing the gate with the default token safe in the first place. It is bounded by the build rather than open-ended. What was wrong was only the claim that it does not exist. Say so in all three places that carried the overstatement: the section itself, the opener's own comment, and the contract test, which pins that each lane has an opener and cannot express when it lands. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c45179a9d0 |
docs(ci): correct the reason the build matrix keeps the rename-detected list
The comment justified the split by saying a path that no longer exists has nothing to build. That is only half the picture: on a rename between packages the source package is still there and still needs rebuilding, and rename detection drops it from the list the matrix reads, so the matrix under-builds. The behaviour predates this change and is out of scope for a docs-only decision, but the comment presented it as deliberate and correct rather than as inherited and left alone. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9c68b1f83e |
fix(ci): stop a cancelled fork run from publishing a verdict
The invariant that a cancelled run publishes nothing was written down in three places and held in two of them. The fork lane's terminal reporter ran under `always()`, which includes cancellation. When a second privileged run supersedes the first, the dying one still posted `failure` on a head SHA it no longer owned, for a suite that never finished. It self-corrected once the superseding run reported, but that can be two hours later, and until then a legitimate pull request reads red. The same-repo lane was moved off `always()` for exactly this reason. Move this one too: a failed publish or e2e still reports, since neither is a cancellation, and a cancelled run now leaves the `pending` it opened standing until its successor concludes. The same-repo guard's rationale cited a scenario that no longer exists. It described a label event cancelling the run that opened the pull request, which the concurrency split made impossible for every label the workflow discards. The guard is still needed, for a push, for a `full-e2e` label superseding the head run, and for a manual cancel, so the comment now names those instead of an extinct one. The reference section listed the pieces this invariant rests on and omitted the reporters, which is the half a reader is most likely to touch, and counted them, which was already wrong and would go wrong again on the next addition. It now enumerates without counting. The contract pins both reporters, because both lanes have had this defect at different times in the same shape. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8a21085324 |
fix(ci): decide docs-only from both sides of a rename
The two lanes answered "is this docs-only" from different sources and could disagree. `plan` reads `git diff --name-only`, which applies rename detection and reports a rename as its new path alone, so moving a code file into docs/ read as docs-only. The fork lane reads the PR file list, where a rename carries `previous_filename`, and reads the same change as code. Disagreement is not a near-miss here. The same-repo lane builds nothing and uploads no patch artifact; the fork lane concludes e2e is required and then fails closed because that artifact is missing, reporting that the fork build did not complete. It completed — it correctly built nothing. There is no push the contributor can make to clear it and no label that overrides it, so a legitimate contribution is permanently red. Point the docs-only decision at the rename-blind list, which the job already computes for the downstream-trigger-map flag and whose comment names this exact trap. Both lanes now agree, and the same-repo side stops shipping a removed code path untested — until now such a change skipped every build and reported e2e as not required. The build matrix keeps reading the rename-detected list: it picks packages to rebuild, and a path that no longer exists has nothing to build. The contract file gains the agreement as a pin, and grows a header to match what it now covers. The fork half is pinned on the docs-only expression rather than on `previous_filename`, which also appears in the file list a few lines down and would have kept the assertion satisfied with the check deleted. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d3515509c8 |
fix(ci): let a status-API blip skip the opener instead of failing the run
The step that claims the required context is the first in `plan`, and `checks`, `build`, `build-talos`, `finalize` and `e2e` all hang off that job. A transient failure from the status API would therefore discard a full build and suite for a call that is an optimisation, not a gate. Mark it `continue-on-error`. Losing the opener degrades exactly to the behaviour that preceded it — the head SHA keeps whatever status it had until `e2e-report` concludes — while a run that dies before reporting leaves the merge blocked either way. The terminal post keeps its own error handling, because that one failing silently is what would let a run finish with no verdict at all. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
ee844c98a5 |
fix(ci): open the "E2E Tests" status when the same-repo run starts
Moving the gate from a job to a commit status lost a property the job had for free. A commit status is latest-wins per (sha, context) and never expires, so a head SHA that already went green keeps reading green from the moment a new run starts until `e2e-report` posts at the end. Re-running all jobs, reopening, or adding full-e2e therefore leaves the pull request mergeable for the whole of a suite that may be about to fail. While the gate was a job, GitHub created a fresh non-success check run each time and the window did not exist. Claim the context at the start of `plan`, the way the fork lane already claims it in `resolve`, and let `e2e-report` conclude it as before. The call is same-repo only: a fork's `pull_request` token is read-only, so posting from there would 403 and fail the job on every fork PR — and that read-only token is what makes the whole commit-status design safe. Fork PRs get their opening status from the privileged workflow instead. The contract test grows a case pinning that each lane writes the context from two points, one of them pending. Both assertions on the same-repo opener are scoped to the `plan` job: the bare fork guard occurs on several steps in other jobs, so a whole-file "at least one" passed with the opener's guard deleted, pinning unrelated guards and reporting that as coverage. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7c5e96947e |
fix(ci): stop two runs from publishing "E2E Tests" on one SHA
Giving every `labeled` event its own concurrency group stopped a discarded label from cancelling a live run, but it also stopped a `full-e2e` label from doing so — and that label is not discarded. `plan` proceeds for it and `e2e-report` publishes with it, so the labelled run and the head run it was meant to widen both post the `E2E Tests` context, on the same head SHA, with no ordering between them. That was a deliberate trade at the time: commit statuses are latest-wins per context, so the reasoning went, the full-suite verdict lands last whenever its suite finishes last, and the label is an explicit opt-in to a longer run. This commit reverses that decision, because the premise only holds while the full run succeeds. The full suite is the superset, so a full-e2e run that fails early — a build flake, a suite failing at minute twenty — finishes before the narrower head run it was meant to widen, and that run's later green erases the failure. The required status then reads success for a SHA whose suite did not pass. One publisher per head SHA is worth more than the minutes of runner time a cancelled narrow run costs. Move aside only the label events `plan` discards, by making the concurrency condition the exact complement of `plan`'s guard. A `full-e2e` run stays in the main group and supersedes the head run the way a push does, so exactly one publisher of the context is alive per head SHA. Opting into the full suite now costs one suite instead of two, and a second label landing afterwards can no longer cancel the full-e2e run, because the two are no longer grouped together. The contract test gains the complement — the group key must exclude the publishing label rather than every label — and a check that the label name is one and the same string in all three guards that read it, since renaming it in one place splits the decision without touching a line that reads load-bearing. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
954be15341 |
fix(e2e): bound every read in the report collector and the step that runs it (#3596)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A single hung read in `hack/cozyreport.sh` did not truncate `cozyreport.tgz`, it lost the artifact outright. The collector writes its tarball on its last line, the `Collect report` step carried no `timeout-minutes` of its own, and `Upload cozyreport.tgz` is the step directly after it — so a read that never returned spent whatever was left of the job budget, the job was cancelled, and the upload never ran. The runs where the report is the only evidence left are the runs that produced none. `|| true` guards against a non-zero exit, not against a read that does not return. Two halves, independent of each other. **The step gets a ceiling of its own**, 30 minutes, in all three workflows that run it. The number is measured against the step rather than divided out of the job: pulling the `Collect report` timings out of recent e2e runs gives 3 to 42 seconds, on failed jobs as much as on green ones, and 30 minutes leaves five sixths of the job budget for the steps that upload what this one produced. It carries `continue-on-error: true` beside it, because the step's `|| true` is its standing statement that collecting diagnostics is never fatal and a step killed at its ceiling never reaches that `|| true` — without the tolerance, a passing run whose collection ran long would be published as failed, and every later `failure()` in the job would start meaning "the tests failed or the report was slow". That tolerance also renders the overrun green, so a following step turns the collect step's `outcome` into a `::warning::` annotation: a bound that fires is recorded, and the outer bound is not exempt from the rule the inner ones follow. **Every read in the collector is bounded**, against four sections before, and gates and host commands are reads too. An `if kubectl get deploy …` that decides whether a module runs costs more unbounded than anything it guards, because the module below it never starts; `df -h` blocks indefinitely on a stale mount and takes the tarball with it by exactly the mechanism above. A gate cut off short is recorded, since skipping the module leaves the same empty tree as a cluster that never ran the component, while a gate answering "not installed" is recorded nowhere — that is the cluster answering correctly, and a marker firing on every install without KubeVirt is one a reader learns to skip. The six LINSTOR reads run a command inside a pod and go through a reader of their own. They are the only reads whose duration a workload sets rather than the apiserver, and `kubectl exec` returns the remote command's exit status — a 124 there is either the collector's deadline or a command that itself exited 124, so the existing classifier would have named this script's own timeout as the cause of something that may have happened inside a pod. The new note states what the status leaves open, and sits beside the output rather than inside it, since two of the six stream a gzipped tarball. A cut-off bundle is kept rather than dropped, and the note says so: the emptiness check at those call sites asks whether the archive lists a first member, which a truncated gzip does. What this does not do is bound the report. The pod, VM, VMI and services walks carry a count cap and a time budget; every other per-object walk carries neither, so their cost is the cluster's to set, and a failed install with a hundred not-Ready HelmReleases spends four hundred bounded reads in one of them. A per-read ceiling removes the read that never returns, not the thousand reads that each return slowly. The file header says so where it previously implied the opposite, and the step ceiling is what covers the rest. Two frozen lists in the tests are gone, replaced by one invariant derived from the code: no read below the library guard runs without a ceiling. Both lists were already wrong. The header named four bounded sections while the file had ten, and the per-section counts of unbounded reads only ever matched a redirect into `$VAR/`, which no gate, selector or command substitution has — so they described the scan rather than the file. The guard that held the old exec assumption was satisfied by leaving every exec unbounded, which is the state the file was in. It now also requires each exec to be bounded, and requires that reader not to borrow the classification written for reads whose 124 is unambiguous. Matching on the verb rather than on the string `kubectl exec` is part of it: half of these calls spell it `kubectl -n cozy-linstor exec`, which that string never saw. Two properties worth knowing before this lands. `kubectl get events -A --sort-by=.lastTimestamp` sorts client-side, so kubectl buffers the whole event list and writes nothing until it finishes — under a flat per-read ceiling a cut-off there leaves a file holding only the marker, not a short tail. That read had no ceiling at all before, where the same slowness could take the whole tarball, so this is a narrower failure rather than a new one. And `kubectl logs -n cozy-kamaji deployment/kamaji` gains `--tail=2000`, matching every other controller log in the file: it was the only unbounded-in-size log, which under a wall-clock ceiling is the one most likely to be cut at its *end*, where a control-plane bootstrap failure sits. Longer kamaji history than 2000 lines is no longer collected. Two accepted imperfections in how the change is split across commits, named rather than left to be found. `docs/agents/e2e-testing.md` keeps this material in a single shared paragraph that all three commits touch, so in the middle commit one sentence about the module gates lands a commit ahead of the gates themselves. And two corrections to the exec reader — it now keeps the remote command's whole message in the listing rather than quoting the line `kubectl exec` always appends last, and its cut-off note names both of the two things that can happen to a partial bundle instead of asserting one — sit in the last commit rather than in the one that introduced that reader. Both are true of the branch tip, where every statement is correct and appears once; the cost of moving them is another rebase across the one shared paragraph, which is not worth what it buys. Refs #3568. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(e2e): every read in the diagnostic report collector now carries a wall-clock ceiling — module gates, the reads that run a command inside a LINSTOR pod, and the host commands included — and a ceiling that fires is recorded in the artifact rather than only in the job log. The workflow step that runs the collector carries a ceiling of its own, so a read that does not return costs one file and a warning annotation instead of consuming the job budget and losing cozyreport.tgz entirely. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added bounded diagnostic collection across Kubernetes, Talos, LINSTOR, host, controller, and command outputs. - Reports preserve partial results and identify timeouts, failures, missing output, and truncated artifacts. - Added warnings when report collection fails or exceeds its time limit. - **Bug Fixes** - Corrected resource targets, selectors, deployment extraction, and command matching. - Improved handling for empty selections, unavailable resources, and missing timeout tools. - **Documentation** - Expanded guidance for diagnostic markers, collection limits, timeouts, and incomplete output. - **Tests** - Added coverage for bounded reads, failure reporting, markers, selectors, logs, and subprocess mocking. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7e2db4da7e |
fix(e2e): bound every remaining read in the report collector
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and the rest -- nodes, namespaces, events, HelmReleases, packages, packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads and every module gate -- read with none. The tarball is written on the collector's last line, so a single read that never returns does not truncate the report, it loses it, on the run where it is the only evidence left. Each read now goes through the shape that fits it: the existing bounded reader for a read whose output is a file, the existing selector for one that feeds a loop, and a new probe for the `if kubectl get deploy ...` gates. A gate is a read like any other, and an unbounded one is worse than the reads it guards, since the module below it never starts. Its cut-off is recorded, because skipping the module leaves the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. The reads that stay on the bare wrapper say so where they sit; what they share is that the output is not the reader's to write, going into a variable or through a pipeline that transforms it, so there is no moment at which a marker could attach. The header claimed a per-section list of what was bounded, and the tests froze that list plus a per-section count of the unbounded reads. Both are gone. The list had already gone stale, naming four sections while the file had ten, and the counts were an undercount by construction -- they matched a redirect to `$VAR/`, which no gate, selector or command substitution has. One test replaces both: no read below the library guard runs without a ceiling, checked by looking for a counterexample. A ceiling per read is not a ceiling on the report, and the header now says so where it used to imply the opposite. The pod, VM, VMI and services walks carry a count cap and a time budget; the walks over nodes, namespaces, HelmReleases, packages, PVCs and certificates carry neither, so their cost is the cluster's to set. What this removes is the read that never returns. The bounded reader now names the tool it ran and describes what that tool returned. It served kubectl alone until this change gave it talosctl and the four host commands, so a `df` that hit a stale mount left a marker reading "kubectl exited 1 ... not because the object ended" -- two false statements in one line, in an artifact whose premise is that its notes can be believed. The exec reader keeps the remote command's whole message in the listing it belongs to. `kubectl exec` appends its own `command terminated with exit code N` after whatever the command said, so quoting the last line there drops the cause and keeps the trailer, and sending the message away from the file leaves a zero-byte listing -- the "returned nothing, or never ran" question this collector exists to answer. Its cut-off note also stops asserting that the archive is beside it: a stream that got members across is kept and truncated, one killed before its first byte leaves zero bytes and the call site's emptiness check removes it, and the reader runs before that check, so it names both. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
96e33fe656 |
ci(e2e): give the report collection step a ceiling of its own
The step that runs hack/cozyreport.sh carried no timeout-minutes in any of the three workflows that run it, so it inherited only the job's 180. The collector writes its tarball on its last line and the upload is the next step, so a read that never returns spends what is left of the job budget, the job is cancelled, and the upload never runs -- the artifact is lost entirely rather than truncated, on exactly the run where it is the only evidence left. `|| true` guards against a non-zero exit, not against a read that does not return. 30 minutes is measured against the step rather than divided out of the job: it takes 3-42s across sampled runs, on failed e2e jobs as well as green ones, and 30 leaves five sixths of the job budget for the steps that upload what this one produced. A test pins the ceiling in all three workflows and requires it to stay under the job's own, since a step ceiling at or above the job's cap is the same as no ceiling at all. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8aa30472b6 |
docs(ci): correct the job-level permissions comment
The comment above the top-level `permissions` block claimed that jobs request the minimum extra scopes on top of it. A job-level block does not add to the workflow-level one: it replaces it, and every scope the job does not name is set to none. A job needing one extra scope has to restate `contents: read` alongside it or lose read access to the repository. The corrected wording states the replace semantics, so the next job block written against this comment grants the whole set it needs rather than only the delta. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
df157da618 |
ci(workflows): open the E2E breakpoint for maintainer-authored PRs (#3158)
## What this PR does Decouples opening the E2E SSH breakpoint from the `debug` label. Today, the only way to open a breakpoint on a failed feature PR is to add the `debug` label. That same label also flips every job's `runs-on` onto self-hosted runners, so you cannot get a breakpoint on the normal ephemeral runner — debugging always drags the whole pipeline onto self-hosted infrastructure. This adds a maintainer-authorship path to the breakpoint's `if` condition: a failed PR whose author is an `OWNER`, `MEMBER`, or `COLLABORATOR` now opens the breakpoint automatically, with no label required and without moving the jobs onto self-hosted runners. The breakpoint simply holds the default `oracle-vm` ephemeral runner the job already runs on. The existing `debug` and `release` label paths are unchanged, and the `authorized-users` list still controls who may actually SSH into an open breakpoint (second defense layer). Workflow triggers are untouched — this is purely an `if`-condition change. ### Release note ```release-note ci(workflows): open the E2E SSH breakpoint automatically on failed maintainer-authored PRs, without requiring the `debug` label or moving the jobs onto self-hosted runners ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated E2E failure debugging so breakpoints can now open for pull requests authored by repository maintainers, not just when specific labels are applied. * Clarified the conditions and runner behavior for breakpoint access in the workflow documentation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
f1f383626e |
fix(ci): overlay images from the PR base branch, and publish per-line artifacts (#3471)
## What this PR does
The PR finalize job overlays image refs for every package a PR did not
rebuild, so e2e and the installer do not test last-release images for
everything outside the PR's build matrix. It could only ever read
`cozystack-packages:main`, which meant a **release-line PR was handed
main's binaries to run against its own line's charts**.
cozystack/cozystack#3437 is the demonstration: a one-line change on
`release-1.6` deactivating an app failed install deterministically,
twice, with
```
helmrelease/backupstrategy-controller: Helm install failed …
error validating data: SchemaError(github.com/cozystack/cozystack/pkg/apis/core/v1alpha1.Option.spec):
unknown model in reference: "github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec"
```
main's `cozystack-controller` served an aggregated OpenAPI that
`release-1.6`'s charts could not validate against. Nothing in that PR
was broken; the lane was. Both branches carry `option_types.go` and key
`OptionSpec` identically in the committed generated OpenAPI, so this is
a generation mismatch at runtime, not a codegen drift.
This PR fixes it by giving every base branch its own artifact to overlay
from, rather than by turning the overlay off.
### The change
1. **`pull-requests.yaml`** reads `cozystack-packages:${BASE_REF}`
(`github.base_ref`) instead of a hardcoded `:main`.
2. **`build-release.yaml`** (new) publishes that artifact for maintained
`release-<major>.<minor>` branches exactly as `build-main.yaml` does for
main: every image tagged with the branch, and the whole packages tree
pushed with each reference digest-pinned to what the run just built.
An earlier revision of this branch simply skipped the overlay for
non-main bases. That fixed the wrong-images problem but left
release-line PRs testing their line's *last release*: for any package
the PR did not rebuild, the committed ref is the released digest, so a
component changed by an earlier backport was still exercised as its
pre-backport binary until the next rc. Per-line artifacts remove that
gap too, which is why the guard was replaced rather than kept.
### Three deliberate choices
**The trigger matches line branches only** (`release-[0-9]+.[0-9]+`).
The per-release and rc staging branches `promote-rc.yaml` and
`tags.yaml` create — `release-1.6.1`, `release-1.6.0-rc.4` — must not
trigger a full rebuild; their images come from the tag build and
rebuilding them is waste.
**`WRITE_CACHE` stays `0`.** `CACHE_REGISTRY/<img>:buildcache` is a
single ref per image and `build-main.yaml` is deliberately its only,
serialized writer so concurrent builds cannot race on the cache manifest
— the 409 class #2711 fixed for image tags. A line build can overlap a
main build, so writing here would reintroduce that race. Line builds
read the cache.
**A missing artifact still degrades to committed refs, but says so.** On
a release line it emits a `::warning::` naming the branch. Silent
degradation is indistinguishable from a working overlay, which is how a
mis-specified branch filter could hide for a whole release cycle.
### Cost
One `make build` per push to a maintained line — in practice per merged
backport. That is the price of release-line PRs testing their line's tip
instead of its last release.
### Verification
`hack/overlay-main-images_test.bats` pins the artifact tag to the base
branch, rejects a hardcoded `:main` in either overlay step, and pins
`build-release.yaml`'s branch filter, image tag and `WRITE_CACHE: '0'`.
Mutation-checked: restoring `:main`, setting `WRITE_CACHE: '1'`, and
broadening the filter to `release-*` each fail a test. 13/13 green;
`actionlint` and `zizmor` clean.
Worth an explicit ack in review: the branch-filter pattern is the one
thing no local test can prove, since only GitHub evaluates it. If it
does not match, `build-release` never runs and the new `::warning::` is
what surfaces it on the next release-line PR.
### Backport
`release-1.6` needs this too — for `pull_request` events GitHub builds
the workflow from the merge ref, so a release-line PR only stops
receiving main's images once the change is on its base branch.
cozystack/cozystack#3472 carried the interim guard and is closed in
favour of backporting this instead.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a “Build release line” workflow that builds and publishes images
and the packages artifact for maintained `release-<major>.<minor>`
branches.
* **Bug Fixes**
* Updated PR workflow finalization to use base-branch–specific package
overlays, avoiding incorrect main-branch image references when targeting
release branches.
* Improved fallback behavior when the base-branch packages artifact is
unavailable.
* **Tests**
* Added workflow wiring tests to verify base-branch artifact usage and
that each maintained release line publishes its own packages artifact.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Myasnikov Daniil
|
6a73a6caa9 |
docs(agents): call the E2E gate a commit status, not a check-run
The §10 heading, four workflow comments and both TOCTOU comments still called the required "E2E Tests" context a check-run, while the body of that same section — and the implementation — insist it is a commit status. The distinction is the whole reason the section exists: a check-run made with the default token floats into an arbitrary check-suite, which is why the gate is a status. Leaving the wrong primitive in the heading steers a maintainer wiring branch protection at exactly the wrong thing. The paragraph explaining why it is NOT a check-run keeps that wording, since there the term is the rejected alternative. Also adds the blank line missing before the "Chainsaw v0.2.15 gotchas" heading. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
c46037ce7b |
fix(ci): stop a cancelled run from wedging the "E2E Tests" gate
Now that the required "E2E Tests" context is a commit status rather than a job, a run that dies before posting leaves the head SHA at "Expected" — which blocks the merge instead of passing it. Two paths reached that state. Same-repo: a `labeled` event shared the PR's concurrency group, so labelling an open PR cancelled the live opened/synchronize run, and the replacement run was then discarded by `plan`'s full-e2e guard -- so `e2e-report` skipped with it and nothing posted the status the cancelled run was going to. Observed on live CI: run 30614258694 (cancelled) followed by 30614280603 (skipped) on the same SHA, 24s apart, from a maintainer adding kind/* labels just after a push. Label events now get their own concurrency group, so they can only supersede each other. Release promotion is unaffected: `gh pr create --label` attaches `release` at creation, so a promote PR gets a single run that already sees the label (verified on #3397 and #3425, one run each, build/finalize gated off). Fork: `resolve` had no equivalent, so a cancelled triggering run fell through to the `CONCLUSION != 'success'` branch and posted `failure` — a false red on a SHA whose e2e never ran, which the replacement run for that same SHA would not flip back (it takes the discarded-label path and posts nothing). It now returns early on a cancelled or wholly-skipped conclusion. Checked on the conclusion rather than the job list because a cancelled run can report no jobs at all, which would make the `Plan build` lookup miss. Both stay fail-closed: declining to post can only preserve a verdict a real run reached, never invent one. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
507336dba0 |
fix(ci): report "E2E Tests" green on a release promotion PR
Under main's rc-gate model a promote PR carries `release` and deliberately not `full-e2e`, so the `e2e` job skips: the immutable rc whose digests the tree pins was already verified, which is the cost the rc gate exists to remove. While the required gate was a job, that skip satisfied branch protection. Now that it is a commit status, e2e-report saw PLAN_RESULT=success, CODE=true (a promotion rewrites ~38 non-docs image refs) and E2E_RESULT=skipped, fell to the last branch and posted failure — so every release would have needed a full suite re-run or an admin override of the very gate this PR strengthens. Give e2e-report the carve-out the `e2e` job already has: release without full-e2e reports success and says why. Adding full-e2e still forces a real run and a real verdict. Close the second route to the same red as well. `gh pr create --label` attaches the label just after opening, so the `labeled` event cancels the `opened` run through cancel-in-progress; under always() that dying run still posted failure with PLAN_RESULT=cancelled, and the `labeled` run replacing it is skipped by the allow-list guard, so nothing flipped the status back. !cancelled() makes a cancelled run report nothing, which is the same fail-closed posture used elsewhere here: not posting leaves the required status "Expected", which blocks. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
8b79a0f20c |
fix(ci): don't clobber a green "E2E Tests" status on a discarded label event
main gained a `labeled` pull_request trigger whose `plan` guard discards every label event except full-e2e. e2e-report carried no such guard, so labelling an already-green open PR started a run in which `plan` was skipped, e2e-report still ran (always()), read PLAN_RESULT != 'success' and posted "E2E Tests" = failure — overwriting the passing required status and making a merge-ready PR unmergeable until the next push. Give e2e-report the same `labeled` guard as `plan`, so it runs if and only if `plan` runs. Skipping posts nothing, which is fail-closed: an unposted required status stays "Expected" and blocks the merge, so the guard can only ever preserve a verdict, never invent one. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
3b70fda870 |
fix(ci): make fork OCI_EXPORT_DIR absolute so archives land where the upload reads
The fork build sets OCI_EXPORT_DIR to a relative `_out/oci`, but each image is built with `make -C <pkg> image`, so buildx resolves the relative `dest=` against the package directory — the archive landed in <pkg>/_out/oci/ while the `if-no-files-found: error` upload reads repo-root `_out/oci/*.oci.tar`. Every non-docs fork build would have failed at that upload (never caught, since same-repo PRs leave OCI_EXPORT_DIR empty). Root it at github.workspace so the dest resolves to repo-root _out/oci regardless of the sub-make cwd. Reported by gemini-code-assist on #3262 (its abspath suggestion resolves against the package dir under `make -C`, so this fixes it at the workflow level instead). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
0f6e99eff8 |
fix(ci): publish the "E2E Tests" gate as a commit status, not a check-run
Route the required "E2E Tests" merge gate through a GitHub commit status
posted with the default GITHUB_TOKEN (+ statuses:write), replacing the
COZYSTACK_CI app check-run. The app path needed the app granted checks:write
(it isn't — checks.create 403s "Resource not accessible by integration"), and
the earlier default-token check-run floated under a random check-suite
("PR size label"). A commit status has neither problem: branch protection
matches it by context name, it carries no check-suite grouping, and it is safe
with the default token because a fork's pull_request token is read-only and
cannot post one — only same-repo runs (e2e-report, guarded !fork) and the
privileged workflow_run in e2e-fork.yaml can set it.
The security property is unchanged: a never-posted required status stays
"Expected" and blocks the merge, so a skipped job can't smuggle a green
(#3257). Removes all COZYSTACK_CI app-token machinery from e2e-fork.yaml —
`resolve` posts the status directly, no `check_run_id` threading, no `open`
step.
Branch protection must require the "E2E Tests" status context (no GitHub App
and no app-id pin needed).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
9cddb532bf |
fix(ci): publish "E2E Tests" check-run as the COZYSTACK_CI app
The synthetic "E2E Tests" required check-run was created with the default GITHUB_TOKEN (the "GitHub Actions" app). A token-created check-run has no owning job, so GitHub files it under an arbitrary GitHub Actions check-suite for the head SHA — in practice the earliest one, the PR-size labeler — so "E2E Tests" rendered under "PR size label". Publish it as the COZYSTACK_CI GitHub App instead (create-github-app-token, scoped to permission-checks: write) so the check lands in the app own dedicated suite. This is also stricter: only the CI app can satisfy the gate. - pull-requests.yaml (e2e-report): mint the app token, pass it to checks.create. - e2e-fork.yaml: split resolve into a verdict decision on the default workflow_run token (keeps the actions:read / pull-requests:read lookups) and an "open" step that publishes the check as the app; report re-mints the token so checks.update runs as the same app that opened the pending run. - docs/agents/e2e-testing.md: document the change and the branch-protection requirement. Requires main branch protection to require "E2E Tests" from the COZYSTACK_CI app id (not app_id 15368). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
76c8f890fa |
fix(ci): export OCI archives for buildx calls that bypass image-tags (#3257)
Fork PRs build images to per-image OCI archives instead of pushing, gated on OCI_EXPORT_DIR. The `--output type=oci` was appended only inside the `image-tags` macro, so `packages/system/capi-providers-cpprovider`, whose `image:` recipe uses a hardcoded `--tag` and never calls `image-tags`, ran buildx with no exporter under export mode (PUSH/LOAD forced to 0). No archive was written, and the fork build's `if-no-files-found: error` upload then failed the whole build. Its `--cache-to type=inline` also needs an image exporter. - hack/common-envs.mk: extract the OCI-output snippet from `image-tags` into a reusable `oci-output <image-name>` macro so any buildx call can opt in; `image-tags` now ends with `$(call oci-output,$(1))` (no-op for the ~33 conforming packages). Move `comma :=` above its first user. - packages/system/capi-providers-cpprovider/Makefile: add `$(call oci-output,cluster-api-control-plane-provider-kamaji)` (archive base name equals the OCIR repo the privileged push keys on) and drop the inline cache under export (`--cache-to type=inline` gated off). - .github/workflows/pull-requests.yaml: wrap the `E2E Tests` check-run creation in try/catch. On error, `core.setFailed` so the required merge gate fails loudly and fail-closed instead of silently succeeding. Repo-wide audit of every `docker buildx build` reachable from the root `build:` matrix confirms capi was the only recipe bypassing both `image-tags` and `oci-output`. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
e0ac2f9e12 |
ci(e2e): run fork-PR e2e via privileged workflow_run (#3257)
Fork PRs cannot push images (pull_request runs from a fork receive no secrets), so `make image` pushed anonymously and died with `denied`. That failed every build, collapsed the build->finalize->e2e chain, and left the required "E2E Tests" job skipped — which branch protection treats as satisfied, so an approved fork PR could merge with e2e never having run. pull-requests.yaml (fork PRs): - build / build-talos / finalize export images to per-image OCI archives (OCI_EXPORT_DIR) and upload them instead of pushing, so the red "denied" build wall disappears and the chain no longer collapses. - the in-tree e2e job is renamed off "E2E Tests" and guarded to same-repo PRs; a new e2e-report job concludes the required "E2E Tests" context as an explicit check-run (success on a docs-only PR or when the in-tree e2e passed, failure otherwise). A skipped job can no longer satisfy the required check. e2e-fork.yaml (new, privileged workflow_run from the default branch): - resolves the fork PR, opens a pending "E2E Tests" check-run, and fails-closed if the fork build failed; - pushes the pre-built image archives to OCIR by digest and flux-pushes the digest-pinned packages tree — the OCIR token is live only for trusted binaries (git/skopeo/flux/yq) over data, never for `make` or any fork-authored script; - runs the e2e suite against the pushed closure with no push token and concludes the "E2E Tests" check-run on the PR head SHA. Fork build code never receives registry credentials; the fork's images and test scripts run only in the credential-less, ephemeral e2e job. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
2330d6f3ca |
fix(ci): overlay images from the PR base branch, and publish per-line artifacts
Supersedes the interim guard in this branch`s first commit, which skipped the overlay entirely for release-line PRs. Skipping fixed the wrong images but left those PRs testing their line`s last release: for any package the PR did not rebuild, the committed ref is the released digest, so a component changed by an earlier backport was exercised as its pre-backport binary until the next rc. The overlay now reads `cozystack-packages:<base branch>` instead of always `:main`, and build-release.yaml publishes that artifact for every maintained `release-<major>.<minor>` branch the way build-main.yaml does for main: images tagged with the branch, and the whole packages tree pushed with each reference digest-pinned to what the run just built. Each base branch therefore has its own generation to overlay from, which is what the original bug was really about — #3437 failed install because main`s cozystack-controller served an aggregated OpenAPI release-1.6`s charts could not validate against. Three deliberate choices: * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The per-release and rc staging branches promote-rc.yaml and tags.yaml create (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their images come from the tag build, and rebuilding them would be waste. * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per image and build-main.yaml is deliberately its only, serialized writer so concurrent builds cannot race on the cache manifest (the 409 class #2711 fixed for image tags). A line build can overlap a main build, so writing here would reintroduce that race. Line builds read the cache. * A missing artifact still degrades to committed refs, but on a release line it now emits a ::warning:: naming the branch. Silent degradation is indistinguishable from a working overlay, which is how a mis-specified branch filter would hide for a whole release cycle. Cost: one `make build` per push to a maintained line, i.e. per merged backport. hack/overlay-main-images_test.bats pins the artifact tag to the base branch, rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main, setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test. 13/13 green; actionlint and zizmor clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
cc44f6c981 |
ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449)
## What this PR does Moves release validation from promote time to rc time, making promotion itself a pure retag operation. This is the follow-up to the v1.6.0 retrospective: the promote flow's checks ran on the promote PR (coupled to promotion, skippable by never promoting), the changelog and website docs were produced only after the release went public, and the finalize tag push silently failed to trigger `tags.yaml` at all (persisted-credential bug, root-caused below). The release lifecycle after this PR: 1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes the staging branch — and now runs a **mandatory full e2e** for every `-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and any published tag) get the same validation as a **manual button** (`workflow_dispatch`). 2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**: the release changelog is generated from the rc tag (reusable `changelog-generate.yaml`) and committed to the rc staging branch, ready for review long before promotion. 3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence: the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages the release, picks up the rc-time changelog (or generates one), and **opens the website docs PR** on cozystack/website from the staging branch — parked with an explicit DO-NOT-MERGE-until-finalize contract. 4. The promote PR **no longer gets `full-e2e` automatically** — its tree is the rc's already-validated digests with only tag strings rewritten; a maintainer can still add the label manually to force the suite. 5. Finalize performs the irreversible side effects (tag, publish, retag, chart) — and with `persist-credentials: false` its tag push now actually authenticates as the CI app, so the `tags.yaml` backstops (`generate-changelog`, `update-website-docs`) fire for promoted tags. This is the one-line fix for the v1.6.0 incident where the tag triggered nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader` silently defeats the app token injected via `git remote set-url`, and `GITHUB_TOKEN` pushes create no workflow runs. Everything is pinned by executable contract tests (`hack/promote-gate-contract.bats`, extended `hack/release-changelog-contract.bats` / `release-changelog-behaviour.bats` — 48 tests total, including behaviour tests that execute the new `hack/parse-rc-tag.sh` and `hack/select-changelog-source.sh` for real), and `docs/release.md` is rewritten to describe the new lifecycle end-to-end. ### Commit map (review commit-by-commit) - `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing changelog-before-publication rework (changelog generated at promote, committed to the staging branch before the PR opens, finalize publishes it as the release body; tags.yaml job becomes a verbatim-port backstop), rebased onto main. - `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane (prerelease → staging branch, stable → tag; draft releases fail fast; dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts). - `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut rc tags (also on `release_exists` re-runs); fixes the `cut-prerelease.yaml` comment that falsely claimed e2e already existed. - `2dfbeea77` — `changelog-generate.yaml` (reusable core) + `changelog-rc.yaml` (rc-time button committing to the staging branch); promote-rc gains a top-level `parse` job with a strict no-whitespace tag policy (`hack/parse-rc-tag.sh`); source selection is validate-then-fallthrough (`hack/select-changelog-source.sh`) so an invalid higher-priority changelog can never silently win. - `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override + removal of the automatic `full-e2e` label (pull-requests.yaml release-path e2e now requires an explicit label; a post-open label now actually starts the run). - `936190d4b` — `website-docs` job at promote time behind a FETCH_REF-support guard (see dependency below), with the DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before `release-next`; `tags.yaml::update-website-docs` stays as the backstop. - `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0 root cause). - `717c8860f` — `docs/release.md` coherence pass over the whole new lifecycle. ### Dependency The promote-time website docs job needs cozystack/website#631 (`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard detects the missing support and degrades to a ⚠️ line in the promote PR body — promotion is unaffected and the tag-time backstop backfills, so this PR is safe to merge first. ### Merge instructions Please **merge-commit, do not squash**: the 13 commits are deliberately atomic (the credential root-cause fix is separable from the feature work) and squashing would collapse ~3k lines into one commit, hurting bisectability. ### First live run The e2e lane is verified by lint, contract tests and dry-traces, but has not yet executed against a real tag; the intended first live run is a manual `e2e-tag.yaml` dispatch against an existing published rc tag before the gate is relied on for a real promotion. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added reusable changelog generation with validation, “present” signaling, and safe artifact upload behavior (including RC-time support). * Added an E2E release-tag workflow with tag parsing, deterministic sandboxing, and comprehensive report collection. * Improved RC promotion with a stricter parse front-door, verified E2E green gate (optional bypass), and earlier parallel changelog/asset staging. * **Documentation** * Refreshed release-process docs to match the updated RC-to-stable, changelog, and website-docs timing rules. * **Bug Fixes** * Improved PR finalization to source release bodies from verified changelogs and warn/fallback on missing or empty content. * Tightened label gating so E2E runs only under the intended conditions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
78fc3f9178 |
refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows)
The E2E job once ran as several state-sharing self-hosted-runner jobs that rendezvoused through a sha256(REPO:WORKFLOW:REF) key and a per-run /tmp/$SANDBOX_NAME copy of the checkout. Since the move to the ephemeral pool it is a single job: there is no cross-job rendezvous and the checkout is never reused, so copying it into /tmp and removing it afterwards is dead weight. Run every step in $GITHUB_WORKSPACE instead: drop the Prepare/Remove workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes, and point the upload paths at _out/. This deletes the unguarded `rm -rf /tmp/$SANDBOX_NAME` rather than guarding it: when an upstream step fails or the job is cancelled, the unconditional `Set sandbox ID` is skipped, SANDBOX_NAME stays empty, and the `if: always()` steps interpolate it — the removal expanding to `rm -rf /tmp/` and failing the cleanup step. `Set sandbox ID` stays, since the container still needs a name. Apply the same treatment to nightly.yaml, and drop the Makefile dead code the single-runner retirement left behind: the world-writable mkdir/chmod of /tmp/$SANDBOX_NAME (backed a bind mount removed long ago) and the `-e SANDBOX_NAME` container env (nothing in the e2e-sandbox image reads it). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Myasnikov Daniil
|
fea40d6314 |
fix(ci): only overlay current-main images on main-based PRs
The PR finalize job pulls cozystack-packages:main and repoints every package the PR did not rebuild at current-main images. On a main-based PR that is the whole point: it stops e2e from testing last-release images for everything outside the build matrix. On a release-line PR it is wrong — the committed refs there are not stale, they are that line`s released digests, and the charts are written against exactly those. So a release-line PR was installing main`s binaries onto its own charts, and the mismatch grows with every commit main gains. #3437 is the demonstration: a one-line change on release-1.6 that deactivates an app failed install deterministically, twice, with SchemaError(...core/v1alpha1.Option.spec): unknown model in reference: "...core~1v1alpha1.OptionSpec" from main`s cozystack-controller serving an aggregated OpenAPI that branch`s charts cannot validate against. Nothing in the PR was broken; the lane was. Left alone this makes every 1.6 backport look red, which is when release-branch PRs are busiest. Both overlay steps are now gated on `github.base_ref == main`, so a release-line PR keeps its committed digests — the behaviour that predates the overlay. Retargeting the overlay at a per-line artifact would be better but is not possible today: build-main.yaml publishes only cozystack-packages:main, and the registry carries no release-* equivalent (verified against the packages repo`s tag list: `main` plus per-PR tags, nothing else). hack/overlay-main-images_test.bats pins the wiring per step, so adding a third overlay step without the guard fails the suite. Mutation-checked by removing one guard. 12/12 green; actionlint clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ff7ddd3c9a |
ci(release): gate promotion on a green rc e2e and drop the automatic full-e2e label
Promotion now assumes checks passed at rc time instead of re-running them. At dispatch, before any staging happens, promote-rc verifies the rc tag has a green full e2e via either evidence path: the rc's tags.yaml push run containing a successful 'rc-e2e / E2E <tag> (full suite)' workflow_call job, or a successful manual e2e-tag.yaml dispatch correlated by its exact tag-bearing job name (immutable rc digests make past evidence valid). No evidence fails the dispatch with both paths and the remediation named. A skip_e2e_gate boolean input bypasses the gate loudly for emergencies; the promote PR body always carries the verification state (verified or bypassed), and 'verified' can only originate in the gate's green branch. The promote PR keeps the release label but no longer gets full-e2e automatically: its tree is the rc's already-validated digests with only tag strings rewritten, so pull-requests.yaml now runs e2e on the release path only when a maintainer opts in by adding full-e2e. The regular-PR path and the ordinary full-e2e forcing are untouched, and the new labeled trigger makes a post-open label actually start the run. A new promote-gate contract suite pins the gate's DAG position, the override default, the label set, the body lines, and the label-gate condition with comment-proof executable-line counts. Assisted-By: GPT-5 <noreply@openai.com> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
792951b9c6 |
fix(ci): make workflow cancellation effective
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
44507d2cd0 |
docs(agents): add downstream repository checklist to the PR template
Cozystack is upstream for repositories that nothing keeps in sync with it, and no CI job compares the two sides, so a change here breaks them in silence. Only two narrow paths are automated, and both fire on a release rather than on a PR. It has already happened. The website docs generator works from an app list hardcoded in its own Makefile, and packages/apps/opensearch was never added to it, so an application we ship has no reference page at all. The Terraform provider still offers a Kubernetes version this repo removed. The plugin repo still sends operators to hack/ scripts that were deleted, one of them rewritten as a Go command. Add a Downstream Repositories trigger map to the contributing guide, naming per repository which change here forces a change over there and which file to touch, and a checklist in the PR template that links to it. The template addresses AI agents directly: they author a large share of PRs and will otherwise tick a box without opening the follow-up it claims. The coupling is wider than the obvious ones, and mostly invisible. talm restates the node contract, overlapping ours but already diverged, so it must be diffed rather than overwritten. cozyhr reads an annotation the operator writes and silently merges the wrong values if it is renamed. cozy-proxy is vendored in from its own repo, yet consumes labels produced here and quietly stops handling external VM services if they move. The telemetry server queries metric names emitted here, and a rename just empties its dashboards. None of these fail loudly. Guard the map with a unit test, because a map about silent drift must not drift silently itself. It pins every in-repo path the map cites to a file that exists, pins the map and the checklist to the same repository list, and pins the workflow carve-out below to the map's location, to the job that reads the flag, and to the output that carries it: cutting any one of those three leaves the other two looking wired up. The header states plainly what the suite cannot check, so a green tick is not mistaken for a correct map: whether a coupling is real, and whether it still holds over there, stays a human's job. Editing the map alone is a docs-only PR, which skips the unit tests and would exempt the map from its own guard. Rather than widen the docs-only rule, which also gates the Talos image build, the plan step exports a separate flag that only the unit-test job reads: a docs PR now runs the tests and still builds nothing. It detects the map by its old path too, since a rename is reported as the new path alone. Document that gh pr create --body and --body-file replace the body wholesale, which would otherwise drop the checklist from every CLI-authored PR. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
302a72f14a |
Merge origin/main into feat/e2e-chainsaw-example
Reconciles the talos-image-cache conflicts created when #3254 merged to main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh, a superset of #3254's logic + the node-join diagnose helper), drop the BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path versions of the mirror manifest, install bats, and unit test. Also picks up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and Talos v1.13.6 (#3240). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
814a47f23e |
ci: move remaining workflows off the persistent self-hosted runner
Finishes the runner decommission tracked in #2937. PR builds, the main cache warmer, nightly, promote-rc and pull-requests-release already run on the ephemeral pool; this moves the stragglers so no workflow depends on the persistent runner anymore: - tags.yaml prepare-release: large ephemeral shape (same as the cache warmer), flux CLI bootstrap, and read-only OCIR access so release builds warm-start from the shared mode=max build cache while still pushing images to GHCR. Without the cache wiring every tag build on an ephemeral runner would be 100% cold. - tags.yaml generate-changelog / update-website-docs, backport.yaml, update-releasenotes.yaml: GitHub-hosted runners — these jobs only need git and the GitHub API, no docker or repo toolchain. - pull-requests.yaml: the debug label no longer reroutes jobs to the self-hosted runner (that path would queue forever once the runner is gone). The label keeps gating the SSH breakpoint on e2e failure, which works from ephemeral runners too. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Myasnikov DaniilandClaude Fable 5
|
1f84d2d49e |
Merge origin/main into feat/e2e-chainsaw-example (post-#3244)
Fold the in-sandbox Talos image cache work into the Chainsaw layout: the new shared helper relocates to hack/e2e-chainsaw/_lib/ talos-image-cache.sh and run-kubernetes.sh sources it from there (the ouroboros DNS-check no-retry rework merges clean). The OIDC bats are resolved as deleted again; their Chainsaw fixtures deliberately skip the talos.imageFactoryURL injection — the helper only emits it when the cache is up, and the zero-worker render-side tests instantiate no DataVolume imports for it to redirect. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
08ac063d6d |
ci(e2e): move install gate to 32vCPU runner to end CPU oversubscription
The three sandbox QEMU guests each run with `-smp 8`, demanding 24 vCPU in total. On the 24-vCPU runner that is a 1:1 CPU commit with no host headroom for QEMU's emulation/IO threads, vhost-net, the host NAT every guest image pull traverses, containerd and the kernel. Under the peak concurrency of "Install Cozystack" the guests lose cycles to CPU steal, so in-guest readiness/liveness probes miss their deadlines and a HelmRelease intermittently flips InProgress -> Failed with dependents cascading to DependencyNotReady — on a different component each run, the signature of resource thrash rather than a product bug. Move the job to the 32-vCPU / 128 GiB runner class. Guest sizing is unchanged (24 vCPU / 72 GiB), so the install does not slow down, while the host keeps 8 vCPU and 56 GiB of headroom that absorbs the emulation and network overhead the probes were starving against. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov DaniilandClaude Fable 5
|
efbf372d25 |
test(e2e): fix post-merge seams the conflict markers missed
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed test-apps-% target; switch it to the single chainsaw invocation (empty CHAINSAW_SUITES = full suite) and upload the JUnit report - pin the chainsaw-report upload-artifact step to the same SHA the rest of the workflows use (scorecard/zizmor gate) - repoint hack/run-kubernetes-drain_test.bats at the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh - replicate cozytest.sh's NotReady-pod data-plane capture (hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so chainsaw-suite failures keep diagnostic parity with the BATS lanes - refresh stale bats path comments Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |