mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-23 11:37:29 +09:00
main
523
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 myasnikovdaniil
|
c6fcb4e7c6 |
fix(ci): dispatch frozen-line builds with GITHUB_TOKEN (#4888)
## What this PR does The first RC freeze hit HTTP 403 when dispatching `build-release.yaml` because the CI app has no Actions write permission ([run](https://github.com/cozystack/cozystack/actions/runs/38229949424)). I switched this dispatch to `GITHUB_TOKEN` with job-scoped `actions: write`. Tag and branch pushes still use the app token. Verified with `make unit-tests`, zizmor 1.26.1 and the commit trailer check. The regression test fails on the old workflow and passes with the fix, including warning and manual recovery on dispatch failure. Live dispatch with `GITHUB_TOKEN` has not been tested. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): use GITHUB_TOKEN to start the initial packages build when freezing a release line ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated release automation to use appropriate permissions and authentication for CI triggers. * Improved warnings and recovery guidance when automated package builds cannot be started. * **Tests** * Added automated checks covering release dispatch permissions, branch targeting, successful execution, and non-blocking failure handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
a6cb84673e |
docs(release): check changelog claims against the code (#4936)
## What this PR does Changelog generator sees only `docs/agents/changelog.md`, two inline prompts and `AGENTS.md`. Failure modes and pre-publish checks live in `docs/release.md`, which it never reads. The prompt has detailed rules for authors and PR numbers and none for checking a claim against the code. I reviewed the generated v1.7.0 draft against the code and found 22 statements wrong or missing, while all authors and PR references were right. Several were in breaking changes: a talm fix credited to a talm release that doesn't contain it, worker pools said to be adopted without churn while Talos workers are rolled once because of the hash-named TalosConfigTemplate (#3523), migrations said to run on a variant that has them disabled. Three changes: - `docs/agents/changelog.md`: new section 8b about verifying claims against the code. Identifiers, migration behaviour and variant scope come from the tree at the upper-bound tag, required actions come from release notes and newly added upgrade sections, not only from `!` commits. Section 9 now asks to run the `docs/release.md` checks and `hack/validate-changelog.sh` on the written file before exit. - `.github/workflows/changelog-generate.yaml`: refresh `docs/release.md` and `hack/validate-changelog.sh` from the dispatch ref, next to the existing `changelog.md` refresh. Job checks out the rc tag detached, so without this generator runs older copies from that tag. Permissions, tokens and tool flags are unchanged. - `hack/validate-changelog.sh`: reject an entry cut off with an ellipsis right before its attribution, and any link to `cozystack.io/docs/next/` (live site answers 404). Reason names the offending lines in one line, because workflows quote it into a single annotation. How I checked it: - Old validator accepts the original v1.7.0 draft, new one rejects it on three truncated entries (lines 335, 393, 518) and, after those are repaired, on the `/docs/next/` link (line 25). All 29 shipped changelogs from v1.3.0 to v1.6.4 still pass. - New contract test plants each defect into a shipped changelog, checks the reason names the line, and checks that near misses real changelogs carry still pass (`content/en/docs/next/` in prose, ellipsis inside a sentence, three dots of a compare link). It goes red on the old validator and on widened patterns. - Overlay test pins both new copies and the cone-mode sparse checkout that brings `docs/release.md` in. It goes red if either copy is removed, cone mode is turned off or `docs/agents` is dropped from sparse paths. - I ran every command the new prompt rules prescribe on v1.6.0..v1.7.0-rc.1, each one finds an error the draft shipped. - `make bats-unit-tests bats-posix-compat-tests` pass (2424 and 1120), zizmor and actionlint are clean on the workflow. Not included: a script that gives the generator component versions and migration headers, and a networked check for references, authors, link liveness and companion-release containment. Both are separate changes. Things I noticed and left alone: - validator summary in the Phase 2 section of `docs/release.md` doesn't list the two new rejections - `docs/agents/changelog.md` runs `docs/changelogs/hack/check-optional-repos.sh`, but the script lives at `hack/check-optional-repos.sh` and still resolves repo root from the old location - a changelog the validator rejects is still not uploaded as an artifact, so rejection costs a regeneration or a hand-commit, that is why section 9 has the generator run the validator first - current v1.7.0 draft still has the three truncated entries, a promotion that picks it up will discard it and generate a new one unless the entries are rewritten - three unrelated bats tests (in `check-gpu-recording-rules.bats` and `promote-rewrite-tags_test.bats`) fail under en_US.UTF-8 collation on main too, they pass under C.UTF-8 that runners use, I ran the suite that way ### Screenshots Not applicable, there is no UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I walked the trigger map in `docs/agents/contributing.md` against each changed file: - `docs/agents/changelog.md` is the changelog prompt, not the commit convention or the PR template that ccp and community restate. Its list of release-window repositories is unchanged. - `.github/workflows/changelog-generate.yaml` is not `tags.yaml`, the release-prep workflow ccp relies on. - `hack/validate-changelog.sh` is changed in place, not moved or renamed, and it is not `hack/package.mk` or `hack/update-crd.sh`. - `hack/release-changelog-contract.bats` is a test. No package, values key, CRD, release asset, make target or node prerequisite changes. Code search across the organization finds no reference to any of these files outside this repository. ### Release note ```release-note docs(release): check changelog claims against the code. The changelog generator now verifies operator-facing statements against the code and runs the release checks on its own output, and the changelog validator rejects entries cut off with an ellipsis and links to cozystack.io/docs/next/. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Release guidance now emphasizes verifying operator-facing claims against the release version and carrying required actions into upgrade instructions. * Documentation links in changelogs should point to versioned documentation. * **Chores** * Changelog validation now flags truncated entries and links to unversioned “next” documentation. * Release guidance and validation checks are refreshed automatically during changelog generation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d1d4e92acb |
feat(release): sign the release assets and attach a source SBOM (#4861)
## What this PR does Implements the release-asset part of #4759: a release cut from a tree that has this change gets an SBOM of its source and one keyless signature that covers all of its assets. These are the two CLOMonitor checks still failing for this repository, `sbom` and `signed_releases`, after #4757 and #4899 brought the project score to 96.69. `hack/upload-assets.sh` now collects its asset list into an array and hands it to the new `hack/sign-release-assets.sh` before the first upload. The script writes three files into `_out/assets`, and they are uploaded with the rest: | File | What it is | |---|---| | `cozystack-source-sbom.spdx.json` | SPDX 2.3 SBOM made with syft, over `git archive HEAD` (the committed release tree, without `_out/`, `.release-tooling` or runner leftovers) and the unpacked `cozypkg-*.tar.gz` binaries. It does **not** describe the container images that `packages/` pins by digest (their SBOMs are #2937, A2 in #3278) or the Talos ISO, disk, kernel and initramfs images. It is named by what it covers, so `cozystack-sbom` stays free for an SBOM of the release as a whole. CLOMonitor's `sbom` check matches `sbom` in an asset name. | | `cozystack-checksums.txt` | SHA-256 of every uploaded asset and of the SBOM. | | `cozystack-checksums.txt.sigstore.json` | Sigstore bundle of `cosign sign-blob` over the checksums, keyless, signed with the job's GitHub OIDC identity. | Before it returns, the script runs the same `cosign verify-blob` that `docs/release.md` gives users, with `--certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)'`: `tags.yaml` on a `v*` tag push, `promote-rc.yaml` dispatched from `main`. A signature that check would reject therefore fails the job before anything is uploaded. It also refuses its own three outputs as inputs: `promote-rc.yaml` downloads every rc asset, and a stable release has to get freshly computed checksums and a fresh signature, since promotion restamps several assets. Workflows: `tags.yaml::prepare-release` and `promote-rc.yaml::promote`, the two jobs that call `make upload_assets`, get `id-token: write` and two setup steps: - syft v1.54.1 from its release tarball, checked against a committed SHA-256 (`c069905b…`, the release's checksums entry; the checksums file verifies against anchore/syft's keyless release signature). Not `anchore/sbom-action/download-syft`: pinned by commit, it still runs syft's `install.sh` fetched from a tag at run time, in a job that holds the app token, the GHCR login and `id-token: write`. v1.54.1 is past GHSA-mw2c-m758-9v5q (fixed in v1.52.0). `hack/ci-tool-installs-contract.bats` now fails on `anchore/sbom-action`, on a syft install script, on a syft download without `sha256sum -c` and on version or checksum drift between the two workflows. - `sigstore/cosign-installer` v4.1.2, pinned by SHA, with `cosign-release: v3.1.3`: the version `docs/release.md` asks users for, so the job checks its own signature the way they will. The action verifies that binary against cosign's keyless and KMS bundles with its digest-pinned bootstrap cosign. `hack/ci-tool-installs-contract.bats` holds the action commit and the release equal in both workflows and fails on a step without `cosign-release`. `docs/release.md` gets a "Release signatures and SBOM" section with the user-side verification: cosign v3.1.3 or later (v2.6.5 on the v2 line), the two commands joined with `&&`, what the SBOM covers and what the certificate identity proves. It now also says, at each place `promote-rc.yaml` is run, that it is dispatched from `main` only. ### Decisions for the release owners - **Fail closed.** If Fulcio or Rekor is down, the release job fails and nothing is uploaded. I chose that so that no release goes out unsigned. If you would rather publish unsigned than block, the change is a `|| echo ::warning::` around the call in `upload-assets.sh`. - **Promotion runs the rc tree's script.** `promote-rc.yaml` calls `make upload_assets` from the rc checkout, not from `.release-tooling`. The first signed stable release is therefore the promotion of the first rc tagged after this merges. `release-1.6` gets signatures only if this is backported. CLOMonitor reads the latest release, so the score moves only when that release carries the files. - **Certificate identity pins the refs** (decided in review): `tags.yaml@refs/tags/v…` and `promote-rc.yaml@refs/heads/main`. A `promote-rc.yaml` dispatched from another branch fails the job's own check at the upload step, after the staging branch, the `promotion-*` candidate and the empty draft release exist; nothing in `parse` refuses a non-`main` dispatch up front. Until the `v*` tag rulesets exist, a hand-pushed `v*` tag whose tree has an edited `tags.yaml` still signs, which `docs/release.md` says. - **Follow-ups after the first signed release** (#4759 stays open for them): - `project.documentation.signature-verification` in `security-insights.yml` (#4757); - the SBOM row in `docs/security/self-assessment.md`, which says "Not published at this time"; the image caveat above stays in it. I left both alone so that they do not describe releases that don't exist yet. ### How it was checked - Bats 1.14.0 (the version CI installs), with gh, syft and cosign stubbed: - `hack/sign-release-assets.bats`, 8 tests: what the checksums cover; that syft sees the committed tree and the cozypkg binaries but not untracked files; that the verify identity accepts `tags.yaml` on a `v*` tag and `promote-rc.yaml` on `main`, and rejects a fork, another workflow, `tags.yaml.evil`, `tags.yaml` on a branch, and `promote-rc.yaml` on `release-1.6`, on `main-evil` and on a tag; that `docs/release.md` shows the same regexp as the script; that a failed verification fails the script; that it refuses a missing asset, assets split over two directories (with a same-named file in both, so the test fails if the directory check is removed) and its own outputs as input. - `hack/upload-assets.bats`, 4 tests, new: the exact upload list and order (the assets, then the SBOM, the checksums and the bundle); that the checksums name every upload but the last two; zero `gh` calls when verification fails or an asset is missing. The old `upload-assets.sh` fails all four, a version that signs after uploading fails the last two. - `hack/ci-tool-installs-contract.bats`, 8 tests. Putting `download-syft` back, dropping the `sha256sum -c` line, letting `SYFT_VERSION` drift between the workflows, dropping `cosign-release` from one step, letting it drift, unpinning the installer or giving one step another installer commit each fails it. - Putting the old any-ref regexp back in the script fails the identity test and the docs tie; putting it back in the docs alone fails the tie. - The meta-suites `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage` and `bats-runner-contract` pass: 90 tests across these eight files. - The pinned regexp gives the same accept/reject results under `grep -E` and Go's `regexp` (what cosign uses). cosign-installer's install script at the pinned commit, run as written with `cosign-release: v3.1.3`, verifies the binary's keyless and KMS bundles and installs v3.1.3 (its SHA-256 matches `cosign_checksums.txt`). `--yes`, `--bundle`, `--certificate-identity-regexp` and `--certificate-oidc-issuer` are listed, not deprecated, in cosign v3.1.3's help. - The `Set up syft` step's script, run as written with `RUNNER_TEMP`/`GITHUB_PATH` set: the checksum passes and `syft version` reports 1.54.1. A real syft v1.54.1 run through `hack/sign-release-assets.sh` (cosign stubbed) over this tree and the v1.6.5 `cozypkg-linux-amd64.tar.gz` took 10 s and wrote an SPDX 2.3 document with 920 packages: 440 golang, 373 npm, 103 github, 4 without a purl; 60 come from the cozypkg binary. - cosign behaviour behind the docs, with a key-signed legacy bundle whose `cert` holds the bare public key, against the documented identity regexp (tlog ignored, since an attacker can upload their own entry): v2.6.4 and v3.1.2 print "Verified OK", v2.6.5 and v3.1.3 reject it. `--new-bundle-format` makes v2.6.4 reject it but not v3.1.2, so the docs give the minimum version and not the flag. A real keyless protobuf bundle (syft's) verifies on v2.6.5 and v3.1.3. - zizmor 1.26.1 `--offline` reports no findings. actionlint 1.7.12 (with `.github/actionlint.yaml`) reports the same 13 pre-existing info-level shellcheck notes in `tags.yaml`/`promote-rc.yaml` as before, nothing new. shellcheck passes on both scripts. `hack/check-commit-trailers.sh` passes. - Not checked: a real keyless signature. It needs the OIDC token of a job in this repository, so the first rc after the merge is the end-to-end test. Cutting an `-alpha.N` first would exercise it without touching an rc. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I checked the trigger map file by file. The only entry `hack/upload-assets.sh` can trigger is renaming a release asset, because the website install pages hardcode asset names. This PR adds three assets and renames none of the released ones, `cozypkg-checksums.txt` included (the SBOM was renamed to `cozystack-source-sbom.spdx.json` in review, before any release carried it). The workflow and docs changes match no entry. ### Release note ```release-note feat(release): releases carry cozystack-source-sbom.spdx.json, an SPDX SBOM of the source tree and the cozypkg binaries (not of the container or Talos images), cozystack-checksums.txt with the SHA-256 of each asset, and cozystack-checksums.txt.sigstore.json, a keyless Sigstore signature over the checksums made by the release workflow; docs/release.md shows how to verify a download with cosign v3.1.3 or later. ``` Refs #4759 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Releases now include an SPDX software bill of materials, SHA-256 checksums, and a keyless Sigstore signature bundle. * Release assets are verified against the expected workflow identity before upload; uploads stop if verification fails. * **Documentation** * Added guidance on release signatures, SBOM coverage, asset verification, and which branch dispatches release workflows. * **Tests** * Added coverage for generated release files, upload behavior, and signing and verification failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Marian Koreniuk
|
bc24b588b2 |
feat(release): sign the release assets and attach a source SBOM
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed. hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order. The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only. Refs #4759 Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
Myasnikov Daniil
|
e2ca1b7bc5 |
ci(release): overlay the release checks the changelog agent runs
The changelog job checks out the rc tag detached, so every file the agent reads by its working-tree path is the rc tag's copy. Only the agent instructions were refreshed from the dispatch ref. The instructions now tell the agent to run the checks in docs/release.md and hack/validate-changelog.sh on its own output before it exits, and the rc tag's copies of those predate any check added since, or for an old enough tag do not exist at all. Refresh both alongside the instructions. docs/release.md is under neither sparse-checkout path and reaches the tooling checkout only because cone mode also carries the files of every parent directory of docs/agents. The contract test pins both copies and that dependency, so a switch to non-cone patterns fails here rather than as a hard failure of the refresh step at promote time. The AI step keeps its read-only token and its tool flags. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Alexey Artamonov
|
ecdc043e63 |
build(seaweedfs): keep Renovate and git whitespace off the in-tree build
Renovate would bump the builder, the base image or their digests one at a time into a combination upstream never shipped, and the next `make update` would silently put them back, so it leaves that Dockerfile to the update recipe. The patches are unified diffs of Go sources, whose context lines start with a marker before tabs and blank lines, which git's whitespace checks would otherwise flag or rewrite. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
cfa9c5a5c7 |
fix(e2e): keep chainsaw asserts polling through transient API errors (#4867)
## What this PR does Chainsaw asserts in the e2e suites can fail long before their timeout. Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other API error. When etcd changes its leader, kube-apiserver answers the GET with a 500, and whichever suite happens to be polling goes red. Between 2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached chainsaw, which is 6 of the 23 assert failures in that window. Each one has `etcdserver: leader changed` in the apiserver logs within 0.2 s of the failure. Five of the six came after 10-09, and four of them were the computeplane ingress-nginx assert. The fix is kyverno/chainsaw#2761, which is still open and not in any release. Until it is released, the e2e sandbox image builds chainsaw v0.2.15 from source with that PR's commit applied unchanged, using the same Go minor and build flags as the v0.2.15 release binary. The build runs the tests of the patched packages. It also runs chainsaw against a fake apiserver that answers 500: a short run of 500s must not fail the assert, and an endless one must fail it at its timeout. Plain v0.2.15 fails both cases after the first GET. The Dockerfile header lists what to remove once a chainsaw release carries the fix. The README sentence added here has to go too, and so does the Renovate rule that keeps the builder on Go 1.26. The upstream PR also breaks one chainsaw test outside the packages it touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks the new `; operation terminated with error: context deadline exceeded` suffix. I found it with a full `go test ./...` on v0.2.15 with and without the patch. The image build tests only the patched packages, so that test does not run here. One trade-off: when the API stays broken, an assert or `error` step now runs to its timeout instead of failing within seconds, so a red run caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw installed on its runner, not the sandbox image, so it does not get this fix. Known gaps, left for later: the new test covers only an assert on a named object, not one that lists objects or an `error` operation, and its skip guard does not notice if no test runs at all. The patch file has no `-whitespace` entry in `.gitattributes`, so `git diff --check` reports whitespace inside the upstream patch. A multi-platform build of the image would run the builder tests once per platform; the image is built for amd64 only. Closes #4720 ### Screenshots Not a UI change. ### Downstream repositories None of them builds or uses the e2e sandbox image; the change stays inside this repo's CI. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * End-to-end sandbox assertions now retry after transient Kubernetes API errors instead of failing immediately. Persistent errors still cause assertions to fail when their timeout expires. * **Testing** * Added checks that verify assertions recover after temporary API errors and fail within the expected timeout when errors persist. * **Maintenance** * Updated sandbox image build and dependency update settings for the Chainsaw testing tool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
785cbc391b |
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery (#4293)
## What this PR does Fixes #4060. `flux-plunger` recovers a HelmRelease stuck with `has no deployed releases` by briefly suspending it, deleting the stale Helm storage Secret, recording a processed-version annotation, and unsuspending. helm-controller v1.5.x skips the Helm uninstall and drops its finalizer when it handles a delete on a suspended HelmRelease, and it wakes on the same delete event flux-plunger does, so a delete arriving inside that window orphaned every resource the release owns (for a VMDisk: DataVolume, PVCs, the CDI importer pod, RBAC, WorkloadMonitor and the storage Secret itself). Lifting the suspension on the delete event cannot win that race reliably. The bug is not specific to VMDisk. The change makes such a delete reach the uninstall: - **An uninstall fence.** The suspension is applied together with a flux-plunger finalizer, `flux-plunger.cozystack.io/uninstall-fence`, in one server-side apply, so flux-plunger never holds a suspension without it. If a delete arrives while the release is suspended, the fence keeps the object whether helm-controller or flux-plunger handles the delete first: flux-plunger lifts its suspension and keeps the fence, helm-controller reconciles the unsuspended generation and runs the uninstall, and flux-plunger removes the fence once helm-controller's finalizer is gone and it has observed that generation. A recovery that completes lifts the suspension and the fence together. A terminating HelmRelease never begins recovery, and termination is re-checked immediately before deleting the storage Secret. - **Only flux-plunger's own suspension is acted on.** Ownership is decided by which field manager owns `spec.suspend` in `metadata.managedFields`: the apiserver transfers the field away the moment an operator (or `cozyhr resume`, or a backup/restore controller) writes a different value to it, so a later suspension is never misread as flux-plunger's. A suspension set through the shared `flux-client-side-apply` manager (platform migrations, the tenant delete hook, an operator's manual `kubectl apply`) is never claimed. A release deleted under another manager's suspension is that manager's decision: flux-plunger does not hold it with the fence, and helm-controller leaves its resources in place, as the migrations that suspend and then delete rely on. - **No suspension taken over in a race.** Acquiring applies without forcing first and, on a conflict, re-fetches and forces only if `spec.suspend` is still `false`: a backup controller owns the field at `false` via a dynamic-client `Update`, which a non-forced apply can never take. Every apply flux-plunger sends, acquiring as well as lifting the suspension or the fence, is preconditioned on the `resourceVersion` and `uid` of the read that decided on it. The `resourceVersion` makes a write landing in between fail the apply instead of being absorbed: acquire then defers rather than adopt or take over a suspension set meanwhile, and lifting the suspension and the fence from a release read as live cannot delete it without an uninstall if helm-controller has handled a delete in the meantime. The `uid` stops an apply to a release that has gone from creating an empty one, since server-side apply creates what does not exist. A conflict is retried through the requeue. The apply carries only `spec.suspend` and the fence, so it cannot leak the required `spec.interval` and conflict on it. - **Fresh reads.** The release, every re-check and the storage Secret list are read from the apiserver (an uncached `APIReader`), not the informer cache. A lagging cache would let the processed-version guard delete the revision below the one just reclaimed, and the ClusterRole grants secrets `get`/`list` but not `watch`, which an informer needs to stay synced; flux-plunger deliberately does not cache Secrets. Every step requeues with backoff on failure. - **Events.** A Warning when flux-plunger holds a suspension it failed to clear, and a Warning when a delete has been held by the fence for more than ten minutes waiting for helm-controller (the held state is re-checked on a timer, not only on helm-controller's next status write). A release deleted under another manager's suspension gets a Normal event when that is a migration's choice, and a Warning when the release carries flux-plunger's processed-version annotation, because then the skipped uninstall is an accident, not a decision. Normal events also mark an externally-owned suspension left alone and a recovery deferred because the release changed between read and write; an external suspension on a release flux-plunger has recovered before is a Warning instead, because it is most likely one an earlier flux-plunger left mid-recovery (see Operating notes). Events from the shutdown drain are written straight to the apiserver, since the manager's recorder stops with the manager. - **The fence is released when flux-plunger goes.** Once its manager has stopped, and with it every reconcile, flux-plunger drains its fences under a deadline inside the pod's termination grace period; a stop that overran its 10s bound is drained too and still reported: a recovery in progress is aborted, the suspension and the fence lifted together for the next flux-plunger to start again, and a release being deleted has flux-plunger's suspension lifted and keeps the fence until helm-controller has run its uninstall. A release helm-controller does not handle in time keeps its fence. The drain lives in the controller, so the envtest suites hold it against a real apiserver's generations and finalizers. A pre-delete hook on the chart is the backstop for what the drain could not finish, so uninstalling the component never leaves a release it cannot finish deleting: it stops flux-plunger, waits out its grace period, then lifts any remaining suspension and fence without waiting again, naming each release it lets go of before helm-controller uninstalled it. It lifts a suspension only where flux-plunger alone holds it, the outcome the controller's own relinquish has, with a JSON patch of `spec.suspend` preconditioned on the `resourceVersion` and `uid` it read; not with `kubectl apply --server-side`, which on a release last written by a client-side `kubectl apply` moves that apply's fields to the applying manager and strips every one the body leaves out. helm-controller runs a pre-delete hook under the release's uninstall timeout, 300s by default and not set by the operator, so the Job's deadline sits under it. A Deployment that cannot be stopped or a listing that never succeeds ends the hook before any fence is touched, and a fence it cannot remove ends it after the sweep; either way it exits non-zero, because a failed pre-delete hook leaves the release untouched and helm-controller retries the uninstall, hook included, whereas exiting 0 would let the component go and strand the fence. A flux-plunger pod still there after its grace period, typically one stuck `Terminating` on a `NotReady` node, is only reported, and the pod wait is cut to what is left of the hook's retry window; because such a pod may still be setting fences, the fences are listed again after the sweep and any found fails the hook. The Job runs once and is kept after success so its report stays readable until its TTL; `helm.sh/hook-output-log-policy` is deliberately not used, because helm-controller never sets Helm's hook output writer and calls a nil one for it (fluxcd/helm-controller#1562). The hook's kubectl image is digest-pinned in the chart values, and its RBAC is only what `kubectl scale`, the listing and the patches issue. The chart's README documents the finalizer and how to clear it by hand. A single writer is kept by leader election, not by the rollout. A second process acting under the same field manager could clear a suspension the first is still recovering, and no Deployment strategy prevents one: the next pod starts as soon as the old one is terminating (Kubernetes v1.35 has no pod-replacement policy for Deployments), and an eviction or a node drain replaces a pod the same way, while the old one stops and drains. The chart now passes `--leader-elect` with a Role on `leases` in the release namespace. The Lease is not given up on stop and outlasts the last renewal, the renew deadline, the manager's bounded stop (10s) and the drain, and only a flux-plunger that led drains at all, so the next one acts only once the old one is done; the price is that a new pod waits for the Lease to run out, a little under two minutes at the default. The timings derive from one value, `shutdownGracePeriodSeconds` (90 by default, refused outside 40 to 140 at render): the pod's termination grace period, the drain deadline (25s less: the manager's 10s stop plus a 15s margin before the kubelet's SIGKILL, both tied to the Go constants by a contract test) and the hook's pod wait. The strategy keeps `maxSurge: 0` and `maxUnavailable: 1` and stays `RollingUpdate` rather than `Recreate` on purpose: the chart shipped without a strategy block, so every existing install's live Deployment carries the apiserver-defaulted `rollingUpdate` object and helm-controller applies server-side, where laying `type: Recreate` over that populated block is rejected on upgrade. ### Operating notes A HelmRelease carries the fence only while flux-plunger is recovering it or waiting for helm-controller to uninstall it. A fenced release whose uninstall keeps failing stays `Terminating` exactly as any HelmRelease does under helm-controller's own finalizer, and raises the Warning above. Stopping flux-plunger drains its fences, and uninstalling it releases whatever the drain left through the pre-delete hook; a fenced release that is deleted while flux-plunger is merely scaled to zero, or killed without a graceful stop, and that helm-controller has not uninstalled, stays `Terminating` until flux-plunger runs again. To release one by hand, after confirming its uninstall has run or is not wanted: ```bash kubectl patch helmrelease NAME --namespace NAMESPACE --type json --patch '[{"op":"test","path":"/metadata/finalizers/INDEX","value":"flux-plunger.cozystack.io/uninstall-fence"},{"op":"remove","path":"/metadata/finalizers/INDEX"}]' ``` where `INDEX` is the position of `flux-plunger.cozystack.io/uninstall-fence` in `metadata.finalizers`; the `test` operation makes the patch fail rather than remove another finalizer if the list has changed. Earlier flux-plunger releases set `spec.suspend` through the shared `flux-client-side-apply` manager, so a suspension one of them left behind cannot be told apart from a migration's or an operator's and is treated as external: it is not cleared automatically. Such a leftover exists only if the old pod stopped in the middle of a recovery, and flux-plunger raises a `SuspensionNotOwned` Warning on it. The chart README carries this section too. To list candidates, suspended releases carrying the flux-plunger annotation whose `spec.suspend` is not owned by flux-plunger's apply: ```bash kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output json | jq --raw-output '.items[] | select(.spec.suspend == true) | select(.metadata.annotations["flux-plunger.cozystack.io/last-processed-version"] != null) | select([.metadata.managedFields[]? | select(.manager == "flux-plunger" and .operation == "Apply" and .fieldsV1["f:spec"]["f:suspend"] != null)] | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"' ``` Clearing one is an operator action: confirm nothing else (a migration, a restore) is holding it, then resume it with `flux resume helmrelease NAME --namespace NAMESPACE`. ### Testing The regression the issue asks for is `TestEnvtest_DeleteDuringRecoveryReachesUninstall`: against a real apiserver, a delete lands while flux-plunger holds its recovery suspension, and helm-controller's v1.5.0 delete handling is reproduced step for step (uninstall only when not suspended, finalizer dropped either way, observed generation recorded). It covers both orders, helm-controller handling the delete first and flux-plunger handling it first; without the fence the first order leaves the release gone with no uninstall. The same suite pins the managedFields ownership dynamics a fake client cannot represent: the field transfer, the forced acquire over a dynamic-`Update` owner, the `resourceVersion` precondition rejecting a write that races either apply, the boundary that a `flux-client-side-apply` suspension is never claimed, and interval preservation. The fence mechanics were also checked against a live helm-controller: with another finalizer holding a deleted, suspended release, lifting the suspension made helm-controller run the uninstall and remove the release's resources. The envtest suite runs on every pull request: a new `make test-controllers-envtest` target builds a pinned `setup-envtest` and fetches a kube-apiserver and etcd into `_out/envtest`, which CI caches and the target reuses without asking the release index, takes the HelmRelease CRD from the in-tree Flux manifests, and CI calls it next to `make test-controllers`; the control plane gets a longer start timeout because it starts beside the other test goals. The drain is held there too: it aborts a live recovery, carries a delete through the uninstall and lifts the fence only once helm-controller has observed the unsuspended generation, keeps the fence when helm-controller never answers, releases a fence left without a suspension at once, and leaves another manager's suspension alone. `TestEnvtest_TheNextFluxPlungerLeadsOnlyAfterTheDrain` runs two real managers on one Lease: a clean stop still drains, and the second manager becomes leader only after the first one's drain has ended; it goes red if the Lease is given up on stop. Unit tests cover the controller logic (every branch, ordering, requeue, and the ownership decision through an injected seam), the Lease timing relations, that only a leader drains and through the manager's API reader rather than its stopped cache, that a failed listing is retried, that a stop which overran its grace period is still drained and its events reach the apiserver, and contract tests tie `main.go`, the chart's flags, the drain's arithmetic and the hook's names to the Go code. The pre-delete hook is covered by helm-unittest and by `hack/flux-plunger-fence-hook_test.bats`, which runs its rendered script against a fake `kubectl` that keeps each release's finalizers, suspension owner, generation and observed generation, bumps the generation on a spec write as the apiserver does, answers NotFound without `--ignore-not-found`, spends simulated time on waits, lists a release once however many copies of the fence it carries, refuses `kubectl apply`, lifts a suspension only through the preconditioned JSON patch and never one another manager also holds, reads a negative `kubectl wait` timeout as a week as kubectl does, and applies a finalizer removal only when it is the guarded test-then-remove at an index that holds the fence. Every mutation in a set covering each predicate conjunct, each retry, each exit path, the re-listing, the timings and the selector turns the hook suite red, as does every mutation in the Go set covering the drain's branches, the Lease timings and the events; three drain mutations that still survive are tracked in #4782. The hook script was also run as a Job in the hook's own image on a live cluster, succeeding, with the Deployment already gone, and with patch denied by RBAC, where it exited non-zero and named the release it left; that run caught that a per-call `--request-timeout` makes kubectl drop its in-cluster config, so the hook sets none and the fake rejects it. A later live run caught `kubectl apply --server-side` stripping the finalizers and spec of a release created by a client-side apply; with the JSON patch the same releases kept every field, the suspension flux-plunger alone held was lifted, a co-owned and a foreign one stayed, and every fence went. ### Remaining limitations - A delete that arrives after recovery has deleted the storage Secret of a release whose only revision was that one finds no revision to uninstall; helm-controller then treats the release as already gone. - A writer that sets `spec.suspend: true` while flux-plunger already holds it at `true` does not take the field over, because the value does not change. If such a writer then deletes the release inside the recovery window (the suspend-then-delete that the tenant delete hook and some platform migrations use to skip the uninstall), flux-plunger still sees its own suspension, lifts it, and helm-controller runs the uninstall that writer meant to skip. The same held before this change, when the end of recovery unsuspended unconditionally. - The already-processed guard compares revision numbers only. When Helm re-creates a revision under the same number after recovery deleted it (a release whose only revision fails again the same way), recovery runs again for it. - `listHelmReleaseSecrets` reads the local cluster only, so a HelmRelease that targets a remote cluster via `spec.kubeConfig` (whose storage Secrets live in the remote cluster) is not recovered; it is skipped, never mis-handled. - A new flux-plunger pod waits for the old one's Lease to run out before it acts, on every restart and after a crash: a little under two minutes at the default. Recovery is not latency-sensitive, and this is what keeps a single writer while the old pod drains. - The leader Lease is left in the release namespace after the chart is uninstalled, as with any controller-runtime leader election; it is inert. ### Screenshots Not applicable: no UI change. ### Downstream repositories The trigger map was walked against the diff: it touches `internal/controller/fluxplunger`, `cmd/flux-plunger`, the `packages/system/flux-plunger` chart (templates including RBAC for the leader Lease, a pre-delete hook, values, tests, its package `Makefile`, README), a new bats test under `hack/`, the root `Makefile` (a new `test-controllers-envtest` target; existing targets are unchanged) and the pull-request workflow that calls it and caches its binaries. No row matches: ccp gates on existing `hack/` files and make targets, and nothing there is moved, renamed or changed in behaviour; a new test file and a new target reach none of them. The website's FluxPlunger section describes the recovery steps, which still hold; it does not describe the fence or the drain, which the chart README now documents. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(fluxcd): a HelmRelease deleted while flux-plunger is recovering it now reaches the Helm uninstall instead of leaving its resources (DataVolume, PVCs, RBAC, storage Secret) behind ``` |
||
|
Myasnikov Daniil
|
e05cef4d48 |
fix(ci): dispatch frozen-line builds with GITHUB_TOKEN
The CI app lacks Actions write access, so freezing a new release line leaves its initial packages build undispatched. Use the repository token for workflow_dispatch without expanding the app installation permissions. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
myasnikovdaniil
|
ebceef4105 |
ci(retention): sweep packages named by the published nightly tree (#4826)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The Artifact Retention workflow has failed every day since mid-August, and it never pruned anything. It lists the org's container packages with `GET /orgs/cozystack/packages?package_type=container`, and that endpoint answers `400 Invalid argument` to the app installation token the job uses. Others hit the same in actions/github-script#449 and cli/cli#9606. An old `|| true` used to hide it. This is item A19 in #3278. The per-package endpoints accept the same token: a probe run confirmed it, and the dry-run below read every package with it. So the job no longer lists the org. The new `hack/retention-packages.sh` takes the package names from the published `cozystack-packages:nightly` tree, reading refs through the shared `hack/lib/image-refs.sh`. It also adds `cozy-installer` and `cozystack-nocloud`, which the nightly publishes outside the tree. The committed tree can't be the source: it misses images whose digest only the build pins, and those still get dated nightly tags every day. Failures now turn the run red instead of passing silently: - if the name list is empty, the nightly sweep is skipped and the run fails, but abandoned promotion candidates are still handled, because their protection doesn't depend on that list - if a package's versions can't be read, that package is skipped and the run fails at the end - if a DELETE is refused, `hack/retention-delete.sh` logs an error with the HTTP status, goes on with the remaining versions and fails the run (before, this was only a warning) - if names resolve but none of the packages is linked to this repository, nothing gets swept, so the run fails too, after the promotion sweep has run (that sweep itself is skipped only when candidate protection can't be resolved) Promotion-candidate protection itself is unchanged. The contract test that pinned the old org listing now pins the new behaviour, and both new scripts have bats tests. Known gap: a component that disappears from the nightly tree drops off the list, so its old dated nightlies are never pruned. Today none of the packages outside the list has more dated nightlies than the keep window. Dry-run on this branch is green: https://github.com/cozystack/cozystack/actions/runs/38075159900. It resolves 59 packages, skips `cozystack/cozystack-ui` with the warning, and would delete 418 old nightly versions plus one abandoned promotion candidate. Deleting needs a different token. The app token lists versions, but a one-version apply run (https://github.com/cozystack/cozystack/actions/runs/37939905343) got `404 Package not found` on DELETE for a version it had just listed. The same DELETE with this repository's `GITHUB_TOKEN` and the job's `packages: write` worked in a second run (https://github.com/cozystack/cozystack/actions/runs/37941084456), with no change to package settings. So `hack/retention-delete.sh` now deletes with `GITHUB_TOKEN`, and listing stays on the app token. That delete was shown on a package linked to this repository. One of the swept packages, `cozystack/cozystack-ui`, is linked to another repository, and I didn't try deleting there. Such a package is skipped before its versions are read and named in a warning, so it can't turn every apply run red. The follow-up for it is in #3278. `RETENTION_APPLY` stays off, so the scheduled run is a green dry-run until someone turns it on. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> N/A ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ci(retention): the GHCR retention workflow takes package names from the published nightly tree instead of the org package listing, which rejects the app token, and fails the run whenever names, versions or deletions cannot be resolved. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Nightly package retention now targets packages referenced by the published nightly tree and keeps the 14 newest eligible versions per package. * Promotion candidates older than 30 days are eligible for pruning only when their digests are not protected by qualifying release pull requests or branch tips. * **Bug Fixes** * Packages linked to another repository are skipped, and sweep failures are reported without preventing other eligible packages from being processed. * **Documentation** * Clarified which nightly versions and promotion candidates qualify for retention. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
34488db766 |
ci(e2e): never tear down the Proxmox tenant on the runner's default context (#4807)
## What this PR does The Proxmox e2e teardown could delete from the wrong cluster. It runs under `always()` and got its kubeconfig only from `GITHUB_ENV`, which the "Stage credentials" step writes. If a step before that fails, for example a missing stand variable, `KUBECONFIG` is never set. The three `kubectl delete` calls then go to whatever cluster the self-hosted runner's own kubeconfig selects. Every teardown call now passes the staged file with `--kubeconfig`. Stage fails unless that file's current context resolves to a server, and prints kubectl's own error when it does not. Before, an empty `COZY_KUBECONFIG` secret let client-go fall back to its defaults, in-cluster credentials among them. The suite step gets the staged path in its own `env`. The `GITHUB_ENV` export is gone, so no step in this workflow picks its cluster up implicitly. Tear down decides whether to skip from how Stage ended, not from the file. If Stage did not succeed, the suite never ran and there is no tenant to remove, so the step prints a notice and exits 0. If Stage succeeded and the file no longer resolves, the step fails with kubectl's error, because the tenant and its VMs may still be there. A side effect: a run whose Stage fails no longer cleans up a tenant left by an earlier `keep_tenant` run. The next run that passes Stage without `keep_tenant` removes it. The Stage and Tear down scripts now have unit tests in `hack/proxmox-teardown-contract.bats`, run against a stub `kubectl`. I also ran both against a real kubectl v1.37.1 for every Stage outcome. The files were missing, empty, garbage YAML, without a current context, with a dangling context, without a server, and valid. One more case was a runner without kubectl. The deletes also no longer stop at the first failure. Under `set -e`, a failed first delete skipped the tenant delete, and the tenant's VMs stayed on the hypervisor. Now each delete runs, and the step fails at the end if any of them failed. With a recording `kubectl` that fails one chosen delete, all three ran in every case and the step exited 1. With the change reverted, a failing first delete was the only one that ran. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): the Proxmox e2e teardown always uses the staged kubeconfig, so it can no longer delete from the runner's default cluster. It skips only when Stage did not succeed and fails when the kubeconfig breaks after Stage. A failed delete no longer skips the tenant delete. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Improved reliability of automated Proxmox test setup and cleanup. Tests now stop setup when Kubernetes configuration is unavailable, skip cleanup when setup fails, and attempt all resource deletions before reporting cleanup failures. * **Tests** * Added checks for configuration validation, cleanup behavior, and handling of failed deletion attempts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
813c618086 |
fix(e2e): keep chainsaw asserts polling through transient API errors
Chainsaw v0.2.15 ends an assert or error operation on the first API error other than NotFound. kube-apiserver answers a GET with 500 when etcd changes its leader, so a leader change that lands inside a poll fails the step long before its own timeout, on whichever suite happens to be polling. The sandbox image now builds v0.2.15 from source with the fix proposed upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw release carries it. The build runs the tests of the patched packages and a test that drives chainsaw built from the patched tree against a fake apiserver: a run of 500s must not fail an assert, and a persistent one must fail it at its timeout rather than at the first error. Closes: #4720 Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
da1f1cc387 |
ci(retention): delete with the workflow token
The app installation token lists package versions but its DELETE of a listed version is answered with 404, while the repository's GITHUB_TOKEN with packages: write deleted the same version without any change to package settings. Deletes therefore go through GITHUB_TOKEN; listing keeps the app token. That deletion was shown on a package linked to this repository. A package linked elsewhere is not attempted: it is skipped before its versions are read and named in a warning, so it cannot turn every apply run red. A failed read of the link still fails the run, and so does a sweep in which no resolved package is linked, since that sweep prunes nothing. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
274f3f7023 |
ci(retention): fail the run when a version cannot be deleted
A refused DELETE only logged a warning, so a token without delete rights left an apply run green with nothing removed. Move the delete into hack/retention-delete.sh, which reports each refusal as an error carrying the API status, still tries the remaining versions, and exits non-zero; the workflow turns that into a failed run after the sweep. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f1c0f6f3f6 |
ci(retention): sweep packages named by the published tree
The org container-package listing answers 400 to the app installation token, so the job has never reached its sweep. Take the names from the published nightly tree via hack/retention-packages.sh and keep calling only the per-package versions endpoints, which accept that token. An unresolved name list now fails the run: the nightly publishes into these packages daily, so an empty list cannot mean there is nothing to prune. It skips only the nightly sweep. The promotion sweep reads one fixed package and is kept safe by the protected set, which the name list does not feed, so it still runs. A versions read that fails, in either sweep, is reported with the API error and fails the run once the rest is swept, instead of aborting the step through set -e. A failed candidate read leaves no ids behind. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
3cf2583acf |
ci(e2e): never tear down the Proxmox tenant on the runner's default context
The teardown runs under always() and took its kubeconfig from GITHUB_ENV, which only Stage credentials writes. When a step before it fails, KUBECONFIG is unset and the three `kubectl delete` calls go to whatever cluster the self-hosted runner's own kubeconfig selects. Pass the staged file explicitly to every teardown call. Stage now fails unless the file's current context resolves to a server, instead of letting client-go fall back to its defaults, which include in-cluster credentials. The suite step gets the same path in its own env, so no step depends on GITHUB_ENV for the cluster. Tear down skips with a notice only when Stage did not succeed, read from the step's outcome: the suite cannot run without it, so the run created no tenant. When Stage succeeded but the file no longer resolves, the step fails rather than reporting a skip with the tenant and its VMs still live. Both guards print kubectl's own error, so a malformed secret or a missing kubectl does not read as a wrong context. The deletes also no longer stop at the first failure: under set -e a failed first delete skipped the tenant delete, leaving behind the VMs the step exists to remove. Each one runs, and the step fails at the end if any of them did. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
be2387323b |
feat(gateway)!: reserve a passthrough listener's name by the spec alone (#4638)
## What this PR does This keeps the TLS-passthrough services on port 443 of the tenant Gateway and makes the passthrough rules hold on Cilium 1.20.2. It is stacked on #3343 and has to land with it (stack #4717). On 1.20, upstream Cilium marks an HTTPS-terminate listener and a TLS-passthrough listener on one port `Conflicted` when their hostnames overlap, and drops both from Envoy. That took `*.<apex>` and the `api`, `vm-exportproxy` and `cdi-uploadproxy` passthrough listeners down together on every tenant Gateway in `dns01` and `existingSecret` mode (#3912). #3343 now ships a patched operator that conflicts the pair only when the hostnames are equal, so the layout stays as it is on main: one Gateway, one address, everything on 443. What remains is the equal-hostname case and the cleanup that 1.20 allows. Under `http01`, a name a passthrough listener answers is now reserved by the spec alone. Before, the controller dropped the terminate listener for such a name only once a TLSRoute was attached to the passthrough listener and its backend Service resolved, which copied how Cilium builds filter chains. Without a working TLSRoute, an HTTPRoute on `api.<apex>` kept its terminate listener next to `tls-api`, and on 1.20 that equal-hostname pair takes both listeners down. Now such an HTTPRoute gets no listener and no certificate, TLSRoute or not. If all its hostnames are reserved it reads `Accepted=False` with `NoMatchingListenerHostname`. If it also claims a free hostname it stays `Accepted` and is served there. The backend lookup, its Service and ReferenceGrant watches and their RBAC rules are gone. Other changes: - `tlsPassthroughListeners` is accepted under `dns01` and `existingSecret`. The old refusal relied on 1.19.5 serving all ports from one Envoy listener; 1.20 splits ports whose SNI overlaps. - A TenantGateway reads `Ready=False` with reason `GatewayClassUnsupported` when its GatewayClass lists features in `status.supportedFeatures` but not `TLSRoute`, while it renders passthrough listeners. A class that lists nothing is not judged. Cilium 1.20.2 lists `TLSRoute`. - Comments, the README and one error message no longer describe Cilium 1.19.5 as the shipped release. - The gateway chart writes `tlsPassthroughServices` on the publishing tenant only, behind the same `gateway.enabled` switch and `publishing.exposedServices` filter as the platform TLSRoutes. Before, every tenant Gateway got the three passthrough listeners with nothing routed to them, and the reservation above would have taken those names from a child tenant's own HTTPRoutes. The uniform `allowedRoutes.kinds` on port 443 from #3087 stays. The Cilium bug behind it is fixed since 1.19.6, but several tests pin it, so removing it is a separate cleanup. ### Upgrade notes Only the publishing tenant keeps the `api`, `vm-exportproxy` and `cdi-uploadproxy` passthrough listeners. That is `tenant-root` unless `publishing.ingressName` says otherwise, and only for names in `publishing.exposedServices` while `gateway.enabled` is on. Other tenants lose these listeners on upgrade. Nothing shipped was attached to them, so a child tenant's own HTTPRoute on `api.<child-apex>` keeps its listener and certificate. Two narrow cases do change. On the publishing tenant under `http01`, an HTTPRoute on an exposed passthrough name loses its listener and certificate when the platform TLSRoute for that name forwards nowhere (the owning package is not installed, or its backends do not resolve). A TLSRoute attached by hand to a child tenant's `tls-*` listener loses its parent. To keep that listener, merge-patch `tlsPassthroughServices` into the tenant's `TenantGateway` after the upgrade. From then on the chart does not render the key there, and later upgrades leave the patch alone. `dns01` and `existingSecret` do not change: the wildcard listener still covers such a name, and the passthrough listener answers it by its exact name. Tested locally: Go tests for the controller, `go vet`, helm-unittest for `extra/gateway` and `cozystack-controller`, `make generate` and `make manifests` with no diff. The gateway e2e scenario checks that a child tenant renders no passthrough listener. It then declares `tls-api` and a passthrough listener on a native port on that tenant, checks that the tenant stays Ready with no route behind either, and checks the reservation against `tls-api`. Unit tests catch a regression in either change. In e2e, dropping the reservation turns the gateway suite red, and dropping the chart scoping does not, because CI never sets `gateway.enabled` (#4846). The controller was not run on a live cluster; the operator patch it depends on was, see #3343. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/725 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(gateway)!: the tenant Gateway renders the api, vm-exportproxy and cdi-uploadproxy passthrough listeners only on the publishing tenant, for the names publishing.exposedServices lists while gateway.enabled is on. Under certMode http01 a name a TLS-passthrough listener answers no longer gets an HTTPS listener on 443, with or without a TLSRoute behind it. On the publishing tenant, an HTTPRoute on such a name whose TLSRoute forwards nowhere loses its listener and certificate. A TLSRoute attached by hand to a tls-* listener of another tenant loses its parent. An HTTPRoute with some reserved and some free hostnames used to be rejected whole; it is now Accepted and its free hostnames get a listener and a certificate. tlsPassthroughListeners is accepted under dns01 and existingSecret. A TenantGateway whose GatewayClass does not list TLSRoute in its supported features reads Ready=False while it renders passthrough listeners. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * TLS passthrough listeners are supported with HTTP-01, DNS-01, and existing-secret certificate modes. Edge-terminated Gateways remain unsupported. * Declaring a passthrough hostname reserves it from HTTPS termination and certificate creation, even when no route is attached. * Passthrough listeners are rendered only for exposed services on the enabled publishing Gateway. * **Bug Fixes** * Gateways that declare passthrough listeners but explicitly lack TLSRoute support are now reported as not ready. * Routes targeting a hostname served by a passthrough listener on a different Gateway are correctly withdrawn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
e816522366 |
chore(renovate): leave the cilium-operator image pins to make update
The operator rebuild compiles one Cilium release with the Go toolchain upstream's builder image pins at that tag, and make update re-pins the builder line from there, along with the operator-generic base, its digest and the source archive checksum. A Renovate bump of that line, by tag or by digest, would build the release with a toolchain upstream never shipped it with, and the next make update would revert it without notice. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c1fe1e9cb2 |
ci(e2e): say what the 215-minute e2e budget actually covers
The comment on every lane claimed the budget fits two tenant Kubernetes suites failing at their node-join deadline. A green full suite on the container lane already spends most of it, and a suite that runs into the deadline costs about 25 minutes more than one that passes, so the claim does not hold. State the real bound so a cancelled run is read as a red suite, not as a budget to raise. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
da28ea7ee8 |
ci(e2e): run nightly and release-tag e2e on the container lane
Nightly and the rc gate still booted the management nodes as QEMU guests, the substrate the merge-gating lanes left behind. On it the tenant Kubernetes suites have missed their node-join deadline on most nights since 2026-10-02, each failure spends the whole deadline, and the job reaches its timeout before the alphabetically later suites run. Over the same days those suites passed on every PR run that selected them, where the nodes are Talos containers and a tenant worker sits one virtualization level shallower. Both lanes now use the PR lanes' container substrate. Nightly's e2e no longer waits for the nocloud disk, which it does not boot; the disk is still built and published. A manual e2e-tag dispatch against a tag that predates the container lane fails early and names the ref to dispatch from instead. DRBD, the replicated StorageClass and the Cozystack Talos node image lose their last e2e coverage with this. The docs and comments that pointed at nightly and the rc gate as that coverage now say so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
6192d5814b |
feat(local-storage): add opt-in OpenEBS LocalPV hostpath package (#3266)
## What this PR does Adds `packages/system/local-storage`, an optional package that vendors OpenEBS Dynamic LocalPV (chart 4.5.1) and registers it as the `cozystack.local-storage` PackageSource. A cluster without the DRBD and ZFS kernel modules can't run LINSTOR, so it has no dynamic PV provisioning, hostpath volumes cover that. They also fit deliberately node-local workloads. - StorageClass is `cozystack-local-hostpath`, `WaitForFirstConsumer`, `Delete` reclaim. It is not the default and doesn't take any existing class name. - Opt-in: a system bundle can list the package in `bundles.enabledPackages`, no built-in variant enables it. Alone it installs with `cozypkg add cozystack.local-storage`. - Vendored RBAC is cut down by `patches/rbac.diff` (reapplied by `make update`) to what volume reconciliation calls. Helper Pods, leader election Leases and analytics state stay in the install namespace, helpers are still privileged because they work on host directories. - Images stay on docker.io, pinned by index digest with `make update-image-pins`, nothing is rebuilt or mirrored. - `local-storage` scope is mapped to `area/storage` in `pr-labeler.yaml`, without it the title lands in `area/uncategorized` on every run. Tested with helm-unittest (StorageClass, PackageSource registration) and `tests/rbac_test.py` (effective permissions across the bindings). Not tried on a live cluster yet. Nothing installs the package by default, the container cluster e2e lane that will use it is a follow-up ([#3238](https://github.com/cozystack/cozystack/issues/3238)). ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/726 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Website follow-up is an issue, the docs there describe a package that isn't merged yet. The diff adds a platform component and touches no platform values, ApplicationDefinition, tooling or `hack/` path, so nothing else in the trigger map matches. ### Release note ```release-note feat(local-storage): add opt-in cozystack.local-storage package that installs OpenEBS LocalPV hostpath and provides the cozystack-local-hostpath StorageClass, it is not enabled by any built-in variant and is not the default StorageClass. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional OpenEBS LocalPV Hostpath storage for provisioning persistent volumes on node-local storage. * Added a configurable `cozy-local-hostpath` StorageClass, disabled by default as the cluster’s default class. * Added controls for storage class creation, reclaim policy, hostpath quotas, and node deployment. * **Documentation** * Added guidance on installation, configuration, storage behavior, and uninstall considerations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
387760b94e |
test(tests): run unit suites with Bats and retain POSIX coverage (#3849)
## What this PR does This runs the unit suite with Bats and keeps a separate `dash` pass for tests that source production shell helpers. Unit files are discovered from the filesystem, and each loads the audited helper that restores `set -u` in test bodies without changing subprocess shell options. CI pins Bats 1.14.0, keeps JUnit reports, and runs Bats for docs-only changes too. Code changes also run Helm, Go, controller and POSIX compatibility tests. Per-file Make targets and the `--keep-going` sub-make are removed because one `bats -j` run goes through all files anyway, so a red file doesn't stop the others, and `hack/bats-unit-keep-going.bats` checks this with real bats. `-k` stays in the workflow because without it a red Bats file keeps `test-controllers` from starting. Local tests require bats-core 1.5 or newer. GNU parallel is optional, and the complete local hook runs with `pre-commit run bats-unit-tests --hook-stage manual --all-files`. Live-cluster suites keep their existing runner. ### Validation `make unit-tests test-controllers -j4 -k --output-sync=target BATS_JOBS=4 BATS_POSIX_SHELL=dash` passed, including every discovered Bats file and the POSIX pass. Mutation checks reject omitted files, disabled or exported nounset, missing CI steps and missing or excessive POSIX coverage. ### Downstream repositories The developer guide update is tracked separately in [cozystack/website](https://github.com/cozystack/website/issues/717). CCP has no caller of the changed targets. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - documentation follow-up: [issue](https://github.com/cozystack/website/issues/717) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Testing & Reliability** * Bats unit tests run as a dedicated suite, with optional parallel execution and JUnit reports. * Pull request checks run Bats tests for documentation-only changes, and unit and controller tests for code changes. * POSIX-shell compatibility tests are included in the standard unit-test run and continue after individual failures. * Test reports are uploaded even when checks fail. * Added a manual local test hook for running Bats and POSIX compatibility tests. * Shared strict setup is applied across Bats unit tests. * **Documentation** * Updated testing guidance with current commands, runner compatibility, and CI coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
34b1533a1a |
ci(labeler): map the local-storage scope to area/storage
The labeler falls back to area/uncategorized for a scope it does not know, and re-applies the fallback on every run, so a manually fixed label does not stay. The local-storage package is a storage component, and its PR title scope should land on area/storage. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
3e078b79a9 |
ci(e2e): refuse a new fork image before the build starts (#4767)
## What this PR does A fork PR that adds a first-party image cannot pass the fork e2e lane, and today it finds that out only after the whole build. `e2e-fork.yaml` pushes the fork's image archives only under names the base tree already exports, so a new image is refused at the push with `image '<name>' is not in the base-tree allowlist`. Nothing the contributor does on the PR changes that. This PR keeps the push allowlist as strict as it is and moves the failure to the start of the run. On fork PRs, `plan` in `pull-requests.yaml` now runs `hack/fork-new-images.sh`. The script derives the image names of the PR tree and of its base parent with the same `make -n` derivation the push step uses. If the PR tree exports a name the base does not, it fails before any build, names the images and says what has to happen: a maintainer lands the image on the base branch first, or the change runs from a branch in this repository. The check runs from the fork's own tree, so a fork can remove it. That is fine, because it only saves an honest contributor a wasted build. The push allowlist in `e2e-fork.yaml` is still the boundary. That workflow cannot call the script, since it runs from the default branch over an untrusted checkout. So `hack/fork-new-images.bats` pins the two derivations to each other, and also covers the script's behaviour and the fork-only gating of the step. I replayed the script against the merge commit of #4291 and its base parent. It fails with `cluster-api-provider-kubevirt`, the same name the push step refused there. The workflow_run lane itself cannot be exercised locally. This does not lift the limitation itself: a fork still cannot get a green run for a new image. Part of #4329. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): a fork PR that adds a new first-party image now fails in the plan job, naming the image, instead of after the whole build at the image push. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Fork pull requests that add image names not present in the base branch are now rejected during planning, with the new names reported before builds begin. * **Documentation** * Updated guidance describes the early check and clarifies how it relates to the image allowlist used during publishing. * **Tests** * Added coverage for new, unchanged, reduced, and empty image sets, along with checks that the pull request workflow applies the check only to fork submissions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
cc942fed68 |
test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting. The unit lane needs complete TAP results and JUnit reports without changing how live-cluster suites run. Sourced production helpers still need a dash pass, and unit test bodies need nounset without exporting shell options into subprocesses. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
bcfa50d7e9 |
test(e2e): install only the packages the selected suites need
The E2E job installs the whole platform on a fresh cluster before it knows which suites will run, so every PR pays install time and flake surface for packages its suites never use. Select the suites first and pass the complement computed by select-install.sh to the platform Package as disabledPackages. The full-e2e label and an empty selection keep the full install, and the nightly and tag lanes pass no list, so they are unchanged. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
80eb2e5037 |
ci(e2e): refuse a new fork image before the build starts
The fork e2e lane pushes a fork's image archives only under names the base tree already exports, which keeps a fork from minting repositories under the CI registry. An image the PR itself adds is therefore refused, but only at the push, after the whole build has run, and nothing the contributor does on the PR can clear it. Compare the image names of the PR tree and its base in the plan job of fork runs, with the same make -n derivation the push allowlist uses, and fail there naming the new images and what has to happen first. The push allowlist is unchanged and remains the boundary: this check runs from the fork's own tree, so it only saves an honest contributor the wasted build. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c854e054b9 |
ci(workflows): install flux in the isp-hosted lane through the pinned action
The isp-hosted lane fetched fluxcd.io/install.sh and ran it as root, which the CI tool-install contract forbids for every workflow: it runs whatever the site serves at that moment. The lane landed next to the contract and kept the old step, leaving the unit tests red on main. It now uses the same SHA-pinned fluxcd/flux2 action and version as the other workflows. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2af8b0c229 |
ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does One transient error from proxy.golang.org is enough to fail an image build or the `Unit & controller tests` job today. The error is an HTTP/2 stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or falls back after it. This PR adds three things: a direct fallback, a bounded retry and a module cache. Image builds now fall back to fetching modules from their origin when the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a build arg from `hack/common-envs.mk`, and every Go builder stage declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error. With Go's default comma it does that only on a 404 or 410. The value comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because buildkit can't reach a mirror on localhost or a `file://` proxy. Every workflow job that runs Go on the runner sets the same value as env. Explicit module downloads in Dockerfiles are retried three times with a growing pause. The loop is inline, in one form that the contract test pins. I tried a shared script from a named build context first, but buildx refuses any `--build-context` for the four Dockerfiles still pinned to the `dockerfile:1.2` frontend. The images also build from different contexts, so no single file can be copied in. Raising those pins changes how those images build and needs a real build to check, so I left it for a follow-up. Two alpine stages, token-proxy and shadowbox's `ss-server`, left the download to `go build`. The alpine golang images have no git, so the direct fallback can't fetch for them, and they now get the same download step. The runner jobs use `hack/go-mod-download.sh` with the same bound. The unit test job now starts from a module cache instead of downloading everything. It restores `GOMODCACHE` with `actions/cache`, keyed on every `go.sum` in the tree, and then prefetches all tracked modules through the retry, so a cache miss still works. A cache saved by a PR run is visible only to that PR, but one saved on the default branch can be restored by any PR ([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)). That is why the build cache warmer on main writes the same key before its image build. This way the first run of a PR, which is the one that flakes, starts warm. The warmer restores only an exact key, so old module versions don't pile up, and its Go steps can't fail the image build. Both runner labels are Linux, so the key matches. Both also check out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the same cache path, but I have not seen it on the warmer's runner yet. The release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env only, no cache. `hack/go-module-fetch-contract.bats` covers all three. Each test is red on main, and I checked each guard with a mutation. Closes #4591 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Reliability** - Go module downloads now fall back to direct access when the public proxy encounters errors. - Builds and automated checks retry failed module downloads, helping them recover from temporary network issues. - **Performance** - CI caches Go modules to reduce repeated downloads and speed up builds and tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7ea71042b7 |
ci(e2e): add a daily isp-hosted install lane on kind (#4712)
## What this PR does
This adds a daily CI lane that installs the `isp-hosted` platform
variant through the `hosted` installer variant. No lane ran either of
them before. The PR, nightly and tag lanes all install `isp-full` on
Talos, where Cozystack brings its own CNI and storage, and that is how
#3376 shipped unnoticed.
What `isp-hosted` changes is what it leaves to the host. The system
bundle swaps kube-ovn and Cilium for a `noop` networking Package and
drops multus, LINSTOR, MetalLB, cozy-proxy, snapshot-controller and the
rest of the shared data-plane set. The iaas bundle refuses to render.
Migrations are not a difference, `isp-full` enables them too.
So the lane runs on kind. kindnet, kube-proxy and local-path come from
the cluster, and there is no Cilium, no LINSTOR and no `replicated` or
`local` StorageClass. It installs the latest GHCR nightly instead of
building anything, at 05:00 UTC after the nightly mirror publishes, and
on `workflow_dispatch`. There is no pull request trigger, because a run
would install the nightly images and not the PR's.
A run passes when every HelmRelease is Ready, `cozystack-platform` and
`tenant-root` included. It also checks that the platform emitted the
`noop` networking Package and none of LINSTOR, MetalLB, multus or
KubeVirt, and that the cluster has no Cilium CRD and no `replicated` or
`local` class.
The lane reuses the e2e sandbox container for tooling and
`collect-report`, attached to the kind network.
`hack/isp-hosted-lane_test.bats` pins the wiring: the Package selects
`isp-hosted`, the install uses `variant=hosted`, the kind DNS domain
matches `networking.clusterDomain`, and kind is pinned by checksum and
digest. Six of its seven tests were red before the implementation
existed.
`hack/sandbox-runner-headroom.bats` gets a named exemption. That check
treats any workflow touching `packages/core/testing` as a lane that
boots the QEMU guests, and this one boots none. The exemption is checked
after the job match, and removing it turns the test red again.
The pins come from the kind v0.32.0 release: the binary sha256
`50030de2…` from its `kind-linux-amd64.sha256sum`, and
`kindest/node:v1.33.12@sha256:3f5c8443…` from its release notes.
v1.33.12 is the Kubernetes version the Talos lanes use, so the variant
is the only thing that differs. All three are plain `env` values in the
workflow, where a Renovate custom manager can match them.
### Real runs
The first real run on this branch,
[36961209755](https://github.com/cozystack/cozystack/actions/runs/36961209755),
went red on `tenant-root`:
```text
tenant-root False Helm install failed for release tenant-root/tenant-root with chart tenant@0.0.0+629b61300902: unable to build kubernetes objects from release manifest: resource mapping not found for name: "allow-external-communication" namespace: "tenant-root" from "": no matches for kind "CiliumNetworkPolicy" in version "cilium.io/v2"
```
This is #3387 reproduced live: `packages/apps/tenant` renders Cilium
policies on a cluster without Cilium. The other releases went Ready,
except `cozystack-basics`, which was still in an upgrade at the last
listing. The fix landed in #4710.
The run after it,
[37089699259](https://github.com/cozystack/cozystack/actions/runs/37089699259),
is green on the first attempt. It installed nightly
`0.0.0-nightly.20261003.37089458387.1`, built from
|
||
|
IvanHunters
|
1621c471f2 |
ci(dashboard): run eslint on the console and clear its errors (#4692)
## What this PR does
The console now passes its own eslint config, and CI runs it. On main
`pnpm lint` exits 1 with 54 problems, 51 of them errors, and
`ui-test.yaml` never ran it. On this branch it exits 0 with no errors,
and a Lint step sits between Typecheck and Test.
How each rule was handled:
- `no-explicit-any` (38 errors): replaced with real types. These commits
build a vite bundle byte-identical to main's.
- `react-refresh/only-export-components` (6 errors): the hooks, context
objects, `resolveSource` and the RJSF registry moved out of component
files into their own modules. `useTenantContext` stays in
`tenant-context.tsx`, so its importers and the tests that mock that
module are untouched.
- `no-unused-vars` (1 error): dropped the unused `ad` binding in
`OverviewTab`.
- `react-hooks/set-state-in-effect` (6 errors): relaxed per line with a
reason, not in the config.
The last rule is the only relaxation. Each of these effects re-syncs
local state to something that changes outside the component. The rule
wants the state derived during render, which changes when the value is
committed or what the first render shows, so it would be a behaviour
change. Per file:
- `components/ResourceQuotasField.tsx` re-syncs the inputs after the
form value is reset from outside.
- `components/SourceWidget.tsx` re-syncs the selected option when the
value loads asynchronously in edit forms.
- `components/command-palette/command-palette.tsx` resets query and
level whenever the palette closes.
- `components/command-palette/use-keyboard-nav.ts` moves the highlight
back to the top when the item list changes.
- `hooks/use-is-mac.ts` would return `true` on the first render on macOS
with a lazy initial state. Today it returns `false`.
- `lib/tenant-provider.tsx` falls back to an existing tenant once the
tenant list has loaded.
The eslint config itself is unchanged, and `no-explicit-any` stays on
everywhere.
`pnpm typecheck` and `pnpm test` (442 tests) pass at every commit. After
the file moves the bundle has the same chunk set, and only the entry
chunk differs, by reordered and renamed minified functions.
Three `react-hooks/exhaustive-deps` warnings from main are left as they
are, because fixing them changes when the effects run. One is in
`SourceWidget.tsx`, where the effect reads `options` but depends only on
`value`. The other two come from `ads = adList?.items || []` in
`use-command-items.tsx`, which gives two `useMemo` hooks a new
dependency on every render.
Things I did not do here:
- The step has no `--max-warnings`, so a new warning will not fail CI.
Once the three warnings are fixed, `--max-warnings 0` can go in.
- `SourceField.tsx` and `SourceWidget.tsx` each declare the same small
`SourceOptionSchema` interface. I kept them local since the two widgets
share nothing else.
- The job name stays `Typecheck and test`, so a required check with that
name keeps matching.
- `pnpm -r lint` only reaches `apps/console`.
`packages/{k8s-client,ui,types}` have no `lint` script, same as on main.
Closes #3911
Closes #4566
### Screenshots
No visual change: types, module layout and CI only.
### Downstream repositories
The diff touches only the console workspace and `ui-test.yaml`. The
console has no downstream copy since `cozystack-ui` was archived, and no
trigger map entry matches.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
ci(dashboard): the console's eslint now runs in the UI test job, and the console passes it with no errors.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Improvements**
* Tenant selection now uses available tenant namespaces, restores a
saved selection when possible, and falls back to the root tenant or the
first available tenant.
* Tenant options are sorted, and the selected tenant is kept in sync
with the available choices.
* **Chores**
* Automated checks now include linting alongside type checks and unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
IvanHunters
|
9239555e6a |
ci(workflows): pin flux, yq and helm-unittest installs (#4645)
## What this PR does Workflows no longer run the flux install script as root. Every step that needed the flux CLI downloaded `https://fluxcd.io/install.sh` and ran it with `sudo`, so whatever that host served at the moment ran on runners that later get registry credentials. PR #4442 pinned the version, but the script was still fetched and executed. All 12 of those steps now use `fluxcd/flux2/action`, pinned to the v2.8.6 commit with `version: "2.8.6"`. The action downloads the release tarball, checks it against that release's checksums file, and adds the binary to `PATH`. Two workflows still took yq from the floating latest release, nightly and the fork e2e lane. They now use the same install the promotion steps already have: yq v4.53.3 checked against a committed SHA-256. I verified that hash against the release's `checksums` file and against the binary itself. The helm-unittest plugin was installed at whatever version was newest. The plugin embeds its own Helm, so a plugin release could change chart test results on a PR that touched no chart. It's pinned to v1.1.2 now, the version CI already runs. A runner that carries a different version gets it replaced. A new contract test scans every workflow for these three tools, because an unpinned install usually comes back by copying an older step. On main it fails on the install script, the missing setup action, the latest-release yq and the unpinned plugin. I ran one mutation per guard, and each one failed only the test meant to catch it. The existing contract tests that checked `FLUX_VERSION` inside the old steps now check for the action step. The e2e sandbox image still runs `install.sh` in its Dockerfile. That's an image build, not a workflow step, so I left it for a separate change. Closes #3619 Closes #3640 Closes #4510 ### Screenshots Not applicable, no UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only workflow tool-install steps and `hack/*.bats` contract tests. Nothing in the trigger map reaches them. ### Release note ```release-note ci(workflows): install flux through the commit-pinned upstream setup action instead of running fluxcd.io/install.sh as root, pin yq with checksum verification in the nightly and fork e2e workflows, and pin the helm-unittest plugin version. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Reliability** * CI workflows now use a consistent, pinned Flux version. The yq and Helm plugin setup also verifies or enforces specific versions. * **Tests** * Added checks to ensure workflow tool versions are pinned and consistent, and that downloaded yq releases are checksum-verified. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
6e4c9c4b6a |
fix(ci): harden the PR labeler's title parsing and map recurring scopes (#4666)
## What this PR does Fixes three problems in `.github/workflows/pr-labeler.yaml`. Each one either killed the job or put a PR in the wrong area, and none of them showed up anywhere. - Both maps are now looked up with `Object.hasOwn`. A title typed or scoped `constructor`, `toString`, `hasOwnProperty` or `__proto__` used to resolve to an `Object.prototype` member and crash the step on `l.startsWith`, which left the PR with no labels at all. - Every leading `[Backport ...]` prefix is now stripped, not only the first one. With two prefixes, the second was read as a bracket scope, so the real scope and the `kind/*` label were both lost. The `area/release` label hid the problem by keeping `area/uncategorized` from firing. - Scopes that recur three or more times over the full PR history now map to an area. So does each area's own name (`networking`, `storage`, `database`, `virtualization`, `testing`, `dependencies`). Scope-to-area decisions: - `area/database`: `database`, `etcd-operator`, `opensearch`, `opensearch-operator` - `area/networking`: `networking`, `kubeovn`, `multus`, `kilo`, `hetzner-robotlb`, `ouroboros` (it rewrites the CoreDNS Corefile to fix hairpin NAT, so it is networking, not platform) - `area/virtualization`: `virtualization`, `vm`, `kubevirt-operator`, `kubevirt-cdi`, `kubevirt-instancetypes` - `area/platform`: `operator`, `cozystack-operator`, `controller`, `cozystack-controller`, `cozystack-basics`, `flux-shard-operator`, `migrations`, `scheduler`, `cozystack-scheduler`, `reloader`, `bootbox`, and `lineage`. I put `lineage` here on purpose: no existing label covers it more precisely, and it is too narrow for an area of its own. - `area/storage`: `storage`, `piraeus-operator`, `backupstrategy-controller` - `area/release`: `changelog`, `changelogs` - `area/ci`: `workflows`, `labeler` - `area/dependencies`: `dependencies`, `renovate` - `area/api`: `registry` (the cozystack-api storage layer) - `area/dashboard`: `dashboard-controller` - `area/monitoring`: `monitoring-agents` - `area/keycloak`: `keycloak-configure` - `area/testing`: `testing` - New `area/dx`, added to `.github/labels.yml`: `dx`, `hack`, `cozypkg`. These are developer tooling (hack/ scripts, package Makefiles, cozypkg). `area/build` covers image builds and `area/testing` covers tests, and neither describes this work. Left unmapped on purpose: - `http-cache`: the application was removed from the tree. - `openbao`: already mapped on main. - `apps`, `system`, `docs`: catch-all scopes that don't point to one area. - `fix`, `bugfix`: malformed titles. - `security`, `rbac`, `governance`, `codeowners`, `maintenance`, `images`, `readme`, `telemetry`, `cluster-autoscaler`: no single area fits any of them. I did not add spelling normalisation. `kubeovn` got its own row, since it is the only alternate spelling that recurs. A new contract test, `hack/pr-labeler-contract.bats`, pulls the workflow's own `script:` block out of the file and runs it under node with stubbed `context`, `core` and `github`, so it tests the shipped code, not a copy. On main it is red on the prototype titles (`THROW: l.startsWith is not a function`), on the double backport (only `area/release`), on the newly mapped scopes and on the area-name scopes. It also checks that every map target exists in `labels.yml` and that no key is written twice. The suite fails rather than skips when node is missing. The older node-based suites skip, so this is the first one that needs node on the unit-test runner. Some things stay out of this PR. The scope table in `docs/agents/contributing.md` is a summary. It has no row for `area/dx` and does not mention that an area's own name works as a scope. Separately, the singular `migration` still maps to `area/virtualization` as it did before, while the new `migrations` maps to `area/platform`. A backport PR whose scope is unmapped still never gets `area/uncategorized`, even with a single prefix. The `area/release` added by the strip satisfies the fallback check. This behaviour is the same on main and is not changed here. The test that checks map targets against `labels.yml` passes by itself if the map literal cannot be extracted. The duplicate-key and area-name tests go red in that case, so the suite as a whole still fails. Closes #4125, closes #4124, closes #4122 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): the PR labeler no longer crashes on titles scoped like Object.prototype members, handles nested backport prefixes, and maps more recurring scopes, including a new area/dx. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added the `area/dx` label for issues and pull requests related to developer tooling. * **Improvements** * Pull requests can now be categorized using a broader set of scope aliases. * Repeated leading backport prefixes are recognized and route the pull request to the release area. * Label mapping now avoids treating inherited property names as valid mappings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
f32e0693c1 |
ci(dashboard): run the console's eslint in the UI test job
The console defines a lint script next to typecheck and test, but no workflow ran it, so new findings landed unseen next to the old ones. The backlog is cleared down to three react-hooks/exhaustive-deps warnings. eslint exits zero on warnings alone, so the lint script caps them at that count: the step gates from its first run, and a new warning fails it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bd504e0b4e |
ci(e2e): run the isp-hosted install daily against the nightly
The lane installs the latest GHCR nightly on a kind cluster once a day, after the nightly has published. It takes no pull request trigger: a run would install the nightly's images, not the pull request's, and report on code the pull request did not change. The unit test holds the lane's wiring in place, since a lane that quietly installed another variant would test something other than its name says. The sandbox headroom check reads any workflow touching packages/core/testing as one that boots the QEMU guests; this lane starts only the sandbox container, so it is exempted by name, after the job match, and is sized like the others if it ever starts calling the guest targets. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bf03c84711 |
chore(deps): move to the Kubernetes 1.37 libraries without the apimachinery fork (#4473)
## What this PR does This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and controller-runtime from v0.23 to v0.25.1, and drops the `k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and #4472; those were a stack of the same commits. The four commits are meant to be read one by one, and each builds and passes the tests on its own: 1. **Plain watches stop getting the initial-events-end bookmark.** The apiserver defaults `sendInitialEvents=true` onto any watch with an empty or `0` resourceVersion and leaves bookmarks off, and our aggregated storages took that flag alone as a WatchList request. From `k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api panics on that bookmark, so `kubectl get -w` on any cozystack-api resource would die after the bump. The marker now needs both flags, the same condition the watch cache uses. There are tests per storage and one through the real handler chain. 2. **The apimachinery fork goes.** Every apps kind shares the `Application` Go type, and stock `Scheme.ConvertToVersion` picks the first kind registered for it. The fork patched that; the apps group now encodes through a serializer that keeps the kind the object carries. A handler-chain test covers create, get, list, update, watch, apply and table for two kinds and fails without it. 3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the plugin is on by default and refuses every write until it has synced `mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and which clusters before 1.36 do not serve. Without this, every create, update and delete would fail after the bump. It is also re-applied in `Complete`, since `--disable-admission-plugins` replaces the list. 4. **The bump itself**, with the regenerated clients, openapi and strategy CRDs. Commits 2 and 4 are where the review effort goes; 1 and 3 are small. Known changes in behaviour: - Without the fork, the row objects of a Table requested with `includeObject=Object` carry the first registered app kind. kubectl asks for that only with `--sort-by`, and it sorts on fields, not on the kind. Nothing at this layer can fix it, because the handler converts through `APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks and ValidatingAdmissionPolicy still see the right kind. - The alpha PodSpec field `workloadRef` became `schedulingGroup` upstream. Once the new strategy CRDs are applied, the apiserver prunes `workloadRef` from stored strategies it serves, and their next write erases it. It only ever worked behind an alpha feature gate. The branch is rebased onto the Renovate rollup (#4447), with `go mod tidy` and codegen rerun on top of it. ### Screenshots No UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Application API responses now preserve the requested resource kind across operations, including lists and watches. * Backup strategy resources support additional Kubernetes pod settings, including probe protocols, volume ownership options, and scheduling groups. * Backup strategy schemas include updated Kubernetes pod fields and descriptions. * **Bug Fixes** * Initial-event watch bookmarks are sent only when requested; plain watches continue without an ending bookmark. * Mutating Admission Policy is disabled by default, including when admission plugin options are customized. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6df5d076b1 |
ci(workflows): pin flux, yq and helm-unittest installs
Every flux install fetched fluxcd.io/install.sh and ran it as root, so whatever that host served at the moment executed on runners that later hold registry credentials. Use the upstream setup action instead, pinned by commit and given an explicit version: it checks the release tarball against that release's checksums file and runs no script fetched at run time. nightly and the fork e2e lane still took yq from the floating latest release. They now use the pinned, checksum-verified install the promotion steps already have. The helm-unittest plugin embeds its own Helm and was installed at whatever version was newest, so a plugin release could change chart test results on a PR that touched no chart. Pin it, and replace a different version a persistent runner may carry. A contract test scans every workflow for these three tools, so an unpinned flux, yq or helm-unittest install copied from an older step fails the unit lane. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2d0c359f4c |
test(e2e): stop bringing up the monitoring instance nothing looks at (#4023)
## What this PR does This PR turns off monitoring instance of root tenant in e2e install and removes waits on it. Install patched root tenant with `monitoring: true` and then waited up to 15m each on vmalert and vmalertmanager, `vlclusters/generic`, shortterm and longterm vmclusters, `grafana-db` CNPG cluster and grafana deployment, but nothing in the run looks at any of it. No chainsaw suite touches tenant-root monitoring objects. Only one that mentions monitoring is `foundationdb`, it asserts `WorkloadMonitor` whose CRD comes from `cozystack-controller` and exists without monitoring stack. Monitoring OIDC wiring is covered by `packages/system/monitoring/tests/oidc_test.yaml` since #4019 removed orphaned BATS suites. Kubernetes suites run with `monitoringAgents.enabled: false` and tracing is off by default (`tracingStorages` is empty), so they don't depend on this instance either. Those waits were not free. Monitoring stack competes with rest of install for node capacity, and removed block own comment said slow victorialogs start can fail install on PR that never touched monitoring - that's why it was moved to 15m budget. ### What stays Monitoring package stays installed, because `tenant-application` has `dependsOn cozystack.monitoring-application` and without the package platform release is never resolved. Only tenant-root instance is off. Container lane `logsStorages` override for victorialogs also stays. It does nothing while monitoring is off, but keeps a run that turns monitoring back on from creating immutable replicated PVC on lane that has only `local`. Its comment, paragraph in `docs/agents/e2e-testing.md` and comment on Install step env in `pull-requests.yaml` now say that instead of calling root monitoring install-time workload. ### What is lost `curl` through `root-ingress-controller` with `Host: grafana.example.org`, it was the only check that root ingress actually routes to backend. Grafana was the target just because it was installed. Moving it to seaweedfs host needs expected response checked on live stand first, so it's tracked in #4615. ### Savings Not measured. The happy path never spent the 15m budgets, so what this returns is real bring-up time of the stack plus tail risk, and only CI runs can show how much it is. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` |
||
|
Aleksei Sviridkin
|
30689f8109 |
fix(ci): map the recurring unmapped scopes and add area/dx
Over the full PR history a set of scopes recurring three or more times still fell through to area/uncategorized. Most of them have an obvious home and were only missing a row. Subsystem names such as `networking` or `storage` are now accepted as the scope of their own area, since contributors reach for those as often as for a package name. `dx`, `hack` and `cozypkg` describe developer tooling, which no existing area covers: area/build is image builds, area/testing is tests. They get a new area/dx rather than being filed under either. The contract test also pins that every map target exists in labels.yml, since the label sync workflow creates only what that file defines, and that no key is written twice, since a repeated key silently overrides the earlier one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a8c3eb5260 |
fix(ci): strip every leading backport prefix from a PR title
A backport of a backport is titled with two `[Backport ...]` prefixes. Only the first was stripped, so the second was parsed as a bracket scope: the real scope and the kind were lost, and the area/release added by the strip suppressed the area/uncategorized fallback, so the PR looked labelled correctly. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
cd6476ccb7 |
fix(ci): look up the labeler's type and scope maps by own key only
Both maps are plain object literals, so a title typed or scoped `constructor`, `toString` or `__proto__` resolved to an Object.prototype member. That value is truthy, landed in the label set and killed the step on `l.startsWith`, leaving the PR with no labels at all. The new contract test runs the workflow's own script under node with stubbed context, so it exercises the code as shipped rather than a copy of it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5eee22fe28 |
ci(workflows): cache Go modules and fall back to direct in Go jobs
The unit and controller test job has no module cache and runs on an ephemeral runner, so every run fetches every module from proxy.golang.org, and one transient stream reset fails the job. It now restores the module cache keyed on every go.sum in the tree and prefetches the modules of every tracked go.mod through a bounded retry, hack/go-mod-download.sh. The cache is written from main by the build cache warmer under the same key, because a cache saved by a PR run is visible to that PR only, while one saved on the default branch is readable from every PR. Every job that runs Go on the runner sets GOPROXY=https://proxy.golang.org|direct so any proxy error falls back to the origin. The release jobs get only that, not a cache. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
312409ecd2 |
test(e2e): fail the suite whose pods outlive its cleanup (#4492)
## What this PR does A Chainsaw suite could pass while the workloads it created kept running after its own cleanup, and nothing noticed until a later suite failed. On one E2E run the brokers of four Kafka releases from suites that had passed and deleted their applications (#4488), plus a TenantControlPlane sitting on its finalizer with two apiserver pods (#4489), pushed pod memory requests to the node limit. Harbor then failed on `Insufficient memory` two suites later, under the wrong name. This makes the suite that leaked fail, and name what it left behind. - Chainsaw v0.2.15 has no hook that runs after a test's cleanup: its `Configuration` carries `error.catch`, which runs only on failure. So `make test-chainsaw` now runs `hack/e2e-chainsaw-run.sh`, which invokes `chainsaw test` once per suite directory and checks the cluster between invocations with `hack/e2e-chainsaw/_lib/leak-guard.sh`. - Attribution: a pod is charged to a suite when all three hold. It is not terminal. Its UID was not in the pod list taken before the suite. The root of its ownerReferences chain was created at or after the suite started. A platform pod replaced mid-suite by a rollout, an eviction or a crash traces back to an object the install created, so it is not charged. A pre-existing pod adopted by a controller recreated mid-suite is excluded by the UID baseline. A leak is charged once, to its own suite. - The suite start is read off an apiserver, from the `creationTimestamp` of a server-side dry-run create, rather than off the sandbox. With several control-plane nodes the reading and a root's timestamp can come from different apiservers, so the rule assumes their clocks agree more closely than the gap between the reading and the suite's first object; no skew margin is added. Source for that behaviour: the generic registry `Store.create` stamps it through `rest.FillObjectMetaSystemFields`, then `DryRunnableStorage.Create` copies it back without persisting (`staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go` and `dryrun.go` at Kubernetes v1.33.0). - Budget: charged pods get 300s to disappear, the default `timeouts.cleanup` in `.chainsaw.yaml`; a unit test keeps the two equal. A suite that overrides its own cleanup timeout (kafka sets 8m) is covered the same way, because the check runs only after `chainsaw test` has returned, which is after every cleanup in the suite has finished under its own timeout; the 300s start from there. The leaks this targets sat about 110 minutes (Kafka brokers) and 30+ minutes (the TenantControlPlane) past their suites. Every green suite logs `leak-guard: suite=<dir> clear after Ns`, so the number can be checked against later runs. - Past the budget the run prints `LEAKED-PODS suite=<dir> budget=300s count=N`, then one line per pod with its phase, deletion state, root and `app.kubernetes.io/instance`. It raises a `leaked-pods` error annotation and adds a failed `leak-guard` test case to `chainsaw-report.xml`. The run continues through the remaining suites and exits non-zero at the end, as the single invocation did on a red test. - A suite that already failed on its own is checked once without the wait. Its leftovers are reported as a `::warning` with `LEAKED-PODS suite=<dir> checked-once=chainsaw-failed`, since some may still be tearing down. - Each pod gets one `kubectl wait` inside the remaining budget, and nothing is retried. A timeout or a spent budget gives the verdict from the last good pod list, also when the re-list after it fails, so a leak fails the suite. A wait that fails only after running for its whole `--timeout` counts as a timeout whatever it printed: kubectl maps an expired context to "timed out waiting for the condition" only inside the watch, so a deadline that passes during its initial lookup surfaces as "context deadline exceeded" (`wait.go` and `delete.go` in kubectl v1.33.0). A spent budget is caught before calling kubectl and never passed as `--timeout=0`, which means "check once". - Any other error on a wait or a re-list, before the budget is spent and before any timeout, stops the waiting and is reported as `LEAKED-PODS suite=<dir> count=N inconclusive="<error>"`. That is a warning, not a leak failure, so an apiserver blip cannot become a leak red. - A failed first reading after a suite, including a pod list that cannot be parsed, leaves the suite unchecked. It fails with a `leak-guard` annotation and a report case saying the cluster could not be read, not that it leaked. Errors that are not leaks carry the annotation title `leak-guard`, not `leaked-pods`. - Known leaks are listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt` as `<suite-dir> <issue-ref>`: `kubernetes-latest` and `kubernetes-previous` against #4489 (both suites create a TenantControlPlane through `_lib/run-kubernetes.sh`). `kafka` and `kafka-metadata` were listed against #4488 until #4490 fixed it; those entries are gone. An allowlisted suite prints its leak as a `::warning` and does not fail. An entry that names a directory that is not a suite stops the run before the first suite, so a rename cannot keep an exemption alive. The PR fixing each issue deletes its lines. Suite selection is unchanged. The runner runs the named directories when there are any, and otherwise every directory directly under `hack/e2e-chainsaw/`, each handed to Chainsaw's own discovery; it refuses a tree with a test at its top level, where that split would stop being exact. I verified this with Chainsaw v0.2.15's own `discovery.DiscoverTests`, built from source at that tag. For the full tree, for `kafka redis harbor` and for `kubernetes-latest`, the union of the per-suite discoveries equals what a single `chainsaw test [suites]` discovers: 57, 7 and 2 tests, identical lists. 57 matches the test count in an earlier full run's report. A unit test pins that the runner passes chainsaw the suites and flags it was given, in order. The order changes too: suites used to run in Chainsaw's own order and now run one directory at a time in alphabetical order, which moves the kubernetes suites after the kafka ones. Before #4490 fixed #4488 this starved them (see the E2E criterion below). With the fix merged the order has no effect, and a future leak that lands before a memory-heavy suite is named at the suite that leaked. One behaviour does change: a selected name that is not a directory used to fail the single invocation before any test ran; now only that suite fails and the others run. `chainsaw-report.xml` keeps its path and shape: one `<testsuites>` root with the summed counts, one `<testsuite>` per suite directory. Its consumers are the `Collect chainsaw report` / `Upload chainsaw report` steps in `pull-requests.yaml`, `e2e-fork.yaml`, `nightly.yaml` and `e2e-tag.yaml`, which `docker cp` the file and upload it as an artifact. Nothing in the repository parses it. The merge is unit-tested on two per-suite reports in Chainsaw's real shape, one of them failed, and the result is checked to parse as XML. Tests: `hack/chainsaw-leak-guard_test.bats` (31 tests, in `make unit-tests`; named without the `e2e-` prefix, which the unit lane excludes) covers the attribution rule, the budget, the allowlist, the merge and the runner, all against stub `kubectl` and `chainsaw`. Every rule was mutation-checked: removing the guard call, having the Makefile bypass the runner, inverting the allowlist match, flipping or tightening the root-timestamp comparison, removing the UID baseline, removing the terminal-phase filter, skipping the budget wait, changing the budget, lowering or raising the depth cap, accepting a stale allowlist entry, reading an unreadable pod list as empty, dropping the leak case from the report, skipping directories in the selection, treating any failed owner read as a missing owner, dropping the separate report for a suite that already failed, dropping the not-run reason from the report, reading any wait error as a timeout, removing the inconclusive branch, letting a failed re-list after a timeout turn a leak inconclusive, ignoring a failed re-list, reporting an unreadable cluster as a leak, reading an unparsable pod list as clean, reporting a kubectl warning instead of the error, passing the report name without its extension, wording a failed re-list as a failed wait, dropping an allowlist's last line when it has no newline, dropping the check that `--set-string` carries a value, and reading a wait that ran its whole timeout but printed a context error as inconclusive. Each one turns its named test red, and restoring turns it green. E2E criterion, registered before the run: in the `Run E2E tests` log of this PR, every suite that ran prints `leak-guard: suite=<dir> clear after`, a `LEAKED-PODS suite=<dir>` line carrying `allowlisted=`, or one carrying `checked-once=` (a suite that failed on its own) or `inconclusive=` (waiting stopped on a failed read), and no line matches `LEAKED-PODS suite=.* budget=`. With #4490 merged, `kafka` and `kafka-metadata` are no longer allowlisted and must print `leak-guard: suite=kafka clear after` and `leak-guard: suite=kafka-metadata clear after`, which is the live check that #4490 fixed #4488. The first run of this PR (before #4490, run 36223896793) is the positive control: it printed `LEAKED-PODS suite=kafka count=15` and `suite=kafka-metadata count=8`, allowlisted, and every other suite cleared in 0-37s. That run also went red on both kubernetes suites, because those 23 leaked brokers left no memory for the second tenant worker VM (`0/3 nodes are available: 3 Insufficient memory`), which the order change described above made possible. The guard has not yet run against a real cluster, so the first run of this PR should carry the `full-e2e` label; its `clear after` lines are the first measured teardown times, and any unexpected `LEAKED-PODS` line is either a real leak or a false positive to understand before merge. Follow-ups, known and out of scope here: - The allowlist exempts a whole suite, so a new leak in an allowlisted suite stays hidden until the issue's fix removes the line. - Attribution is per suite directory, not per Chainsaw `Test`; the leak lines carry the release instance, which usually names the test. - Only pods are checked. A leaked PVC, LoadBalancer IP or bucket holds no node memory and is not seen. - The 300s budget rests on Chainsaw's cleanup timeout, not on a measured pod-gone latency; the `clear after` lines are where that measurement will come from. - A platform object recreated mid-suite (for example a HelmRelease uninstall/reinstall remediation, or a force-replace on an immutable field) has a root newer than the suite start, and its pods would be charged to whichever suite was running as a leak; the first CI runs are where to look for this. - The per-pod wait recomputes the remaining budget before each pod, and no unit test pins that. - `kubernetes-previous` is allowlisted for #4489 although the stuck finalizer was seen only in `kubernetes-latest`; both suites create a TenantControlPlane the same way, and the exemption hides any new leak there until #4489 is fixed. - Each clean suite now waits for its own pods to finish terminating before the next starts, where the tail used to overlap the next suite. The first full run's `clear after` lines say what that costs against the E2E job's 215-minute cap. - After a timeout followed by a failed re-list, the leak lines come from the list taken before the wait, so a pod whose own wait had already succeeded is still named and the count can be too high; the verdict is unaffected. - Likewise, when every wait succeeded and the re-list then fails, the inconclusive report still lists the pods whose deletion was just confirmed. In the boundary case where the budget runs out during the last successful wait and the re-list then fails, the result is inconclusive rather than a verdict from the last good list, because the guard only treats the budget as spent when it finds it exhausted at a check. - A mistyped `COZY_LEAK_ALLOWLIST` path passes allowlist validation; it fails closed, since no suite is then exempt. - An allowlisted suite is not waited on, so `kubernetes-latest` and `kubernetes-previous` will print `LEAKED-PODS ... allowlisted=` on most runs even for pods still in normal teardown; waiting out the budget first and applying the allowlist only to what remains would stop that noise. - A slow teardown in each of several suites can add up to 300s apiece, against the E2E job's 215-minute cap. On the first full run of this PR (run 36223896793) the `clear after` values of the suites that were waited on summed to 210s; the last full run before this change took 158 of the 215 minutes. - A pod whose owner's resource type is no longer served gets "the server doesn't have a resource type" rather than NotFound, so its suite is reported unchecked instead of leaked; it still fails. - `kubectl wait --for=delete` waits for deletion, while the guard skips pods in `Succeeded` or `Failed`; a pod that finishes but is never deleted (a Job under a leaked root) spends the rest of the budget before the suite shows `clear`. The verdict is right; only job time is lost. - The unit tests do not pin the one-line empty report Chainsaw writes for a directory with no tests (`_lib`), nor the branch that catches a spent budget before calling `kubectl wait`; both are verified against the Chainsaw and kubectl sources rather than by a test. - The runner's per-suite report directory from `mktemp -d` is not removed; the sandbox is discarded after the job. - Each non-suite directory under `hack/e2e-chainsaw/` (today `_lib`) costs one empty Chainsaw invocation, because the runner leaves the decision of what is a test to Chainsaw. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff adds files under `hack/`, changes the `test-chainsaw` target, and rewords one comment and the failure line in the four E2E workflows. The trigger map's `hack/` entry is for cozystack/ccp, whose skills anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`, none of which change here. A GitHub code search for `chainsaw` across the cozystack organisation finds hits only in this repository. ### Release note ```release-note test(e2e): the E2E app suite now fails the suite whose pods outlive its own cleanup by more than 300s, naming each leftover pod and its owner; suites with a filed leak are listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt and only warn. ``` |
||
|
Aleksei Sviridkin
|
4d7f9e485a |
docs(ci): say why the overlay's ancestry tie goes to main (#4469)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does This fixes the comment on the overlay's ancestry tie, which described the wrong reason for the tie. No behaviour changes. The comment said a line fresh off main is the same as main. That holds only for the graph. `cozystack-packages:main` is built from main's tip, so once main moves past the cut, a stack on such a line gets newer images than the line. The comment now says so. It also says why the tie still goes to main: a branch forked from main before the cut ties the same way, and that case is far more common. This follows up on a review note in #4460 and is stacked on it. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note docs(ci): explain why the base-branch overlay sends an ancestry tie to main. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified how artifact selection works for stacked pull requests on release lines without line-specific commits, including why `main` is preferred in a tie and how that may affect image freshness. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d8ec8f0af1 |
fix(ci): run every bats unit file after a failure and cover build-*.sh in the cross-compile check (#4633)
## What this PR does Two fixes to the tooling of the unit lane. A local run of the bats unit tests stopped at the first red file and never ran the files after it, while the output looked like a finished suite. Each bats file is its own make target, and without `--keep-going` make stops scheduling targets after the first failure. CI passes `-k`, so it was not affected. The target now runs the files in a `--keep-going` sub-make. That sub-make still exits non-zero if any file failed, and it shares the caller's `-j` slots and `--output-sync`. The new `hack/bats-unit-keep-going.bats` drives the real Makefile against two fixture files. It checks that both run and the target fails when the first one fails, that the target exits 0 when both pass, and that the sub-make still finds the root Makefile when `MAKEFILES` names another file. I split the exact-line check on this target in `hack/bats-runner-coverage.bats` in two to match the new recipe. A workflow comment and a test name that still called each file a prerequisite now call it a target. The cross-compile check never looked at the multus-cni builder. It treated only a script named exactly `*build.sh` as a compile step, and multus runs upstream `./hack/build-go.sh`. The pattern now allows a suffix after `build` but no `/`, and each case has a fixture test. Among the Dockerfiles the check scans under `packages/`, this matches exactly one new line, the multus stage, and that stage passes. Closes #3692 for the make-level truncation. The in-file half, where `hack/cozytest.sh` ends a file at its first failing test, is tracked in #4620. Closes #4500 Things I left out: - The multus stage passes because a `curl ... TARGETARCH` step sits in the same stage. The check does not see that `build-go.sh` reads `TARGETPLATFORM`. The test header already documents this per-stage limit. - The sub-make reads only the root Makefile, so an extra `make -f other.mk` given to the parent no longer reaches the bats targets. Command-line variables and `MAKEFILES` still do. Nothing in the repo calls it that way. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): `make bats-unit-tests` runs every bats file after a failing one and still fails at the end, and the image cross-compile check now covers build scripts such as `build-go.sh`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Unit tests now continue running across individual test-file failures while still reporting an overall failure if any file fails. * Build-stage detection now recognizes build scripts with names such as `build-go.sh`. * **Tests** * Added coverage for unit-test results and build-script detection. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
71a943277f |
build(make): keep running bats unit files after one fails
Each bats file is its own make target, so a plain `make bats-unit-tests` stopped scheduling files at the first red one and never ran the rest, while the output looked like a finished suite. CI passes -k and was not affected; a local run was. The target now runs the files in a --keep-going sub-make, which still exits non-zero when any file fails and shares the caller's job slots and output sync. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
3111fa7802 |
chore(ci): add area/openbao to the label set and the scope mapping
Every openbao PR lands on area/uncategorized, because the labeler has no scope entry for it and there would be no label to apply if it did. #4168 and #4177 are both sitting on the fallback right now. labels.yml sets the bar for a new area at three open issues on the topic, and openbao is at three: #2787, #2483 and #2793. It also follows the shape already used for a single shipped component, next to area/kamaji, area/keycloak and area/cilium. Maps the plain `openbao` scope and `openbao-central` alongside it, since the platform instance lives in its own package and gets its own scope. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
57bf10acae |
test(e2e): stop bringing up the monitoring instance nothing looks at
The install turned monitoring on in the root Tenant and then waited up to 15m each on vmalert and vmalertmanager, vlclusters/generic, the shortterm and longterm vmclusters, the grafana-db CNPG cluster and the Grafana Deployment. No suite inspects any of it. The one Chainsaw suite that mentions monitoring, foundationdb, asserts a WorkloadMonitor, whose CRD ships with cozystack-controller and exists without the monitoring stack. The OIDC wiring of the monitoring chart is covered by its helm-unittest suite rather than by the e2e run. Those waits were not idle. The stack competes with the rest of the install for node capacity, and their own comment recorded that a slow VictoriaLogs start could fail the install on a pull request that never touched monitoring. The run was paying bring-up time and flake risk for a component it never looked at. The monitoring package stays installed: tenant-application declares dependsOn cozystack.monitoring-application, so dropping the package would leave the platform release unresolved. Only the tenant-root instance is turned off. The container lane's VictoriaLogs storage override stays. It does nothing while monitoring is off, and it keeps a run that turns monitoring back on from creating a replicated PVC on a lane that only has local storage. One check goes with the Grafana waits: the curl through root-ingress-controller at grafana.example.org, the only assertion that the root ingress routes to a backend. Grafana was its target because it happened to be installed, and pointing it at a seaweedfs host needs the expected response checked on a live stand first. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
941101b65e |
ci: drop unused packages:write from the pull-request workflow
The build, build-talos and finalize jobs each requested `packages: write`, but none of them writes to GHCR. They authenticate to OCIR with OCIR_USER/OCIR_TOKEN and push to REGISTRY, which the workflow sets to OCIR. The only GHCR login in the file is in verify-release-candidate, and that job holds `packages: read`. GHCR pushes elsewhere in CI authenticate with GITHUB_TOKEN plus packages:write rather than a stored secret, so on those paths the scope is the whole credential. Granting it on the pull_request lane hands every same-repo PR run a token with GHCR write authority that the lane has no use for. Whether a package honours that write depends on its Actions access setting, which lives outside this repo, so this is least-privilege hygiene rather than a fix for a demonstrated exploit. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Marian Koreniuk
|
7500a9fbae |
test(kubernetes): add the Proxmox e2e suite, parked disabled
Upstream CI has no hypervisor, so this suite cannot run there; it ships
`.disabled` and is run by a lane that owns a Proxmox stand.
Named for what the chart actually creates. The resource definition prefixes the
release, so a Kubernetes CR called k8s-e2e produces
ProxmoxCluster/kubernetes-k8s-e2e in the tenant's own namespace, and the admin
kubeconfig Secret is kubernetes-k8s-e2e-admin-kubeconfig.
Stand-specific values are variables rather than hardcoded. Every one of them is
a bare ${COZY_PVE_*} that run-proxmox.sh expands with envsubst, supplying the
defaults itself — GNU envsubst leaves a ${VAR:-default} in the text exactly as
it is, set or not, and that reaches the apiserver as a string where the schema
wants an integer — and refusing a fixture with a variable still in it. The
StorageClass and the PVC probe read the same COZY_PROXMOX_STORAGE, so the volume
lands where the probe looks. The runner reads the tenant kubeconfig's .conf key,
the address a runner outside the management cluster can reach.
The stand's resource pool comes in through COZY_PVE_POOL, empty by default.
hack/proxmox-e2e-fixture_test.bats pins the shape of the fixture and the render.
The PVC probe reads the storage again after the claim is deleted: a deleted
claim whose disk is still there is the leak the pre-delete hook's Step 1b
exists to prevent. keep_tenant is honoured by the suite's own teardown, not
only by the workflow's. The workflow carries no fallback for the stand's
variables: an unset one fails the job before a key is sent anywhere. The
schedule runs only where vars.PROXMOX_NIGHTLY is true: the job targets a
self-hosted runner with the `proxmox` label, and a scheduled run on a repository
without one would sit in the queue until GitHub cancels it.
The suite also asserts the half of the deployment that lives in the management
cluster — the CSI controller and the cloud-controller-manager — because that is
the property the pair is designed around and a tenant-only assertion would pass
with the token in the wrong place.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|