mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
705e76f5dbca284887d78d87bbc06e63a239778b
5218
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov Daniil
|
705e76f5db |
fix(ci): fail closed on renamed and truncated fork PR file lists (#3262)
Address review feedback from coderabbitai on .github/workflows/e2e-fork.yaml: the docs-only decision read only `filename`, so a rename reported the NEW path alone. A PR renaming packages/<app>/values.yaml -> docs/whatever.md therefore looked docs-only, concluded "E2E Tests" green, and never ran the suite — while in fact removing a code path. Both sides of a rename must be under docs/ for a PR to count as docs-only. `pulls.listFiles` is also capped at 3000 files, and a truncated list could omit the only non-docs path. Fail closed at the cap instead of ruling docs-only on a list known to be incomplete. Feed both sides of a rename to TIA as well: the old path is a removal the selector must see. Extra paths can only widen the selection, so this cannot cause a suite to be skipped. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
4f9f599cce |
test(hack): cover the default (non-export) build path (#3262)
Address review feedback from IvanHunters on hack/common-envs_test.bats: every existing case set OCI_EXPORT_DIR, so the OFF path — the one every same-repo PR, main and release build still takes — had no regression test at all. A refactor of the export wiring could stop pushing entirely, or strip the release tags, and the suite would stay green. Add a case asserting that with OCI_EXPORT_DIR unset buildx still pushes (--push=1) and writes no archive, image-tags expands all three release tags, and talos' skopeo copies still target docker:// with the `[ -z "$(OCI_EXPORT_DIR)" ]` release gates inert. Mutation-checked: making the PUSH/LOAD override unconditional (so export mode leaks into the default path) fails this case. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
914d75f06f |
test(hack): count --tag occurrences, not matching lines (#3262)
Address review feedback from IvanHunters on hack/common-envs_test.bats:70: `grep -c` counts matching LINES, not occurrences. The image-tags macro expands onto a single line, so every "exactly one tag survives under OCI_EXPORT_DIR" assertion returned 1 no matter how many --tag flags were actually emitted. Proven vacuous by mutation: removing the OCI_EXPORT_DIR guard from image-tags restores three --tag flags on that one line and the suite stayed green. Switching to `grep -o … | wc -l` counts occurrences, and the same mutation now fails the suite as it should. Applied to the --output type=oci,dest= assertions too, which count occurrences the same way and had the same blind spot. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
65cd5f629c |
fix(ci): gate release tags/pushes off under OCI_EXPORT_DIR (defense-in-depth)
Belt-and-suspenders for the B2-class trap lexfrei flagged as latent in
image-tags. PUBLISH_* are 0 on fork PRs, so today the export path never emits a
versioned/floating tag — but if it ever did:
- image-tags: multiple --tag under `--output type=oci` write >1 manifest into
the archive's index.json, and `skopeo copy oci-archive:<f>` then refuses it
("more than one image in oci"). Suppress the versioned/floating tags when
OCI_EXPORT_DIR is set (macro-level $(if …), so `make -n` prints only the
taken branch).
- packages/core/talos: gate its versioned/floating `skopeo copy docker://…`
release copies on an empty OCI_EXPORT_DIR too.
The release (non-export) path is unchanged — versioned + :latest still emitted.
Covered by a new common-envs_test.bats case that forces PUBLISH_*=1 under
export.
Reported by gemini-code-assist on #3262.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
1a1f3696ee |
fix(ci): fix the nullglob-defeated OCI-archive guard in the fork publish
`count=$(ls _out/oci/*.oci.tar | wc -l)` runs under `shopt -s nullglob`, so when nothing matches the glob expands to zero words and `ls` lists the current directory instead — count is always >=1 and the "at least one archive" guard never fires. A zero-archive build then falls through to `flux push` and defers the real failure to an obscure image-pull error deep in the ~180-minute e2e run. Glob into an array (empty under nullglob) and test its length; reuse it for the push loop. Reported by coderabbitai on #3262. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
3b70fda870 |
fix(ci): make fork OCI_EXPORT_DIR absolute so archives land where the upload reads
The fork build sets OCI_EXPORT_DIR to a relative `_out/oci`, but each image is built with `make -C <pkg> image`, so buildx resolves the relative `dest=` against the package directory — the archive landed in <pkg>/_out/oci/ while the `if-no-files-found: error` upload reads repo-root `_out/oci/*.oci.tar`. Every non-docs fork build would have failed at that upload (never caught, since same-repo PRs leave OCI_EXPORT_DIR empty). Root it at github.workspace so the dest resolves to repo-root _out/oci regardless of the sub-make cwd. Reported by gemini-code-assist on #3262 (its abspath suggestion resolves against the package dir under `make -C`, so this fixes it at the workflow level instead). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
0f6e99eff8 |
fix(ci): publish the "E2E Tests" gate as a commit status, not a check-run
Route the required "E2E Tests" merge gate through a GitHub commit status
posted with the default GITHUB_TOKEN (+ statuses:write), replacing the
COZYSTACK_CI app check-run. The app path needed the app granted checks:write
(it isn't — checks.create 403s "Resource not accessible by integration"), and
the earlier default-token check-run floated under a random check-suite
("PR size label"). A commit status has neither problem: branch protection
matches it by context name, it carries no check-suite grouping, and it is safe
with the default token because a fork's pull_request token is read-only and
cannot post one — only same-repo runs (e2e-report, guarded !fork) and the
privileged workflow_run in e2e-fork.yaml can set it.
The security property is unchanged: a never-posted required status stays
"Expected" and blocks the merge, so a skipped job can't smuggle a green
(#3257). Removes all COZYSTACK_CI app-token machinery from e2e-fork.yaml —
`resolve` posts the status directly, no `check_run_id` threading, no `open`
step.
Branch protection must require the "E2E Tests" status context (no GitHub App
and no app-id pin needed).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
03b519dc75 |
fix(ci): fork e2e — Chainsaw+TIA run and image-push hardening
The fork e2e path in e2e-fork.yaml still ran the pre-#2826 BATS app loop (`hack/e2e-apps/*.bats` + `test-apps-%`), which the merged Chainsaw migration deleted — so the fork run was already broken. Rewrite it to the same Chainsaw + Test-Impact-Analysis path as the in-tree job, and clear the remaining review follow-ups on the publish job. - Run E2E via a single `make test-chainsaw` invocation (no per-app retry — e2e-testing.md §1). Suite selection and the full-e2e override come from `resolve`'s TRUSTED base-repo listFiles/labels (new `changed_b64` / `full_e2e` outputs), never a git diff of the fork-controlled merge ref, so TIA cannot be shaped into skipping the suite (keeps the #3257 property). Collect + upload the Chainsaw JUnit report like the in-tree job. - Restrict fork image pushes to a trusted allowlist derived from the base parent (HEAD^1) Makefiles, so a fork cannot mint an arbitrary repo path under the CI registry by renaming an image. The charset guard stays as defence in depth. - `skopeo copy --all` so a multi-platform build's index digest (the one baked into pr.patch) is preserved. - Pin the flux CLI (FLUX_VERSION) in the privileged publish job instead of install.sh's rolling latest. Docs: e2e-testing.md §10 records that the fork path now runs Chainsaw + TIA and that fork suite selection must stay sourced from trusted data. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
9cddb532bf |
fix(ci): publish "E2E Tests" check-run as the COZYSTACK_CI app
The synthetic "E2E Tests" required check-run was created with the default GITHUB_TOKEN (the "GitHub Actions" app). A token-created check-run has no owning job, so GitHub files it under an arbitrary GitHub Actions check-suite for the head SHA — in practice the earliest one, the PR-size labeler — so "E2E Tests" rendered under "PR size label". Publish it as the COZYSTACK_CI GitHub App instead (create-github-app-token, scoped to permission-checks: write) so the check lands in the app own dedicated suite. This is also stricter: only the CI app can satisfy the gate. - pull-requests.yaml (e2e-report): mint the app token, pass it to checks.create. - e2e-fork.yaml: split resolve into a verdict decision on the default workflow_run token (keeps the actions:read / pull-requests:read lookups) and an "open" step that publishes the check as the app; report re-mints the token so checks.update runs as the same app that opened the pending run. - docs/agents/e2e-testing.md: document the change and the branch-protection requirement. Requires main branch protection to require "E2E Tests" from the COZYSTACK_CI app id (not app_id 15368). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
06d734cf53 |
docs(agents): document the fork-PR e2e split and check-run gate (#3257)
docs/agents/e2e-testing.md is the mandated reference for E2E CI changes but described a single pull-requests.yaml pipeline. Document the split (same-repo e2e in pull-requests.yaml; fork e2e in the privileged e2e-fork.yaml via workflow_run), that the required "E2E Tests" context is a synthetic check-run rather than a job (so renaming the e2e job must preserve it), that fork builds export OCI archives instead of pushing, and the repo prerequisite Actions → "Require approval for all outside collaborators". Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
056ff92e16 |
test(hack): cover OCI_EXPORT_DIR export mode in common-envs (#3257)
Add hack/common-envs_test.bats (runs under make bats-unit-tests). For a representative slice of the root build: matrix — including the two recipes that previously slipped through, capi-providers-cpprovider and core/talos — it asserts that under OCI_EXPORT_DIR each built image emits exactly one --output type=oci and no ungated docker:// push, and that capi emits a single --tag. This would have caught the talos-push and capi-two-tag regressions. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
eba55976a0 |
fix(ci): decide fork docs-only from the PR file list and harden publish (#3257)
resolve inferred "docs-only" from the absence of a pr-patch artifact, but artifact presence is decided by pull-requests.yaml running from the fork's merge commit — a fork could suppress it (code=false, or if:false on finalize) and conclude the required E2E Tests check green with e2e never run, re-opening the very bypass this feature closes. Derive docs-only from trusted base-repo data (pulls.listFiles): green only when every changed file is under docs/; any non-docs file makes e2e required, and a missing pr-patch then fails the check closed. Resolve the PR open-only (no closed/merged fallback). Also harden the privileged jobs: - drop `packages: write` from publish (it authenticates to OCIR, not GHCR); - assert at least one OCI archive before pushing, and validate each archive basename against a strict charset before using it as a registry repo; - install flux from a downloaded script rather than piping curl to sudo bash; - assert `git rev-parse HEAD^2` equals the resolved head SHA in publish and e2e, so a moved merge ref cannot operate on a different tree than the check. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
68cdeb8541 |
fix(ci): emit a single tag for capi-providers under OCI export (#3257)
capi-providers-cpprovider passes two --tag flags; under --output type=oci buildx writes both refs into the archive index, and `skopeo copy oci-archive:<f> docker://…` in the privileged push loop then fails with "more than one image in oci, choose an image". Gate the secondary :$(IMAGE_TAG)-$(TAG) tag off when OCI_EXPORT_DIR is set so the exported archive holds a single manifest. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
ab14922495 |
fix(ci): export talos image as an OCI archive on fork builds (#3257)
packages/core/talos `image-talos` pushed to the registry via `skopeo copy docker://…` unconditionally — it is not a buildx call, so neither OCI_EXPORT_DIR nor PUSH reached it. On a fork PR the registry login is skipped, so this was an anonymous push that died with `denied`, collapsing build-talos → finalize and the whole chain. Under OCI_EXPORT_DIR, copy the installer to oci-archive:<dir>/talos.oci.tar instead of pushing; keep the docker:// push otherwise. A make-level $(if …) is used (not a shell if/else) so `make -n` prints only the taken branch. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
076658e70f |
fix(ci): avoid zizmor github-env finding on fork e2e sandbox name
The `Set sandbox ID` step wrote a computed value to $GITHUB_ENV, which the zizmor github-env audit flags as high severity (write to GITHUB_ENV may allow code execution). Set SANDBOX_NAME as a job-level env var instead — no $GITHUB_ENV write, so the audit does not apply. Uses github.run_id, which is unique per run; in workflow_run context GITHUB_REF is the default branch and cannot disambiguate concurrent fork runs sharing a self-hosted host. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
76c8f890fa |
fix(ci): export OCI archives for buildx calls that bypass image-tags (#3257)
Fork PRs build images to per-image OCI archives instead of pushing, gated on OCI_EXPORT_DIR. The `--output type=oci` was appended only inside the `image-tags` macro, so `packages/system/capi-providers-cpprovider`, whose `image:` recipe uses a hardcoded `--tag` and never calls `image-tags`, ran buildx with no exporter under export mode (PUSH/LOAD forced to 0). No archive was written, and the fork build's `if-no-files-found: error` upload then failed the whole build. Its `--cache-to type=inline` also needs an image exporter. - hack/common-envs.mk: extract the OCI-output snippet from `image-tags` into a reusable `oci-output <image-name>` macro so any buildx call can opt in; `image-tags` now ends with `$(call oci-output,$(1))` (no-op for the ~33 conforming packages). Move `comma :=` above its first user. - packages/system/capi-providers-cpprovider/Makefile: add `$(call oci-output,cluster-api-control-plane-provider-kamaji)` (archive base name equals the OCIR repo the privileged push keys on) and drop the inline cache under export (`--cache-to type=inline` gated off). - .github/workflows/pull-requests.yaml: wrap the `E2E Tests` check-run creation in try/catch. On error, `core.setFailed` so the required merge gate fails loudly and fail-closed instead of silently succeeding. Repo-wide audit of every `docker buildx build` reachable from the root `build:` matrix confirms capi was the only recipe bypassing both `image-tags` and `oci-output`. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
e0ac2f9e12 |
ci(e2e): run fork-PR e2e via privileged workflow_run (#3257)
Fork PRs cannot push images (pull_request runs from a fork receive no secrets), so `make image` pushed anonymously and died with `denied`. That failed every build, collapsed the build->finalize->e2e chain, and left the required "E2E Tests" job skipped — which branch protection treats as satisfied, so an approved fork PR could merge with e2e never having run. pull-requests.yaml (fork PRs): - build / build-talos / finalize export images to per-image OCI archives (OCI_EXPORT_DIR) and upload them instead of pushing, so the red "denied" build wall disappears and the chain no longer collapses. - the in-tree e2e job is renamed off "E2E Tests" and guarded to same-repo PRs; a new e2e-report job concludes the required "E2E Tests" context as an explicit check-run (success on a docs-only PR or when the in-tree e2e passed, failure otherwise). A skipped job can no longer satisfy the required check. e2e-fork.yaml (new, privileged workflow_run from the default branch): - resolves the fork PR, opens a pending "E2E Tests" check-run, and fails-closed if the fork build failed; - pushes the pre-built image archives to OCIR by digest and flux-pushes the digest-pinned packages tree — the OCIR token is live only for trusted binaries (git/skopeo/flux/yq) over data, never for `make` or any fork-authored script; - runs the e2e suite against the pushed closure with no push token and concludes the "E2E Tests" check-run on the PR head SHA. Fork build code never receives registry credentials; the fork's images and test scripts run only in the credential-less, ephemeral e2e job. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
529efa7e01 |
build(hack): add OCI_EXPORT_DIR archive-export build mode
When OCI_EXPORT_DIR is set, every image builds to a per-image OCI archive (<dir>/<name>.oci.tar) instead of being pushed, and PUSH/LOAD are forced off. This lets a credential-less build (a fork PR's pull_request run) produce the images as artifacts that a privileged workflow later pushes to the registry by digest. The image digest is captured via --metadata-file regardless of output type, so the values.yaml/.tag refs baked during the build match exactly what the privileged run pushes. The output flag is threaded through the image-tags macro, which every package's buildx call already uses, so no per-package Makefile change is needed. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
cc44f6c981 |
ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449)
## What this PR does Moves release validation from promote time to rc time, making promotion itself a pure retag operation. This is the follow-up to the v1.6.0 retrospective: the promote flow's checks ran on the promote PR (coupled to promotion, skippable by never promoting), the changelog and website docs were produced only after the release went public, and the finalize tag push silently failed to trigger `tags.yaml` at all (persisted-credential bug, root-caused below). The release lifecycle after this PR: 1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes the staging branch — and now runs a **mandatory full e2e** for every `-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and any published tag) get the same validation as a **manual button** (`workflow_dispatch`). 2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**: the release changelog is generated from the rc tag (reusable `changelog-generate.yaml`) and committed to the rc staging branch, ready for review long before promotion. 3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence: the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages the release, picks up the rc-time changelog (or generates one), and **opens the website docs PR** on cozystack/website from the staging branch — parked with an explicit DO-NOT-MERGE-until-finalize contract. 4. The promote PR **no longer gets `full-e2e` automatically** — its tree is the rc's already-validated digests with only tag strings rewritten; a maintainer can still add the label manually to force the suite. 5. Finalize performs the irreversible side effects (tag, publish, retag, chart) — and with `persist-credentials: false` its tag push now actually authenticates as the CI app, so the `tags.yaml` backstops (`generate-changelog`, `update-website-docs`) fire for promoted tags. This is the one-line fix for the v1.6.0 incident where the tag triggered nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader` silently defeats the app token injected via `git remote set-url`, and `GITHUB_TOKEN` pushes create no workflow runs. Everything is pinned by executable contract tests (`hack/promote-gate-contract.bats`, extended `hack/release-changelog-contract.bats` / `release-changelog-behaviour.bats` — 48 tests total, including behaviour tests that execute the new `hack/parse-rc-tag.sh` and `hack/select-changelog-source.sh` for real), and `docs/release.md` is rewritten to describe the new lifecycle end-to-end. ### Commit map (review commit-by-commit) - `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing changelog-before-publication rework (changelog generated at promote, committed to the staging branch before the PR opens, finalize publishes it as the release body; tags.yaml job becomes a verbatim-port backstop), rebased onto main. - `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane (prerelease → staging branch, stable → tag; draft releases fail fast; dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts). - `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut rc tags (also on `release_exists` re-runs); fixes the `cut-prerelease.yaml` comment that falsely claimed e2e already existed. - `2dfbeea77` — `changelog-generate.yaml` (reusable core) + `changelog-rc.yaml` (rc-time button committing to the staging branch); promote-rc gains a top-level `parse` job with a strict no-whitespace tag policy (`hack/parse-rc-tag.sh`); source selection is validate-then-fallthrough (`hack/select-changelog-source.sh`) so an invalid higher-priority changelog can never silently win. - `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override + removal of the automatic `full-e2e` label (pull-requests.yaml release-path e2e now requires an explicit label; a post-open label now actually starts the run). - `936190d4b` — `website-docs` job at promote time behind a FETCH_REF-support guard (see dependency below), with the DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before `release-next`; `tags.yaml::update-website-docs` stays as the backstop. - `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0 root cause). - `717c8860f` — `docs/release.md` coherence pass over the whole new lifecycle. ### Dependency The promote-time website docs job needs cozystack/website#631 (`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard detects the missing support and degrades to a ⚠️ line in the promote PR body — promotion is unaffected and the tag-time backstop backfills, so this PR is safe to merge first. ### Merge instructions Please **merge-commit, do not squash**: the 13 commits are deliberately atomic (the credential root-cause fix is separable from the feature work) and squashing would collapse ~3k lines into one commit, hurting bisectability. ### First live run The e2e lane is verified by lint, contract tests and dry-traces, but has not yet executed against a real tag; the intended first live run is a manual `e2e-tag.yaml` dispatch against an existing published rc tag before the gate is relied on for a real promotion. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added reusable changelog generation with validation, “present” signaling, and safe artifact upload behavior (including RC-time support). * Added an E2E release-tag workflow with tag parsing, deterministic sandboxing, and comprehensive report collection. * Improved RC promotion with a stricter parse front-door, verified E2E green gate (optional bypass), and earlier parallel changelog/asset staging. * **Documentation** * Refreshed release-process docs to match the updated RC-to-stable, changelog, and website-docs timing rules. * **Bug Fixes** * Improved PR finalization to source release bodies from verified changelogs and warn/fallback on missing or empty content. * Tightened label gating so E2E runs only under the intended conditions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
5d420475f4 |
refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows) (#3454)
## What this PR does
The E2E job's `if: always()` cleanup ran `rm -rf /tmp/$SANDBOX_NAME`
with no check that `SANDBOX_NAME` was ever set. `Set sandbox ID` carries
no condition, so it inherits the implicit `success()` every
unconditional step gets: any failure ahead of it — a conflicting
`pr.patch`, a `download-artifact` miss, `Checkout code`, the
release-path `curl` — or a cancellation that reaches the job after it
has started (the job condition is `!cancelled()`) skips it, and
`SANDBOX_NAME` is never written to `$GITHUB_ENV`. The `if: always()`
steps then run anyway and interpolate the empty value; `Remove
workspace` is the destructive one, expanding to `rm -rf /tmp/` and
failing the cleanup step against root-owned entries in the runner's
`/tmp`. This is not a "re-run skips the setup" bug: a re-run restarts
the job at step one and re-runs the unconditional `Set sandbox ID` — the
trigger is incomplete initialization, by any of the routes above.
The `/tmp/$SANDBOX_NAME` workspace is a vestige of the retired multi-job
self-hosted-runner design, where several `runs-on: [self-hosted]` jobs
shared no state and rendezvoused through a `sha256(REPO:WORKFLOW:REF)`
key, with a per-run `cp -r` of the checkout into `/tmp`. Since CI moved
onto the ephemeral pool (`runs-on: oracle-vm-32cpu-128gb-x86-64`) it is
a single job: there is no cross-job rendezvous and the checkout is never
reused across runs, so copying it into `/tmp` and removing it afterwards
is dead weight. (An earlier review justified the change by "wiping the
entire `/tmp` of the self-hosted runner" — that describes the topology
we retired; the hazard is real, but not for that reason.)
So this drops the workspace rather than guarding the `rm`. Both the PR
(`pull-requests.yaml`) and nightly (`nightly.yaml`) E2E jobs now run in
`$GITHUB_WORKSPACE`: the `Prepare workspace` and `Remove workspace`
steps are gone, the `cd /tmp/$SANDBOX_NAME` prefixes are dropped, and
the upload paths point at `_out/…` instead of
`/tmp/${SANDBOX_NAME}/_out/…`. That takes the empty-value expansion out
of the `cd`s, the stray `mkdir`, the upload paths and the destructive
`rm` in one move, across both workflows rather than one step of many.
`Set sandbox ID` stays: the job still reads `$SANDBOX_NAME` directly
(failure diagnostics, the chainsaw `docker cp`, teardown, and every
`make SANDBOX_NAME=…` argument) — the container needs a name, just not a
directory.
It also removes the dead code the single-runner retirement left in
`packages/core/testing/Makefile`: the world-writable `mkdir -p
/tmp/${SANDBOX_NAME}` + `chmod 777` in the `apply` target (they backed a
`-v /tmp/${SANDBOX_NAME}:/workspace/hosttmp` bind mount that was later
removed — nothing under `images/e2e-sandbox/` references
`/workspace/hosttmp`), and the `-e SANDBOX_NAME=${SANDBOX_NAME}` passed
into the container (nothing under `images/e2e-sandbox/` reads it). Every
other use of `SANDBOX_NAME` — the container name and the `make`
arguments — is kept.
Note this cannot make a job with an empty `SANDBOX_NAME` go green: such
a job already failed initialization or was cancelled, so its conclusion
is fixed before cleanup runs. It removes a second, destructive failure
the old cleanup piled on top, and it does not change whether re-runs
work in general — they do.
The diagnosis of the real trigger, the corrected blast radius, and the
scope across both workflows plus the Makefile are credited to the
maintainer review on this PR.
### Evidence
Run [30067779342 attempt
3](https://github.com/cozystack/cozystack/actions/runs/30067779342/attempts/3)
exhibits the mechanism: `Download Talos image (regular PR)` failed on
`GetSignedArtifactURL` with a 404, which skipped `Set sandbox ID`, and
the `always()` steps then ran on an empty value — `Collect chainsaw
report` created `/tmp/_out`, the uploads resolved to `/tmp//_out/…`, and
`Remove workspace` did:
```
rm: cannot remove '/tmp/systemd-private-…-systemd-timedated.service-1oADZP': Operation not permitted
rm: cannot remove '/tmp/.X11-unix': Operation not permitted
##[error]Process completed with exit code 1.
```
The step ran unprivileged in `/home/ubuntu/_work/…`, so all it could hit
was `Operation not permitted` on root-owned entries in the sticky `/tmp`
— it deleted nothing. That run is a later, independent occurrence (it
started well after this PR was opened), not this change's motivating
case. The 404 that triggered it is a separate, non-deterministic problem
— the artifact listing succeeded and only the signed-URL call failed
~190 ms in, `download-artifact` treats 404 as non-retryable, and attempt
2 of the same run fetched the artifact fine — and it deserves its own
issue rather than being folded in here.
### Screenshots
Not applicable — CI workflow change only.
### Downstream repositories
The change is confined to this repo's E2E CI
(`.github/workflows/pull-requests.yaml`,
`.github/workflows/nightly.yaml`) and its testing Makefile
(`packages/core/testing/Makefile`). No downstream repository consumes
it.
- [x] No downstream repository is affected by this change
### Release note
```release-note
refactor(ci): drop the vestigial /tmp/$SANDBOX_NAME workspace from the E2E PR and nightly jobs — they now run in $GITHUB_WORKSPACE, removing an unguarded `rm -rf /tmp/$SANDBOX_NAME` cleanup that failed the job on cleanup when SANDBOX_NAME was empty
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved end-to-end test environment setup and sandbox cleanup.
* Updated test reporting and artifact collection for more reliable
results.
* Ensured OpenAPI and full-suite tests run from the correct prepared
environment.
* **Chores**
* Simplified container and workflow configuration for sandbox-based
testing.
* Clarified checkout token handling in nightly workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
9ed5483260 |
feat(computeplane): provision a ComputePlane tenant module for untrusted-code apps (#3280)
## What this PR does Implements the ComputePlane per the merged design proposal ([cozystack/community design-proposals/compute-plane](https://github.com/cozystack/community/tree/main/design-proposals/compute-plane); supersedes the earlier #3150 which was built on the pre-revision surface and came from a fork): a managed, isolated environment for running code-executing (untrusted-code) catalog applications. ComputePlane is delivered as an operator-owned **tenant module**: - **`packages/extra/computeplane`** — the module chart. Provisions a single-tenant, Cozystack-managed cluster by deploying the ordinary `apps/kubernetes` chart (Kamaji control plane + KubeVirt-VM workers) with operator-fixed values: the Cozystack-enablement addons (`fluxcd` + pinned distribution artifact, `certManager`, `ingressNginx` with `exposeMethod: Proxied`) are inline `spec.values`, so nothing injected via `valuesFrom` can disable them. The tenant can set cluster *shape* (`nodeGroups`, schema-validated) and receives no admin kubeconfig. - **Release naming**: neither HelmRelease sets `spec.releaseName` — the aggregated API rebuilds HelmRelease specs without that field, so both releases derive names from their object names. The cluster HelmRelease object is `computeplane-cluster`, and Kamaji writes the admin kubeconfig to the derived Secret **`computeplane-cluster-admin-kubeconfig`** — the contract future `placement: ComputePlane` routing consumes. Tenant-visible secrets are an allowlist (`include: []` withholds everything), so the credential is withheld by construction. - **`packages/system/computeplane-rd`** — registers the `ComputePlane` module-kind (`ApplicationDefinition`, `dashboard.module: true`). - **`packages/core/platform/sources/computeplane-application.yaml`** — self-contained package source: re-declares `apps/kubernetes` as its own source-only component; `dependsOn` cozystack-engine (ApplicationDefinition CRD) and kubernetes-application (registration ordering). Ships in the `iaas` bundle (variant `kubevirt`). - **Tenant toggle** — `computeplane: false` bool. The tenant chart fails loudly when the module is toggled on without the iaas package. Note: automatic `placement: ComputePlane` routing is a deferred follow-up and is not part of this PR; until it lands, external catalogs target the cluster via its kubeconfig Secret. - **`apps/kubernetes` fix** — the Proxied ingress `Service` renders only when the ingress-nginx addon is enabled **and** the tenant has etcd (mirroring the addon HelmRelease gates), independent of the `hosts` list. Follow-ups tracked separately: `placement` routing in cozystack-api, scoped per-service egress, visibility/mutation access control, e2e (#3369), release-name guards for the remaining extra charts (#3371). Tests: helm-unittest suites pin the module contract (object names, no explicit releaseName, tamper-proof inline addons, nodeGroups validation + ingress-role guard, unlisted-field pass-through), the tenant toggle, the package-source/bundle wiring, and the ingress Service gates. ### Screenshots Not a UI change. ### Release note ```release-note feat(computeplane): ComputePlane tenant module — a single-tenant, Cozystack-managed cluster for untrusted-code applications, delivered as an operator-owned extra module wrapping the kubernetes app. Enable with `computeplane: true` on the Tenant; the tenant receives no admin kubeconfig for it, and the cluster's admin kubeconfig Secret is `computeplane-cluster-admin-kubeconfig`. Catalog routing (`placement: ComputePlane`) ships separately. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a new **ComputePlane** tenant option (`computeplane`, default off) to provision an isolated single-tenant environment for untrusted-code applications, including cluster creation and configurable **node groups**. * **Bug Fixes** * Refined ingress-nginx proxied rendering rules to avoid emitting incomplete resources: Ingress/Services are now gated on addon enablement (and etcd-backed tenancy), with correct behavior when hosts aren’t provided. * **Documentation / Tests** * Updated tenant and ComputePlane documentation; added/extended rendering tests for the new ComputePlane module and ingress-nginx gating (including release-name and etcd-less cases). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
e6460dbefe |
docs(governance): document the Reviewer role and align the maintainer roster (#3462)
## What this PR does
Makes the documented roster match who actually holds write access, which
CNCF Incubation checks directly through the API ("Code and Doc ownership
in GitHub matches documented governance roles", Required).
The two lists currently describe different projects. Ten people hold
write or admin on this repository without appearing in `MAINTAINERS.md`;
three people listed there — `@kingdonb`, `@matthieu-robin`,
`@mattia-eleuteri` — hold only read, one of them while carrying "CSI,
Storage, Networking & Security" as a documented responsibility. There is
no in-between state a reviewer will accept: someone with write is either
a maintainer or a reviewer, and should be findable as one.
**Maintainers, +3.** `@lexfrei`, `@myasnikovdaniil` and `@IvanHunters`
work across the whole tree rather than in one area, and already review
other people's changes at maintainer volume.
**Reviewers, new table.** `@sircthulhu`, `@androndo` and `@scooby87` own
a specific area with write access, without the project-wide
responsibilities of a maintainer. The role is not invented here, and it
is not new in practice either: `CONTRIBUTOR_LADDER.md` defines a
Reviewer as someone named in the owners file for one or more
directories, and `.github/CODEOWNERS` has named people since August 2024
— these three among them since February and April. What was missing is
that `MAINTAINERS.md` never said so, which is why the roster and the
access rights read as unrelated. The authoritative area-to-owner mapping
stays in `.github/CODEOWNERS`; this table only names the people.
**Emeritus, +2.** `@gecube` and `@nbykov0` move to the emeritus section.
Neither is carrying the responsibilities the ladder attaches to the role
— reviewing at least 40 pull requests a year and taking part in the
project's technical direction — and the section exists to record exactly
that, as a note on past contribution rather than a withdrawal of it.
**Security contacts.** `SECURITY.md` already routes reports correctly;
this names the maintainers who triage them, so the Required "contact
information" item points at people.
### What this PR does not do, and depends on
- `.github/CODEOWNERS` on `main` is still a single wildcard line naming
seven people, five of whom are not in `MAINTAINERS.md`. Aligning it is
#3228, which already scopes ownership by area and has been in draft
since 7 July. **This PR should land with or after it** — until then the
Reviewers table points at a file that does not yet reflect it. The area
additions still missing from #3228 (`@kingdonb` on
`packages/system/fluxcd*`, GPU/HAMi, `metrics-server`/`seaweedfs`) are
prepared and not pushed, because that branch is not mine to force.
- Permission changes themselves are not in any PR: raising `@kingdonb`,
`@matthieu-robin` and `@mattia-eleuteri` to write, and dropping write
from the four contributors who are neither maintainers nor reviewers, is
done by hand in the org.
- Two cells need confirmation from the people involved: `@scooby87`
publishes no name or company, and `@androndo`'s "Independent" is
inferred rather than stated. Both are placeholders, not claims.
### Screenshots
N/A — documentation change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
Walked the trigger map against the diff: no row matches. The change
touches no package, no schema, no `ApplicationDefinition`, no platform
variant, no release asset and no node prerequisite. The website carries
no page mirroring `MAINTAINERS.md` — only blog posts that mention
maintainers in passing, which are historical and not regenerated.
The CNCF `.project` metadata does mirror this file, and is being updated
in a companion PR in
[cozystack/.project](https://github.com/cozystack/.project):
`project_lead` there still points at an emeritus maintainer, and the
roster needs the same three additions plus the reviewers team. That
repository is not on this checklist because it is CNCF automation
metadata rather than a downstream consumer of the platform.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Expanded maintainer and reviewer listings, including area ownership
guidance.
* Clarified vulnerability reporting instructions with specific private
contact options and security triage details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
93338cc434 |
refactor(computeplane): drop the in-cluster Flux pin
The fluxcd addon pin was a holdover from the #3150-era model where app charts rendered nested HelmReleases into the target cluster and needed an in-cluster helm-controller. Consumers now keep their releases on the management cluster and remote-apply via spec.kubeConfig, so the management Flux resolves charts and applies manifests — nothing inside the ComputePlane reconciles HelmReleases. This also decouples the module from the FluxCD addon removal (#3379). certManager and ingressNginx stay pinned: Certificate and Ingress objects land inside the cluster and need their controllers there. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
2ff9c30d04 |
docs(kubernetes): note the etcd gate in the Proxied Service comment
The Proxied ingress Service is gated on _namespace.etcd as well as the addon toggle and exposeMethod, but the comment above it still described only the addon+exposeMethod gate — comment drift from the etcd-gate change. State both gates so the comment matches the condition it sits on. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8fc8d23dfd |
fix(computeplane): address review round three
Gate the Proxied ingress Service on _namespace.etcd like the addon HelmRelease it fronts, so an etcd-less tenant with the addon pinned on does not get a selector-less Service (+ test). Correct the secrets framing: the tenant-visible surface is an allowlist (include: [] withholds everything), so drop the never-load-bearing exclude entry, the bats cross-check and the rename-coupling comments, and state the real mechanism. Fix the etcd prerequisite docs (an ancestor's etcd satisfies the wrapped chart; the awaiting-etcd beacon is the diagnostic), reword the tenant toggle to stop advertising the not-yet-shipped placement routing, require maxReplicas >= 1 on the ingress-role node group, and fix comment drift (Service name after the release rename, placement-hook label, package-guard lookup semantics). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
366268d9f5 |
test(kubernetes): pin that a disabled ingress addon renders nothing with hosts set
hosts with the addon disabled used to render the Ingress; the addon gate makes 'the addon is off' mean nothing renders. Pin the combination so the behaviour change reads as a decision. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
5a3d25b8f5 |
docs(computeplane): document the etcd prerequisite and unlisted-field pass-through
The tenant Kubernetes machinery — including the cluster-autoscaler that brings up the default scale-from-zero md0 group — is backed by the tenant's etcd module, so a ComputePlane without it provisions no workers and gives no diagnostic; state the prerequisite in the README. Also state plainly that unlisted NodeGroup fields pass through to the wrapped chart unvalidated (the schema generator cannot close the object), with a test pinning that behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
6dcf5373e7 |
fix(computeplane): let Flux derive both release names
The aggregated API cannot represent spec.releaseName: the conversion rebuilds HelmRelease specs without it, so the first tenant edit through the API dropped the module's explicit computeplane-module release name, let it default to the object name computeplane, and collided with the Helm storage record the cluster release owned — with the tenant chart writing the old name back every interval. Drop releaseName from both HelmReleases and derive names from object names, like every other tenant module. Kamaji now writes the admin kubeconfig to computeplane-cluster-admin-kubeconfig; the computeplane-rd secrets.exclude entry follows in the same change, and a bats test pins the exclusion to the name the cluster release actually produces so a future rename cannot silently re-expose the credential. Also add the check-release-name guard the other extra modules carry, and pin the fluxcd addon's distribution artifact to the flux-operator version vendored in system/fluxcd-operator instead of :latest. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
ebeeab77aa |
docs(platform): correct the computeplane dependsOn and variant comments
The dependsOn comment claimed the kubernetes-application edge orders the addon artifacts the wrapped chart references; variant dependsOn only gates install: components — the operative effect is ordering the computeplane-rd install so the ComputePlane kind is not registered before the kubernetes machinery exists. Also document why the variant is named kubevirt. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
96842d4a94 |
fix(computeplane): validate nodeGroups and require an ingress-nginx role
nodeGroups was a bare object: typos surfaced as a stuck release deep in the wrapped chart instead of an admission error, and groups without the ingress-nginx role silently left the pinned ingress-nginx controller (and with it the ComputePlane ingress) Pending forever, since supplied groups are not merged with the wrapped chart's default md0. Reuse the kubernetes app's nodeGroups typedefs for the schema and refuse rendering when no group carries the role. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
76374f6472 |
fix(tenant): fail loudly when computeplane is enabled without its package
The computeplane-application package ships in the iaas bundle while the toggle arrives with tenant-rd in the system bundle, and bundles are independent booleans. Refuse the render with a clear diagnostic instead of producing a HelmRelease that retries a nonexistent artifact forever. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
bca5b3f9fb |
fix(kubernetes): render the Proxied ingress Service only when the addon is enabled
exposeMethod defaults to Proxied while the addon defaults to off, so gating the Service on exposeMethod alone emitted a selector-less Service — surfaced in the tenant dashboard — on every cluster that runs no ingress-nginx at all. Gate on the addon instead; ComputePlane sets enabled: true explicitly, and the Service still renders without a hosts list. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
aa5198b758 |
fix(platform): add cozystack-engine dependency to the computeplane package
The computeplane-rd component installs an ApplicationDefinition, whose CRD is served by cozystack-engine — hack/check-applicationdefinition-crd-ordering.bats enforces the edge for every package with a system/*-rd component. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
9cb4f2922d |
fix(kubernetes): create the Proxied ingress Service independently of hosts
The proxy Service was only rendered when addons.ingressNginx.hosts was non-empty, but external consumers — e.g. a ComputePlane app's per-app management Ingress — need to route to the tenant cluster's ingress-nginx without the module knowing every app host in advance. Render the Service whenever exposeMethod is Proxied; the Ingress still requires hosts. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
b2232904b6 |
feat(tenant): add the computeplane module toggle
Wire the computeplane module into the tenant chart as a bool toggle, matching the other tenant modules. The module's own Helm release takes the -module suffix so the canonical release name computeplane (and with it the computeplane-admin-kubeconfig Secret) stays reserved for the cluster HelmRelease the module renders. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
4c3262a4ee |
feat(computeplane): deliver ComputePlane as a tenant module wrapping the kubernetes app
Add the operator-owned computeplane tenant module per the revised design (cozystack/community design-proposals/compute-plane): an extra/computeplane chart that provisions a single-tenant, Cozystack-managed cluster for untrusted-code apps by deploying the ordinary apps/kubernetes chart with operator-fixed values (fluxcd, cert-manager, ingress-nginx Proxied). The package re-declares apps/kubernetes as its own source-only component, so the module is self-contained; the cluster release is named computeplane so Kamaji writes the admin kubeconfig to computeplane-admin-kubeconfig — the contract consumed by placement: ComputePlane applications. The tenant holds no admin kubeconfig and can set cluster shape (nodeGroups) only. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8b461f0c0b |
feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins (#3378)
## What this PR does Cozystack builds and publishes `ghcr.io/cozystack/cozystack/grafana` — a rebuilt Grafana image — and uses it by default in the monitoring stack. Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a redistributor of an AGPL Grafana build. This is a governance concern for CNCF Incubation (there is no precedent for a CNCF project redistributing an AGPL binary). The only reason the image was ever rebuilt was to bundle the VictoriaLogs datasource plugin, which was not in the official Grafana plugin catalog at the time. It is now published there as a signed plugin (`victoriametrics-logs-datasource`), so the rebuild is no longer necessary. This PR moves the deployment to the upstream `grafana/grafana` image (digest-pinned, routed through `cozy-lib.image` so mirrored-registry installs keep working) and installs the plugins at container startup via `GF_INSTALL_PLUGINS`, with versions pinned: - `victoriametrics-logs-datasource 0.29.0` — the VictoriaLogs datasource (signed; the reason the image was rebuilt) - `natel-discrete-panel 0.1.1`, `grafana-worldmap-panel 1.0.6`, `marcusolsson-dynamictext-panel 6.2.0` — reproduce the panel set the removed Dockerfile vendored **Air-gapped installs keep working.** The plugin zips are not downloaded from the internet at pod startup: they are mirrored from the official catalog into the existing `grafana-dashboards` image at build time and served by that in-cluster static file server, next to the dashboards it already serves. `GF_INSTALL_PLUGINS` points at `http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…`, so the runtime dependency stays inside the cluster and the artifact is mirrored into private registries together with every other first-party image. A new `allow-to-grafana-dashboards` CiliumNetworkPolicy (mirroring `allow-to-keycloak`) opens tenant egress to that server, since tenant egress is default-deny. As a result, `ghcr.io/cozystack/cozystack/grafana` is no longer built or published: the Dockerfile, the pinned image tag, the `image` targets in both monitoring Makefiles, and the build step in the root Makefile are removed. Notes: - The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored plugin zips are Apache-2.0 licensed and are served unmodified from the official catalog builds (signatures intact). - `allow_loading_unsigned_plugins` and the custom `/var/lib/grafana-plugins` path are dropped: the catalog datasource is signed, and plugins install into the writable default `GF_PATHS_PLUGINS` (`/var/lib/grafana/plugins`). - The VictoriaLogs datasource provisioning (the `GrafanaDatasource` CRs pointing at `vlselect`) is unchanged. - Plugins are re-installed on every pod start from the in-cluster server (no PVC involved, ~77 MB per start per replica over the cluster network); a failed download fails the pod loudly instead of silently dropping the datasource. - `natel-discrete-panel` and `grafana-worldmap-panel` are Angular plugins; Angular is disabled by default in Grafana 11, so they do not load — this is unchanged from the current rebuilt image (same Grafana 11.6.15, same config). They are kept for parity and can be dropped as a follow-up. ### How this was verified - `helm unittest` across the repo passes, including new regression tests: the monitoring chart asserts the exact upstream image reference (with and without `images-registry` set) and the exact `GF_INSTALL_PLUGINS` value; the tenant chart asserts the new egress policy. - Local end-to-end of the full chain in Docker: built the modified `grafana-dashboards` image, served it on a shared network, and started upstream `grafana/grafana:11.6.15` with the rendered `GF_INSTALL_PLUGINS` value. All four plugins download from the in-cluster-style URL (no internet egress from Grafana), and `GET /api/plugins` reports `victoriametrics-logs-datasource` v0.29.0 as `enabled: true` with `signature: valid`. ### Screenshots Not applicable — no UI change. Grafana renders the same datasources and panels; only the image source and plugin delivery mechanism change. ### Downstream repositories Walked the trigger map against the diff. This is an internal change to how the monitoring stack sources the Grafana image and its plugins (`packages/system/monitoring`, `packages/system/grafana-operator`, `packages/apps/tenant` network policy). No package is added, renamed or removed; no `values.schema.json`, `ApplicationDefinition`, platform values, release asset, shared `hack/` tooling or `package.mk` contract changes; the added `grafana.image` key is an internal system-chart value, not a user-facing app field. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Grafana in monitoring is now based on a digest-pinned upstream image for more consistent deployments. * Required Grafana plugins are installed automatically at startup via `GF_INSTALL_PLUGINS`; unsigned-plugin configuration was removed. * Tenant networking now includes a new egress rule specifically for Grafana dashboards. * Grafana dashboards image now includes required plugin assets prepared during image build. * **Tests** * Added/updated checks covering digest pinning, mirrored-registry behavior, plugin-installation behavior, and the new Grafana dashboards network policy. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
78fc3f9178 |
refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows)
The E2E job once ran as several state-sharing self-hosted-runner jobs that rendezvoused through a sha256(REPO:WORKFLOW:REF) key and a per-run /tmp/$SANDBOX_NAME copy of the checkout. Since the move to the ephemeral pool it is a single job: there is no cross-job rendezvous and the checkout is never reused, so copying it into /tmp and removing it afterwards is dead weight. Run every step in $GITHUB_WORKSPACE instead: drop the Prepare/Remove workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes, and point the upload paths at _out/. This deletes the unguarded `rm -rf /tmp/$SANDBOX_NAME` rather than guarding it: when an upstream step fails or the job is cancelled, the unconditional `Set sandbox ID` is skipped, SANDBOX_NAME stays empty, and the `if: always()` steps interpolate it — the removal expanding to `rm -rf /tmp/` and failing the cleanup step. `Set sandbox ID` stays, since the container still needs a name. Apply the same treatment to nightly.yaml, and drop the Makefile dead code the single-runner retirement left behind: the world-writable mkdir/chmod of /tmp/$SANDBOX_NAME (backed a bind mount removed long ago) and the `-e SANDBOX_NAME` container env (nothing in the e2e-sandbox image reads it). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
d598e672f2 |
fix(reloader): add startupProbe to survive slow first start under load (#3470)
## What this PR does Adds a startup probe to the reloader deployment, rendered from package values through a patch on the vendored chart. Reloader went into CrashLoopBackOff on a loaded install. Events show `Liveness probe failed: ... connect: connection refused`, repeating on the liveness cadence until the kubelet gave up. A refusal rather than a timeout closes two questions at once. It is an RST from the target netns, so the packet arrived and the pod network is fine. It also rules out the mode in [stakater/Reloader#252](https://github.com/stakater/Reloader/issues/252), where the listener is already up and `/metrics` just answers slowly; that one produces probe timeouts. Nothing was listening on `:9090` yet. `http.ListenAndServe` is the last statement of `startReloader`, so the whole prologue runs before the socket opens. The slow part is not the informers, those are lazy and do their List in a goroutine. It is `IsOpenshift = isOpenshift()` in `pkg/kube`: a package-level var, so it runs at init before `main()`, and it does a `GET /apis/project.openshift.io` on a `context.TODO()` carrying no deadline. The latency is the API server's, and the container waits it out under a 100m CPU limit. Without a startup probe the only grace period is the liveness budget itself, `initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5`. A start that overruns it gets killed mid-prologue, and the restart re-runs the prologue from the top against an API server that is no less busy, so the loop feeds itself. This reaches past reloader. `packages/core/platform/sources/linstor.yaml` lists `cozystack.reloader` in the LINSTOR stack's `dependsOn`, and `dependsOn` fans out to every component of that stack, so a reloader stuck in this loop blocks that whole install. The probe is `/live` at `periodSeconds 5` and `failureThreshold 45`, about 225s, well inside the 10m install timeout that cozystack-operator stamps on generated HelmReleases. Liveness is suspended while it runs and gets its tight budget back as soon as the listener binds. `timeoutSeconds` is 5 to match the other two probes instead of the kubelet default of 1s, since a throttled process at 100m can hold the socket open and still need more than a second to answer. Both liveness knobs the chart already exposes would work without any patch. Widening `failureThreshold` buys the grace permanently and loosens hang detection for the life of the container. `initialDelaySeconds` is start-up-only and would genuinely do, but it is a fixed blind window on every start, including one that binds in three seconds, and nothing restarts a container that wedges inside it. Adaptive against fixed is the trade. The win is smaller than it sounds while `reloader.enableHA` is unset, since `/live` is a constant 200 then and liveness only catches the listener dying outright. A chart bump does not fix this. 2.2.14 still renders no `startupProbe` anywhere, and v1.4.19 added a second eager package-level call next to `IsOpenshift`, so the prologue got longer instead of shorter. `packages/system/velero` already carries a startupProbe patch on its vendored chart for the same reason. `make update` now drives the shared `%-update` rule with an exact `CHART_VERSION`. It has to be exact. With a stale local repo cache `helm pull --version 2.0.0` fails with version-not-found, while `--version "^2.0.0"` quietly resolves to 2.2.14, whose `deployment.yaml` has the ports block at line 179 instead of 161, so the hunk would fuzz or land somewhere unintended. The patch runs with `--forward` because `patch(1)` defaults diverge between implementations over a tree that already carries the change. GNU 2.7.6 refuses and exits 1. Apple 2.0-12u11 takes its `Assume -R? [y]` default, strips the hunk back out and exits 0. GNU under `--batch` does the same, which is the trap, since `--batch` is what you reach for to make a script non-interactive. `--forward` refuses in all three and leaves a `.rej`. Two things keep the probe from disappearing quietly, and they fire at different times. `--forward` catches it at the moment someone re-runs the patch by hand. The test only catches it at the next run of the suite, late enough for a commit to travel in between. `make test` also checks that the vendored `Chart.yaml` still records the pinned version, so bumping `CHART_VERSION` without re-running `make update` fails instead of leaving the pin and the tree disagreeing. Checked by doing the violation and watching it go red. ### Screenshots Not applicable, no UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. All five files sit inside one `packages/system` package. No package is added, renamed or removed, nothing under `hack/` is touched, and no CRD, namespace, `packages/core` values key, version enum or release asset changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(reloader): add a startup probe so a slow first start is not killed by the liveness probe and left in CrashLoopBackOff, which also unblocks the LINSTOR stack that depends on reloader ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a configurable startup health probe for the Reloader service. * The probe allows additional time for the service to start before health checks begin. * Documented the probe’s timing and behavior in the chart configuration. * **Bug Fixes** * Preserved existing liveness and readiness probe behavior. * **Tests** * Added coverage for enabled, disabled, and unaffected health probes. * Added validation to ensure the chart version matches the pinned release. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
9fe6ae2ec1 |
fix(keycloak-configure): patch HelmRelease in release namespace on teardown (#3372)
## What this PR does
The `keycloak-configure` pre-delete teardown Job clears the Flux
`HelmRelease` finalizers as its final step so the release can be
uninstalled. That final `kubectl patch` targeted a hardcoded namespace
that does not match where the release actually installs. The release
installs into `cozy-keycloak`, and the teardown Job's own RBAC — a
`Role` + `RoleBinding` granting `patch` on the `HelmRelease` by release
name — is created in the release namespace, but the patch was aimed at
`cozy-system`.
The result is a deadlock: the Job's `ServiceAccount` is `Forbidden` to
patch the `HelmRelease` in the mismatched namespace, so the Job errors
and retries forever, the `HelmRelease` sticks in `Terminating`, and the
Helm release wedges in status `uninstalling`. This blocks any teardown
or reinstall of the component.
The fix templates both the release name and the namespace from `{{
.Release.Name }}` / `{{ .Release.Namespace }}`, so the patch targets the
namespace where the `HelmRelease` lives and where the teardown RBAC
grants access. I also added a helm-unittest that pins the rendered
target, guards against the hardcoded namespace regressing, and asserts
the patch tracks the release name and namespace.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(keycloak-configure): fix the pre-delete teardown Job so uninstalling the release no longer wedges — it now patches the HelmRelease in the release namespace instead of a hardcoded one
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved Helm chart deletion cleanup to patch the correct HelmRelease
using the chart’s release name and namespace instead of fixed values.
- Reduced risk of teardown targeting resources in an unintended
namespace.
- **Tests**
- Added a new delete/teardown YAML test suite to verify correct
parameterization and that only the intended namespace is modified.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
07cefee158 |
fix(kubernetes): raise kubevirt-csi API client rate limits (#3428)
## What this PR does The kubevirt-csi-driver builds its infra and tenant cluster clients from a `rest.Config` with QPS/Burst left unset, so client-go uses its default 5 QPS / 10 burst. The wrapper's NFS `ControllerPublishVolume` polls the infra PVC once per second for up to two minutes. When a tenant scales out and several volumes attach at once, that shared token bucket starves and the poll returns `client rate limiter Wait returned an error: context deadline exceeded`, which the tenant kubelet reports as `FailedAttachVolume`. It looks like an intermittent E2E failure attaching a tenant volume, with nothing wrong in the storage backend. This sets QPS/Burst to 100/200 on both configs, before any client is built. The new flags `--kube-api-qps` and `--kube-api-burst` make the limits tunable. The compiled-in defaults are the real fix: the deployment passes no client flags, so both configs resolve to the in-cluster config and pick up the new defaults. Non-positive values are rejected, since client-go would read `0` as the starving default and a negative value as no rate limiting. Includes a regression test: a burst of 200 waiters times out against the default 5 QPS / 10 burst limiter and passes against the configured one. ### Screenshots None. No UI changes. ### Downstream repositories The diff touches only the driver's Go source in `packages/apps/kubernetes/images/kubevirt-csi-driver/`. No package was added, renamed or removed, and no `values.schema.json`, chart values, `ApplicationDefinition`, node prerequisite, metric or label changed. Nothing in the downstream trigger map is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): raise the kubevirt-csi-driver API client rate limits (QPS/Burst, tunable via --kube-api-qps and --kube-api-burst) above the client-go defaults so a burst of concurrent volume attaches no longer times out as FailedAttachVolume. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable Kubernetes API request rate limits for improved handling of high-volume operations. * Added validation to prevent invalid or non-positive rate-limit settings. * **Bug Fixes** * Improved reliability during bursts of concurrent volume attachment requests by reducing request throttling and timeout risk. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
b22b7b2431 |
test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race (#3430)
## What this PR does The split-horizon-routing-proof step ran two external-dns probes as bare Pods and polled them by fixed name. When the Cilium host-scope IPAM plus Gateway API restore race (cilium/cilium#46799) hands a probe the per-node reserved:ingress IP, the sandbox is rejected with "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting a controller to reschedule it onto a free IP. A bare Pod has no controller, so that delete was terminal: the probe vanished and the step failed on a NotFound pod. It surfaces as an intermittent kuberture E2E failure with a `FailedCreatePodSandBox` event on a probe pod. This runs the probes as Jobs. The Job re-creates its pod on a fresh IP after the healer deletes it, so the proof survives the race. Each Job is asserted to reach Complete instead of polling a fixed pod name, and its logs are read by the job-name label. The failure path also gains a catch podLogs selector for the probe pods, so their stdout is captured when a probe fails for a different reason. ### Screenshots None. No UI changes. ### Downstream repositories The diff touches only the kuberture E2E suite under `hack/e2e-chainsaw/kuberture/`. No package, values schema, chart, CRD or platform manifest changed. Nothing in the downstream trigger map is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved end-to-end test reliability by hardening the split-horizon routing proof against probe workload churn. * Strengthened route isolation validation between public and internal probes. * Enhanced failure diagnostics by capturing logs from the completed probe Jobs. * **Testing** * Updated DNS EDNS probes to run as Kubernetes Jobs (instead of bare Pods) to support retries and rescheduling. * Added explicit assertions that both public and internal probe Jobs reach **Complete** before running hostname/route isolation checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
2c453778f1 |
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors (#3436)
## What this PR does Fixes #3431. `hack/seaweedfs-naming-audit.sh` was fail-open: any kubectl failure produced an empty table indistinguishable from an honestly clean fleet — a failed namespace LIST meant zero namespaces walked, failed secret/PVC/STS LISTs meant zero findings, all silenced by `2>/dev/null` with no error handling anywhere in the script. The runbook uses this output as the gate before PVC deletion, as post-deletion verification, and as an upgrade precondition, so a transient API error could green-light destroying data. Every kubectl call now routes through a `run_kubectl` helper: a non-zero exit prints a FATAL line naming the failed query and the code propagates up the whole chain (enumerations restructured to capture-then-check, since an `exit` inside `$(...)` dies with the subshell). By-name GETs separate existence from extraction: a `--ignore-not-found -o name` probe keeps legitimate absence (Helm-pruned revision, unbound PVC) a clean skip, while everything else fails loudly — including a release Secret that exists but has no decodable payload or decodes without a chart name, which previously dropped the whole tenant from the audit. Incomplete PV-age evidence now marks the generation incomplete and falls to the safe "direction cannot be established" branch, so a failed GET can no longer flip an OVERLAP verdict into a wrong Step-3 deletion candidate. A successful audit prints the same bytes as before. Tests: 14 new cozytest cases — failure injection for every LIST and by-name GET site, corrupt-payload variants (missing `.data.release`, undecodable base64/gzip, chartless `{}`), and full-output golden diffs proving the success paths stay byte-identical to the pre-change script. 25/25 green via `hack/cozytest.sh`. ### Release note ```release-note fix(seaweedfs): the naming-audit script fails loudly when a kubectl query or a Helm release payload read fails, instead of printing an empty table indistinguishable from a clean fleet ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Audits now fail clearly (non-zero exit + fatal messaging) when Kubernetes queries or Helm release data is unavailable or corrupt, instead of producing incomplete results. * Missing release revisions are skipped safely; missing/unreadable PV age evidence no longer leads to incorrect deletion candidates, and MIXED now reports conservative “direction cannot be established” when evidence is incomplete. * Non-SeaweedFS releases remain silently ignored. * **Tests** * Added end-to-end acceptance coverage for fail-closed behavior and byte-for-byte stable output across key naming scenarios. * **Documentation** * Updated the runbook to require relying on the audit script exit code before proceeding, with a dedicated MIXED re-check step. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
49222768ec |
fix(release): make promote-retag digest verification media-type-agnostic (#3435)
## What this PR does
The finalize post-copy verification runs `skopeo inspect --format
'{{.Digest}}'` on every freshly retagged stable ref. For OCI artifacts
that command prints nothing and exits non-zero — `cozystack-packages` is
a flux artifact with `config.mediaType:
application/vnd.cncf.flux.config.v1+json` — so the v1.6.0 finalize (run
29932878820) aborted right after both copies of that artifact had
already succeeded, leaving 32 of 43 repos without stable tags and
skipping the cozy-installer publish. `--dry-run` never reaches this
branch, which is why rehearsals did not catch it.
This PR computes the digest as sha256 over the raw manifest instead
(`skopeo inspect --raw`), which is how registries define the manifest
digest and works identically for container images and OCI artifacts. The
helper gates the hash on skopeo's exit status via a temp file rather
than a pipeline, so a missing tag still resolves to an empty string (the
pre-check reads that as "tag absent, proceed to copy") and never to the
empty-input hash `sha256:e3b0c442…`, and never aborts under `set -eu`.
Verified against live GHCR: the raw-manifest hash of
`cozystack-packages:v1.6.0` matches the digest finalize expected
(`bf68208730860fa8…`), and the old and new methods agree byte-for-byte
on regular multi-arch images (`cozystack-operator`, `kubeovn`,
`cozystack-controller`, `kubevirt-cloud-provider`). New cozytest cases
cover the OCI-artifact digest resolution, a post-copy mismatch, and the
missing-tag path; the mock skopeo asserts `--raw`, so the suite fails
against the old code.
### Release note
```release-note
fix(release): promote's post-copy digest verification no longer breaks on OCI artifacts (cozystack-packages), which aborted the v1.6.0 finalize mid-retag with 32 of 43 repos left untagged
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved OCI artifact retagging by deriving stable manifest digests
from raw manifests to ensure accurate “already correct” and mismatch
detection.
* Prevented incorrect skips when the stable tag points to an unexpected
digest.
* Added safer handling for missing or unknown stable tags and clearer
failure behavior for unexpected empty manifest responses and transient
registry errors.
* **Tests**
* Expanded promotion/retag test coverage for digest-based resolution,
post-copy verification mismatches, missing-tag behavior, and transient
registry failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
bea9a2b829 |
fix(harbor): guarantee CPU for redis sentinels (#3446)
## What this PR does The harbor e2e suite intermittently times out on the `harbor-test` HelmRelease because the bundled Redis Sentinel set does not converge in time. Sentinels are NotReady by design until the redis-operator elects a master and rewrites their monitored-master config. Normal convergence is 2-3 minutes; on a CPU-contended e2e runner the tail stretches past the old 6m assert. Three changes: - `packages/system/harbor`: the inline RedisFailover ran its three sentinels at 50m CPU request / 100m limit. Now they run at Guaranteed QoS (requests == limits, 250m CPU / 128Mi), so master election gets scheduling shares and is not throttled under node contention. A helm unit test pins the sizing. This reserves 750m CPU / 384Mi per Harbor instance for the sentinel trio, up from 150m / 192Mi requested, paid by every install. It also diverges from the standalone redis app on purpose: its preset-based sentinels are Burstable with a 25m CPU request, and matching that would lower the request and make contention worse. - harbor e2e suite: on failure the catch now collects the redis-operator log, a describe of the RedisFailover, and the rfr-/rfs- pod states and logs. None of these were collected before, so a non-converging sentinel set was undiagnosable from the artifacts. The `harbor-test` HelmRelease wait grows from 6m to 10m, same as the `harbor-test-system` budget in the same suite and the documented longer-wait allowance for harbor. - docs and hack: three copies of a wrong claim that the pinned Chainsaw podLogs collector "exposes only container/tail/selector" are corrected. It accepts name/namespace/selector/container/tail/timeout plus the cluster fields; only `previous` is missing. The wrong list steered catch authors toward imperative scripts for captures podLogs handles declaratively. Caveat: the failing run carried no CPU metrics, so the resource fix is mechanism-backed (CFS shares cycles proportionally to requests), not metric-proven. If the flake recurs, the new diagnostics show which reconcile gate stalled. Closes #3445 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Trigger-map walk against the diff: `packages/system/harbor/` (template and tests) is not in the website app lists (those cover `packages/apps/` and `packages/extra/`) and touches no `values.schema.json`, no values keys, no ApplicationDefinition. `hack/e2e-chainsaw/` and `hack/e2e-capture-previous-logs.sh` appear in no trigger. `docs/agents/e2e-testing.md` is repo-internal agent documentation. Node prerequisites, `hack/package.mk`, and namespaces are untouched. ### Release note ```release-note fix(harbor): the bundled Redis Sentinel pods now run with guaranteed resources (250m CPU / 128Mi, requests == limits) so the HA set converges reliably on CPU-contended nodes ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased Redis Sentinel CPU and memory requests/limits to improve reliability during node contention. * Extended Harbor readiness timing for RedisFailover to reduce false failures during initialization. * **Diagnostics** * Improved Harbor e2e failure output by adding RedisFailover-specific details and collecting relevant Redis operator/pod logs, plus best-effort runtime log capture during readiness stalls. * **Tests** * Added a Redis Sentinel sizing test suite to verify the updated request/limit values. * **Documentation** * Clarified guidance on capturing logs for previous container instances in e2e runs, including what gets captured and when. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7f67d02fb7 |
fix(reloader): add startupProbe to survive slow first start under load
Reloader binds its :9090 listener as the last statement of startReloader, so /live and /metrics answer connection refused until everything before it has finished. The blocking part of that prologue is the package-level IsOpenshift = isOpenshift() in pkg/kube, which issues a GET /apis/project.openshift.io against the API server on a context carrying no deadline, before main() is even entered. That latency belongs to the API server, and the container waits it out at a 100m CPU limit. The chart renders no startupProbe, so the only grace period is the liveness budget: initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5, about 60 seconds. A first start that needs longer is killed mid-prologue, and the restart does not resume it, it runs it again from the beginning, so the loop sustains itself. The blast radius reaches past reloader: the LINSTOR stack lists cozystack.reloader in dependsOn, which fans out to every component of that stack, so a reloader stuck in this loop blocks the whole install. Render a values-driven startupProbe on /live so liveness only begins evaluating once the listener is up. Budget = periodSeconds 5 times failureThreshold 45, so about 225 seconds. timeoutSeconds matches the 5s the chart already gives liveness rather than the kubelet default of 1s, which a throttled process can exceed while holding the socket open. Both liveness knobs the chart already exposes could buy the same grace with no patch to carry. Widening failureThreshold buys it permanently, loosening hang detection for the life of the container. initialDelaySeconds is start-up-only and would genuinely do, but it is a fixed blind window on every start, whereas a startup probe returns liveness to its tight budget the moment the listener binds. The chart is vendored, so the template change is carried as patches/deployment-startup-probe.diff and reapplied by `make update`, which now drives the shared %-update rule with an exact chart version pin instead of pulling whatever is latest. The patch is applied with --forward because patch(1) defaults diverge by implementation over a tree that already carries the change: GNU 2.7.6 refuses and exits 1, Apple 2.0-12u11 takes its "Assume -R? [y]" default and strips the hunk back out with exit 0, and GNU under --batch does the same. --forward refuses in all three. A helm unittest pins the probe, its budget, the no-op path the template promises for an unset value, and the liveness numbers the budget is sized against, so the documented arithmetic cannot drift away from the chart without a red test. The same target asserts that the vendored Chart.yaml still records the pinned version, so raising the pin without re-running `make update` fails instead of leaving the tree and the pin quietly disagreeing. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
1925a6b7d5 |
docs(governance): fill in the reviewers' names and affiliations
Both rows carried placeholders: @androndo was marked Independent on a guess, and @scooby87 had no name at all because neither is published on their GitHub profile. Both work at Ænix. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
fd568dee87 |
fix(keycloak-configure): patch HelmRelease in release namespace on teardown
The pre-delete teardown Job clears the HelmRelease finalizers as its final step, but targeted a hardcoded namespace that does not match the namespace where the release and its RBAC live. The Job's Role and RoleBinding are created in the release namespace and grant patch on the HelmRelease by release name, so the ServiceAccount was Forbidden to patch the HelmRelease in the hardcoded namespace. The Job errored and retried forever, the HelmRelease stuck in Terminating, and the Helm release wedged in "uninstalling", blocking any teardown or reinstall. Target the release name and namespace so the patch matches the RBAC that grants it. Add a helm-unittest pinning the rendered namespace. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2dfe380ac3 |
fix(kubernetes): raise kubevirt-csi API client rate limits
The kubevirt-csi controller built its infra- and tenant-cluster Kubernetes clients on a rest.Config that left QPS/Burst unset, so client-go applied its 5 QPS / 10 burst default. ControllerPublishVolume polls the infra PVC once per second for up to two minutes; when a tenant scales out and several volumes attach at once, the default limiter starves and the poll fails with "client rate limiter Wait returned an error: context deadline exceeded", which the tenant kubelet surfaces as FailedAttachVolume. Set QPS/Burst to 100/200, overridable via --kube-api-qps and --kube-api-burst, on both the infra and tenant configs so a concurrent-attach burst is admitted promptly. Reject non-positive flag values, which client-go would otherwise turn into the starving default (0) or unlimited (<0). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
436a19df1b |
test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race
The split-horizon-routing-proof step ran two external-dns probes as bare Pods and polled them by fixed name. When the Cilium host-scope IPAM + Gateway API restore race (cilium/cilium#46799) hands a probe the per-node reserved:ingress IP, the sandbox is rejected with "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting a controller to reschedule it. A bare Pod has no controller, so the delete was terminal: the probe vanished and the step failed polling a NotFound pod. Run the probes as Jobs so the pod is re-created on a fresh IP after the healer's delete, assert each Job reaches Complete instead of polling a fixed Pod name, and read the logs by job-name label. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |