mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-13 08:05:19 +09:00
f48a94de6d53d23801f4a27b016090e03ff24632
1559
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov DaniilandClaude Opus 4.8
|
056ff92e16 |
test(hack): cover OCI_EXPORT_DIR export mode in common-envs (#3257)
Add hack/common-envs_test.bats (runs under make bats-unit-tests). For a representative slice of the root build: matrix — including the two recipes that previously slipped through, capi-providers-cpprovider and core/talos — it asserts that under OCI_EXPORT_DIR each built image emits exactly one --output type=oci and no ungated docker:// push, and that capi emits a single --tag. This would have caught the talos-push and capi-two-tag regressions. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
76c8f890fa |
fix(ci): export OCI archives for buildx calls that bypass image-tags (#3257)
Fork PRs build images to per-image OCI archives instead of pushing, gated on OCI_EXPORT_DIR. The `--output type=oci` was appended only inside the `image-tags` macro, so `packages/system/capi-providers-cpprovider`, whose `image:` recipe uses a hardcoded `--tag` and never calls `image-tags`, ran buildx with no exporter under export mode (PUSH/LOAD forced to 0). No archive was written, and the fork build's `if-no-files-found: error` upload then failed the whole build. Its `--cache-to type=inline` also needs an image exporter. - hack/common-envs.mk: extract the OCI-output snippet from `image-tags` into a reusable `oci-output <image-name>` macro so any buildx call can opt in; `image-tags` now ends with `$(call oci-output,$(1))` (no-op for the ~33 conforming packages). Move `comma :=` above its first user. - packages/system/capi-providers-cpprovider/Makefile: add `$(call oci-output,cluster-api-control-plane-provider-kamaji)` (archive base name equals the OCIR repo the privileged push keys on) and drop the inline cache under export (`--cache-to type=inline` gated off). - .github/workflows/pull-requests.yaml: wrap the `E2E Tests` check-run creation in try/catch. On error, `core.setFailed` so the required merge gate fails loudly and fail-closed instead of silently succeeding. Repo-wide audit of every `docker buildx build` reachable from the root `build:` matrix confirms capi was the only recipe bypassing both `image-tags` and `oci-output`. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
529efa7e01 |
build(hack): add OCI_EXPORT_DIR archive-export build mode
When OCI_EXPORT_DIR is set, every image builds to a per-image OCI archive (<dir>/<name>.oci.tar) instead of being pushed, and PUSH/LOAD are forced off. This lets a credential-less build (a fork PR's pull_request run) produce the images as artifacts that a privileged workflow later pushes to the registry by digest. The image digest is captured via --metadata-file regardless of output type, so the values.yaml/.tag refs baked during the build match exactly what the privileged run pushes. The output flag is threaded through the image-tags macro, which every package's buildx call already uses, so no per-package Makefile change is needed. Refs: #3257 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
2330d6f3ca |
fix(ci): overlay images from the PR base branch, and publish per-line artifacts
Supersedes the interim guard in this branch`s first commit, which skipped the overlay entirely for release-line PRs. Skipping fixed the wrong images but left those PRs testing their line`s last release: for any package the PR did not rebuild, the committed ref is the released digest, so a component changed by an earlier backport was exercised as its pre-backport binary until the next rc. The overlay now reads `cozystack-packages:<base branch>` instead of always `:main`, and build-release.yaml publishes that artifact for every maintained `release-<major>.<minor>` branch the way build-main.yaml does for main: images tagged with the branch, and the whole packages tree pushed with each reference digest-pinned to what the run just built. Each base branch therefore has its own generation to overlay from, which is what the original bug was really about — #3437 failed install because main`s cozystack-controller served an aggregated OpenAPI release-1.6`s charts could not validate against. Three deliberate choices: * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The per-release and rc staging branches promote-rc.yaml and tags.yaml create (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their images come from the tag build, and rebuilding them would be waste. * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per image and build-main.yaml is deliberately its only, serialized writer so concurrent builds cannot race on the cache manifest (the 409 class #2711 fixed for image tags). A line build can overlap a main build, so writing here would reintroduce that race. Line builds read the cache. * A missing artifact still degrades to committed refs, but on a release line it now emits a ::warning:: naming the branch. Silent degradation is indistinguishable from a working overlay, which is how a mis-specified branch filter would hide for a whole release cycle. Cost: one `make build` per push to a maintained line, i.e. per merged backport. hack/overlay-main-images_test.bats pins the artifact tag to the base branch, rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main, setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test. 13/13 green; actionlint and zizmor clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
cc44f6c981 |
ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449)
## What this PR does Moves release validation from promote time to rc time, making promotion itself a pure retag operation. This is the follow-up to the v1.6.0 retrospective: the promote flow's checks ran on the promote PR (coupled to promotion, skippable by never promoting), the changelog and website docs were produced only after the release went public, and the finalize tag push silently failed to trigger `tags.yaml` at all (persisted-credential bug, root-caused below). The release lifecycle after this PR: 1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes the staging branch — and now runs a **mandatory full e2e** for every `-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and any published tag) get the same validation as a **manual button** (`workflow_dispatch`). 2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**: the release changelog is generated from the rc tag (reusable `changelog-generate.yaml`) and committed to the rc staging branch, ready for review long before promotion. 3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence: the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages the release, picks up the rc-time changelog (or generates one), and **opens the website docs PR** on cozystack/website from the staging branch — parked with an explicit DO-NOT-MERGE-until-finalize contract. 4. The promote PR **no longer gets `full-e2e` automatically** — its tree is the rc's already-validated digests with only tag strings rewritten; a maintainer can still add the label manually to force the suite. 5. Finalize performs the irreversible side effects (tag, publish, retag, chart) — and with `persist-credentials: false` its tag push now actually authenticates as the CI app, so the `tags.yaml` backstops (`generate-changelog`, `update-website-docs`) fire for promoted tags. This is the one-line fix for the v1.6.0 incident where the tag triggered nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader` silently defeats the app token injected via `git remote set-url`, and `GITHUB_TOKEN` pushes create no workflow runs. Everything is pinned by executable contract tests (`hack/promote-gate-contract.bats`, extended `hack/release-changelog-contract.bats` / `release-changelog-behaviour.bats` — 48 tests total, including behaviour tests that execute the new `hack/parse-rc-tag.sh` and `hack/select-changelog-source.sh` for real), and `docs/release.md` is rewritten to describe the new lifecycle end-to-end. ### Commit map (review commit-by-commit) - `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing changelog-before-publication rework (changelog generated at promote, committed to the staging branch before the PR opens, finalize publishes it as the release body; tags.yaml job becomes a verbatim-port backstop), rebased onto main. - `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane (prerelease → staging branch, stable → tag; draft releases fail fast; dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts). - `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut rc tags (also on `release_exists` re-runs); fixes the `cut-prerelease.yaml` comment that falsely claimed e2e already existed. - `2dfbeea77` — `changelog-generate.yaml` (reusable core) + `changelog-rc.yaml` (rc-time button committing to the staging branch); promote-rc gains a top-level `parse` job with a strict no-whitespace tag policy (`hack/parse-rc-tag.sh`); source selection is validate-then-fallthrough (`hack/select-changelog-source.sh`) so an invalid higher-priority changelog can never silently win. - `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override + removal of the automatic `full-e2e` label (pull-requests.yaml release-path e2e now requires an explicit label; a post-open label now actually starts the run). - `936190d4b` — `website-docs` job at promote time behind a FETCH_REF-support guard (see dependency below), with the DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before `release-next`; `tags.yaml::update-website-docs` stays as the backstop. - `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0 root cause). - `717c8860f` — `docs/release.md` coherence pass over the whole new lifecycle. ### Dependency The promote-time website docs job needs cozystack/website#631 (`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard detects the missing support and degrades to a ⚠️ line in the promote PR body — promotion is unaffected and the tag-time backstop backfills, so this PR is safe to merge first. ### Merge instructions Please **merge-commit, do not squash**: the 13 commits are deliberately atomic (the credential root-cause fix is separable from the feature work) and squashing would collapse ~3k lines into one commit, hurting bisectability. ### First live run The e2e lane is verified by lint, contract tests and dry-traces, but has not yet executed against a real tag; the intended first live run is a manual `e2e-tag.yaml` dispatch against an existing published rc tag before the gate is relied on for a real promotion. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added reusable changelog generation with validation, “present” signaling, and safe artifact upload behavior (including RC-time support). * Added an E2E release-tag workflow with tag parsing, deterministic sandboxing, and comprehensive report collection. * Improved RC promotion with a stricter parse front-door, verified E2E green gate (optional bypass), and earlier parallel changelog/asset staging. * **Documentation** * Refreshed release-process docs to match the updated RC-to-stable, changelog, and website-docs timing rules. * **Bug Fixes** * Improved PR finalization to source release bodies from verified changelogs and warn/fallback on missing or empty content. * Tightened label gating so E2E runs only under the intended conditions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
847c83a2f6 |
fix(e2e): boot CI QEMU guests with cache=unsafe to keep etcd unstalled
The ephemeral CI VM disks ran with QEMU's default cache=writeback, so every guest fsync propagated to the host-backed raw files. During the VM-heavy suites the management-cluster etcd fsync on each Raft commit serialized behind the concurrent image-unpack and CDI import I/O, the apiserver returned context-deadline-exceeded, and operators lost their leader-election leases together, failing a different suite each run. The VMs are recreated per run, so the traded-away durability is unused. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Andrei Kvapil
|
6cb579338b |
fix(nightly): rewrite an image host that is the whole scalar value
The nightly host rewrite is a literal "<src-registry>/" substring replace, so it only reaches a host sitting contiguously in front of the repository. kubeovn keeps its host in global.registry.address with the bare repository in a sibling key: no trailing slash, no match. The image was mirrored to the public registry and the tree kept pointing at the private build registry. The gap was dormant while kubeovn's image came from a separate repository, because the baked ref carried the public host and the ownership filter skipped it. Building the image in-tree stamps the build-registry host into that key and turns it live, so fix it here. Add an end-of-line-anchored expression that fires only when the source registry is the complete value; a contiguous ref continues past the host, so the two expressions cannot both fire on one reference. keycloak-operator splits at a different boundary and stays unfixed -- neither of its keys holds the source registry as a whole. Documented, and dormant while no CI path rebuilds it. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8b461f0c0b |
feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins (#3378)
## What this PR does Cozystack builds and publishes `ghcr.io/cozystack/cozystack/grafana` — a rebuilt Grafana image — and uses it by default in the monitoring stack. Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a redistributor of an AGPL Grafana build. This is a governance concern for CNCF Incubation (there is no precedent for a CNCF project redistributing an AGPL binary). The only reason the image was ever rebuilt was to bundle the VictoriaLogs datasource plugin, which was not in the official Grafana plugin catalog at the time. It is now published there as a signed plugin (`victoriametrics-logs-datasource`), so the rebuild is no longer necessary. This PR moves the deployment to the upstream `grafana/grafana` image (digest-pinned, routed through `cozy-lib.image` so mirrored-registry installs keep working) and installs the plugins at container startup via `GF_INSTALL_PLUGINS`, with versions pinned: - `victoriametrics-logs-datasource 0.29.0` — the VictoriaLogs datasource (signed; the reason the image was rebuilt) - `natel-discrete-panel 0.1.1`, `grafana-worldmap-panel 1.0.6`, `marcusolsson-dynamictext-panel 6.2.0` — reproduce the panel set the removed Dockerfile vendored **Air-gapped installs keep working.** The plugin zips are not downloaded from the internet at pod startup: they are mirrored from the official catalog into the existing `grafana-dashboards` image at build time and served by that in-cluster static file server, next to the dashboards it already serves. `GF_INSTALL_PLUGINS` points at `http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…`, so the runtime dependency stays inside the cluster and the artifact is mirrored into private registries together with every other first-party image. A new `allow-to-grafana-dashboards` CiliumNetworkPolicy (mirroring `allow-to-keycloak`) opens tenant egress to that server, since tenant egress is default-deny. As a result, `ghcr.io/cozystack/cozystack/grafana` is no longer built or published: the Dockerfile, the pinned image tag, the `image` targets in both monitoring Makefiles, and the build step in the root Makefile are removed. Notes: - The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored plugin zips are Apache-2.0 licensed and are served unmodified from the official catalog builds (signatures intact). - `allow_loading_unsigned_plugins` and the custom `/var/lib/grafana-plugins` path are dropped: the catalog datasource is signed, and plugins install into the writable default `GF_PATHS_PLUGINS` (`/var/lib/grafana/plugins`). - The VictoriaLogs datasource provisioning (the `GrafanaDatasource` CRs pointing at `vlselect`) is unchanged. - Plugins are re-installed on every pod start from the in-cluster server (no PVC involved, ~77 MB per start per replica over the cluster network); a failed download fails the pod loudly instead of silently dropping the datasource. - `natel-discrete-panel` and `grafana-worldmap-panel` are Angular plugins; Angular is disabled by default in Grafana 11, so they do not load — this is unchanged from the current rebuilt image (same Grafana 11.6.15, same config). They are kept for parity and can be dropped as a follow-up. ### How this was verified - `helm unittest` across the repo passes, including new regression tests: the monitoring chart asserts the exact upstream image reference (with and without `images-registry` set) and the exact `GF_INSTALL_PLUGINS` value; the tenant chart asserts the new egress policy. - Local end-to-end of the full chain in Docker: built the modified `grafana-dashboards` image, served it on a shared network, and started upstream `grafana/grafana:11.6.15` with the rendered `GF_INSTALL_PLUGINS` value. All four plugins download from the in-cluster-style URL (no internet egress from Grafana), and `GET /api/plugins` reports `victoriametrics-logs-datasource` v0.29.0 as `enabled: true` with `signature: valid`. ### Screenshots Not applicable — no UI change. Grafana renders the same datasources and panels; only the image source and plugin delivery mechanism change. ### Downstream repositories Walked the trigger map against the diff. This is an internal change to how the monitoring stack sources the Grafana image and its plugins (`packages/system/monitoring`, `packages/system/grafana-operator`, `packages/apps/tenant` network policy). No package is added, renamed or removed; no `values.schema.json`, `ApplicationDefinition`, platform values, release asset, shared `hack/` tooling or `package.mk` contract changes; the added `grafana.image` key is an internal system-chart value, not a user-facing app field. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Grafana in monitoring is now based on a digest-pinned upstream image for more consistent deployments. * Required Grafana plugins are installed automatically at startup via `GF_INSTALL_PLUGINS`; unsigned-plugin configuration was removed. * Tenant networking now includes a new egress rule specifically for Grafana dashboards. * Grafana dashboards image now includes required plugin assets prepared during image build. * **Tests** * Added/updated checks covering digest pinning, mirrored-registry behavior, plugin-installation behavior, and the new Grafana dashboards network policy. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
330acfe6e2 |
fix(e2e): widen tenant node-join deadline to 18m
The kubernetes-latest and kubernetes-previous suites flaked on the worker node-join wait under host load. In run 30260770694 both failed at exactly 12m with zero tenant Nodes registered and the tenant cilium HR still mid-install: a transient drbd.linbit.com/lost-quorum taint delayed the worker DataVolume imports, so the KubeVirt VMIs were only created ~2m into this wait and the guests then had ~10m to import Talos, boot, register a kubelet and let cilium turn the nodes Ready. This single deadline absorbs the entire worker bring-up (the machinedeployment replicas=2 gate clears before the VMIs even exist), so under storage pressure 12m is too tight. A less-loaded fleet run passed both suites unchanged, confirming load-induced slowness rather than a stuck bring-up. Widen the deadline to 18m, which restores margin and still sits well inside the 40m chainsaw step timeout. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
ad35afa534 |
fix(e2e): disable guest fsync on ephemeral CI VM disks
The e2e QEMU guests boot with three raw virtio -drive lines that carry no cache= option, so QEMU defaults to cache=writeback and honors every guest fsync. The management-cluster Talos etcd fsyncs on each Raft commit, while containerd unpacks dozens of images and CDI imports multi-GB VM disks onto the same host-backed raw files during the VM-heavy suites. The fsync storm serializes etcd's writes into multi-second stalls, the mgmt apiserver times out lease PUTs, and many operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji) lose leadership at once — failing whichever chainsaw suite is mid-flight, a different one each run. Add cache=unsafe to each guest drive so guest flushes become no-ops against the host page cache, removing the fsync-gated etcd stalls. This is safe only because these CI VMs are ephemeral and destroyed every run; never use it for real data. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
b22b7b2431 |
test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race (#3430)
## What this PR does The split-horizon-routing-proof step ran two external-dns probes as bare Pods and polled them by fixed name. When the Cilium host-scope IPAM plus Gateway API restore race (cilium/cilium#46799) hands a probe the per-node reserved:ingress IP, the sandbox is rejected with "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting a controller to reschedule it onto a free IP. A bare Pod has no controller, so that delete was terminal: the probe vanished and the step failed on a NotFound pod. It surfaces as an intermittent kuberture E2E failure with a `FailedCreatePodSandBox` event on a probe pod. This runs the probes as Jobs. The Job re-creates its pod on a fresh IP after the healer deletes it, so the proof survives the race. Each Job is asserted to reach Complete instead of polling a fixed pod name, and its logs are read by the job-name label. The failure path also gains a catch podLogs selector for the probe pods, so their stdout is captured when a probe fails for a different reason. ### Screenshots None. No UI changes. ### Downstream repositories The diff touches only the kuberture E2E suite under `hack/e2e-chainsaw/kuberture/`. No package, values schema, chart, CRD or platform manifest changed. Nothing in the downstream trigger map is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved end-to-end test reliability by hardening the split-horizon routing proof against probe workload churn. * Strengthened route isolation validation between public and internal probes. * Enhanced failure diagnostics by capturing logs from the completed probe Jobs. * **Testing** * Updated DNS EDNS probes to run as Kubernetes Jobs (instead of bare Pods) to support retries and rescheduling. * Added explicit assertions that both public and internal probe Jobs reach **Complete** before running hostname/route isolation checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
2c453778f1 |
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors (#3436)
## What this PR does Fixes #3431. `hack/seaweedfs-naming-audit.sh` was fail-open: any kubectl failure produced an empty table indistinguishable from an honestly clean fleet — a failed namespace LIST meant zero namespaces walked, failed secret/PVC/STS LISTs meant zero findings, all silenced by `2>/dev/null` with no error handling anywhere in the script. The runbook uses this output as the gate before PVC deletion, as post-deletion verification, and as an upgrade precondition, so a transient API error could green-light destroying data. Every kubectl call now routes through a `run_kubectl` helper: a non-zero exit prints a FATAL line naming the failed query and the code propagates up the whole chain (enumerations restructured to capture-then-check, since an `exit` inside `$(...)` dies with the subshell). By-name GETs separate existence from extraction: a `--ignore-not-found -o name` probe keeps legitimate absence (Helm-pruned revision, unbound PVC) a clean skip, while everything else fails loudly — including a release Secret that exists but has no decodable payload or decodes without a chart name, which previously dropped the whole tenant from the audit. Incomplete PV-age evidence now marks the generation incomplete and falls to the safe "direction cannot be established" branch, so a failed GET can no longer flip an OVERLAP verdict into a wrong Step-3 deletion candidate. A successful audit prints the same bytes as before. Tests: 14 new cozytest cases — failure injection for every LIST and by-name GET site, corrupt-payload variants (missing `.data.release`, undecodable base64/gzip, chartless `{}`), and full-output golden diffs proving the success paths stay byte-identical to the pre-change script. 25/25 green via `hack/cozytest.sh`. ### Release note ```release-note fix(seaweedfs): the naming-audit script fails loudly when a kubectl query or a Helm release payload read fails, instead of printing an empty table indistinguishable from a clean fleet ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Audits now fail clearly (non-zero exit + fatal messaging) when Kubernetes queries or Helm release data is unavailable or corrupt, instead of producing incomplete results. * Missing release revisions are skipped safely; missing/unreadable PV age evidence no longer leads to incorrect deletion candidates, and MIXED now reports conservative “direction cannot be established” when evidence is incomplete. * Non-SeaweedFS releases remain silently ignored. * **Tests** * Added end-to-end acceptance coverage for fail-closed behavior and byte-for-byte stable output across key naming scenarios. * **Documentation** * Updated the runbook to require relying on the audit script exit code before proceeding, with a dedicated MIXED re-check step. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
49222768ec |
fix(release): make promote-retag digest verification media-type-agnostic (#3435)
## What this PR does
The finalize post-copy verification runs `skopeo inspect --format
'{{.Digest}}'` on every freshly retagged stable ref. For OCI artifacts
that command prints nothing and exits non-zero — `cozystack-packages` is
a flux artifact with `config.mediaType:
application/vnd.cncf.flux.config.v1+json` — so the v1.6.0 finalize (run
29932878820) aborted right after both copies of that artifact had
already succeeded, leaving 32 of 43 repos without stable tags and
skipping the cozy-installer publish. `--dry-run` never reaches this
branch, which is why rehearsals did not catch it.
This PR computes the digest as sha256 over the raw manifest instead
(`skopeo inspect --raw`), which is how registries define the manifest
digest and works identically for container images and OCI artifacts. The
helper gates the hash on skopeo's exit status via a temp file rather
than a pipeline, so a missing tag still resolves to an empty string (the
pre-check reads that as "tag absent, proceed to copy") and never to the
empty-input hash `sha256:e3b0c442…`, and never aborts under `set -eu`.
Verified against live GHCR: the raw-manifest hash of
`cozystack-packages:v1.6.0` matches the digest finalize expected
(`bf68208730860fa8…`), and the old and new methods agree byte-for-byte
on regular multi-arch images (`cozystack-operator`, `kubeovn`,
`cozystack-controller`, `kubevirt-cloud-provider`). New cozytest cases
cover the OCI-artifact digest resolution, a post-copy mismatch, and the
missing-tag path; the mock skopeo asserts `--raw`, so the suite fails
against the old code.
### Release note
```release-note
fix(release): promote's post-copy digest verification no longer breaks on OCI artifacts (cozystack-packages), which aborted the v1.6.0 finalize mid-retag with 32 of 43 repos left untagged
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved OCI artifact retagging by deriving stable manifest digests
from raw manifests to ensure accurate “already correct” and mismatch
detection.
* Prevented incorrect skips when the stable tag points to an unexpected
digest.
* Added safer handling for missing or unknown stable tags and clearer
failure behavior for unexpected empty manifest responses and transient
registry errors.
* **Tests**
* Expanded promotion/retag test coverage for digest-based resolution,
post-copy verification mismatches, missing-tag behavior, and transient
registry failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
436a19df1b |
test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race
The split-horizon-routing-proof step ran two external-dns probes as bare Pods and polled them by fixed name. When the Cilium host-scope IPAM + Gateway API restore race (cilium/cilium#46799) hands a probe the per-node reserved:ingress IP, the sandbox is rejected with "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting a controller to reschedule it. A bare Pod has no controller, so the delete was terminal: the probe vanished and the step failed polling a NotFound pod. Run the probes as Jobs so the pod is re-created on a fresh IP after the healer's delete, assert each Job reaches Complete instead of polling a fixed Pod name, and read the logs by job-name label. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
3733ebd736 |
fix(e2e): gate tenant patch on authenticated aggregated-API readiness
The Configure Tenant test issued its first kubectl patch of tenants/root as soon as the previous test saw the aggregated APIService report Available. That condition tracks endpoint availability, not the serving pod's auth stack: a freshly rolled cozystack-api pod can answer while it has not yet loaded the requestheader client-CA, so it drops the front-proxy identity and rejects the request as system:anonymous with a 403, failing the test 0.3s in on its very first command. Add an authenticated GET against the same resource, bounded to 120s, before the patch. It proves the aggregated API both serves and authenticates a request against the actual resource before the patch's own initial GET runs, closing the inter-test race without a blind retry; if the aggregation layer is genuinely wedged past two minutes the gate fails loud, surfacing a real regression. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8a44b3e408 |
fix(e2e): skip crust-gather host-log collection rejected by PodSecurity
The diagnostic snapshot captured on a failed e2e run spawns a privileged debug pod per node for its host-log leg. Baseline PodSecurity rejects every such pod with a 403, and crust-gather's retry loop keeps trying until it burns the entire --duration budget, then exits 1 with a deadline error. The snapshot verdict therefore printed FAILED on every failure capture even though the namespaced object state had been collected, and the host-log leg itself gathered nothing usable here. Pass --disable-additional-logs at all three collect sites: the host snapshot from the BATS runner, the host snapshot from the Chainsaw error.catch, and the nested tenant snapshot. The object collection now finishes cleanly, the happy path reports complete, and FAILED once again means a genuine collection failure. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a7ce4371c2 |
docs(agents): correct the chainsaw podLogs capability note
The e2e conventions, the previous-logs capture script, and the global chainsaw catch all claimed the v0.2.15 podLogs collector exposes only container/tail/selector. It also accepts name, namespace, timeout and the cluster fields; only the previous option is genuinely missing. The wrong list steers catch authors toward imperative scripts for cross-namespace log capture that podLogs handles declaratively. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
824a799c6a |
test(e2e): capture redis convergence diagnostics in harbor suite
When harbor's redis sentinels never converge, the suite times out on the HelmRelease assert with no evidence of why: the catch pod-log selector matches neither the redis-operator (it runs in another namespace) nor the rfr-/rfs- pods, and the RedisFailover CRD ships no status subresource to describe-assert on. Add a RedisFailover describe plus a best-effort script that dumps the operator log and the rfr-/rfs- pod states and logs, so the stalled reconcile gate is visible in the failure output. Also widen the harbor-test HR wait from 6m to 10m: readiness is gated on the operator electing a redis master, which normally takes 2-3m but runs long on a CPU-committed runner. This matches the harbor-test-system budget in the same suite and the justified longer waits allowed by the e2e conventions. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
fea40d6314 |
fix(ci): only overlay current-main images on main-based PRs
The PR finalize job pulls cozystack-packages:main and repoints every package the PR did not rebuild at current-main images. On a main-based PR that is the whole point: it stops e2e from testing last-release images for everything outside the build matrix. On a release-line PR it is wrong — the committed refs there are not stale, they are that line`s released digests, and the charts are written against exactly those. So a release-line PR was installing main`s binaries onto its own charts, and the mismatch grows with every commit main gains. #3437 is the demonstration: a one-line change on release-1.6 that deactivates an app failed install deterministically, twice, with SchemaError(...core/v1alpha1.Option.spec): unknown model in reference: "...core~1v1alpha1.OptionSpec" from main`s cozystack-controller serving an aggregated OpenAPI that branch`s charts cannot validate against. Nothing in the PR was broken; the lane was. Left alone this makes every 1.6 backport look red, which is when release-branch PRs are busiest. Both overlay steps are now gated on `github.base_ref == main`, so a release-line PR keeps its committed digests — the behaviour that predates the overlay. Retargeting the overlay at a per-line artifact would be better but is not possible today: build-main.yaml publishes only cozystack-packages:main, and the registry carries no release-* equivalent (verified against the packages repo`s tag list: `main` plus per-PR tags, nothing else). hack/overlay-main-images_test.bats pins the wiring per step, so adding a third overlay step without the guard fails the suite. Mutation-checked by removing one guard. 12/12 green; actionlint clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
a5a0449c9b |
test(hack): re-point the shape-2 image-ref exemplar at grafana-dashboards
The promote-retag and image-refs enumeration tests used the monitoring grafana.tag as their canonical example of a reference living outside a values.yaml. That file is gone now that the Grafana rebuild is dropped, so use the grafana-dashboards tag — the same storage shape, still in the tree — as the exemplar. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
IvanHunters
|
eeee87f0b1 |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Resolve the migrations pin conflict in packages/core/platform/values.yaml: keep targetVersion 55 (this branch owns migration 54) and take the newer platform-migrations image v1.6.0 from main. The in-tree image pin intentionally lags targetVersion until the release retag, and run-migrations.sh fails loud if a targeted migration is missing from the image, so the lag is safe by design. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Myasnikov Daniil
|
1eded27900 |
test(release): scope the trim pin to the parse step, fix two stale references
Three review nits, all verified against the tree, plus a tighter scope than the first attempt at them. The behaviour suite asserted "promote-rc does not trim the rc tag" against the whole file. promote-rc.yaml has no .trim() today, so it passed, but any unrelated .trim() added anywhere in the workflow would have failed a test about tag policy. Scoping it to the parse JOB was not enough either: that job is ~180 lines, most of them the e2e gate's github-script, whose string handling has nothing to do with tag policy. Both greps are now scoped to the parse STEP, and run through a code_lines filter copied from the contract suite so a YAML comment mentioning .trim() cannot fail them. Mutation-checked in both directions: a .trim() injected into the parse step fails the test, one injected into the e2e-gate step of the same job does not. - The Step 2 checklist entry still said "must be main" while Step 2 itself now lists three valid starting points (main, detached at the release tag, detached at the rc tag being promoted). The checklist is what an agent ticks off, so it was the half that mattered; it now points at Step 2 rather than re-encoding the count, so a fourth configuration cannot desync it. The heading said "Checking current branch", which two of the three states do not have. - hack/testdata/needs-semantics.yaml named release-changelog-contract.bats as its consumer; the only consumer is release-changelog-behaviour.bats. 50/50 tests green across the three suites (11 + 24 + 15). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
a2667c58ca |
fix(ci): grant rc-e2e the checks:write ceiling e2e-tag.yaml declares
A caller job's permissions are the ceiling for every job in the workflow it calls, and GitHub validates that ceiling when it CREATES the run — before the calling job's `if:` is evaluated, let alone before a step runs. e2e-tag.yaml's e2e job declares `checks: write` for its workflow_dispatch-only breakpoint, so granting only `contents: read` here made the run fail validation with "requesting 'checks: write', but is only allowed 'checks: none'". A startup failure takes the whole run with it, and an `if:` cannot gate it, so this was not limited to rc cuts: EVERY tag push — rc and stable, including the one finalize pushes to publish a release — would have produced no build, no draft release and no staging branch. The comment claimed the opposite: that because workflow_call can never reach the breakpoint branch, the grant was unnecessary. Runtime reachability has no bearing on a static check. e2e-tag.yaml now states the contract for its callers, next to the permission block that was being read as "contents: read is all this needs". hack/promote-gate-contract.bats pins both halves of the pair, and separately pins the `E2E <tag> (full suite)` job name the gate correlates evidence by. Neither is reachable from any PR lane — tags.yaml runs only on tag pushes — so a regression here would otherwise surface at the next release. Mutation-checked: dropping the grant fails the first pin, renaming the e2e job fails the second. changelog-rc.yaml's reusable-workflow call now pins `contents: read` explicitly, matching promote-rc.yaml's identical call. The outcome is the same either way, but the docs describe an omitted block as the repository default rather than the workflow's own top-level block, and this is the file class where that ambiguity just cost a release lane. Reported by CodeRabbit on #3449; blast radius and the missing pins found by an adversarial re-check. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ae39475789 |
fix(release): prove a stable tag is absent before copying over it
Review notes from #3435, and a correction to my first attempt at them. manifest_digest gated on skopeo`s exit status alone, so a zero exit with no bytes hashed the empty string to sha256:e3b0c442... — a digest that looks real and belongs to no manifest. The first fix simply required non-empty bytes, which was worse than it looked: the caller turns an empty result into "not published, safe to copy", so a registry answering 200 with an empty body went from an accidental refusal (the bogus digest mismatched, so the write-once check refused) to an actual copy over whatever the tag really held. Verified against real skopeo 1.23: a 200 with Content-Length 0 does exit 0 with no stdout. The helper is now explicitly three-state. A digest, or empty output for a tag the registry PROVED absent, or non-zero for anything indeterminate — which under set -e aborts the promotion before it writes. Absence has to be reported as "manifest unknown"; 429, 5xx, auth and network failures all exit non-zero with no bytes too, and reading those as "unpublished" is the same fail-open one level down. finalize retags ~42 refs with no retry wrapper, so a single rate-limit was enough to reach that path. skopeo`s stderr is captured instead of discarded, so the diagnostics now name the real failure. Tests: the write-once refusal branch the review found untested (an existing stable tag at a different digest must be refused before any write), the empty- body case (refuses, writes nothing, and the empty-input hash never appears in the diagnostics), and a 429 that must not read as an absent tag. The mock now emits realistic registry errors, since a bare non-zero exit is no longer proof of absence. Both new guards are mutation-checked: accepting any non-zero exit as absence fails the 429 test, dropping the empty-bytes check fails the empty-body test. 11/11 tests green, hack/image-pin-consistency.bats still 2/2, sh -n clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e02adad38f |
fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls
Review findings from #3436, plus what an adversarial re-check of the first attempt at them turned up. The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals "config" (the release`s values) after "chart", so a values subtree that spells chart.metadata.name shadowed the real chart name: the release then read as non-SeaweedFS, the tenant silently vanished from the report, and the script exited 0 -- the precise false clean this PR exists to prevent, reachable without any corruption at all. It now takes the FIRST match. The path stays adjacent on purpose: a looser "any name after metadata" matches chart.templates[].name, which Helm emits immediately after metadata on every healthy release, so the review`s suggested relaxation would have returned a template path for every tenant. Over-strictness fails loudly and recoverably; over-looseness deletes data quietly. Newlines are now folded before matching, so a pretty-printed payload parses at all rather than aborting, and whitespace around the punctuation is tolerated. first_deployed got the same treatment -- it had the identical greedy, single- line-only shape, and a spaced payload silently dropped the PV-vintage row from the report. The FATAL message names the path it read and the shape it expected, so a future Helm format change is diagnosable instead of looking like real corruption. Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy) each byte-compared against the same golden as the compact payload, so the shape must make no difference to the report. Mutation-checked -- restoring last-match extraction fails the decoy test on its own, restoring the single-line matcher fails the pretty test. The test fake answered any unmodelled kubectl invocation with exit 0 and empty stdout, the fail-open shape this script was rewritten to reject, and enough to let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97 naming the invocation. Verified inert first by instrumenting the fake: 88 invocations across the suite, none unmodelled. Also added the missing test for the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it. The runbook now tells the operator to read the exit code: a non-zero exit means the table is incomplete and no step may be taken on it. That contract lived only in the script header, while the runbook is what the operator follows. 29/29 tests green; sh -n clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ee838261b1 |
test(release): pin the promote gate with POSIX grep, not ripgrep
The rc-E2E gate contract used ripgrep for every pin. The unit-test runner has no ripgrep, so `code_lines()` hit "rg: not found", the `|| true` on it swallowed the 127, and every pin degraded to "0 matches" — the suite failed on the first assertion with no hint that the tool, not the workflow, was missing. No other hack/*.bats uses ripgrep. Swap all pins to POSIX grep and make the filter fail closed: grep exits 1 when nothing is selected (legitimate for an empty block) and 2 on an actual error, so only the latter propagates instead of silently yielding no lines. Verified 13/13 green both normally and with `rg`/`fd` shimmed out of PATH to reproduce the runner, plus the eleven suites the aborted CI run never reached (all green, zero shim hits). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Timofei Larkin
|
730d006ce8 |
fix(platform): skip pinning names no object can have
The pin path composed "kubernetes-<cluster>-<group>" and read it back, but a group key that cannot form a valid object name (a slash, a space, an underscore...) provably never had objects applied under it -- the old chart's apply would have been rejected by the apiserver. Worse, kubectl rejects a name containing '/' client-side, deterministically, so pin_keep's fail-closed read classified it as a real failure and the migration exited 1 on every retry: a permanent fleet-wide pre-upgrade deadlock over a pool that cannot exist. Lean on the invariant instead of the read: pin_pool_objects statically checks the composed name against the DNS-1123 subdomain shape and skips with a note when it cannot name an object -- nothing to pin, no kubectl call to misread. Keys that are label-invalid but still valid object names (e.g. dotted) may have live objects and keep today's pin behavior; the pin-path transient-error test now uses a dotted key so it still exercises a genuine read. The fake kubectl reproduces real kubectl's client-side slash rejection, and a new bats case pins the skip: a "pool/a" key must warn, pin nothing, apply nothing, and still reach the version stamp. Red against the previous script, which deadlocked at the first pin read. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
a54d5ca4a5 |
fix(kubernetes-nodes): unpin absence via --ignore-not-found
Same two holes as the migration's read classification, in the unpin hook. Capturing stderr into $pol (2>&1) contaminated the success path: kubectl warns on successful reads, so an unannotated object read as "pinned" and drew a spurious annotate call -- one per named object. And the *"not found"* pattern classified a transient conversion-webhook failure as absence, letting the Job report success while the keep-annotated objects leaked past the uninstall. --ignore-not-found makes the whole classification unnecessary here: absence folds into the same empty-output no-op branch as "present but unannotated", which is exactly the semantics unpin() wants, and any non-zero exit is a real failure that set -e turns into a failed Job. The case/esac disappears; stderr stays on the Job log where it is visible instead of parsed. This also avoids tempfile juggling under the container's readOnlyRootFilesystem. The bats suite gains a previously missing failure-path case and regression tests for both holes, red against the previous commit: a failed KMT listing must fail the Job (FAKE_LIST_FAIL), a read failure whose message contains "not found" must fail the Job, and a success-path stderr warning must not fake a pinned state (FAKE_WARN_STDERR). Absence is now modeled the way real kubectl behaves under --ignore-not-found: success with empty output. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
bdadb3ff04 |
fix(platform): detect absence via --ignore-not-found
The stderr-based classification this branch introduced had two holes of its own. Capturing stderr into the data variable (2>&1) contaminates the SUCCESS path: kubectl prints warnings on successful reads (API deprecation headers -- deterministic for a deprecated GVK, and CAPI v1beta1 is one -- plus client-side throttling notices), so the warning glued onto the owner annotation would trip adopt_one's foreign-owner refusal on every tenant and permanently deadlock the pre-upgrade hook. And the *"not found"* pattern over-matches: a transient conversion-webhook outage fails a read with '... service "capi-webhook-service" not found', which the classifier would treat as absence -- skipping the keep pin and re-opening the exact fail-open prune hazard this branch exists to close. Detect absence with --ignore-not-found instead: a missing object is SUCCESS with empty output, so a non-zero exit can only be a real failure and aborts before the version stamp, with no stderr parsing at all. adopt_one reads -o json and extracts the owner annotation with jq (already a dependency), keeping absence distinct from present-but- unannotated; pin_keep reads -o name. list_kmts still needs stderr for the one legitimate classification -- the two deterministic missing-CRD messages -- so it captures stderr to a temp file, never into data, and no longer accepts NotFound-shaped text for a list. The fake kubectl honors --ignore-not-found the way real kubectl does, gains FAKE_WARN_STDERR (success-path stderr warnings) and FAKE_GET_FAIL_MSG (transient error with arbitrary text). Two new bats tests, each red against the previous commit: a deprecation warning must not break adoption or fake a foreign owner, and a failure whose message contains "not found" must still abort without stamping. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
1ca947e2fb |
fix(kubernetes-nodes): fail unpin reads closed
The pre-delete unpin hook's existence check swallowed every kubectl get failure (`... >/dev/null 2>&1 || return 0`), four lines below the comment declaring "Failures are NOT swallowed". kubectl exits 1 for NotFound and for a transient apiserver error alike, so a timeout during uninstall read as "object absent": the unpin was skipped, the Job succeeded, and Helm then skipped the still-keep-annotated MachineDeployment -- leaking the adopted pool's worker VMs into the tenant namespace, running and unmanaged, after the pool was deleted. The KMT-listing path was already hardened for exactly this class; the existence check was the remaining instance. Classify by stderr: NotFound (native pool, or already gone) stays a no-op; any other failure fails the Job, bounded by backoffLimit and activeDeadlineSeconds, with logs retained. The existence check and the annotation read fold into a single request. The embedded script's behavior cannot be asserted from the manifest, so a new bats suite renders the chart, extracts the Job's script, and executes it against a fake kubectl: NotFound no-ops and succeeds, a transient named-get error fails the Job (red against the previous script, which printed "unpin done" and exited 0), keep-annotated objects are unpinned with the 6-hex hash anchor respected, and an unpinned existing object is left untouched. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
f8258c8859 |
fix(platform): fail migration 54 reads closed
kubectl exits 1 for NotFound and for transient server failures (timeout, 429, etcd hiccup) alike, and migration 54 treated every non-zero read as "object absent". A transient error while reading a worker object would skip the keep pin and the ownership rewrite, let the migration complete and stamp version 55 (so it never re-runs), and the parent's control-plane-only upgrade would then prune the un-pinned live MachineDeployment -- deleting every running worker VM in the pool. The doubly-masked KubevirtMachineTemplate listings (2>/dev/null plus `|| true`) had the same shape: a failed list adopted or pinned zero KMTs, permanently stranding a preserved older-revision KMT still referenced by an in-flight MachineSet, since the child chart only re-emits KMTs already annotated with its own release name. The `cur=$(... || true)` owner read additionally bypassed the foreign-owner refusal guard. Classify reads by stderr, the migration-47 precedent: NotFound means genuinely absent and is skipped; anything else exits 1 so run-migrations.sh leaves CURRENT_VERSION at 54 and retries on the next upgrade. adopt_one and pin_keep fold the existence check into the annotation read (one request instead of two); KMT listings go through a capture-first list_kmts helper because `for x in $(...)` discards the substitution's exit status even under set -e, leaving `|| true` to cover only the empty-grep-match case, mirroring the unpin hook. The child-HR existence check stays deliberately tolerant: its fallback is an idempotent apply that itself fails closed. adopt_one now also stamps meta.helm.sh/release-namespace so an object whose Helm annotations were stripped out-of-band cannot fail the child's ownership validation. The fake kubectl now reports absent objects the way the real apiserver does (NotFound on stderr, exit 1) and gains FAKE_GET_FAIL to inject transient errors; four new bats tests pin the classification: NotFound still completes and stamps, while a transient error on the adopt path, the KMT listing, or the pin path aborts before the version stamp. Each red against the previous script. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Myasnikov Daniil
|
01e1e7188e |
ci(release): stop finalize persisting credentials so the tag push triggers tags.yaml
pull-requests-release.yaml's finalize job checked out the repo with the default persist-credentials, leaving GITHUB_TOKEN embedded as http.extraheader. Every later tag/branch push re-injects the CI app token via `git remote set-url`, but a persisted extraheader silently wins over the URL credential — so those pushes authenticated as GITHUB_TOKEN. A GITHUB_TOKEN-authenticated push creates no workflow run (GitHub anti-recursion), which is exactly why v1.6.0's stable tag never triggered tags.yaml: the generate-changelog and update-website-docs backstops never ran. Set persist-credentials: false on the finalize checkout, so the set-url app token actually authenticates the pushes — the same handling promote-rc.yaml already uses. Audited every git/gh step in the job: both tag steps set-url the app token before pushing, and Ensure-maintenance-branch / Publish-draft-release / the retag+chart steps use the app token via github-script or a separate registry login — none depends on the persisted GITHUB_TOKEN extraheader. Pin it in hack/promote-gate-contract.bats. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
936190d4b6 |
ci(release): open the website docs PR at promote time
The cozystack/website 'update managed apps reference' PR used to be opened only at tag time by tags.yaml::update-website-docs — after finalize pushed the stable tag, i.e. after the release was already published (too late to review) — and it fetched README content from that stable tag (BRANCH=RELEASE_TAG), which finalize only creates post-merge. Add a promote-time website-docs job to promote-rc.yaml (needs [parse, promote]) that opens the same update-docs-vX.Y.Z PR from the release-X.Y.Z staging branch via the website Makefile's FETCH_REF variable, so the docs are reviewable alongside the promote PR while the stable tag is still unborn. It refreshes the trunk pins from the staging ref before release-next so a new minor inherits fresh Talos/cozystack pins (the v1.6 talos-pin incident class), then runs update-all with FETCH_REF pinned to the staging branch. The job is non-blocking: a FETCH_REF-support guard fails loudly (warn + exit 1) when cozystack/website main predates the FETCH_REF Makefile variable — otherwise make would silently ignore the unknown FETCH_REF override, fall back to the not-yet-created stable tag, and update_apps.sh would warn-and-continue into STUB docs. Any failure surfaces as a website-docs warning line in the promote PR body (plumbed through open-pr via needs.website-docs.result) and never blocks promotion; the tag-time job stays as the backstop and no-ops when promote already did the work. MERGE-TIMING CONTRACT: the website PR must NOT be merged before finalize publishes the release — doing so flips the site's latest-version pointer to a version whose docs and openapi are not on GitHub yet, 404ing every version-pin. The website PR body and the promote PR body both say so in bold. Pin the contract in hack/promote-gate-contract.bats (needs shape, FETCH_REF invocation, the guard, non-persisted checkout credentials, the DO-NOT-MERGE body wording, the promote-PR status line, and the tags.yaml backstop comment) and relax the release-changelog-contract needs pin to tolerate the appended website-docs dependency. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ff7ddd3c9a |
ci(release): gate promotion on a green rc e2e and drop the automatic full-e2e label
Promotion now assumes checks passed at rc time instead of re-running them. At dispatch, before any staging happens, promote-rc verifies the rc tag has a green full e2e via either evidence path: the rc's tags.yaml push run containing a successful 'rc-e2e / E2E <tag> (full suite)' workflow_call job, or a successful manual e2e-tag.yaml dispatch correlated by its exact tag-bearing job name (immutable rc digests make past evidence valid). No evidence fails the dispatch with both paths and the remediation named. A skip_e2e_gate boolean input bypasses the gate loudly for emergencies; the promote PR body always carries the verification state (verified or bypassed), and 'verified' can only originate in the gate's green branch. The promote PR keeps the release label but no longer gets full-e2e automatically: its tree is the rc's already-validated digests with only tag strings rewritten, so pull-requests.yaml now runs e2e on the release path only when a maintainer opts in by adding full-e2e. The regular-PR path and the ordinary full-e2e forcing are untouched, and the new labeled trigger makes a post-open label actually start the run. A new promote-gate contract suite pins the gate's DAG position, the override default, the label set, the body lines, and the label-gate condition with comment-proof executable-line counts. Assisted-By: GPT-5 <noreply@openai.com> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
2dfbeea776 |
ci(release): dispatch changelog generation at rc time
Extract the promote-time changelog generation into a reusable workflow (changelog-generate.yaml, workflow_call) and add a dispatchable wrapper (changelog-rc.yaml) so a maintainer can generate the release changelog the moment an rc goes green, while there is still time to review it. The reusable core keeps promote-rc's observable behaviour: detached rc-tag checkout, read-only app token for the AI step, the validate-changelog.sh gate, non-blocking AI failure, and the same changelog-vX.Y.Z artifact. .release-tooling is overlaid from github.ref_name, which under workflow_call is the caller's dispatch ref (and coincides with the reusable workflow's own ref for a local uses:), so "current tooling against an old rc tag" is preserved. Secrets are declared explicitly rather than inherited, so a caller that forgets one fails at parse time. The rc tag is parsed by a shared hack/parse-rc-tag.sh that REJECTS whitespace outright (a stripped 'v1. 6.0-rc.2' would silently become a different tag). promote-rc parses it ONCE in a cheap `parse` job that both `promote` and `changelog` depend on — so they still run in parallel with each other while consuming one validated tag, and a whitespace tag fails the whole promotion rather than letting promote succeed while the changelog job rejects the same input and the PR opens empty. The reusable core mirrors the reject-whitespace policy inline because its parse must run before any checkout. changelog-rc.yaml verifies the rc staging branch release-X.Y.Z-rc.N exists UP FRONT (in parse, which generate depends on), so a missing branch fails in seconds rather than after a 30-minute AI run, then commits the result there as docs/changelogs/vX.Y.Z.md idempotently (compare-before-push). promote-rc's changelog job is a thin uses: call. Source selection — validate-then-fall-through across the rc tag tree, the rc staging branch (the rc-time pickup), then AI generation, discarding any invalid source with a loud warning so it never suppresses a valid lower-priority one — lives in hack/select-changelog-source.sh, executed after the tooling overlay. open-pr carries the picked-up file onto release-X.Y.Z and finalize publishes it. Contract and behaviour bats are repointed and extended: the shared parser, the source-selection script, and promote's parse policy are all exercised by execution on adversarial inputs; the remaining grep pins match executable lines with exact counts so a comment or dead block cannot satisfy them. docs/release.md lists the new workflow and the rc-time entry point. actionlint and zizmor (default and pedantic) are clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
928df3c2d3 |
fix(platform): harden migration 54 against non-object and whitespace nodeGroups
The RFC-1123 key guard added in the previous round closed hostile map
keys, but migration 54 still trusted the shape of nodeGroups and of each
group value, and word-split keys on whitespace. Because it runs as the
platform pre-upgrade hook, either gap can deadlock the upgrade for every
tenant off one hand-mangled parent HelmRelease (kubectl edit bypasses the
aggregated API schema, the same vector the key guard addresses).
Shape: a non-object nodeGroups (array/string/number) crashes jq under
set -e (keys[] on an array yields indices, then .[$idx] fails); a
non-object group value aborts the value-mapping jq on `$grp | has(...)`.
Normalize both: skip the cluster for a non-object nodeGroups map, and
route a non-object group value through the same warn + pin + skip path as
an invalid name. {} / null still fall through to the implicit md0 default.
Whitespace: an unquoted `for group in $(... keys[])` word-splits (and
globs) a key like 'my pool' into fragments that each pass the RFC-1123
guard, fabricating garbage child releases. Iterate keys newline-safely so
the intact key reaches the guard. Fail-closed exit semantics are
preserved through the while pipeline.
Covered by three new bats fixtures in migration-54-adopt-path.bats, each
red against the current script and green with this change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
Myasnikov Daniil
|
dd308644c8 |
ci(release): execute the release-changelog logic in tests, not just grep it
The existing suite asserts on workflow YAML. That catches a rename or a deleted `if:` and nothing else: it cannot tell whether the shell inside a `run:` block does what the step name claims, and it stayed green while the logic underneath was wrong — twice, in the two preceding rounds. Add hack/release-changelog-behaviour.bats, which runs things for real. The GitHub Actions semantics the whole non-blocking design rests on are now verified by executing them. hack/testdata/needs-semantics.yaml mirrors the promote / changelog / open-pr shapes with a fatal (non-AI) step in the changelog job, and `act` runs it. Result: open-pr DOES run when a job it needs has failed, `needs.changelog.result` is `failure`, and — the part that could not be read off the YAML — a failed job's outputs come back EMPTY, while a successful job's propagate. That empty-output behaviour is precisely why open-pr re-establishes changelog presence from disk instead of trusting needs.changelog.outputs. Deleting the `if:` from the fixture makes open-pr get skipped, so the gate is demonstrably load-bearing. The git logic moves into hack/changelog-preserve.sh and is exercised against real bare remotes: a branch that exists, one that does not, one with no changelog, one with an empty changelog, and the full preserve/checkout -B/restore/force-push cycle promote actually performs. The last of these directly reproduces the regression the preserve step exists to prevent. An empty changelog is deliberately NOT preserved — it would suppress generation and then publish as blank notes. finalize's release-body selection runs under node across all three inputs (good, whitespace-only, absent) and the resulting decision table is compared as a whole. Replace the shell overlay with a sparse second actions/checkout of the dispatch ref at .release-tooling. The overlay had a bootstrap problem I introduced: the script that fetches missing tooling was itself missing from the rc tree, so it could never have worked for exactly the trees that need it. actions/checkout comes from the runner, not the repo, so it has no such circularity. .release-tooling is gitignored because promote runs `git add -A`, and the contract suite pins that. All 21 tests mutation-checked. Breaking changelog-preserve.sh, making it return an empty file, removing the fixture's `if:`, dropping the tooling checkout, and renaming its path each fail the suite and pass on restore. Not covered, and stated in the suite header rather than implied: the AI generation step, the real GitHub API calls, and whether the sparse checkout resolves on GitHub's runners. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
34e45295fe |
ci(release): fix second-round review findings
Two more review passes (Claude + Codex) on the previous commit. Codex found the one that would have broken the very next promotion. The agent reads its own instructions from the working tree, which is the rc tag. Every rc cut before this change carries the old §2 of docs/agents/changelog.md — two valid configurations, "stop" on anything else — and a detached rc-tag checkout is exactly that "anything else". So promoting v1.6.0-rc.4 would have presented a compliant agent with instructions telling it to refuse, silently, producing no changelog on every promotion of an existing rc. Verified against the tag. The same applies to hack/validate-changelog.sh, absent from those trees entirely, which would have failed the Verify step with "command not found" and, in open-pr, taken the promote PR down with it. Both are now overlaid from the dispatch ref, best-effort, in both jobs. The structural validation added last round rejected every PATCH changelog. It asserted a `# Cozystack vX.Y.Z` header and a 20-line floor; patch releases use `# vX.Y.Z (date)` and v1.5.1 is a complete 19-line release. Good AI output would have been discarded and the promote PR mislabelled "generation failed". Measured across the tree: the header form holds for 23 of 115 changelogs, so it was never an invariant. The predicates now live in hack/validate-changelog.sh, tested directly against real changelogs rather than through YAML greps — the reason the defect shipped is that the suite pinned finalize's guard but nothing on the promote-side validator, and stayed green when the validator was mutated away. It accepts both header conventions, has no line floor, and checks the version in the header, the release-link comment and the compare link, so a changelog generated for the wrong version is caught. All 16 current-era changelogs pass; whitespace, empty, truncated and wrong-version inputs are rejected. Also fixed: - A fatal artifact download in open-pr would abort the job before the PR step — the same wedge as a skipped job, one step lower. Download is now best-effort and the commit and PR body key off what is on disk, not an upstream job output that says nothing about whether the transfer worked. - A pre-existing changelog (hand-written, or preserved across the force-push) reached the PR without ever being validated, was advertised with ✅ and published verbatim. It now goes through the same validator, as does the tags.yaml backstop. - `gh pr view <branch>` matches closed PRs, so re-dispatching after closing an abandoned promote PR edited the dead PR and opened nothing. Branch on state explicitly. - `gh pr edit --body` unconditionally overwrote the PR body, destroying the upstream breaking-change notes docs/release.md instructs reviewers to paste there. Refresh only while the body is still the generated one. - open-pr consumed raw dispatch input while its siblings used the validated parse; the changelog job's bash parse did not trim, so an input with a stray space passed promote and failed there. Both fixed. - The prompt and §2 said "trailing release-link comment"; it is the first thing in every changelog. Tests grow to 16, each mutation-checked: dropping the overlay, making the download fatal, reverting the existing-changelog check to `-s`, and removing the backstop validator each fail the suite. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f49d54a689 |
ci(release): fix review findings; keep website docs at tag time
Dual-model review (Claude + Codex) of the previous two commits found seven substantive defects, two of them destructive. Both models independently reached three of them; the split is noted per item. Descope the website half (Claude, verified by reproduction). The docs job was moved to promote time, but the website Makefile pins the fetch ref with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that cannot exist at promote time, since promote's own validate step hard-fails if it does. update_apps.sh logs the failed fetch and exits 0 AFTER truncating each destination file, so every app doc would become a stub, `changed=true` would open the PR, and the new auto-merge would land it unreviewed. Not fixable from this repo. update-website-docs returns to tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize auto-merge is dropped. Opening it earlier needs a cozystack/website change first and is left as a follow-up. This also removes the unordered-sibling race Codex found, where the release PR could merge before the docs job opened its PR, leaving finalize nothing to merge and no retry. Fix the non-blocking guarantee (both models). Only the AI step carried continue-on-error; the changelog JOB did not, so a flaky `npm i -g @github/copilot` failed it and `needs` skipped open-pr entirely — staging branch pushed, release drafted, no promote PR. Gate open-pr on promote alone. Generate from a checkout the contract permits (both models). The job checked out the rc staging branch, which docs/agents/changelog.md §2 rejects — a compliant agent stops and writes nothing, so the normal path produced no changelog at all. Check out the rc tag detached and document that configuration in §2. The checkout ref now comes from the validated parse output rather than raw dispatch input. Port, don't regenerate, in the backstop (both models). For a patch release promoted onto release-X.Y the changelog never reaches main, so the backstop fired and ran a second AI pass; once its PR merged, update-releasenotes.yaml overwrote the published body with unreviewed text. Detect the file at the tag and port it verbatim; the AI runs only when it is absent from both main and the tag. Survive re-dispatch (Codex). promote force-pushes the staging branch, erasing a hand-written changelog — the documented recovery when generation fails. Preserve an existing one across the rebuild, and stop open-pr overwriting it with generated output. Validate structurally, not by size (Codex). A truncated or whitespace-only AI run passed `[ -s ]`, was committed, advertised as valid in the PR body, and published verbatim as release notes. Assert the header, the compare link and a plausible line count; finalize additionally refuses whitespace-only content rather than blanking the notes. Add hack/release-changelog-contract.bats — nine tests pinning each link in the chain, mutation-checked: removing open-pr's if:, breaking finalize's body read, and restoring paths-ignore each fail the suite. Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which described the old flow throughout. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
6ecb36ffc7 |
feat(cozystack-controller): extract application CA into a key-free tenant Secret (#3407)
## What this PR does Adds an engine-agnostic controller that publishes a key-free `<release>.tenant-ca` Secret (only `ca.crt`) for a managed application, so a tenant can verify the application's TLS without ever seeing a private key. The source is declared, not guessed: a chart renders a namespaced `TenantProjection` sentinel (group `internal.cozystack.io`) naming the CA Secret to lift; the controller watches the sentinel, extracts `ca.crt`, and writes the projection owner-referenced to the sentinel, so garbage collection is native. The projection carries the `internal.cozystack.io/tenant-ca` label the lineage webhook turns into tenant visibility. Postgres is the first consumer. The trust boundary is RBAC: no tenant role grants any verb on `internal.cozystack.io`, so a tenant cannot forge a sentinel. A companion `ValidatingAdmissionPolicy` that pins the writer to helm-controller ships separately as defence in depth. This is the controller-and-CRD slice, split out from #3299 for focused review. It is the only piece required for the feature to work; the render-time helper, the writer policy, the capability-gate, and the release-prefix validation land as their own PRs. ### Downstream repositories Walked the trigger map against the diff. This adds an internal CRD and controller and a sentinel to the Postgres chart; no app is added or renamed and no `values.schema.json` changes, so the typed provider and Ansible are unaffected. The one to confirm is the Postgres README edit — if the website mirrors it, it needs a refresh. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: ### Release note ```release-note feat(cozystack-controller): publish a key-free per-application CA trust anchor (`<release>.tenant-ca`) that tenants read to verify managed-application TLS, declared through a new `TenantProjection` CRD. Postgres is the first consumer. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added tenant-facing CA certificate projections for PostgreSQL deployments. * Tenant secrets now provide only the CA certificate, excluding private keys. * Added support for configuring and monitoring tenant CA projections. * **Documentation** * Updated PostgreSQL TLS instructions to retrieve certificates through tenant secrets. * **Bug Fixes** * Improved protection against invalid certificates, secret collisions, and accidental exposure of private key material. * Added validation to ensure projected certificates remain synchronized with their sources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
8aeedde088 |
feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported, tested capability for the new `backups.cozystack.io` API. Fixes #2774. Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the `recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime` plumbing; #3313 has since merged and this branch is rebased on `main`. This PR adds the reliability + verification layer #2774 asked for. ### Changes - **Precise failure reason for an unreachable target.** The restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the sole authority for "this restore is stuck" — the client waits the full window either way. What changes is the *diagnosis*: once the deadline has elapsed, if `recoveryTime` was set and the recovery Cluster never went healthy, the CNPG driver reads the `full-recovery` pod logs and — when several distinct attempts exited with PostgreSQL's `FATAL: recovery ended before configured recovery target was reached` — fails the RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason `RecoveryTargetUnreachable`) and an actionable message, instead of a bare generic timeout. Timing is unchanged; only the reason/message differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.) Design points: - **We deliberately do NOT trip early off the FATAL.** A valid near-now target emits that same FATAL transiently — repeatedly, for many attempts, on a slow archiver — before it converges, so any earlier trip would wrongly reject a recoverable restore (observed twice in CI). The FATAL is used only to *explain* a failure the deadline has already declared. - **Health wins unconditionally** — the deadline/classification check precedes the healthy check, but a recovery that converges within the window reaches success and is never classified as failed (a live dev7 restore did exactly this). - **Counting distinct FATAL-ing attempts** (not pod failures) tolerates a one-off archive-lag miss on a near-now target and ignores transient recovery-pod crashes (a node blip / brief API-server unreachability never emits that FATAL). - Adds the `pods/log` RBAC grant the log read needs (distinct subresource from `pods`); a `Forbidden` is logged loudly and surfaced in the failure message rather than silently disabling the classification. - **e2e** (`examples/backups/postgres/run-all.sh`, wired into the Chainsaw `postgres-2-backup-roundtrip` suite): - Step 45 — write `before`, capture server timestamp (µs precision), write `after`, restore to that timestamp, assert `before` present and `after` absent. - Step 46 — a `recoveryTime` an hour ahead must fail with reason `RecoveryTargetUnreachable` once the deadline elapses (asserts the *reason*, so it proves the classification fired, not just that a timeout occurred). - Chainsaw budget raised 25m → 40m to cover three sequential restores + the post-deadline classification. - **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a `docs/operations/backup-classes.md` PITR section (recoverable window, discovering earliest/latest restorable time from the backup catalog since `firstRecoverabilityPoint` is unreliable under the plugin, archive-lag tolerance, idempotency under GitOps). ### Verification - `go build` / `go vet` clean; unit tests pass (`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`, `recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level success-path test asserting `Succeeded` + `RecoveryConverged=True`, and — added per review — `TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`, a reconcile-level test driving the deadline branch through all three sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL absent → generic timeout, `Forbidden` log read → generic timeout + missing-`pods/log`-RBAC note. It fails if the call-site reason is reverted to the generic one). - Reviewed via adversarial branch-review; fixes include a critical missing-RBAC dead-on-arrival, a classification false-positive on transient recovery-pod crashes, and a sub-second e2e flake. - **dev7**: backup + WAL archiving + recovery convergence verified live on the #3313 build. The load-bearing assumption of the classification — CNPG **accumulates** failed `full-recovery` pods rather than reusing one — was confirmed (observed 7 simultaneous failed recovery pods). A fully-green live step-46 run was blocked by intermittent COSI bucket-provisioning flakiness on dev7; the negative path is covered by the CI Chainsaw e2e. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The only user-facing surface is the new `docs/operations/backup-classes.md` PITR section; the website keeps a hand-mirrored copy per docs version. - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: PITR docs mirror warranted (not yet opened; same pending mirror as #3313's backup-classes changes) - [x] No other downstream repository is affected by this change ```release-note feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added PostgreSQL point-in-time recovery (PITR) support via RFC3339 `recoveryTime`. * Introduced a PITR RestoreJob example, plus helper variables and an unreachable-target scenario. * **Bug Fixes** * Improved restore deadline handling with deterministic, log-based fail-fast classification for unreachable recovery targets, including clearer conditions. * **Documentation** * Added detailed PITR operational documentation (recovery windows, failure behavior, and GitOps/idempotent behavior). * Expanded README and example comments for PITR verification and failure expectations. * **Tests** * Extended unit tests and end-to-end chainsaw workflow to cover PITR success and unreachable failures. * **Chores** * Updated RBAC to allow pod log reads; refined cleanup and run scripts for additional RestoreJob artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
3dc6cbd4f4 |
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors
hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure produced an empty table indistinguishable from an honestly clean fleet (namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST failures = zero findings, all under 2>/dev/null with no error handling). The runbook uses this output as the gate before PVC deletion, as post-deletion verification, and as an upgrade precondition, so a transient API error could green-light destroying data (#3431). Route every kubectl call through a run_kubectl helper: non-zero exit prints a FATAL line naming the failed query and propagates the code up the whole chain (enumerations restructured to capture-then-check, since an exit inside $(...) dies with the subshell). By-name GETs distinguish legitimate absence from real errors via --ignore-not-found on a separate existence check; a Secret that exists but has no decodable release payload, or one that decodes without a chart name, is corrupt state and fails loudly instead of silently dropping the tenant. Incomplete PV-age evidence now marks the generation incomplete and falls to the safe no-direction branch, so a failed GET can no longer flip OVERLAP into a wrong Step-3 deletion candidate. A successful run with zero findings still prints the same bytes as before. Tests: 14 new cozytest cases — failure injection for every LIST and by-name GET site, corrupt-payload variants, and full-output golden diffs for the success paths (verified byte-identical to the pre-change script). 25/25 green. Fixes #3431 Assisted-By: Claude <noreply@anthropic.com> Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
5d8dd51cab |
fix(release): make promote-retag digest verification media-type-agnostic
skopeo inspect --format '{{.Digest}}' fails on OCI artifacts: for
cozystack-packages (config.mediaType
application/vnd.cncf.flux.config.v1+json) it prints nothing and exits
nonzero, so the v1.6.0 finalize aborted its post-copy verification right
after both copies had already succeeded, leaving 32 of 43 repos without
stable tags and skipping the cozy-installer publish. --dry-run never
executes this branch, so rehearsals could not catch it.
Compute the digest as sha256 of the raw manifest instead, which is how
registries define it and works for any media type. The helper gates the
hash on skopeo's exit status via a temp file rather than a pipeline:
a missing tag still yields an empty string (pre-check reads that as
"tag absent, proceed to copy"), never the empty-input hash
sha256:e3b0c442..., and never an abort under set -eu.
Verified against live GHCR: the raw-manifest hash of
cozystack-packages:v1.6.0 matches the digest finalize expected
(bf68208730860fa8...), and agrees byte-for-byte with the old method on
regular multi-arch images. New cozytest cases cover the OCI-artifact
digest, a post-copy mismatch, and the missing-tag path.
Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
IvanHunters
|
206957610e |
test(platform): cover implicit-md0 migration path and sync jq comment
Both existing adopt-values cases feed an explicit md0, leaving the DEFAULT_MD0
materialisation (empty nodeGroups) uncovered. Add a case asserting a parent HR
with no nodeGroups yields the child HR kubernetes-nodes-<cluster>-md0 with
minReplicas 0 and the ingress-nginx role. Also correct the header comment: the
jq picks ["version","talos"] and narrows images to {kubectl} separately, not
["version","talos","images"].
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
c8ba404646 |
fix(platform): guard migration 54 against RFC-1123-invalid pool names
The 53-char overflow branch exists because exiting the migration deadlocks the platform pre-upgrade hook for every tenant. The equivalent deadlock via name validity was unguarded: the old nodeGroups map key was never schema-constrained (map[string]NodeGroup, no propertyNames pattern), so a stored parent HR can carry a key like 'My_Pool' whose own render already failed at apply time. The migration then built child_name=kubernetes-nodes-<cluster>-My_Pool, kubectl apply rejected the RFC-1123-invalid HelmRelease name, the script exited 1 under set -e, and every subsequent platform upgrade re-ran and re-failed. Validate child_name against the RFC-1123 label pattern alongside the length check and route failures through the same warn + pin + skip path. Factor the pin-and-skip logic shared by both guards into pin_pool_objects. Cover the new branch in migration-54-adopt-path.bats (the fake kubectl apply now rejects an invalid metadata.name, so the case fails without the guard). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Myasnikov Daniil
|
6a41d65f16 |
test(e2e): avoid ripgrep dependency in diagnostics tests
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
4e0cfd994d |
test(e2e): render COSI catch output as YAML
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ad1adbf785 |
test(e2e): fix diagnostic collector blind spots
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
55195ded0c |
test(e2e): capture previous container logs and stop killing the snapshot (#3346)
## What this PR does Root-causing a recent E2E failure was nearly blocked because the diagnostics captured on failure had collected almost nothing. Two independent mechanisms were responsible, and both had been silently in place for a long time. **The failing container's previous logs were never captured.** The pod had crash-looped, so the decisive evidence lived in the first container instance, reachable only through `kubectl logs --previous`. Chainsaw's `podLogs` action has no `previous` field, so this could not be expressed per-suite; the conclusion had to be reconstructed from indirect evidence instead. A new capture runs from the global catch and from the BATS failure hook, selecting containers at runtime by `restartCount > 0` across both init and regular containers, so a healthy run stays quiet and future suites are covered without per-suite wiring. **The catch was structurally guaranteed to be killed.** A Chainsaw `script` op is bounded by `timeouts.exec`, which is 5m, while the catch's own inner budgets summed to roughly 960s. It reported `SCRIPT | ERROR | signal: killed` and produced nothing — and its second leg, the data-plane capture, had therefore never run at all. The kill signature was itself misleading: `timeout` exits 124 and cannot produce that message, which comes from Go's `exec.Cmd` killing the process group. Each catch op now carries an explicit timeout above the budgets it contains, so the inner one fires and leaves a partial, usable artifact instead of losing everything. **The snapshot was truncated at 60 seconds.** `crust-gather --duration` defaults to 60s and was never passed, so every snapshot ever taken was cut short, with nothing in the artifact saying so. The API discovery it performs first is not covered by that budget at all, which is where the outer wall-clock was going on a cluster with a degraded aggregated APIService — precisely the state a failed run is in. Both budgets are now set explicitly, and the collector's output goes to a log inside the snapshot with a completeness verdict, so "is this snapshot complete?" is answerable from the artifact. Failure paths are reported rather than swallowed: an unreachable API says so instead of implying nothing had restarted, a truncated capture is kept and marked, and every dropped item is logged, since silent truncation reads as full coverage. ### What is not verified here The mechanism claims are read from the pinned versions of Chainsaw and crust-gather. Everything cluster-facing is not verified by this PR: the go-template's output shape against a live API server, `--previous` returning the first instance for a real crash-looper, and whether the new collection budgets suffice on a degraded cluster. The bats suite covers the selection helpers only, and those cases are mutation-tested. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Diagnostics** * Improved end-to-end failure evidence by capturing “previous instance” logs for crash-looping pods (including restarted init containers) and prioritizing relevant namespaces. * Added time-bounded diagnostics collection with explicit outer timeouts; collector output is saved to per-snapshot log files. * Classifies collector outcomes as complete, timeout-truncated, or failed, while preserving the original test failure signal. * **Documentation** * Expanded E2E failure-diagnostics guidance with new timeout and evidence-capture conventions. * **Tests** * Added unit-style tests for previous-log selection, prioritization, row limiting, and deterministic log artifact naming. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d0c0f399f8 |
fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404)
## The bug Promotion rewrites the rc version substring to the stable version across the vendored image references. That rewrite lived inline in `promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus `packages/apps/kubernetes/images/*.tag` **only**, on the premise that the kubernetes app was the only one whose `.tag` files carry the cozystack version. Nine other `.tag` files do, and `system/multus` seds its reference straight into a vendored upstream daemonset manifest. #3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33 of 55 first-party references still reading `v1.6.0-rc.4`**. `hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind spot from the other direction — both scanned the depth-2 `values.yaml` alone, so those images were never retagged to the stable version (**30 references selected before, 42 after**) and never mirrored to the public registry for a nightly. A retag miss leaves an image short of a tag, since the digest still resolves inside one registry. A mirror miss is worse: the host rewrite walks the same file list, so a reference that is neither mirrored nor rewritten leaves the published tree pointing at the private build registry. `v1.6.0` is the first release to go through promotion — every earlier release was a full rebuild from the tag, which stamped every file uniformly. The glob has always been wrong; the old path just made it unobservable. ## Why it could not be caught before The rewrite was workflow-inline, so there was nothing to unit test. The only way to observe the miss was to cut a release. It is a script now — `hack/promote-rewrite-tags.sh` — and `hack/promote-rewrite-tags_test.bats` round-trips it against the real tree under `make unit-tests`, with no cluster, registry or release involved. ## The fix The three call sites each carried their own idea of where a reference can live, which is how they drifted apart. That knowledge moves into `hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()` for the files, `collect_image_refs()` for the references inside them — and all three source it, so a new storage location is declared once and reaches every consumer. The rewrite also asserts its own postcondition: it scans **wider** than it rewrites and fails the promotion when a reference survives in a location the enumeration does not know about, rather than shipping it quietly. The wide scan is shape-filtered to image-reference position, because `X.Y.Z-rc.N` is an ordinary version string that other things legitimately contain — `kubevirt-csi-driver/go.sum` pins `github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:` instead would have been wrong: `system/cilium/values.yaml` keeps `tag:` and `digest:` under separate keys. `docs/agents/image-refs.md` records the contract — three tag classes, three storage shapes, the invariants, and which consumer does what. ## Not a bug: kamaji `cluster-api-control-plane-provider-kamaji` stays at `v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild of an upstream component, versioned by that component rather than by the cozystack version line, so no rewrite is owed. A test pins this so a future fix here cannot over-reach. ## Documented known gaps, deliberately not fixed here Both are pre-existing and in the **host** dimension; the version dimension this PR owns is complete, because the version always lives in a `tag` key regardless of where the host sits. - **The nightly host rewrite cannot reach a split host.** It is a literal `<src-registry>/` substring replace, so `keycloak-operator` (host in a sibling `registry:` key) and `kubeovn` (`global.registry.address`, no trailing slash) are mirrored but keep pointing at the build registry in the published tree. Fixing it needs structure-aware rewriting. - **A reference inside a gzip is invisible.** `capi-providers-cpprovider` ships `files/components.gz`, and the promote postcondition cannot catch this class at all since `grep -rIl` skips binaries. Nothing is owed today — kamaji is component-versioned and the image is still mirrored and retagged via its `.tag` — so the residual is that a nightly's kamaji ConfigMap keeps the build-registry host. ## Testing - 17 tests in the new suite; `make bats-unit-tests` green (312 tests, exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow. - Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`, `1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no false negatives. - Every test is mutation-proven. Reverting the enumeration to the old glob, deleting any of the five YAML shapes, corrupting emitted digests, reverting extras to parse-only, dropping the unreadable-file guard, broadening shape 4 beyond `global.images`, or planting an unenumerated first-party reference in any storage shape each fail a named test. ## Reviewer note, out of scope `platform-migrations` is pinned at two different digests — `core/platform/values.yaml` at `v1.5.0` and `backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned pin; no Makefile writes that key, tracked in #3143). A `promote-retag.sh` dry-run emits **two copies to the same destination tag**, so the write-once guard will fail the promotion. This is independent of this PR and needs a decision about the backup-client image version, so it is left alone here — but it blocks the 1.6.0 promote and wants fixing before the re-cut. ## Release note ```release-note Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry. ``` Fixes the promote half of #3143. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Image promotion and nightly mirroring now detect digest-pinned image references across values files, standalone tag files, and supported templates. - Stable release preparation supports RC→stable tag rewriting via a dedicated promotion script, with broader verification to ensure no RC references remain. - **Bug Fixes** - Improved completeness of ref discovery and scanning, including previously missed template/tag locations. - **Documentation** - Added and linked guidance documenting image-reference invariants and how tooling enumerates supported storage shapes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |