admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-04 02:57:27 +09:00
Code Issues Packages Projects Releases Wiki Activity
5,224 Commits 548 Branches 294 Tags
docs/role-nomination-template
Commit Graph
689 Commits
Author SHA1 Message Date
Andrei Kvapil 0f0204a6c2 fix(e2e): disable guest fsync on ephemeral CI VM disks (#3455) 
## What this PR does

The e2e QEMU guests are launched in `hack/e2e-prepare-cluster.bats` with
three `-drive … if=virtio,format=raw` lines that carry no `cache=`
option, so QEMU falls back to its `cache=writeback` default, which
honors every guest fsync against the host-backed raw disk files.

The management-cluster Talos etcd — which holds every operator's
leader-election lease — fsyncs on each Raft commit. During the VM-heavy
suites, containerd is simultaneously unpacking dozens of images and CDI
is importing multi-GB VM disks onto the same host-backed raw files. The
resulting fsync storm serializes etcd's writes into multi-second stalls,
so the management apiserver returns `context deadline exceeded` on lease
PUTs, and several operators (capk, etcd-operator, fdb-operator,
cilium-operator, kamaji) lose leadership at the same moment. Whichever
chainsaw suite is mid-flight then fails — a different one each run
(vmcluster, kubernetes-latest, vmdisk), which is the fingerprint of
shared-resource contention rather than a bug in any single suite.

This adds `cache=unsafe` to each guest drive. That makes guest flushes
no-ops against the host page cache, which removes the fsync-gated etcd
stalls at the root of the flakiness. It is safe here precisely because
these CI VMs are ephemeral and destroyed at the end of every run — a
crash before writeback loses only throwaway state — and it must never be
used for real data. The change is minimal and uniform across all three
guest drives; no non-guest drive is touched (there are none in this
file).

Runner sizing alone (#3251) did not resolve this, because more CPU and
memory do not change the I/O path: the guest fsyncs still propagate
through `cache=writeback` to the host, and etcd still stalls behind the
same fsync storm. This addresses the I/O path directly.

`git log -S 'cache='` on this file is empty, so a cache mode has never
previously been set here.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
actual diff. The only entries that reference
`hack/e2e-prepare-cluster.bats` (ansible-cozystack and ccp) are scoped
to node prerequisites — kernel modules, containerd settings, the LVM
`global_filter`, and the cluster domain. This change touches none of
those: it adds a QEMU host-side hypervisor flag (`cache=unsafe`) to the
CI VM `-drive` options, which is invisible to the guest OS and to any
real node. No downstream repository is affected.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(e2e): boot e2e QEMU guests with cache=unsafe so ephemeral CI VM fsyncs no longer stall the management-cluster etcd, removing the cross-operator leader-election flakiness that failed a different chainsaw suite each run
```

---

This PR was prepared with the assistance of an AI agent (Claude). A
human maintainer reviewed the diff and the reasoning before submission
and remains responsible for the change.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Updated end-to-end virtual machine startup testing to use optimized
QEMU disk caching settings for improved reliability and performance.
* Improved end-to-end Kubernetes cluster readiness checks under load by
extending the deadline for reaching “at least 2 nodes Ready,” and
aligned related messaging and failure diagnostics accordingly.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-29 19:32:45 +02:00
Andrei Kvapil 3dc0f56477 feat(kube-ovn): build the chart and image in-tree, retiring the wrapper repo (#3460) 
## What this PR does

Brings the `kubeovn-chart` wrapper into the tree, so
`cozystack/kubeovn-chart` can be archived.

Until now `packages/system/kubeovn` pulled a chart tarball from the
latest `cozystack/kubeovn-chart` tag at `make update` time, and that
repository also built and pushed the `kubeovn` image. The wrapper
existed only to cache the upstream chart; with the current caching
approach that reason is gone, and the split cost a second repository, a
second release step, and a package whose provenance a reader cannot
follow without leaving the tree.

The package is now shaped like `packages/system/cilium`:

- `update` pulls the upstream `kubeovn/kube-ovn` chart directly and
applies `patches/mtu.diff`, then syncs the version into the Dockerfile.
The MTU patch already carried monorepo-relative paths, so it applies
unchanged.
- `image` builds `images/kubeovn` (carrying the existing
`fix-keepvmip-portgroup` patch) and pins the digest into `values.yaml`,
using the shared `image-tags` / `cache-args` helpers.

It also carries `cozystack/kubeovn-chart#5`, which never got a tag:
`OVN_IPSEC_KEY_DIR` is pinned under `/var/lib`. The kube-ovn default
sits in `/etc/origin`, which a Talos node cannot create, so enabling
IPsec left the node without ESP security associations.

### The nightly host rewrite had to be fixed first

Moving the image build here activates a gap recorded in
`docs/agents/image-refs.md`. The nightly host rewrite is a literal
`<src-registry>/` substring replace, and kubeovn keeps its host in
`global.registry.address` with the bare repository in a sibling key — no
trailing slash, no match. While the image came from a separate
repository the baked reference carried the public host and the ownership
filter skipped it; once CI builds the image here it stamps the
build-registry host into that key, and the result is an image mirrored
to the public registry while the published tree keeps pointing at the
private build registry.

The second commit adds an end-of-line-anchored expression that fires
only when the source registry is the complete scalar value, so a
contiguous reference — which continues past the host — is still handled
by the first expression and the two can never both fire on one
reference. `keycloak-operator` splits at a different boundary and stays
unfixed; that is documented, and dormant while no CI path rebuilds it.

Verified: `hack/nightly-mirror_test.bats` passes with the new case and
the case fails without the fix; `promote-retag` and
`promote-rewrite-tags` suites unaffected; `helm template` and `helm
unittest` on the package pass. The pre-existing `helm lint` complaint
about a missing `kube-ovn` dependency reproduces on `main` and is left
alone.

### Follow-ups this PR does not do

- `cozystack/kubeovn-chart` should be archived once this merges, with a
pointer here. Its open PR #5 is carried by this change and needs no
separate landing.
- `cozystack/website` links the Kube-OVN license at
`cozystack/kubeovn-chart` in `operations/configuration/licenses.md` and
`guides/platform-stack/_index.md` (all versions, including `next`).
Archived repositories stay readable so nothing breaks, but the link
should move upstream or in-tree. **No follow-up has been opened yet** —
flagging rather than ticking the box.

### Screenshots

N/A — build and packaging change, no user-facing or UI impact.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walked the trigger map against the diff. The website row matches — not
through the app-list or platform-component triggers, but because two of
its pages link the repository being archived. Left unticked because no
follow-up exists there yet; a human should decide whether the license
link moves to `kubeovn/kube-ovn` or to this package. No other row
matches: the change touches no `packages/apps/` or `packages/extra/`
package, no `values.schema.json`, no `ApplicationDefinition`, no
platform variant, and no node prerequisite.

### Release note

```release-note
feat(kube-ovn): the Kube-OVN chart and image are now built in this repository instead of cozystack/kubeovn-chart. OVN_IPSEC_KEY_DIR now defaults to /var/lib/ovs_ipsec_keys, so enabling IPsec on Talos nodes no longer leaves them without ESP security associations.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Updated Kube-OVN image build to use upstream chart metadata and keep
registry/version alignment during `update`.
  - Added conditional MTU support for the OVN CNI DS via chart values.
- Introduced `OVN_IPSEC_KEY_DIR` to standardize the OVN IPsec keys
directory.

- **Bug Fixes**
- Preserved VM networking port-group assignments during live migration
when an alive virt-launcher sibling pod exists.
- Improved nightly mirror host rewriting for split and standalone
registry host formats.

- **Tests**
- Added coverage for the end-of-line “whole scalar value” host rewrite
case.

- **Documentation**
- Refreshed “Known gaps” guidance for nightly mirror host rewriting
behavior and limitations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-29 18:19:41 +02:00
myasnikovdaniil cc44f6c981 ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote (#3449) 
## What this PR does

Moves release validation from promote time to rc time, making promotion
itself a pure retag operation. This is the follow-up to the v1.6.0
retrospective: the promote flow's checks ran on the promote PR (coupled
to promotion, skippable by never promoting), the changelog and website
docs were produced only after the release went public, and the finalize
tag push silently failed to trigger `tags.yaml` at all
(persisted-credential bug, root-caused below).

The release lifecycle after this PR:

1. `cut-prerelease.yaml` cuts an rc; `tags.yaml` builds, drafts, pushes
the staging branch — and now runs a **mandatory full e2e** for every
`-rc.N` tag via the new reusable `e2e-tag.yaml` lane. Alpha/beta (and
any published tag) get the same validation as a **manual button**
(`workflow_dispatch`).
2. Optionally, a maintainer presses the **`changelog-rc.yaml` button**:
the release changelog is generated from the rc tag (reusable
`changelog-generate.yaml`) and committed to the rc staging branch, ready
for review long before promotion.
3. `promote-rc.yaml` dispatch is **gated on green rc e2e** (evidence:
the rc's `tags.yaml` run containing a successful `rc-e2e` job, or a
successful manual `e2e-tag` dispatch for the tag; `skip_e2e_gate` input
for emergencies, always surfaced in the PR body as ✅/⚠️). Promote stages
the release, picks up the rc-time changelog (or generates one), and
**opens the website docs PR** on cozystack/website from the staging
branch — parked with an explicit DO-NOT-MERGE-until-finalize contract.
4. The promote PR **no longer gets `full-e2e` automatically** — its tree
is the rc's already-validated digests with only tag strings rewritten; a
maintainer can still add the label manually to force the suite.
5. Finalize performs the irreversible side effects (tag, publish, retag,
chart) — and with `persist-credentials: false` its tag push now actually
authenticates as the CI app, so the `tags.yaml` backstops
(`generate-changelog`, `update-website-docs`) fire for promoted tags.
This is the one-line fix for the v1.6.0 incident where the tag triggered
nothing: the checkout's persisted `GITHUB_TOKEN` `http.extraheader`
silently defeats the app token injected via `git remote set-url`, and
`GITHUB_TOKEN` pushes create no workflow runs.

Everything is pinned by executable contract tests
(`hack/promote-gate-contract.bats`, extended
`hack/release-changelog-contract.bats` /
`release-changelog-behaviour.bats` — 48 tests total, including behaviour
tests that execute the new `hack/parse-rc-tag.sh` and
`hack/select-changelog-source.sh` for real), and `docs/release.md` is
rewritten to describe the new lifecycle end-to-end.

### Commit map (review commit-by-commit)

- `75b5192f1`…`dd308644c` + `6fc970851` — the pre-existing
changelog-before-publication rework (changelog generated at promote,
committed to the staging branch before the PR opens, finalize publishes
it as the release body; tags.yaml job becomes a verbatim-port backstop),
rebased onto main.
- `5aec70b3a` — `e2e-tag.yaml`: reusable published-tag e2e lane
(prerelease → staging branch, stable → tag; draft releases fail fast;
dispatch-only SSH `debug` breakpoint; tag-suffixed artifacts).
- `025564d96` — `tags.yaml::rc-e2e`: mandatory full e2e for freshly cut
rc tags (also on `release_exists` re-runs); fixes the
`cut-prerelease.yaml` comment that falsely claimed e2e already existed.
- `2dfbeea77` — `changelog-generate.yaml` (reusable core) +
`changelog-rc.yaml` (rc-time button committing to the staging branch);
promote-rc gains a top-level `parse` job with a strict no-whitespace tag
policy (`hack/parse-rc-tag.sh`); source selection is
validate-then-fallthrough (`hack/select-changelog-source.sh`) so an
invalid higher-priority changelog can never silently win.
- `ff7ddd3c9` — the promote e2e gate + `skip_e2e_gate` override +
removal of the automatic `full-e2e` label (pull-requests.yaml
release-path e2e now requires an explicit label; a post-open label now
actually starts the run).
- `936190d4b` — `website-docs` job at promote time behind a
FETCH_REF-support guard (see dependency below), with the
DO-NOT-MERGE-until-finalize contract and a pin-freshness pass before
`release-next`; `tags.yaml::update-website-docs` stays as the backstop.
- `01e1e7188` — the `persist-credentials: false` finalize fix (v1.6.0
root cause).
- `717c8860f` — `docs/release.md` coherence pass over the whole new
lifecycle.

### Dependency

The promote-time website docs job needs cozystack/website#631
(`FETCH_REF` + fail-loud fetch). Until that merges, the job's guard
detects the missing support and degrades to a ⚠️ line in the promote PR
body — promotion is unaffected and the tag-time backstop backfills, so
this PR is safe to merge first.

### Merge instructions

Please **merge-commit, do not squash**: the 13 commits are deliberately
atomic (the credential root-cause fix is separable from the feature
work) and squashing would collapse ~3k lines into one commit, hurting
bisectability.

### First live run

The e2e lane is verified by lint, contract tests and dry-traces, but has
not yet executed against a real tag; the intended first live run is a
manual `e2e-tag.yaml` dispatch against an existing published rc tag
before the gate is relied on for a real promotion.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added reusable changelog generation with validation, “present”
signaling, and safe artifact upload behavior (including RC-time
support).
* Added an E2E release-tag workflow with tag parsing, deterministic
sandboxing, and comprehensive report collection.
* Improved RC promotion with a stricter parse front-door, verified E2E
green gate (optional bypass), and earlier parallel changelog/asset
staging.
* **Documentation**
* Refreshed release-process docs to match the updated RC-to-stable,
changelog, and website-docs timing rules.
* **Bug Fixes**
* Improved PR finalization to source release bodies from verified
changelogs and warn/fallback on missing or empty content.
* Tightened label gating so E2E runs only under the intended conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-29 17:14:29 +05:00
Andrei Kvapil 6cb579338b fix(nightly): rewrite an image host that is the whole scalar value 
The nightly host rewrite is a literal "<src-registry>/" substring
replace, so it only reaches a host sitting contiguously in front of the
repository. kubeovn keeps its host in global.registry.address with the
bare repository in a sibling key: no trailing slash, no match. The
image was mirrored to the public registry and the tree kept pointing at
the private build registry.

The gap was dormant while kubeovn's image came from a separate
repository, because the baked ref carried the public host and the
ownership filter skipped it. Building the image in-tree stamps the
build-registry host into that key and turns it live, so fix it here.

Add an end-of-line-anchored expression that fires only when the source
registry is the complete value; a contiguous ref continues past the
host, so the two expressions cannot both fire on one reference.

keycloak-operator splits at a different boundary and stays unfixed --
neither of its keys holds the source registry as a whole. Documented,
and dormant while no CI path rebuilds it.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-28 18:07:34 +02:00
Andrei Kvapil 8b461f0c0b feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins (#3378) 
## What this PR does

Cozystack builds and publishes `ghcr.io/cozystack/cozystack/grafana` — a
rebuilt Grafana image — and uses it by default in the monitoring stack.
Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a
redistributor of an AGPL Grafana build. This is a governance concern for
CNCF Incubation (there is no precedent for a CNCF project redistributing
an AGPL binary).

The only reason the image was ever rebuilt was to bundle the
VictoriaLogs datasource plugin, which was not in the official Grafana
plugin catalog at the time. It is now published there as a signed plugin
(`victoriametrics-logs-datasource`), so the rebuild is no longer
necessary.

This PR moves the deployment to the upstream `grafana/grafana` image
(digest-pinned, routed through `cozy-lib.image` so mirrored-registry
installs keep working) and installs the plugins at container startup via
`GF_INSTALL_PLUGINS`, with versions pinned:

- `victoriametrics-logs-datasource 0.29.0` — the VictoriaLogs datasource
(signed; the reason the image was rebuilt)
- `natel-discrete-panel 0.1.1`, `grafana-worldmap-panel 1.0.6`,
`marcusolsson-dynamictext-panel 6.2.0` — reproduce the panel set the
removed Dockerfile vendored

**Air-gapped installs keep working.** The plugin zips are not downloaded
from the internet at pod startup: they are mirrored from the official
catalog into the existing `grafana-dashboards` image at build time and
served by that in-cluster static file server, next to the dashboards it
already serves. `GF_INSTALL_PLUGINS` points at
`http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…`, so the
runtime dependency stays inside the cluster and the artifact is mirrored
into private registries together with every other first-party image. A
new `allow-to-grafana-dashboards` CiliumNetworkPolicy (mirroring
`allow-to-keycloak`) opens tenant egress to that server, since tenant
egress is default-deny.

As a result, `ghcr.io/cozystack/cozystack/grafana` is no longer built or
published: the Dockerfile, the pinned image tag, the `image` targets in
both monitoring Makefiles, and the build step in the root Makefile are
removed.

Notes:

- The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored
plugin zips are Apache-2.0 licensed and are served unmodified from the
official catalog builds (signatures intact).
- `allow_loading_unsigned_plugins` and the custom
`/var/lib/grafana-plugins` path are dropped: the catalog datasource is
signed, and plugins install into the writable default `GF_PATHS_PLUGINS`
(`/var/lib/grafana/plugins`).
- The VictoriaLogs datasource provisioning (the `GrafanaDatasource` CRs
pointing at `vlselect`) is unchanged.
- Plugins are re-installed on every pod start from the in-cluster server
(no PVC involved, ~77 MB per start per replica over the cluster
network); a failed download fails the pod loudly instead of silently
dropping the datasource.
- `natel-discrete-panel` and `grafana-worldmap-panel` are Angular
plugins; Angular is disabled by default in Grafana 11, so they do not
load — this is unchanged from the current rebuilt image (same Grafana
11.6.15, same config). They are kept for parity and can be dropped as a
follow-up.

### How this was verified

- `helm unittest` across the repo passes, including new regression
tests: the monitoring chart asserts the exact upstream image reference
(with and without `images-registry` set) and the exact
`GF_INSTALL_PLUGINS` value; the tenant chart asserts the new egress
policy.
- Local end-to-end of the full chain in Docker: built the modified
`grafana-dashboards` image, served it on a shared network, and started
upstream `grafana/grafana:11.6.15` with the rendered
`GF_INSTALL_PLUGINS` value. All four plugins download from the
in-cluster-style URL (no internet egress from Grafana), and `GET
/api/plugins` reports `victoriametrics-logs-datasource` v0.29.0 as
`enabled: true` with `signature: valid`.

### Screenshots

Not applicable — no UI change. Grafana renders the same datasources and
panels; only the image source and plugin delivery mechanism change.

### Downstream repositories

Walked the trigger map against the diff. This is an internal change to
how the monitoring stack sources the Grafana image and its plugins
(`packages/system/monitoring`, `packages/system/grafana-operator`,
`packages/apps/tenant` network policy). No package is added, renamed or
removed; no `values.schema.json`, `ApplicationDefinition`, platform
values, release asset, shared `hack/` tooling or `package.mk` contract
changes; the added `grafana.image` key is an internal system-chart
value, not a user-facing app field. No downstream repository is
affected.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Grafana in monitoring is now based on a digest-pinned upstream image
for more consistent deployments.
* Required Grafana plugins are installed automatically at startup via
`GF_INSTALL_PLUGINS`; unsigned-plugin configuration was removed.
* Tenant networking now includes a new egress rule specifically for
Grafana dashboards.
* Grafana dashboards image now includes required plugin assets prepared
during image build.
* **Tests**
* Added/updated checks covering digest pinning, mirrored-registry
behavior, plugin-installation behavior, and the new Grafana dashboards
network policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-28 18:06:57 +02:00
Andrei Kvapil 330acfe6e2 fix(e2e): widen tenant node-join deadline to 18m 
The kubernetes-latest and kubernetes-previous suites flaked on the
worker node-join wait under host load. In run 30260770694 both failed at
exactly 12m with zero tenant Nodes registered and the tenant cilium HR
still mid-install: a transient drbd.linbit.com/lost-quorum taint delayed
the worker DataVolume imports, so the KubeVirt VMIs were only created ~2m
into this wait and the guests then had ~10m to import Talos, boot,
register a kubelet and let cilium turn the nodes Ready.

This single deadline absorbs the entire worker bring-up (the
machinedeployment replicas=2 gate clears before the VMIs even exist), so
under storage pressure 12m is too tight. A less-loaded fleet run passed
both suites unchanged, confirming load-induced slowness rather than a
stuck bring-up. Widen the deadline to 18m, which restores margin and
still sits well inside the 40m chainsaw step timeout.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-28 17:48:40 +02:00
Andrei Kvapil ad35afa534 fix(e2e): disable guest fsync on ephemeral CI VM disks 
The e2e QEMU guests boot with three raw virtio -drive lines that carry
no cache= option, so QEMU defaults to cache=writeback and honors every
guest fsync. The management-cluster Talos etcd fsyncs on each Raft
commit, while containerd unpacks dozens of images and CDI imports
multi-GB VM disks onto the same host-backed raw files during the
VM-heavy suites. The fsync storm serializes etcd's writes into
multi-second stalls, the mgmt apiserver times out lease PUTs, and many
operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji)
lose leadership at once — failing whichever chainsaw suite is
mid-flight, a different one each run.

Add cache=unsafe to each guest drive so guest flushes become no-ops
against the host page cache, removing the fsync-gated etcd stalls. This
is safe only because these CI VMs are ephemeral and destroyed every
run; never use it for real data.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-28 17:48:40 +02:00
Andrei Kvapil b22b7b2431 test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race (#3430) 
## What this PR does

The split-horizon-routing-proof step ran two external-dns probes as bare
Pods and polled them by fixed name. When the Cilium host-scope IPAM plus
Gateway API restore race (cilium/cilium#46799) hands a probe the
per-node reserved:ingress IP, the sandbox is rejected with "IP already
in use" and the e2e leak-healer deletes the wedged pod, expecting a
controller to reschedule it onto a free IP. A bare Pod has no
controller, so that delete was terminal: the probe vanished and the step
failed on a NotFound pod. It surfaces as an intermittent kuberture E2E
failure with a `FailedCreatePodSandBox` event on a probe pod.

This runs the probes as Jobs. The Job re-creates its pod on a fresh IP
after the healer deletes it, so the proof survives the race. Each Job is
asserted to reach Complete instead of polling a fixed pod name, and its
logs are read by the job-name label. The failure path also gains a catch
podLogs selector for the probe pods, so their stdout is captured when a
probe fails for a different reason.

### Screenshots

None. No UI changes.

### Downstream repositories

The diff touches only the kuberture E2E suite under
`hack/e2e-chainsaw/kuberture/`. No package, values schema, chart, CRD or
platform manifest changed. Nothing in the downstream trigger map is
reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved end-to-end test reliability by hardening the split-horizon
routing proof against probe workload churn.
* Strengthened route isolation validation between public and internal
probes.
* Enhanced failure diagnostics by capturing logs from the completed
probe Jobs.
* **Testing**
* Updated DNS EDNS probes to run as Kubernetes Jobs (instead of bare
Pods) to support retries and rescheduling.
* Added explicit assertions that both public and internal probe Jobs
reach **Complete** before running hostname/route isolation checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-28 17:43:26 +02:00
myasnikovdaniil 2c453778f1 fix(seaweedfs): make naming audit fail closed on kubectl and payload errors (#3436) 
## What this PR does

Fixes #3431.

`hack/seaweedfs-naming-audit.sh` was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet —
a failed namespace LIST meant zero namespaces walked, failed
secret/PVC/STS LISTs meant zero findings, all silenced by `2>/dev/null`
with no error handling anywhere in the script. The runbook uses this
output as the gate before PVC deletion, as post-deletion verification,
and as an upgrade precondition, so a transient API error could
green-light destroying data.

Every kubectl call now routes through a `run_kubectl` helper: a non-zero
exit prints a FATAL line naming the failed query and the code propagates
up the whole chain (enumerations restructured to capture-then-check,
since an `exit` inside `$(...)` dies with the subshell). By-name GETs
separate existence from extraction: a `--ignore-not-found -o name` probe
keeps legitimate absence (Helm-pruned revision, unbound PVC) a clean
skip, while everything else fails loudly — including a release Secret
that exists but has no decodable payload or decodes without a chart
name, which previously dropped the whole tenant from the audit.
Incomplete PV-age evidence now marks the generation incomplete and falls
to the safe "direction cannot be established" branch, so a failed GET
can no longer flip an OVERLAP verdict into a wrong Step-3 deletion
candidate. A successful audit prints the same bytes as before.

Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants (missing `.data.release`,
undecodable base64/gzip, chartless `{}`), and full-output golden diffs
proving the success paths stay byte-identical to the pre-change script.
25/25 green via `hack/cozytest.sh`.

### Release note

```release-note
fix(seaweedfs): the naming-audit script fails loudly when a kubectl query or a Helm release payload read fails, instead of printing an empty table indistinguishable from a clean fleet
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Audits now fail clearly (non-zero exit + fatal messaging) when
Kubernetes queries or Helm release data is unavailable or corrupt,
instead of producing incomplete results.
* Missing release revisions are skipped safely; missing/unreadable PV
age evidence no longer leads to incorrect deletion candidates, and MIXED
now reports conservative “direction cannot be established” when evidence
is incomplete.
  * Non-SeaweedFS releases remain silently ignored.

* **Tests**
* Added end-to-end acceptance coverage for fail-closed behavior and
byte-for-byte stable output across key naming scenarios.

* **Documentation**
* Updated the runbook to require relying on the audit script exit code
before proceeding, with a dedicated MIXED re-check step.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-28 16:56:05 +05:00
myasnikovdaniil 49222768ec fix(release): make promote-retag digest verification media-type-agnostic (#3435) 
## What this PR does

The finalize post-copy verification runs `skopeo inspect --format
'{{.Digest}}'` on every freshly retagged stable ref. For OCI artifacts
that command prints nothing and exits non-zero — `cozystack-packages` is
a flux artifact with `config.mediaType:
application/vnd.cncf.flux.config.v1+json` — so the v1.6.0 finalize (run
29932878820) aborted right after both copies of that artifact had
already succeeded, leaving 32 of 43 repos without stable tags and
skipping the cozy-installer publish. `--dry-run` never reaches this
branch, which is why rehearsals did not catch it.

This PR computes the digest as sha256 over the raw manifest instead
(`skopeo inspect --raw`), which is how registries define the manifest
digest and works identically for container images and OCI artifacts. The
helper gates the hash on skopeo's exit status via a temp file rather
than a pipeline, so a missing tag still resolves to an empty string (the
pre-check reads that as "tag absent, proceed to copy") and never to the
empty-input hash `sha256:e3b0c442…`, and never aborts under `set -eu`.

Verified against live GHCR: the raw-manifest hash of
`cozystack-packages:v1.6.0` matches the digest finalize expected
(`bf68208730860fa8…`), and the old and new methods agree byte-for-byte
on regular multi-arch images (`cozystack-operator`, `kubeovn`,
`cozystack-controller`, `kubevirt-cloud-provider`). New cozytest cases
cover the OCI-artifact digest resolution, a post-copy mismatch, and the
missing-tag path; the mock skopeo asserts `--raw`, so the suite fails
against the old code.

### Release note

```release-note
fix(release): promote's post-copy digest verification no longer breaks on OCI artifacts (cozystack-packages), which aborted the v1.6.0 finalize mid-retag with 32 of 43 repos left untagged
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved OCI artifact retagging by deriving stable manifest digests
from raw manifests to ensure accurate “already correct” and mismatch
detection.
* Prevented incorrect skips when the stable tag points to an unexpected
digest.
* Added safer handling for missing or unknown stable tags and clearer
failure behavior for unexpected empty manifest responses and transient
registry errors.
* **Tests**
* Expanded promotion/retag test coverage for digest-based resolution,
post-copy verification mismatches, missing-tag behavior, and transient
registry failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-28 16:55:42 +05:00
Aleksei Sviridkin 436a19df1b test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race 
The split-horizon-routing-proof step ran two external-dns probes as bare
Pods and polled them by fixed name. When the Cilium host-scope IPAM +
Gateway API restore race (cilium/cilium#46799) hands a probe the per-node
reserved:ingress IP, the sandbox is rejected with "IP already in use" and
the e2e leak-healer deletes the wedged pod, expecting a controller to
reschedule it. A bare Pod has no controller, so the delete was terminal:
the probe vanished and the step failed polling a NotFound pod.

Run the probes as Jobs so the pod is re-created on a fresh IP after the
healer's delete, assert each Job reaches Complete instead of polling a
fixed Pod name, and read the logs by job-name label.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-27 22:59:49 +03:00
Aleksei Sviridkin a7ce4371c2 docs(agents): correct the chainsaw podLogs capability note 
The e2e conventions, the previous-logs capture script, and the global
chainsaw catch all claimed the v0.2.15 podLogs collector exposes only
container/tail/selector. It also accepts name, namespace, timeout and
the cluster fields; only the previous option is genuinely missing. The
wrong list steers catch authors toward imperative scripts for
cross-namespace log capture that podLogs handles declaratively.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-27 22:48:25 +03:00
Aleksei Sviridkin 824a799c6a test(e2e): capture redis convergence diagnostics in harbor suite 
When harbor's redis sentinels never converge, the suite times out on
the HelmRelease assert with no evidence of why: the catch pod-log
selector matches neither the redis-operator (it runs in another
namespace) nor the rfr-/rfs- pods, and the RedisFailover CRD ships no
status subresource to describe-assert on. Add a RedisFailover describe
plus a best-effort script that dumps the operator log and the
rfr-/rfs- pod states and logs, so the stalled reconcile gate is
visible in the failure output.

Also widen the harbor-test HR wait from 6m to 10m: readiness is gated
on the operator electing a redis master, which normally takes 2-3m but
runs long on a CPU-committed runner. This matches the
harbor-test-system budget in the same suite and the justified longer
waits allowed by the e2e conventions.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-27 22:48:24 +03:00
Andrei Kvapil a5a0449c9b test(hack): re-point the shape-2 image-ref exemplar at grafana-dashboards 
The promote-retag and image-refs enumeration tests used the monitoring
grafana.tag as their canonical example of a reference living outside a
values.yaml. That file is gone now that the Grafana rebuild is dropped,
so use the grafana-dashboards tag — the same storage shape, still in
the tree — as the exemplar.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-27 13:08:50 +02:00
Myasnikov Daniil 1eded27900 test(release): scope the trim pin to the parse step, fix two stale references 
Three review nits, all verified against the tree, plus a tighter scope than the
first attempt at them.

The behaviour suite asserted "promote-rc does not trim the rc tag" against the
whole file. promote-rc.yaml has no .trim() today, so it passed, but any unrelated
.trim() added anywhere in the workflow would have failed a test about tag policy.
Scoping it to the parse JOB was not enough either: that job is ~180 lines, most
of them the e2e gate's github-script, whose string handling has nothing to do
with tag policy. Both greps are now scoped to the parse STEP, and run through a
code_lines filter copied from the contract suite so a YAML comment mentioning
.trim() cannot fail them.

Mutation-checked in both directions: a .trim() injected into the parse step fails
the test, one injected into the e2e-gate step of the same job does not.

- The Step 2 checklist entry still said "must be main" while Step 2 itself now
  lists three valid starting points (main, detached at the release tag, detached
  at the rc tag being promoted). The checklist is what an agent ticks off, so it
  was the half that mattered; it now points at Step 2 rather than re-encoding the
  count, so a fourth configuration cannot desync it. The heading said "Checking
  current branch", which two of the three states do not have.
- hack/testdata/needs-semantics.yaml named release-changelog-contract.bats as its
  consumer; the only consumer is release-changelog-behaviour.bats.

50/50 tests green across the three suites (11 + 24 + 15).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-27 14:03:49 +05:00
Myasnikov Daniil a2667c58ca fix(ci): grant rc-e2e the checks:write ceiling e2e-tag.yaml declares 
A caller job's permissions are the ceiling for every job in the workflow it
calls, and GitHub validates that ceiling when it CREATES the run — before the
calling job's `if:` is evaluated, let alone before a step runs. e2e-tag.yaml's
e2e job declares `checks: write` for its workflow_dispatch-only breakpoint, so
granting only `contents: read` here made the run fail validation with
"requesting 'checks: write', but is only allowed 'checks: none'".

A startup failure takes the whole run with it, and an `if:` cannot gate it, so
this was not limited to rc cuts: EVERY tag push — rc and stable, including the
one finalize pushes to publish a release — would have produced no build, no
draft release and no staging branch.

The comment claimed the opposite: that because workflow_call can never reach the
breakpoint branch, the grant was unnecessary. Runtime reachability has no bearing
on a static check. e2e-tag.yaml now states the contract for its callers, next to
the permission block that was being read as "contents: read is all this needs".

hack/promote-gate-contract.bats pins both halves of the pair, and separately pins
the `E2E <tag> (full suite)` job name the gate correlates evidence by. Neither is
reachable from any PR lane — tags.yaml runs only on tag pushes — so a regression
here would otherwise surface at the next release. Mutation-checked: dropping the
grant fails the first pin, renaming the e2e job fails the second.

changelog-rc.yaml's reusable-workflow call now pins `contents: read` explicitly,
matching promote-rc.yaml's identical call. The outcome is the same either way,
but the docs describe an omitted block as the repository default rather than the
workflow's own top-level block, and this is the file class where that ambiguity
just cost a release lane.

Reported by CodeRabbit on #3449; blast radius and the missing pins found by an
adversarial re-check.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-27 14:03:49 +05:00
Myasnikov Daniil ae39475789 fix(release): prove a stable tag is absent before copying over it 
Review notes from #3435, and a correction to my first attempt at them.

manifest_digest gated on skopeo`s exit status alone, so a zero exit with no
bytes hashed the empty string to sha256:e3b0c442... — a digest that looks real
and belongs to no manifest. The first fix simply required non-empty bytes, which
was worse than it looked: the caller turns an empty result into "not published,
safe to copy", so a registry answering 200 with an empty body went from an
accidental refusal (the bogus digest mismatched, so the write-once check
refused) to an actual copy over whatever the tag really held. Verified against
real skopeo 1.23: a 200 with Content-Length 0 does exit 0 with no stdout.

The helper is now explicitly three-state. A digest, or empty output for a tag the
registry PROVED absent, or non-zero for anything indeterminate — which under
set -e aborts the promotion before it writes. Absence has to be reported as
"manifest unknown"; 429, 5xx, auth and network failures all exit non-zero with
no bytes too, and reading those as "unpublished" is the same fail-open one level
down. finalize retags ~42 refs with no retry wrapper, so a single rate-limit was
enough to reach that path. skopeo`s stderr is captured instead of discarded, so
the diagnostics now name the real failure.

Tests: the write-once refusal branch the review found untested (an existing
stable tag at a different digest must be refused before any write), the empty-
body case (refuses, writes nothing, and the empty-input hash never appears in
the diagnostics), and a 429 that must not read as an absent tag. The mock now
emits realistic registry errors, since a bare non-zero exit is no longer proof
of absence. Both new guards are mutation-checked: accepting any non-zero exit as
absence fails the 429 test, dropping the empty-bytes check fails the empty-body
test.

11/11 tests green, hack/image-pin-consistency.bats still 2/2, sh -n clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-27 13:56:45 +05:00
Myasnikov Daniil e02adad38f fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls 
Review findings from #3436, plus what an adversarial re-check of the first
attempt at them turned up.

The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.

Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.

The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.

Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.

The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.

The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.

29/29 tests green; sh -n clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-27 13:52:59 +05:00
Myasnikov Daniil ee838261b1 test(release): pin the promote gate with POSIX grep, not ripgrep 
The rc-E2E gate contract used ripgrep for every pin. The unit-test runner
has no ripgrep, so `code_lines()` hit "rg: not found", the `|| true` on it
swallowed the 127, and every pin degraded to "0 matches" — the suite failed
on the first assertion with no hint that the tool, not the workflow, was
missing. No other hack/*.bats uses ripgrep.

Swap all pins to POSIX grep and make the filter fail closed: grep exits 1
when nothing is selected (legitimate for an empty block) and 2 on an actual
error, so only the latter propagates instead of silently yielding no lines.

Verified 13/13 green both normally and with `rg`/`fd` shimmed out of PATH to
reproduce the runner, plus the eleven suites the aborted CI run never reached
(all green, zero shim hits).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-27 12:02:53 +05:00
Myasnikov Daniil 01e1e7188e ci(release): stop finalize persisting credentials so the tag push triggers tags.yaml 
pull-requests-release.yaml's finalize job checked out the repo with the default
persist-credentials, leaving GITHUB_TOKEN embedded as http.extraheader. Every
later tag/branch push re-injects the CI app token via `git remote set-url`, but a
persisted extraheader silently wins over the URL credential — so those pushes
authenticated as GITHUB_TOKEN. A GITHUB_TOKEN-authenticated push creates no
workflow run (GitHub anti-recursion), which is exactly why v1.6.0's stable tag
never triggered tags.yaml: the generate-changelog and update-website-docs
backstops never ran.

Set persist-credentials: false on the finalize checkout, so the set-url app token
actually authenticates the pushes — the same handling promote-rc.yaml already
uses. Audited every git/gh step in the job: both tag steps set-url the app token
before pushing, and Ensure-maintenance-branch / Publish-draft-release / the
retag+chart steps use the app token via github-script or a separate registry
login — none depends on the persisted GITHUB_TOKEN extraheader.

Pin it in hack/promote-gate-contract.bats.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:26 +05:00
Myasnikov Daniil 936190d4b6 ci(release): open the website docs PR at promote time 
The cozystack/website 'update managed apps reference' PR used to be opened only
at tag time by tags.yaml::update-website-docs — after finalize pushed the stable
tag, i.e. after the release was already published (too late to review) — and it
fetched README content from that stable tag (BRANCH=RELEASE_TAG), which finalize
only creates post-merge.

Add a promote-time website-docs job to promote-rc.yaml (needs [parse, promote])
that opens the same update-docs-vX.Y.Z PR from the release-X.Y.Z staging branch
via the website Makefile's FETCH_REF variable, so the docs are reviewable
alongside the promote PR while the stable tag is still unborn. It refreshes the
trunk pins from the staging ref before release-next so a new minor inherits fresh
Talos/cozystack pins (the v1.6 talos-pin incident class), then runs update-all
with FETCH_REF pinned to the staging branch.

The job is non-blocking: a FETCH_REF-support guard fails loudly (warn + exit 1)
when cozystack/website main predates the FETCH_REF Makefile variable — otherwise
make would silently ignore the unknown FETCH_REF override, fall back to the
not-yet-created stable tag, and update_apps.sh would warn-and-continue into STUB
docs. Any failure surfaces as a website-docs warning line in the promote PR body
(plumbed through open-pr via needs.website-docs.result) and never blocks
promotion; the tag-time job stays as the backstop and no-ops when promote already
did the work.

MERGE-TIMING CONTRACT: the website PR must NOT be merged before finalize
publishes the release — doing so flips the site's latest-version pointer to a
version whose docs and openapi are not on GitHub yet, 404ing every version-pin.
The website PR body and the promote PR body both say so in bold.

Pin the contract in hack/promote-gate-contract.bats (needs shape, FETCH_REF
invocation, the guard, non-persisted checkout credentials, the DO-NOT-MERGE body
wording, the promote-PR status line, and the tags.yaml backstop comment) and
relax the release-changelog-contract needs pin to tolerate the appended
website-docs dependency.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:26 +05:00
Myasnikov Daniil ff7ddd3c9a ci(release): gate promotion on a green rc e2e and drop the automatic full-e2e label 
Promotion now assumes checks passed at rc time instead of re-running
them. At dispatch, before any staging happens, promote-rc verifies the
rc tag has a green full e2e via either evidence path: the rc's
tags.yaml push run containing a successful 'rc-e2e / E2E <tag> (full
suite)' workflow_call job, or a successful manual e2e-tag.yaml dispatch
correlated by its exact tag-bearing job name (immutable rc digests make
past evidence valid). No evidence fails the dispatch with both paths
and the remediation named. A skip_e2e_gate boolean input bypasses the
gate loudly for emergencies; the promote PR body always carries the
verification state (verified or bypassed), and 'verified' can only
originate in the gate's green branch.

The promote PR keeps the release label but no longer gets full-e2e
automatically: its tree is the rc's already-validated digests with only
tag strings rewritten, so pull-requests.yaml now runs e2e on the
release path only when a maintainer opts in by adding full-e2e. The
regular-PR path and the ordinary full-e2e forcing are untouched, and
the new labeled trigger makes a post-open label actually start the run.

A new promote-gate contract suite pins the gate's DAG position, the
override default, the label set, the body lines, and the label-gate
condition with comment-proof executable-line counts.

Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:26 +05:00
Myasnikov Daniil 2dfbeea776 ci(release): dispatch changelog generation at rc time 
Extract the promote-time changelog generation into a reusable workflow
(changelog-generate.yaml, workflow_call) and add a dispatchable wrapper
(changelog-rc.yaml) so a maintainer can generate the release changelog
the moment an rc goes green, while there is still time to review it.

The reusable core keeps promote-rc's observable behaviour: detached
rc-tag checkout, read-only app token for the AI step, the
validate-changelog.sh gate, non-blocking AI failure, and the same
changelog-vX.Y.Z artifact. .release-tooling is overlaid from
github.ref_name, which under workflow_call is the caller's dispatch ref
(and coincides with the reusable workflow's own ref for a local uses:),
so "current tooling against an old rc tag" is preserved. Secrets are
declared explicitly rather than inherited, so a caller that forgets one
fails at parse time.

The rc tag is parsed by a shared hack/parse-rc-tag.sh that REJECTS
whitespace outright (a stripped 'v1. 6.0-rc.2' would silently become a
different tag). promote-rc parses it ONCE in a cheap `parse` job that
both `promote` and `changelog` depend on — so they still run in parallel
with each other while consuming one validated tag, and a whitespace tag
fails the whole promotion rather than letting promote succeed while the
changelog job rejects the same input and the PR opens empty. The
reusable core mirrors the reject-whitespace policy inline because its
parse must run before any checkout.

changelog-rc.yaml verifies the rc staging branch release-X.Y.Z-rc.N
exists UP FRONT (in parse, which generate depends on), so a missing
branch fails in seconds rather than after a 30-minute AI run, then
commits the result there as docs/changelogs/vX.Y.Z.md idempotently
(compare-before-push).

promote-rc's changelog job is a thin uses: call. Source selection —
validate-then-fall-through across the rc tag tree, the rc staging branch
(the rc-time pickup), then AI generation, discarding any invalid source
with a loud warning so it never suppresses a valid lower-priority one —
lives in hack/select-changelog-source.sh, executed after the tooling
overlay. open-pr carries the picked-up file onto release-X.Y.Z and
finalize publishes it.

Contract and behaviour bats are repointed and extended: the shared
parser, the source-selection script, and promote's parse policy are all
exercised by execution on adversarial inputs; the remaining grep pins
match executable lines with exact counts so a comment or dead block
cannot satisfy them. docs/release.md lists the new workflow and the
rc-time entry point. actionlint and zizmor (default and pedantic) are
clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 18:16:16 +05:00
Myasnikov Daniil dd308644c8 ci(release): execute the release-changelog logic in tests, not just grep it 
The existing suite asserts on workflow YAML. That catches a rename or a
deleted `if:` and nothing else: it cannot tell whether the shell inside a
`run:` block does what the step name claims, and it stayed green while the
logic underneath was wrong — twice, in the two preceding rounds.

Add hack/release-changelog-behaviour.bats, which runs things for real.

The GitHub Actions semantics the whole non-blocking design rests on are
now verified by executing them. hack/testdata/needs-semantics.yaml mirrors
the promote / changelog / open-pr shapes with a fatal (non-AI) step in the
changelog job, and `act` runs it. Result: open-pr DOES run when a job it
needs has failed, `needs.changelog.result` is `failure`, and — the part
that could not be read off the YAML — a failed job's outputs come back
EMPTY, while a successful job's propagate. That empty-output behaviour is
precisely why open-pr re-establishes changelog presence from disk instead
of trusting needs.changelog.outputs. Deleting the `if:` from the fixture
makes open-pr get skipped, so the gate is demonstrably load-bearing.

The git logic moves into hack/changelog-preserve.sh and is exercised
against real bare remotes: a branch that exists, one that does not, one
with no changelog, one with an empty changelog, and the full
preserve/checkout -B/restore/force-push cycle promote actually performs.
The last of these directly reproduces the regression the preserve step
exists to prevent. An empty changelog is deliberately NOT preserved — it
would suppress generation and then publish as blank notes.

finalize's release-body selection runs under node across all three inputs
(good, whitespace-only, absent) and the resulting decision table is
compared as a whole.

Replace the shell overlay with a sparse second actions/checkout of the
dispatch ref at .release-tooling. The overlay had a bootstrap problem I
introduced: the script that fetches missing tooling was itself missing
from the rc tree, so it could never have worked for exactly the trees that
need it. actions/checkout comes from the runner, not the repo, so it has
no such circularity. .release-tooling is gitignored because promote runs
`git add -A`, and the contract suite pins that.

All 21 tests mutation-checked. Breaking changelog-preserve.sh, making it
return an empty file, removing the fixture's `if:`, dropping the tooling
checkout, and renaming its path each fail the suite and pass on restore.

Not covered, and stated in the suite header rather than implied: the AI
generation step, the real GitHub API calls, and whether the sparse
checkout resolves on GitHub's runners.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 11:34:17 +05:00
Myasnikov Daniil 34e45295fe ci(release): fix second-round review findings 
Two more review passes (Claude + Codex) on the previous commit. Codex
found the one that would have broken the very next promotion.

The agent reads its own instructions from the working tree, which is the
rc tag. Every rc cut before this change carries the old §2 of
docs/agents/changelog.md — two valid configurations, "stop" on anything
else — and a detached rc-tag checkout is exactly that "anything else". So
promoting v1.6.0-rc.4 would have presented a compliant agent with
instructions telling it to refuse, silently, producing no changelog on
every promotion of an existing rc. Verified against the tag. The same
applies to hack/validate-changelog.sh, absent from those trees entirely,
which would have failed the Verify step with "command not found" and, in
open-pr, taken the promote PR down with it. Both are now overlaid from the
dispatch ref, best-effort, in both jobs.

The structural validation added last round rejected every PATCH
changelog. It asserted a `# Cozystack vX.Y.Z` header and a 20-line floor;
patch releases use `# vX.Y.Z (date)` and v1.5.1 is a complete 19-line
release. Good AI output would have been discarded and the promote PR
mislabelled "generation failed". Measured across the tree: the header form
holds for 23 of 115 changelogs, so it was never an invariant.

The predicates now live in hack/validate-changelog.sh, tested directly
against real changelogs rather than through YAML greps — the reason the
defect shipped is that the suite pinned finalize's guard but nothing on
the promote-side validator, and stayed green when the validator was
mutated away. It accepts both header conventions, has no line floor, and
checks the version in the header, the release-link comment and the compare
link, so a changelog generated for the wrong version is caught. All 16
current-era changelogs pass; whitespace, empty, truncated and
wrong-version inputs are rejected.

Also fixed:

- A fatal artifact download in open-pr would abort the job before the PR
  step — the same wedge as a skipped job, one step lower. Download is now
  best-effort and the commit and PR body key off what is on disk, not an
  upstream job output that says nothing about whether the transfer worked.
- A pre-existing changelog (hand-written, or preserved across the
  force-push) reached the PR without ever being validated, was advertised
  with ✅ and published verbatim. It now goes through the same validator,
  as does the tags.yaml backstop.
- `gh pr view <branch>` matches closed PRs, so re-dispatching after
  closing an abandoned promote PR edited the dead PR and opened nothing.
  Branch on state explicitly.
- `gh pr edit --body` unconditionally overwrote the PR body, destroying
  the upstream breaking-change notes docs/release.md instructs reviewers
  to paste there. Refresh only while the body is still the generated one.
- open-pr consumed raw dispatch input while its siblings used the
  validated parse; the changelog job's bash parse did not trim, so an
  input with a stray space passed promote and failed there. Both fixed.
- The prompt and §2 said "trailing release-link comment"; it is the first
  thing in every changelog.

Tests grow to 16, each mutation-checked: dropping the overlay, making the
download fatal, reverting the existing-changelog check to `-s`, and
removing the backstop validator each fail the suite.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 11:34:17 +05:00
Myasnikov Daniil f49d54a689 ci(release): fix review findings; keep website docs at tag time 
Dual-model review (Claude + Codex) of the previous two commits found seven
substantive defects, two of them destructive. Both models independently
reached three of them; the split is noted per item.

Descope the website half (Claude, verified by reproduction). The docs job
was moved to promote time, but the website Makefile pins the fetch ref
with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line
BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that
cannot exist at promote time, since promote's own validate step hard-fails
if it does. update_apps.sh logs the failed fetch and exits 0 AFTER
truncating each destination file, so every app doc would become a stub,
`changed=true` would open the PR, and the new auto-merge would land it
unreviewed. Not fixable from this repo. update-website-docs returns to
tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize
auto-merge is dropped. Opening it earlier needs a cozystack/website change
first and is left as a follow-up. This also removes the unordered-sibling
race Codex found, where the release PR could merge before the docs job
opened its PR, leaving finalize nothing to merge and no retry.

Fix the non-blocking guarantee (both models). Only the AI step carried
continue-on-error; the changelog JOB did not, so a flaky `npm i -g
@github/copilot` failed it and `needs` skipped open-pr entirely — staging
branch pushed, release drafted, no promote PR. Gate open-pr on promote
alone.

Generate from a checkout the contract permits (both models). The job
checked out the rc staging branch, which docs/agents/changelog.md §2
rejects — a compliant agent stops and writes nothing, so the normal path
produced no changelog at all. Check out the rc tag detached and document
that configuration in §2. The checkout ref now comes from the validated
parse output rather than raw dispatch input.

Port, don't regenerate, in the backstop (both models). For a patch release
promoted onto release-X.Y the changelog never reaches main, so the
backstop fired and ran a second AI pass; once its PR merged,
update-releasenotes.yaml overwrote the published body with unreviewed
text. Detect the file at the tag and port it verbatim; the AI runs only
when it is absent from both main and the tag.

Survive re-dispatch (Codex). promote force-pushes the staging branch,
erasing a hand-written changelog — the documented recovery when generation
fails. Preserve an existing one across the rebuild, and stop open-pr
overwriting it with generated output.

Validate structurally, not by size (Codex). A truncated or whitespace-only
AI run passed `[ -s ]`, was committed, advertised as valid in the PR body,
and published verbatim as release notes. Assert the header, the compare
link and a plausible line count; finalize additionally refuses
whitespace-only content rather than blanking the notes.

Add hack/release-changelog-contract.bats — nine tests pinning each link in
the chain, mutation-checked: removing open-pr's if:, breaking finalize's
body read, and restoring paths-ignore each fail the suite.

Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which
described the old flow throughout.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-24 11:34:16 +05:00
Aleksei Sviridkin 6ecb36ffc7 feat(cozystack-controller): extract application CA into a key-free tenant Secret (#3407) 
## What this PR does

Adds an engine-agnostic controller that publishes a key-free
`<release>.tenant-ca` Secret (only `ca.crt`) for a managed application,
so a tenant can verify the application's TLS without ever seeing a
private key. The source is declared, not guessed: a chart renders a
namespaced `TenantProjection` sentinel (group `internal.cozystack.io`)
naming the CA Secret to lift; the controller watches the sentinel,
extracts `ca.crt`, and writes the projection owner-referenced to the
sentinel, so garbage collection is native. The projection carries the
`internal.cozystack.io/tenant-ca` label the lineage webhook turns into
tenant visibility. Postgres is the first consumer.

The trust boundary is RBAC: no tenant role grants any verb on
`internal.cozystack.io`, so a tenant cannot forge a sentinel. A
companion `ValidatingAdmissionPolicy` that pins the writer to
helm-controller ships separately as defence in depth.

This is the controller-and-CRD slice, split out from #3299 for focused
review. It is the only piece required for the feature to work; the
render-time helper, the writer policy, the capability-gate, and the
release-prefix validation land as their own PRs.

### Downstream repositories

Walked the trigger map against the diff. This adds an internal CRD and
controller and a sentinel to the Postgres chart; no app is added or
renamed and no `values.schema.json` changes, so the typed provider and
Ansible are unaffected. The one to confirm is the Postgres README edit —
if the website mirrors it, it needs a refresh.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:

### Release note

```release-note
feat(cozystack-controller): publish a key-free per-application CA trust anchor (`<release>.tenant-ca`) that tenants read to verify managed-application TLS, declared through a new `TenantProjection` CRD. Postgres is the first consumer.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added tenant-facing CA certificate projections for PostgreSQL
deployments.
* Tenant secrets now provide only the CA certificate, excluding private
keys.
  * Added support for configuring and monitoring tenant CA projections.

* **Documentation**
* Updated PostgreSQL TLS instructions to retrieve certificates through
tenant secrets.

* **Bug Fixes**
* Improved protection against invalid certificates, secret collisions,
and accidental exposure of private key material.
* Added validation to ensure projected certificates remain synchronized
with their sources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-23 14:14:28 +03:00
Andrey Kolkov 8aeedde088 feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383) 
## What this PR does

Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes #2774.

Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.

### Changes

- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).

### Verification

- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change

```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-23 14:14:58 +04:00
Myasnikov Daniil 3dc6cbd4f4 fix(seaweedfs): make naming audit fail closed on kubectl and payload errors 
hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet
(namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST
failures = zero findings, all under 2>/dev/null with no error handling).
The runbook uses this output as the gate before PVC deletion, as
post-deletion verification, and as an upgrade precondition, so a
transient API error could green-light destroying data (#3431).

Route every kubectl call through a run_kubectl helper: non-zero exit
prints a FATAL line naming the failed query and propagates the code up
the whole chain (enumerations restructured to capture-then-check, since
an exit inside $(...) dies with the subshell). By-name GETs distinguish
legitimate absence from real errors via --ignore-not-found on a
separate existence check; a Secret that exists but has no decodable
release payload, or one that decodes without a chart name, is corrupt
state and fails loudly instead of silently dropping the tenant.
Incomplete PV-age evidence now marks the generation incomplete and
falls to the safe no-direction branch, so a failed GET can no longer
flip OVERLAP into a wrong Step-3 deletion candidate. A successful run
with zero findings still prints the same bytes as before.

Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants, and full-output golden
diffs for the success paths (verified byte-identical to the pre-change
script). 25/25 green.

Fixes #3431

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 23:25:01 +05:00
Myasnikov Daniil 5d8dd51cab fix(release): make promote-retag digest verification media-type-agnostic 
skopeo inspect --format '{{.Digest}}' fails on OCI artifacts: for
cozystack-packages (config.mediaType
application/vnd.cncf.flux.config.v1+json) it prints nothing and exits
nonzero, so the v1.6.0 finalize aborted its post-copy verification right
after both copies had already succeeded, leaving 32 of 43 repos without
stable tags and skipping the cozy-installer publish. --dry-run never
executes this branch, so rehearsals could not catch it.

Compute the digest as sha256 of the raw manifest instead, which is how
registries define it and works for any media type. The helper gates the
hash on skopeo's exit status via a temp file rather than a pipeline:
a missing tag still yields an empty string (pre-check reads that as
"tag absent, proceed to copy"), never the empty-input hash
sha256:e3b0c442..., and never an abort under set -eu.

Verified against live GHCR: the raw-manifest hash of
cozystack-packages:v1.6.0 matches the digest finalize expected
(bf68208730860fa8...), and agrees byte-for-byte with the old method on
regular multi-arch images. New cozytest cases cover the OCI-artifact
digest, a post-copy mismatch, and the missing-tag path.

Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 22:31:00 +05:00
Myasnikov Daniil 6a41d65f16 test(e2e): avoid ripgrep dependency in diagnostics tests 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:27:02 +05:00
Myasnikov Daniil 4e0cfd994d test(e2e): render COSI catch output as YAML 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:27:02 +05:00
Myasnikov Daniil ad1adbf785 test(e2e): fix diagnostic collector blind spots 
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-22 17:27:02 +05:00
myasnikovdaniil 55195ded0c test(e2e): capture previous container logs and stop killing the snapshot (#3346) 
## What this PR does

Root-causing a recent E2E failure was nearly blocked because the
diagnostics captured on failure had collected almost nothing. Two
independent mechanisms were responsible, and both had been silently in
place for a long time.

**The failing container's previous logs were never captured.** The pod
had crash-looped, so the decisive evidence lived in the first container
instance, reachable only through `kubectl logs --previous`. Chainsaw's
`podLogs` action has no `previous` field, so this could not be expressed
per-suite; the conclusion had to be reconstructed from indirect evidence
instead. A new capture runs from the global catch and from the BATS
failure hook, selecting containers at runtime by `restartCount > 0`
across both init and regular containers, so a healthy run stays quiet
and future suites are covered without per-suite wiring.

**The catch was structurally guaranteed to be killed.** A Chainsaw
`script` op is bounded by `timeouts.exec`, which is 5m, while the
catch's own inner budgets summed to roughly 960s. It reported `SCRIPT |
ERROR | signal: killed` and produced nothing — and its second leg, the
data-plane capture, had therefore never run at all. The kill signature
was itself misleading: `timeout` exits 124 and cannot produce that
message, which comes from Go's `exec.Cmd` killing the process group.
Each catch op now carries an explicit timeout above the budgets it
contains, so the inner one fires and leaves a partial, usable artifact
instead of losing everything.

**The snapshot was truncated at 60 seconds.** `crust-gather --duration`
defaults to 60s and was never passed, so every snapshot ever taken was
cut short, with nothing in the artifact saying so. The API discovery it
performs first is not covered by that budget at all, which is where the
outer wall-clock was going on a cluster with a degraded aggregated
APIService — precisely the state a failed run is in. Both budgets are
now set explicitly, and the collector's output goes to a log inside the
snapshot with a completeness verdict, so "is this snapshot complete?" is
answerable from the artifact.

Failure paths are reported rather than swallowed: an unreachable API
says so instead of implying nothing had restarted, a truncated capture
is kept and marked, and every dropped item is logged, since silent
truncation reads as full coverage.

### What is not verified here

The mechanism claims are read from the pinned versions of Chainsaw and
crust-gather. Everything cluster-facing is not verified by this PR: the
go-template's output shape against a live API server, `--previous`
returning the first instance for a real crash-looper, and whether the
new collection budgets suffice on a degraded cluster. The bats suite
covers the selection helpers only, and those cases are mutation-tested.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Diagnostics**
* Improved end-to-end failure evidence by capturing “previous instance”
logs for crash-looping pods (including restarted init containers) and
prioritizing relevant namespaces.
* Added time-bounded diagnostics collection with explicit outer
timeouts; collector output is saved to per-snapshot log files.
* Classifies collector outcomes as complete, timeout-truncated, or
failed, while preserving the original test failure signal.
* **Documentation**
* Expanded E2E failure-diagnostics guidance with new timeout and
evidence-capture conventions.
* **Tests**
* Added unit-style tests for previous-log selection, prioritization, row
limiting, and deterministic log artifact naming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 17:22:11 +05:00
myasnikovdaniil d0c0f399f8 fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404) 
## The bug

Promotion rewrites the rc version substring to the stable version across
the vendored image references. That rewrite lived inline in
`promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus
`packages/apps/kubernetes/images/*.tag` **only**, on the premise that
the kubernetes app was the only one whose `.tag` files carry the
cozystack version. Nine other `.tag` files do, and `system/multus` seds
its reference straight into a vendored upstream daemonset manifest.

#3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33
of 55 first-party references still reading `v1.6.0-rc.4`**.

`hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind
spot from the other direction — both scanned the depth-2 `values.yaml`
alone, so those images were never retagged to the stable version (**30
references selected before, 42 after**) and never mirrored to the public
registry for a nightly. A retag miss leaves an image short of a tag,
since the digest still resolves inside one registry. A mirror miss is
worse: the host rewrite walks the same file list, so a reference that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

`v1.6.0` is the first release to go through promotion — every earlier
release was a full rebuild from the tag, which stamped every file
uniformly. The glob has always been wrong; the old path just made it
unobservable.

## Why it could not be caught before

The rewrite was workflow-inline, so there was nothing to unit test. The
only way to observe the miss was to cut a release.

It is a script now — `hack/promote-rewrite-tags.sh` — and
`hack/promote-rewrite-tags_test.bats` round-trips it against the real
tree under `make unit-tests`, with no cluster, registry or release
involved.

## The fix

The three call sites each carried their own idea of where a reference
can live, which is how they drifted apart. That knowledge moves into
`hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()`
for the files, `collect_image_refs()` for the references inside them —
and all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite also asserts its own postcondition: it scans **wider** than
it rewrites and fails the promotion when a reference survives in a
location the enumeration does not know about, rather than shipping it
quietly. The wide scan is shape-filtered to image-reference position,
because `X.Y.Z-rc.N` is an ordinary version string that other things
legitimately contain — `kubevirt-csi-driver/go.sum` pins
`github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a
cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:`
instead would have been wrong: `system/cilium/values.yaml` keeps `tag:`
and `digest:` under separate keys.

`docs/agents/image-refs.md` records the contract — three tag classes,
three storage shapes, the invariants, and which consumer does what.

## Not a bug: kamaji

`cluster-api-control-plane-provider-kamaji` stays at
`v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild
of an upstream component, versioned by that component rather than by the
cozystack version line, so no rewrite is owed. A test pins this so a
future fix here cannot over-reach.

## Documented known gaps, deliberately not fixed here

Both are pre-existing and in the **host** dimension; the version
dimension this PR owns is complete, because the version always lives in
a `tag` key regardless of where the host sits.

- **The nightly host rewrite cannot reach a split host.** It is a
literal `<src-registry>/` substring replace, so `keycloak-operator`
(host in a sibling `registry:` key) and `kubeovn`
(`global.registry.address`, no trailing slash) are mirrored but keep
pointing at the build registry in the published tree. Fixing it needs
structure-aware rewriting.
- **A reference inside a gzip is invisible.**
`capi-providers-cpprovider` ships `files/components.gz`, and the promote
postcondition cannot catch this class at all since `grep -rIl` skips
binaries. Nothing is owed today — kamaji is component-versioned and the
image is still mirrored and retagged via its `.tag` — so the residual is
that a nightly's kamaji ConfigMap keeps the build-registry host.

## Testing

- 17 tests in the new suite; `make bats-unit-tests` green (312 tests,
exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow.
- Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`,
`1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no
false negatives.
- Every test is mutation-proven. Reverting the enumeration to the old
glob, deleting any of the five YAML shapes, corrupting emitted digests,
reverting extras to parse-only, dropping the unreadable-file guard,
broadening shape 4 beyond `global.images`, or planting an unenumerated
first-party reference in any storage shape each fail a named test.

## Reviewer note, out of scope

`platform-migrations` is pinned at two different digests —
`core/platform/values.yaml` at `v1.5.0` and
`backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned
pin; no Makefile writes that key, tracked in #3143). A
`promote-retag.sh` dry-run emits **two copies to the same destination
tag**, so the write-once guard will fail the promotion. This is
independent of this PR and needs a decision about the backup-client
image version, so it is left alone here — but it blocks the 1.6.0
promote and wants fixing before the re-cut.

## Release note

```release-note
Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry.
```

Fixes the promote half of #3143.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Image promotion and nightly mirroring now detect digest-pinned image
references across values files, standalone tag files, and supported
templates.
- Stable release preparation supports RC→stable tag rewriting via a
dedicated promotion script, with broader verification to ensure no RC
references remain.

- **Bug Fixes**
- Improved completeness of ref discovery and scanning, including
previously missed template/tag locations.

- **Documentation**
- Added and linked guidance documenting image-reference invariants and
how tooling enumerates supported storage shapes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 16:41:17 +05:00
myasnikovdaniil bbfe9f9a9d fix(backupstrategy-controller): give chBackupClientImage a producer (#3405) 
## This is a workaround

`backupstrategy-controller` should not be running the platform
**migrations** image. This PR does not fix that — it makes the existing
reuse *safe* so the v1.6.0 promote can proceed, and documents in the
tree exactly what is wrong and what should replace it. Treat it as a
stopgap with a TODO, not as an endorsement of the design.

## What is actually wrong

The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its
script uses exactly four binaries — `curl`, `jq`, `sleep`, `date`. For
that it currently pulls `platform-migrations`:

| | |
| --- | --- |
| Size | **349 MB** (a dedicated `alpine + curl + jq + ca-certificates`
measures **23 MB**) |
| Also contains | `kubectl`, `helm`, `git`, `cozyhr`, the `etcd-migrate`
binary, every migration script, vendored etcd CRDs |
| `ENTRYPOINT` | `run-migrations.sh` |

Two consequences worth naming. A tenant-adjacent backup Pod carries
cluster tooling it has no use for. And because the image's own
entrypoint is the migration runner, the `command:` override in the
strategy template is load-bearing — drop that one line and the Pod runs
platform migrations with `kubectl` and `helm` already on the PATH.

It also creates the coupling that caused the bug this PR fixes: bumping
`ETCD_OPERATOR_VERSION` or editing any migration script re-digests the
image, forcing the backup client's pin to move for reasons that have
nothing to do with backups.

## The release blocker

`platform-migrations` was pinned at two different digests in the
committed tree:

| Location | Pin | Producer |
| --- | --- | --- |
| `packages/core/platform/values.yaml` `.migrations.image` |
`v1.5.0@sha256:8bf61f17…` | `packages/core/platform/Makefile` |
| `backupstrategy-controller/values.yaml` `.chBackupClientImage` |
`v1.4.0-rc.2@sha256:17390197…` | **none** |

Promotion retags **by digest**: `hack/promote-retag.sh` copies every
collected `<repo>@<digest>` to `<repo>:<stable-version>`. Two digests
under one repository produce two copies competing for the same
destination tag — the first wins, the second hits the write-once guard,
and the promotion fails. A dry run against `main` emits exactly that
pair:

```
▸ ghcr.io/cozystack/cozystack/platform-migrations  sha256:17390197…
  → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9
▸ ghcr.io/cozystack/cozystack/platform-migrations  sha256:8bf61f17…
  → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9
```

`v1.6.0` is the first release cut through promotion rather than a full
rebuild. Every earlier release ran `make build`, which restamped both
copies as a side effect and hid the missing producer.

## What this PR does

`packages/core/platform/Makefile` now stamps both keys with the same
ref, so they move in lockstep by construction rather than by
instruction, and the committed value is aligned to the digest
`core/platform` already carried. Moving the runner forward two minors is
safe: it is used only as a curl + jq container, and the image built at
`v1.5.0` carries them (`git show v1.5.0:…/migrations/Dockerfile` → `apk
add … jq ca-certificates bash curl`).

## What should replace it

A dedicated `ch-backup-client` image built by **this package's own
Makefile** — measured at 23 MB, 15× smaller, with none of the cluster
tooling and none of the entrypoint hazard. That also deletes the
cross-package stamp this PR adds, because the pin would be produced by
the package that owns it.

The objection that has kept this reuse in place — "a second tag release
CI would have to re-digest each cut" — does not survive contact with the
actual defect. Every other first-party image already pays that cost,
automatically. What broke here was not an extra tag; it was **a pin with
no producer**, which a package-owned image cannot reproduce.

I am deliberately not doing that here: it means a new image, a new build
target and a new digest introduced during a release cut. It belongs in a
follow-up.

## The guard

`hack/image-pin-consistency.bats` asserts the general invariant — no
repository pinned at more than one digest — driven through the real
promotion selector so it tracks whatever set promotion actually acts on,
rather than a reimplementation that could drift from it. Plus a specific
lockstep assertion so a recurrence names its cause. This check could not
have been added before now: it fails on the very duplicate this PR
removes.

## Testing

- `hack/image-pin-consistency.bats` — both tests pass; verified they
fail when the drifted pin is restored.
- `make bats-unit-tests` green (294 tests, exit 0).
- `make -C packages/system/backupstrategy-controller test` — 11 tests, 3
suites, green.
- `helm template … --set backupStorage.bucketNameOverride=test-bucket`
renders the strategy Pod with the new ref.
- The Makefile's `yq` step was simulated directly to confirm the
relative path and key path resolve and that comments survive the
in-place edit.
- Merged locally with #3404 onto `main` and re-run: the expanded ref
collector in that PR surfaces no additional duplicate pins (314 tests,
exit 0).

## Relationship to other work

Relates to #3143 — this is one instance of its unowned-pin class, the
one its discussion flagged as having no producer. The four never-built
packages there remain untouched.

Independent of #3404 (promote/retag/mirror storage-shape coverage); the
two touch no common files and can merge in either order. Both are needed
before the `v1.6.0` promote is re-cut.

## Release note

```release-note
Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Ensured the platform migrations image digest is pinned consistently
between the platform and backup strategy components.
* Updated the backup strategy component to reference the new pinned
`platform-migrations` digest.

* **Tests**
* Added promotion/retag consistency checks to fail on conflicting source
digests for the same stable tag.
* Added drift detection to ensure `platform-migrations` image references
match across consumer configuration files.

* **Documentation**
* Expanded in-chart comments and values documentation clarifying the
coupling and workaround behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-22 16:41:13 +05:00
Andrey KolkovandClaude Opus 4.8 4605bbf3db test(mariadb): pre-clean the backup roundtrip against stale tenant-root state 
Address review feedback on #3384. The TIA rule for
examples/backups/<app>/ that an earlier revision of this commit added
has since landed on main independently (50fb3c87c, "run the postgres
backup round-trip e2e in CI under TIA"), so only the suite change
remains here after the rebase.

mariadb-2-backup-roundtrip runs in the shared, persistent tenant-root
namespace with fixed resource names, so a run killed before the finally
leaves stale state behind; a leftover BackupJob still carrying
status.phase=Succeeded would satisfy the Succeeded wait without a real
backup ever running. Add an idempotent pre-clean step (run cleanup.sh
before run-all.sh), mirroring the etcd suite, and reword the suite
comment to reference the merged etcd-2-backup-roundtrip pattern instead
of postgres-2-backup-roundtrip.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 10:45:32 +04:00
Andrey KolkovandClaude Opus 4.8 d1f2822018 test(mariadb): add backup/restore round-trip e2e (chainsaw) via example harness 
Reworks the never-merged backup-mariadb-e2e branch onto the modern
Chainsaw approach (docs/agents/e2e-testing.md). The bats port that
branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the
whole hack/e2e-apps/*.bats app suite has been replaced by
hack/e2e-chainsaw/, and the mariadb strategy controller + example
manifests it depended on have since landed on main independently. Only
the e2e coverage was still missing.

Convert examples/backups/mariadb/ from static numbered YAML into the
same executable harness the other backup demos use (00-helpers.sh,
run-all.sh, cleanup.sh), keeping the numbered manifests as the
human-readable flow that run-all.sh drives — so the documented flow and
the automated test can never drift. run-all.sh provisions a Bucket,
derives the two per-app Secrets the strategy references from BucketInfo
+ the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires
an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and
asserts the sentinel round-tripped through S3 into the copy while the
source is left untouched.

Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to
hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring
postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant
cannot reach tenant-root's seaweedfs across the Cilium egress policy),
skipDelete since the harness (not Chainsaw) owns the resources, fail-fast
on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics.
Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs
the Cozystack jobs own only by label.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 10:44:42 +04:00
Andrey KolkovandClaude Opus 4.8 6a1e2d0d06 feat(backups): support PostgreSQL point-in-time recovery (PITR) 
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:

- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
  and the recovery Cluster is not healthy, the driver reads the CNPG
  full-recovery pod logs and, once several distinct attempts have exited with
  PostgreSQL's "recovery ended before configured recovery target was reached"
  FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
  Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
  restore deadline. Health is checked first, so a converged recovery always
  wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
  transient archive-lag miss on a near-now target and ignores transient
  recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.

- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
  postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
  the server timestamp, writes an after marker, restores to that timestamp and
  asserts the before row survived and the after row did not; step 46 submits a
  recoveryTime an hour ahead and asserts it fails fast with reason
  RecoveryTargetUnreachable.

- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
  (docs/operations/backup-classes.md) covering the recoverable window, how to
  discover the earliest/latest restorable time from the backup catalog, the
  archive-lag tolerance, and idempotency under GitOps.

Refs #2774

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-22 10:40:18 +04:00
Aleksei Sviridkin 99f04d99bd test(postgres): verify the tenant CA projection end to end 
Assert the live controller publishes <release>.tenant-ca carrying only
ca.crt, that its certificate matches CloudNativePG's CA while the private
key never crosses over, and that the anchor is reachable through the
tenantsecrets API while the key-bearing source Secret is not.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-21 23:59:48 +03:00
Aleksei Sviridkin 37a91f98b8 feat(cozystack-controller): extract application CA into a key-free tenant Secret 
Every catalog engine issues a per-release CA, but the object holding
ca.crt usually holds a private key beside it (cert-manager's tls.key,
CloudNativePG's ca.key), so granting a tenant read on it hands over key
material too. Add a controller that publishes the trust anchor as one
canonical, key-free Secret per release, "<release>.tenant-ca", holding
only ca.crt.

A chart declares the anchor by rendering a namespaced TenantProjection
sentinel (internal.cozystack.io/v1alpha1) that names the source Secret
and key. The controller rebuilds ca.crt from the parsed certificate DER
rather than copying the source, refuses any value carrying private key
material, and owns the projection by owner reference so a foreign Secret
at the same name is never overwritten. No tenant RBAC grants any verb on
internal.cozystack.io; a ValidatingAdmissionPolicy pins writes to
helm-controller as defence in depth.

The controller does not grant tenant read access — the lineage webhook
does, from the ApplicationDefinition's spec.secrets. This side stamps the
engine-agnostic internal.cozystack.io/tenant-ca label and forces a
re-admission when those selectors change.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-21 23:59:48 +03:00
Myasnikov Daniil 427ed68761 fix(backupstrategy-controller): give chBackupClientImage a producer 
platform-migrations was pinned at two different digests: v1.5.0 in
packages/core/platform/values.yaml, stamped by its Makefile, and
v1.4.0-rc.2 as .backupStrategyController.chBackupClientImage, stamped by
nothing at all. The second is the Altinity strategy Pod's curl+jq runner,
deliberately reusing the migrations image rather than shipping a second
one-binary tag that release CI would have to re-digest every cut. That
reuse only holds while the two move together, and the values.yaml comment
said so -- but a comment was the only thing enforcing it, so the copy with
no producer froze two minors back.

This is a release blocker, not just staleness. Promotion retags by digest,
copying every collected <repo>@<digest> to <repo>:<stable>. Two digests
under one repository produce two copies competing for one destination tag:
the first wins and the second hits the write-once guard in
hack/promote-retag.sh, which fails the promotion. A dry run against the
tree emitted exactly that pair. v1.6.0 is the first release cut through
promotion rather than a full rebuild, so it would have been the first to
hit it -- every earlier release ran `make build`, which restamped both
copies as a side effect and hid the missing producer.

packages/core/platform/Makefile now stamps both keys with the same ref, so
they are in lockstep by construction rather than by instruction, and the
committed value is aligned to the digest core/platform already carried.
The image is used only as a curl + jq + ca-certificates runner, so moving
it forward two minors changes no behaviour the strategy depends on; it
renders and the chart's helm-unittest suite passes.

hack/image-pin-consistency.bats adds the general guard -- no repository
pinned at more than one digest, driven through the real promotion selector
so it tracks whatever set promotion acts on -- plus a specific lockstep
assertion so a recurrence names its cause. The general check could not be
added before now: it fails on the very duplicate this commit removes.

Relates to #3143 (the unowned-pin class) and unblocks the v1.6.0 promote.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 23:15:46 +05:00
Myasnikov Daniil 91c076137f test(ci): pin shape 3's per-map digest binding 
The shape-3 fixture held a single repository/digest pair, so a rule that
binds one digest per file and reuses it for every repository satisfied
it -- and passed all three suites. That is not a theoretical mutation:
against the real linstor values it hands piraeus-server's digest to
linstor-csi, which retags and mirrors the wrong image under the right
name. Silently shipping wrong bytes is worse than the missing-tag bug
this branch started from.

The fixture now carries two shape-3 maps with distinct digests in one
file and asserts both exact canonical refs, so the digest must be bound
per map rather than per file.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 22:54:19 +05:00
Myasnikov Daniil a16f218875 test(ci): correct a stale comment and isolate the fail-open fixtures 
Two review nits, both comment- or fixture-level.

The completeness oracle's comment still said the per-shape test "matches
on the repository substring, so a shape that loses its registry rejoin
still satisfies it". That stopped being true in the same commit that
introduced it -- the substring match was replaced with an exact
canonical-ref assertion, which kills both the shape-2 and shape-3 rejoin
mutations. The comment understated coverage.

The fail-open test copied the tree once and looped three rc versions over
it, so the second and third iterations ran against a tree the first
rewrite had already mutated. Each now gets a fresh copy, so each tests
the input it names.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 22:51:00 +05:00
Myasnikov Daniil 539a7b8e0b test(ci): catch shape-4 stapling inside global, and spaced marker keys 
Round-5 review. The shape-4 negative fixture placed its decoy outside
`global` entirely, so it only caught a document-wide broadening. Mutating
the rule to `.global | ..` instead passed the per-shape test and all three
suites while still stapling the document-level registry onto unrelated
maps inside `global` -- manufacturing first-party refs that promotion
would then retag and mirror to destinations nothing ever built. The
fixture now carries two decoys at different depths, one inside `global`
but outside `global.images` and one outside `global`, and the mutation is
caught.

Markers also required the key to be immediately followed by its colon, so
`registry : ghcr.io/...` was missed. No producer writes that spelling and
it is valid YAML rather than anything in the tree, so this is theoretical
-- but tolerating whitespace costs one character class and keeps the
comment's exhaustiveness claim true lexically as well as structurally.
Marker count is unchanged at 43 files with the same two allowlist entries.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 22:26:38 +05:00
Myasnikov Daniil fa4ae65d04 test(ci): assert usable refs, quoted keys, and shape-4 over-collection 
Round-4 review. Both reviewers independently found the declared-extra
test was passing while the thing it guards was broken, and each found one
more.

That test asserted `grep -q "@sha256:<digest>"`, which any token
containing the digest satisfies. Its fixture was a split-host ref, which
a textual scrape can never reconstruct -- only the contiguous tag value
is recoverable, so ref_repo() reduces it to `latest` and every consumer
drops it at the ownership filter. The fixture was therefore
enumerated-but-uncollected, which is exactly the conflation the test was
added to catch, and gutting the scrape to emit a bare digest with no
repository left the whole suite green. The fixture is now a contiguous
ref in an unparseable template (single-quoted, so the scrape's new
quote-tolerance is exercised) and the assertion pins the exact canonical
ref. The limitation that a split-host ref in an unparseable file is
recoverable by neither route is recorded rather than implied away: the
file-level marker check is what covers that case.

The scrape also captured a leading quote, so any quoted ref in a .tag
file or declared extra yielded a token every consumer silently discards.
Nothing in the tree is affected today, but it meant the double-pass
rescued only unquoted refs.

Markers missed quoted YAML keys: an unenumerated template using
"registry": / "repository": passed the entire suite. They now tolerate
quoting on the key as well as the value, still matching 43 files with no
new allowlist entries.

Shape 4 could be broadened from .global.images[] to every map in the
document and pass everything, because asserting that the expected ref
appears cannot see an unexpected one. That scoping is load-bearing -- the
host is a document-level key, so a global binding staples kubeovn's
registry onto unrelated repositories and manufactures owned refs that
were never built, which promotion would then retag and mirror. The shape-4
fixture now carries a digest-pinned sibling outside global.images and the
test asserts its absence.

Two per-shape fixture gaps closed while there: shape 2 had no sibling
`registry:`, so it losing its rejoin was invisible here and caught only
by one nightly-mirror fixture; and shape 1 had no tagless case, so
narrowing it to require a :tag before the digest went undetected -- a
digest-only pin is an ordinary Helm spelling.

Also corrected the residual-limit comment on the markers. Coverage is
exhaustive, not merely broad: the collector joins registry + "/" +
repository, so a split can only fall on a path-separator boundary and
markers 1-3 cover all four possibilities. A split elsewhere does not need
covering because it does not produce a first-party ref.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 22:05:48 +05:00
Myasnikov Daniil 8ad25b970f test(ci): pin digests exactly and generalize the first-party markers 
Round-3 review, both reviewers again converging on the same defects.

The file-level marker check was keyed to the exact ref spellings present
in the tree rather than to the join semantics _collect_yaml_shapes
implements, and three realistic first-party refs evaded it: `registry:
ghcr.io/cozystack/cozystack` + `repository: <name>` (kubeovn's layout
under the key name keycloak-operator and ingress-nginx use, which the old
marker missed because it required a trailing slash a complete-value host
does not have), a host+org split with precedent at fluxcd's values.yaml,
and a single-quoted `repository:` where only double quotes were admitted.
All three carried a version-free latest@sha256: tag -- exactly the
profile the check exists for -- and passed the whole suite. The markers
now follow the collector's join semantics; this matches 43 files and
needs no new allowlist entries. The residual limit (a host split at an
arbitrary point inside cozystack/cozystack) is stated rather than
implied, along with why widening further is not worth it: a bare
cozystack/cozystack marker matches 65 files and would need 23 allowlist
entries for Makefiles and test fixtures.

More seriously, making the completeness oracle share shape logic
introduced a regression the previous independent oracle would have
caught: corrupting the last hex digit of every emitted digest left the
rewrite, retag and mirror suites entirely green, because both sides of
the comparison move together and the per-shape test matched only a
repository substring. The digest is the only part deciding which bytes
get retagged, so the per-shape test now asserts the exact canonical
repo@sha256:digest for every shape.

Declared extras were also narrowed by the refactor: parsing alone means a
ref inside a block scalar yields the enclosing block (which the ownership
filter then discards) and an extra that stops parsing after `make update`
yields nothing at all. Both are silent skips, in the one storage shape
defined as a manifest vendored verbatim from upstream. Extras get the
textual scrape again alongside parsing, with the callers' dedup absorbing
the overlap.

Finally, enumerating a path only proves the path is visited: a declared
extra whose ref the parser cannot reach stayed enumerated but uncollected
while both completeness checks passed. Added a test asserting a declared
extra yields its ref, using an unparseable Helm template with a
single-quoted split-host ref -- the shape that defeated both guards.

Also corrected the claim that the per-shape test compensates for the
shared oracle: it catches a shape being DELETED, not narrowed. A shape
losing its registry rejoin is caught by promote-retag and nightly-mirror,
which assert named packages, and the comment now says so.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 21:38:26 +05:00
Aleksei Sviridkin f3a0f2d323 test(e2e): avoid head -n 0 in prevlog_cap for BSD portability 
A zero container cap reached head -n 0, which GNU head treats as an empty
result but BSD/macOS head rejects ("illegal line count -- 0"), aborting a
local run. Special-case the zero cap to consume stdin instead, so the
helper behaves the same on both coreutils flavours.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-21 19:32:42 +03:00
Myasnikov Daniil 648eb42cae test(ci): make the image-ref guards see split-host and unparseable files 
Both round-2 reviewers independently landed on the same two test-side
holes, and both are the guard being blind in exactly the way the bug it
guards was.

The completeness oracle derived its expected set from a grep for a
contiguous ghcr.io/cozystack/cozystack/...@sha256: string, so it could
not see any split-host shape: cilium, keycloak-operator, kubeovn, the OCI
artifact and seven shape-3 refs, 12 of 43 first-party refs in all. That
includes keycloak-operator, which the test's own rationale names as the
motivating profile. The fixture-discovery grep in the same file had
already been widened for this exact reason; the fix was never carried to
the oracle.

The oracle now applies the same shape knowledge to an independently
discovered file list, which is the axis that actually drifts -- what each
consumer disagreed about historically is which FILES it visits, not how a
ref is spelled inside one. To keep that sharing safe, shape coverage is
now pinned by its own test, so deleting a shape still fails something
even though both sides of the comparison would move together.

Sharing required splitting collect_refs_from_file out of
collect_image_refs. Behaviour is unchanged: same 67 refs collected before
and after, multus and the packages artifact both still present.

A YAML oracle cannot cover everything, though. A reviewer planted a
split-host ref in a Helm template, which yq cannot parse at all, and it
survived every test. A file-level marker check now backstops that: any
file carrying a first-party marker must be enumerated or explicitly
allowlisted. Two files are allowlisted today, each with its reason -- the
talos-csr-signer Dockerfile placeholder, and the documented capi gzip
gap. This also closes the secondary weakness that comparing on repo@digest
masks an unenumerated occurrence when the same image sits at an
enumerated path, which is live for that capi file.

Second hole: the postcondition's key-position branch had no coverage.
Replacing the whole pattern with the naive "${RC}@sha256:" -- the
narrowing rejected because cilium keeps tag: and digest: under separate
keys -- left all 22 tests green, so the branch that makes the split shape
visible was asserted by nothing. Added a cilium-shaped fixture at an
unenumerated path.

Both previously-surviving mutations are now caught. The oracle was also
pre-filtered by registry marker before parsing, cutting it from 32s to 5s.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 21:08:15 +05:00
Aleksei Sviridkin 76f7267ff2 test(e2e): describe --previous as the preceding instance, not the first 
kubectl logs --previous returns the immediately preceding terminated
container instance, not the original first one — after multiple restarts
the first instance is already gone. Reword the helper, the BATS and
Chainsaw failure handlers, and the diagnostics doc to match the actual
--previous semantics.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-21 18:57:01 +03:00