Commit Graph
7777 Commits
Author SHA1 Message Date
Myasnikov Daniil 53a7fc4dcd feat(release): cutting an rc freezes the line into release-X.Y
Cutting the first vX.Y.0-rc.N now creates release-X.Y at the tagged
commit, and every later cut for that line must be dispatched from the
branch — a dispatch from main is refused.

Before this, only the stable promote-PR merge created release-X.Y, so
rc.2 and rc.3 were cut from main's tip and silently picked up every
feature merged since rc.1. The release therefore shipped content no rc
had ever validated. Freezing at the rc closes that: main stays open for
the next minor, and fixes reach the release by cherry-pick onto the
branch.

promote-rc.yaml needs no change — it already prefers release-X.Y as the
promote PR base when that branch exists, so the stable release is now
cut from the frozen tree.

Gated to -rc (alpha/beta are pre-freeze builds off an open main) and to
patch 0 (a patch-line rc is cut from a release-X.Y that already exists).
The branch is create-only and never force-moved: moving it would re-open
the freeze and drag in whatever the new tip holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
myasnikovdaniil 3f967ec7e0 fix(platform): restore the isp-hosted dependency closure (#3391)
## Problem

Fixes #3376. The `isp-hosted` preset shipped `bundles.system.enabled:
false` while `paas` and `naas` were enabled, so it enabled an app
catalog whose dependency closure it simultaneously disabled. Every
emitted Package sat in `DependenciesNotReady`, the `*-rd` HelmReleases
failed with `no matches for kind "ApplicationDefinition" in version
"cozystack.io/v1alpha1"`, and the `cozystack-platform` HelmRelease never
went Ready.

Rendering the preset at `origin/main` shows the scale: **24 emitted
Packages, 24 of them blocked**, by 6 missing dependencies — two more
than the issue lists (`cert-manager` and `prometheus-operator-crds` are
also absent). `isp-full` renders 75 Packages with a fully satisfied
closure, which is a clean control.

## Root cause

`templates/bundles/system.yaml` is gated end to end on
`bundles.system.enabled`, but it carries two separable things: a
variant-selected **data-plane head** (kubeovn/cilium/linstor on
`isp-full`, a noop networking Package on `isp-hosted`) and a
variant-agnostic **control-plane body** — `cozystack-engine`, the sole
source of the `application-definition-crd` component, plus
`cert-manager`, `prometheus-operator-crds`, `postgres-operator` and
`objectstorage-controller`.

"Hosted" means the host already provides CNI, storage and
virtualisation, so only the data-plane head should be dropped. Disabling
the whole bundle also deleted the control-plane body that every variant
needs.

Three things show the disable was the mistake rather than the intent:
the preset still sets `variant: "isp-hosted"`; `system.yaml` contains an
`isp-hosted` branch (noop networking plus default scheduler) that could
never render; and `iaas.yaml` already refuses to render on this variant,
encoding the same "system bundle minus iaas" contract. The preset has
never worked — it was born this way in `2d022e38e` (January 2026), the
refactor that replaced `bundles/paas-hosted.yaml`, whose contents
confirm hosted was always meant to ship the engine, `cert-manager` and
`prometheus-operator-crds`.

## Changes

1. **`values-isp-hosted.yaml`** — enable the system bundle. Hosted now
emits 53 Packages with every `dependsOn` satisfied, installing 59
components against the 60 of the pre-refactor `paas-hosted` bundle, with
no LINSTOR, KubeVirt, CAPI or Kamaji leakage.
2. **Move `cozystack.bucket-application`** from the iaas bundle into the
system bundle. Its own `dependsOn` closure is `networking` +
`objectstorage-controller` + `cozystack-engine`, none of them iaas,
while `cozystack.backupstrategy-controller` — a *default* package of the
system bundle — hard-depends on it. Any system-on/iaas-off configuration
therefore wedged `backupstrategy-controller`; both `isp-full` presets
masked this by enabling iaas. This is a second, independent bug not
reported in the issue.
3. **Guards** — `paas`, `naas` and `iaas` now also require
`bundles.system.enabled`. The pre-existing checks looked only at
`bundles.system.variant`, which keeps its value when the bundle is
disabled, so they passed on exactly this configuration. `iaas` had the
identical hole (12 Packages, 6 unsatisfied deps). This turns a silent
multi-hour `DependenciesNotReady` hang into a render-time failure that
names the remedy.
4. **Two helm-unittest suites** — one pinning the dependency closure the
catalog needs, one pinning the guards.

## Verification

- All four operator-registered variants render dependency-closed:
`default` 0 Packages, `isp-hosted` 53, `isp-full` 75, `isp-full-generic`
75. The two `isp-full` variants are byte-identical before and after, so
the bucket move is a no-op for them.
- Hosted is a strict subset of full with zero data-plane leakage, and
its exclusions match the pre-refactor hosted bundle.
- 23 suites / 97 tests pass in `packages/core/platform`; repo-wide
`hack/helm-unit-tests.sh` is green.
- Every change was **mutation-tested** — reverting each one in turn
makes the suite fail, so the tests are not vacuous.
- The workaround documented in #3376 (`paas`/`naas` disabled) still
renders cleanly; the new guards do not fire on it.
- Hosted runtime spot-checks: `cozystack-api` is a Deployment with only
a *preferred* control-plane affinity plus blanket tolerations, and
`cozystack-scheduler` uses its `default` variant (plain Deployment, no
LINSTOR extender) — both schedule fine on a cluster with no
control-plane nodes.

## What this does not fix

This restores the dependency closure. It is **not** by itself sufficient
for a fully converged hosted install. Review surfaced further
hosted-unaware defaults that the dead bundle had been masking, filed
separately:

- #3387 — `apps/tenant` renders 14 `cilium.io/v2` policies with no
`.Capabilities` guard, so `tenant-root` fails on a non-Cilium cluster.
This is the blocker for actual hosted convergence, and it needs a
deliberate decision rather than a reflexive capability guard, because
those policies implement tenant network isolation.
- #3388 — `extra/monitoring` defaults `logsStorages` to the LINSTOR-only
`replicated` StorageClass.
- #3389 — `bundles.disabledPackages` can still break the closure with no
render-time error; the guards here cover the bundle flags only.
- #3390 — nothing in CI exercises `isp-hosted`, which is why all of the
above accumulated undetected.

## Upgrade note

Existing `*-application` Package CRs carry `helm.sh/resource-policy:
keep`. An operator who applied the #3376 workaround but did not also
delete the retained Package CRs (as that issue instructs) will find them
install once this change makes their dependencies Ready, since the
reconciler re-enqueues dependents on readiness. Deleting the unwanted
Packages before upgrading avoids that.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added fail-fast validation that requires the **System bundle** when
enabling **IaaS**, **NaaS**, or **PaaS** to prevent incomplete catalogs.
* Updated bundle composition so the **bucket application** is provided
by the **System bundle**.
* Enabled the **System bundle** by default for the `isp-hosted` variant.
* **Bug Fixes**
* Improved error messaging to clearly indicate the required
bundle/variant configuration.
* **Tests**
* Added Helm unittest coverage for rendering-gate failures,
dependency/package expectations, and hosted data-plane exclusions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-03 21:27:56 +05:00
Mattia Eleuteri e85501e9f5 fix(backupstrategy-controller): bound each gate check and tolerate an absent Secret
Two defects found in review of the periodic recovery loop.

checkAndLog passed the manager's root context straight into Check, which
makes several sequential API calls. That context is only cancelled at
shutdown, so an API server stalling on any one of them blocked the call
indefinitely. The ticker loop is sequential, so one stuck call stopped
every later check for the life of the pod: the recovery this gate exists
to provide went silent, leaving a stale gauge and no further log line,
with only a restart to bring it back. Each check now runs under a
timeout of half the Period, capped at 30s, so it can never overlap the
next tick.

Check also propagated a NotFound on the projector's source Secret,
although the very next branch treats an empty bucket name as an expected
no-op. Those are the same bootstrap state seen a moment apart, so the
absent Secret logged a check failure on every tick of the whole
bootstrap window for a condition the doc comment frames as graceful.

Both are pinned by tests that fail against the previous behaviour: one
reverts to an error on the absent Secret, the other to a timeout equal
to Period.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-08-03 17:47:36 +02:00
Mattia Eleuteri 51ad226706 fix(backupstrategy-controller): repair lookup-gated backup objects
The default Strategy CRs and the Velero BackupStorageLocation are
Helm-templated behind a `lookup` of the BucketClaim the same chart
creates, and the <bucket>-<user>-credentials Secret behind a `lookup` of
the COSI Secret. When those lookups are empty at install time the objects
were silently skipped, and helm-controller does not re-render a release
whose chart and values did not change (drift detection is off on
operator-generated HelmReleases), so the skip was permanent: clusters ran
for months with only BackupClass cozy-default and no Strategy CRs at all.
That also fail-closes the pre-adoption snapshot in the v1.6.0 etcd
migration, which reads the projected credentials Secret.

Add a DefaultObjectsGate runnable to backupstrategy-controller. Once the
bucket name is resolvable from the projector's source Secret, it checks
that every object cozy-default routes to exists and forces one real Helm
upgrade (reconcile.fluxcd.io/forceAt + requestedAt) when any is missing.
Helm remains the objects' only author; the gate only makes sure the render
that produces them actually happens. It cannot be a render-time `fail` in
this chart: the chart is the producer of the Bucket its own lookup reads,
so a failed render would deadlock the condition.

The bucket chart's user-credentials template CAN fail safely — the
BucketAccess that produces the COSI Secret belongs to the parent release
and that release retries forever — so make it fail loudly instead of
skipping, with requireUserCredentials=false as the offline-render escape.

Also correct the chart and doc comments that claimed Flux re-renders on
its interval, and document the two-release manual recovery for clusters
already affected.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-08-03 17:16:55 +02:00
myasnikovdaniil f1f383626e fix(ci): overlay images from the PR base branch, and publish per-line artifacts (#3471)
## What this PR does

The PR finalize job overlays image refs for every package a PR did not
rebuild, so e2e and the installer do not test last-release images for
everything outside the PR's build matrix. It could only ever read
`cozystack-packages:main`, which meant a **release-line PR was handed
main's binaries to run against its own line's charts**.

cozystack/cozystack#3437 is the demonstration: a one-line change on
`release-1.6` deactivating an app failed install deterministically,
twice, with

```
helmrelease/backupstrategy-controller: Helm install failed …
error validating data: SchemaError(github.com/cozystack/cozystack/pkg/apis/core/v1alpha1.Option.spec):
unknown model in reference: "github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec"
```

main's `cozystack-controller` served an aggregated OpenAPI that
`release-1.6`'s charts could not validate against. Nothing in that PR
was broken; the lane was. Both branches carry `option_types.go` and key
`OptionSpec` identically in the committed generated OpenAPI, so this is
a generation mismatch at runtime, not a codegen drift.

This PR fixes it by giving every base branch its own artifact to overlay
from, rather than by turning the overlay off.

### The change

1. **`pull-requests.yaml`** reads `cozystack-packages:${BASE_REF}`
(`github.base_ref`) instead of a hardcoded `:main`.
2. **`build-release.yaml`** (new) publishes that artifact for maintained
`release-<major>.<minor>` branches exactly as `build-main.yaml` does for
main: every image tagged with the branch, and the whole packages tree
pushed with each reference digest-pinned to what the run just built.

An earlier revision of this branch simply skipped the overlay for
non-main bases. That fixed the wrong-images problem but left
release-line PRs testing their line's *last release*: for any package
the PR did not rebuild, the committed ref is the released digest, so a
component changed by an earlier backport was still exercised as its
pre-backport binary until the next rc. Per-line artifacts remove that
gap too, which is why the guard was replaced rather than kept.

### Three deliberate choices

**The trigger matches line branches only** (`release-[0-9]+.[0-9]+`).
The per-release and rc staging branches `promote-rc.yaml` and
`tags.yaml` create — `release-1.6.1`, `release-1.6.0-rc.4` — must not
trigger a full rebuild; their images come from the tag build and
rebuilding them is waste.

**`WRITE_CACHE` stays `0`.** `CACHE_REGISTRY/<img>:buildcache` is a
single ref per image and `build-main.yaml` is deliberately its only,
serialized writer so concurrent builds cannot race on the cache manifest
— the 409 class #2711 fixed for image tags. A line build can overlap a
main build, so writing here would reintroduce that race. Line builds
read the cache.

**A missing artifact still degrades to committed refs, but says so.** On
a release line it emits a `::warning::` naming the branch. Silent
degradation is indistinguishable from a working overlay, which is how a
mis-specified branch filter could hide for a whole release cycle.

### Cost

One `make build` per push to a maintained line — in practice per merged
backport. That is the price of release-line PRs testing their line's tip
instead of its last release.

### Verification

`hack/overlay-main-images_test.bats` pins the artifact tag to the base
branch, rejects a hardcoded `:main` in either overlay step, and pins
`build-release.yaml`'s branch filter, image tag and `WRITE_CACHE: '0'`.
Mutation-checked: restoring `:main`, setting `WRITE_CACHE: '1'`, and
broadening the filter to `release-*` each fail a test. 13/13 green;
`actionlint` and `zizmor` clean.

Worth an explicit ack in review: the branch-filter pattern is the one
thing no local test can prove, since only GitHub evaluates it. If it
does not match, `build-release` never runs and the new `::warning::` is
what surfaces it on the next release-line PR.

### Backport

`release-1.6` needs this too — for `pull_request` events GitHub builds
the workflow from the merge ref, so a release-line PR only stops
receiving main's images once the change is on its base branch.
cozystack/cozystack#3472 carried the interim guard and is closed in
favour of backporting this instead.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a “Build release line” workflow that builds and publishes images
and the packages artifact for maintained `release-<major>.<minor>`
branches.

* **Bug Fixes**
* Updated PR workflow finalization to use base-branch–specific package
overlays, avoiding incorrect main-branch image references when targeting
release branches.
* Improved fallback behavior when the base-branch packages artifact is
unavailable.

* **Tests**
* Added workflow wiring tests to verify base-branch artifact usage and
that each maintained release line publishes its own packages artifact.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-03 18:22:47 +05:00
Myasnikov Daniil 6a73a6caa9 docs(agents): call the E2E gate a commit status, not a check-run
The §10 heading, four workflow comments and both TOCTOU comments
still called the required "E2E Tests" context a check-run, while the
body of that same section — and the implementation — insist it is a
commit status. The distinction is the whole reason the section exists:
a check-run made with the default token floats into an arbitrary
check-suite, which is why the gate is a status. Leaving the wrong
primitive in the heading steers a maintainer wiring branch protection
at exactly the wrong thing.

The paragraph explaining why it is NOT a check-run keeps that wording,
since there the term is the rejected alternative. Also adds the blank
line missing before the "Chainsaw v0.2.15 gotchas" heading.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:57 +05:00
Myasnikov Daniil 1582eff3f7 docs(ci): correct the stale shared-runner claims in the fork e2e lane
Two comments in e2e-fork.yaml justified themselves with a threat that
no longer exists: the allowlist derivation guarded a `git worktree
add` against "a crashed/concurrent run on the shared self-hosted
runner", and the sandbox name was said to disambiguate "concurrent
fork runs on a shared host". The dedicated runner was decommissioned
in #2937 — these labels are ephemeral Oracle ARC VMs, one per job,
as .github/actionlint.yaml states. A comment that asserts a shared
host next to a step handling a push token misleads exactly the
reader who is auditing that boundary.

Both guards stay; only their stated reason changes to the one that
actually holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:47 +05:00
Myasnikov Daniil 1d668e1fcb fix(ci): scope the fork publish OCIR login to a run-local auth store
The `publish` job ran `skopeo login` and `docker login` with no
DOCKER_CONFIG override and no logout, so the OCIR push credential
landed in the runner's default stores (~/.docker/config.json and
$XDG_RUNTIME_DIR/containers/auth.json). Every other OCIR login in
this repo — pull-requests.yaml's build, build-talos and finalize —
already scopes to $RUNNER_TEMP/.docker, and `Set up Docker config`
in those jobs copies ~/.docker verbatim into the build context, so
a token left in the default store is a token handed to whatever
runs next.

Scope both logins to $RUNNER_TEMP/.docker (skopeo via --authfile,
flux via the docker keychain's DOCKER_CONFIG), carry the same env
into the push step that reads the auth, and drop the credential in
an always() logout step so a failed push does not leave it behind.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:32 +05:00
Myasnikov Daniil c46037ce7b fix(ci): stop a cancelled run from wedging the "E2E Tests" gate
Now that the required "E2E Tests" context is a commit status rather
than a job, a run that dies before posting leaves the head SHA at
"Expected" — which blocks the merge instead of passing it. Two paths
reached that state.

Same-repo: a `labeled` event shared the PR's concurrency group, so
labelling an open PR cancelled the live opened/synchronize run, and
the replacement run was then discarded by `plan`'s full-e2e guard --
so `e2e-report` skipped with it and nothing posted the status the
cancelled run was going to. Observed on live CI: run 30614258694
(cancelled) followed by 30614280603 (skipped) on the same SHA, 24s
apart, from a maintainer adding kind/* labels just after a push.
Label events now get their own concurrency group, so they can only
supersede each other. Release promotion is unaffected: `gh pr create
--label` attaches `release` at creation, so a promote PR gets a
single run that already sees the label (verified on #3397 and #3425,
one run each, build/finalize gated off).

Fork: `resolve` had no equivalent, so a cancelled triggering run fell
through to the `CONCLUSION != 'success'` branch and posted `failure`
— a false red on a SHA whose e2e never ran, which the replacement run
for that same SHA would not flip back (it takes the discarded-label
path and posts nothing). It now returns early on a cancelled or
wholly-skipped conclusion. Checked on the conclusion rather than the
job list because a cancelled run can report no jobs at all, which
would make the `Plan build` lookup miss.

Both stay fail-closed: declining to post can only preserve a verdict
a real run reached, never invent one.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 16:54:07 +05:00
Andrey KolkovandClaude Opus 4.8 9379729b31 fix(clickhouse): address review notes on endpointCA and the CA copy
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings).

1. The backup-s3-ca volume carries no `optional: true` while the adjacent
   useSystemBucket credential refs deliberately do, and the asymmetry was
   undocumented — an operator naming a Secret that does not exist gets the whole
   ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the
   fail-closed behaviour and wrote down why the two cases differ: credentials are
   projected by the platform after install, so a strict ref would wedge every
   fresh release until the first BackupJob, whereas this CA is named by the
   tenant, must already exist, and is trust material — optional would mount an
   empty dir, the Pod would look healthy, and the first backup would fail deep
   inside clickhouse-backup with an x509 error naming neither the Secret nor the
   chart. The comment also states the blast radius plainly.

2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read
   was reported as "CA not found" followed by advice (set S3_CA_SECRET) that
   cannot fix a permissions problem. It now separates present / absent /
   not-allowed-to-look and says so, and the README no longer labels step 03
   simply "tenant": copying the CA reads a Secret in tenant-root, which is a
   platform capability. Added a section spelling out the three ways to satisfy
   that (admin credentials, a one-off admin copy, or skipping the CA entirely on
   a publicly-trusted endpoint).

3. Caveat, and a usability regression this branch introduced: on a cluster with
   a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default
   the user never chose. It now distinguishes an explicitly named Secret (still
   an error when missing — a typo should not be guessed past) from the unset
   default, where it warns and continues with backup.endpointCA unset. Verified
   all four paths against a fake kubectl: Forbidden, absent+default,
   absent+explicit, and present.

4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy
   `schedule` CronJob writes to S3 through restic and does not consume it, so a
   private-CA endpoint combined with `schedule` still fails TLS. Documented on
   the field itself, so it lands in values.schema.json, the README and the
   cozyrds openAPISchema.

The PR body's claim about select-e2e.sh/select-install.sh mappings was stale
after the restructure that removed them; corrected there rather than in code.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-03 15:40:41 +04:00
myasnikovdaniil 04b7426706 chore(codeowners): don't gate per-app generated files on architect review (#3507)
## What this PR does

`.github/CODEOWNERS` scopes `/api/` to the architects (`@kvaps
@lllamnyp`), meant for hand-authored API surface: the aggregated
apiserver, the registry, the core API types. But
`api/apps/v1alpha1/<name>/types.go` and
`api/apps/v1alpha1/<name>/zz_generated.deepcopy.go` are entirely
produced by `make generate` from each app's own
`values.yaml`/`values.schema.json` — 23 of the 32 packages under
`packages/apps/` and `packages/extra/` wire this into their `generate`
target. Since CODEOWNERS applies the last matching pattern and nothing
more specific overrode `/api/` for these paths, any PR that edits an
app's `values.yaml` documentation and reruns `make generate` (mandatory
per the contributing guide) ends up requiring an architect review, no
matter which area actually changed — even though the app's own reviewers
already cover the source file that produced the diff.
`packages/system/<name>-rd/cozyrds/<name>.yaml` has the same problem in
a milder form: it isn't scoped by CODEOWNERS at all, so it falls through
to the generic `/packages/` owners instead of the app's own team, again
just because it's a generated mirror.

This adds a CODEOWNERS entry with no owner for these three generated
paths, so touching them alone does not pull in any codeowner requirement
beyond what the source `values.yaml` already needs. This is intentional
and safe: both are verified by the existing pre-commit CI
(`run-make-generate` / `run-make-generate-root`), which reruns the
generator and fails the check if the committed output does not match — a
stronger, mechanical guarantee against drift or hand-tampering than a
human skim of a generated diff would be.

While in there: `api/.gitattributes` had a stale `linguist-generated`
pattern (`zz_generated_deepcopy.go`) that has never matched any real
file in this repo — the actual generated files are named
`zz_generated.deepcopy.go` (dot, not underscore, matching the standard
controller-gen output). Fixed the pattern and added the same treatment
for the per-app `types.go` files and for
`packages/system/*/cozyrds/*.yaml`, so all of them collapse in the PR
diff view instead of just some.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

This only touches `.github/CODEOWNERS` and `.gitattributes` files; it
changes neither the `values.schema.json` contract, generated CRDs,
version enums, nor anything else in the trigger map in
`docs/agents/contributing.md`.

### Release note

```release-note
chore(codeowners): per-app generated API type mirrors and cozyrds manifests no longer require architect review on their own — only the source values.yaml they're generated from does, with the existing make-generate CI check guarding against drift.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated repository metadata to better identify generated API and
application artifacts.
* Added ownership exceptions for generated files and
application-specific configuration directories.
* Marked generated CozyRDS configuration files for improved repository
classification.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-03 16:29:28 +05:00
Timofei Larkin e5d4940338 chore(codeowners): don't gate per-app generated files on architect review
api/apps/v1alpha1/*/types.go and packages/system/*/cozyrds/*.yaml are
entirely produced by `make generate` from each app's own values.yaml,
but the /api/ codeowner rule pulled every app's regenerated types.go
into an architect-only review, regardless of which area actually
changed. Correctness there is already enforced by the pre-commit
make-generate CI hooks, not by human review, so list no owner for
these generated paths and let that check carry the weight instead.

Also fixes a stale api/.gitattributes pattern that never matched any
real file (zz_generated_deepcopy.go vs the actual
zz_generated.deepcopy.go), and adds a matching linguist-generated
attribute for the cozyrds ApplicationDefinition manifests, so both
collapse in the PR diff view like the Go files already do.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-08-03 14:08:36 +03:00
Timofei Larkin c76a05d14c test(kubernetes): backstop the worker MD wait on the child HR
Since the Phase 2 split the worker MachineDeployment is rendered by the
child kubernetes-nodes-<cluster>-md0 HelmRelease, a separately
dispatched reconcile with its own 20-30s helm-controller latency -- not
by the parent install that produced the deployments the script has just
waited on. A fast parent bringup therefore reached the MD wait before
the child HR had applied anything, and `kubectl wait` on an absent
object exits NotFound immediately instead of polling: the faster the
sandbox, the more certain the failure. CI showed exactly this -- KCP
created 2s after the CR, all four deployments available in ~30s, zero
MachineDeployments in the namespace at failure time.

Add the event-driven existence backstop before the scale wait, the same
sanctioned until-loop pattern the KCP wait above uses for the same
dispatch-latency reason.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-31 16:11:08 +03:00
Timofei Larkin f0b99da43a test(kubernetes): make kubectl_wait_retry set -e safe
The chainsaw scripts source run-kubernetes.sh under `set -eu`, where the
wrapper's bare `_out=$(kubectl wait ... 2>&1)` on a failing wait aborts
the whole sourcing script at the assignment -- before `_rc=$?`, before
the retry loop, before anything prints, with the error text captured
into the substitution and lost. Every `kubectl wait` failure therefore
killed the suite instantly and silently, and the transient-retry
machinery the wrapper exists for (etcd leader flap et al.) has never
been reachable. This is how the kubernetes-latest suite died in CI with
an empty stderr: the wait it was executing failed, and the script
simply stopped.

Guard the assignment (`_rc=0; _out=$(...) || _rc=$?`) so the wrapper's
own logic decides: allowlisted transient errors retry with a visible
notice, everything else prints and propagates, exactly the documented
contract.

New unit suite executes the wrapper in a `sh -eu` subshell against a
fake kubectl, the same way chainsaw runs it: a transient etcd error
must be retried with the script surviving (against the previous lib
this exits 1 after one call with empty output), and NotFound must not
be retried, must fail, and must be printed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-31 16:11:08 +03:00
Myasnikov Daniil 007d0b1a22 chore(ci): re-trigger CI after a label event produced a no-op run
The latest run on this head came from a `labeled` event carrying a label other
than full-e2e. pull-requests.yaml discards those by design — `plan` skips, and
checks, build, build-talos, finalize and e2e all skip behind it — so the run
reported nothing. Two side effects made that worse than a wasted run: the
required "E2E Tests" context was left SKIPPED, which branch protection counts
as satisfied, and cancel-in-progress killed the in-flight api-review-gate run
that was carrying the real signal.

Empty commit to get a genuine run against this head. No content change.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-31 11:22:26 +05:00
Andrey KolkovandClaude Opus 4.8 0c14d6cf43 test(e2e): sharpen round-trip diagnostics and make teardown settle
Recommendations from the third review pass (which returned LGTM; none of these
blocked, all three are cheap and make the failure paths honest).

1. The round-trip's catch named nothing. An unnamed `describe HelmRelease` in
   tenant-root dumps every release on the root tenant of a live cluster and
   buries the two that matter, so name them (mariadb-2-backup-roundtrip names
   its release for the same reason). Raised podLogs tail 100 -> 200, since the
   sidecar logs every action and an S3 error sits well back from the tail by the
   time a step fails, and added the Keeper Pods: chkeeper.yaml labels them
   `app: <release>-keeper`, so the existing app.kubernetes.io/instance selectors
   missed them entirely — and a source ClickHouse that never reaches a ready
   replica is usually Keeper failing to form a quorum, which explains itself
   only there.

2. SSL_CERT_DIR is now `/etc/ssl/cozy-s3-ca:/etc/ssl/certs`. The single-directory
   form works — Go reads the default bundle FILE regardless, which is what the
   earlier measurement showed — but that left the public-CA half of the contract
   resting on a file this image happens to ship. Naming the system directory
   costs nothing and survives a base-image bump that moves the bundle; verified
   against public AWS S3 that both forms still verify (403 auth, no x509). The
   mount path stays the bare directory, and the unit test now pins both halves
   separately — it caught the over-broad edit that had briefly made mountPath a
   colon list.

3. cleanup.sh returned while the uninstalls it triggered were still running, so
   the next suite (etcd) could start with helm-controller still draining this
   one, and the pre-clean could hand step 04 an application whose predecessor
   was still going away. It now waits for both ClickHouse releases and the
   Bucket to actually disappear, and — since the script deliberately runs
   without `set -e` so one absent resource cannot abandon the rest — records
   those failures and exits non-zero, rather than reporting a teardown that did
   not settle as success. wait_deleted polls `get` instead of using
   `kubectl wait --for=delete`, which errors on an already-absent resource: the
   normal case here, and indistinguishable from a real failure.

Sized the two cleanup step budgets to 12m for the same reason the run step is
60m: cleanup.sh's waits now sum to 540s, which the previous 8m did not clear.

The Keeper-sized-from-the-server-resources bug the review found outside this
diff is filed as #3487.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 82aa967674 test(e2e): correct the round-trip budget and its stale comments
Second review pass on this branch.

1. The op timeout equalled the inner sum instead of exceeding it: the comment
   enumerated 3900s of inner waits and then set 65m, and 3900s IS 65m — so there
   was zero headroom for the applies and SQL calls between those waits, and the
   claim of "room" was false. Sized the inner budgets against what CI actually
   takes instead (the bucket suite completes in ~37s and a backup-enabled
   ClickHouse plus Keeper reaches Ready in ~98s, measured on run 30353894332):
   bucket claim/access 300->180 and the three job waits 600->480, summing to
   3300s, under a 60m op timeout. Shrinking the inner budgets rather than
   raising the ceiling also keeps this suite closer to its postgres and mariadb
   siblings on a 180-minute job cap.

2. The strategy Pod's securityContext comment still justified root with "Pod
   runs apk add at startup", which the previous commit made conditional — the
   default path now runs the platform's pre-baked image and installs nothing.
   Scoped the note to the alpine fallback and said explicitly that the field set
   matches cozy-default-altinity, runAsNonRoot deliberately unset in both.

3. clickhouse-backup.yaml still referenced clickhouse-backup-{1-contracts,
   2-roundtrip}: the file moved unchanged, so the Test rename never reached its
   own header and a reader following either name found nothing.

4. wait_sts_ready's timeout diagnostic could not print anything. It selected
   clickhouse.altinity.com/chi=${name#chi-}, which yields
   clickhouse-<app>-clickhouse-0-0 while the label carries the CHI name
   (clickhouse-<app>), and 2>/dev/null then swallowed kubectl's "No resources
   found". Ask for the StatefulSet's own first replica by name instead and let
   both streams through; verified it now surfaces the Pod events that explain
   why the replica never came up.

Also dropped the clickhouse bats case added in the previous commit: it asserted
what the generic examples/backups/<app> -> <app> mapping already guarantees, so
it did not lock the invariant its comment claimed. That invariant is now
recorded where it belongs — in the mapping's own comment in select-e2e.sh.

The dedicated ch-backup-client image this example now inherits is tracked in
#3420; noted there that the command: override is load-bearing in two files.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 c1fd7db51d test(e2e): fold the clickhouse backup Tests into the app's own suite
Review findings on the previous two commits.

1. TIA no longer selected the suite that runs the harness. select-e2e.sh maps
   examples/backups/<app>/** to suite <app>, so with the round-trip living in a
   separate clickhouse-backup/ dir an edit to the harness selected `clickhouse`
   — which did not execute those scripts — while the suite that does was left
   out. Fixed the way postgres/mariadb/etcd already do it: both Tests move into
   hack/e2e-chainsaw/clickhouse/ as clickhouse-1-backup-contracts and
   clickhouse-2-backup-roundtrip. That also deletes every special case the
   earlier commits added (src_to_suites, suite_to_source, and the bats case that
   guarded them — both scripts are now byte-identical to main), and the suite
   list in hack/e2e-chainsaw/README.md stays correct because no dir is added.
   The bats case is replaced by one asserting the invariant that actually
   matters: an examples/backups/clickhouse edit selects the suite that runs it.

2. The docs claimed a fail-fast the harness did not have, and the budget was
   sized as if it did. wait_for_field polled to the full timeout with no
   terminal-value escape, and the HelmRelease/StatefulSet waits were bare
   `kubectl wait` with no existence backstop and no Stalled fail-fast. Ported
   wait_for_field's fail_value parameter and wait_hr_ready from the postgres
   helpers, added wait_sts_ready, and routed all eight waits through them.
   Verified against a fake kubectl: a Failed BackupJob now bails in 3s instead
   of 600s, a Stalled HelmRelease immediately. The inner budgets sum to 3900s,
   which the old 30m op timeout did not cover — a hang would have been SIGKILLed
   with no diagnostics, the failure mode .chainsaw.yaml calls strictly worse — so
   the timeout is now 65m with the arithmetic written out.

3. The strategy Pod ran `apk add --no-cache curl jq` on every backup and
   restore. Harmless while nothing in CI ran it; a per-PR dependency on the
   Alpine CDN once it does, and broken outright air-gapped — which is exactly
   why the shipped strategy bakes curl+jq into its image. Step 01 now reads that
   image off cozy-default-altinity and drops the apk add, falling back to
   alpine only when the platform strategy is absent. Also bounded the two
   polling curls with --max-time; a hung connection previously stalled the loop
   until the outer timeout.

4. 91-scenario-user-backup.md still documented the old tenant-test default.

5. .chainsaw.yaml claimed all app tests run in tenant-test, which the postgres
   and mariadb round-trips already contradicted before this branch.

The round-trip header now also states its scope plainly: it exercises the
example's strategy, not the shipped cozy-default-altinity.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 dac34b7469 test(e2e): actually run the clickhouse backup round-trip in CI
The round-trip Test was skipping: it exited 0 behind CLICKHOUSE_E2E_S3_ROUNDTRIP,
so "PASS (1.01s)" meant the gate fired, not that backup and restore worked. Only
the contracts Test ran, which proves the chart still exposes the sidecar and its
auth Secret — not that a byte ever reaches S3 and comes back.

The gate existed because the sidecar could not verify the in-cluster SeaweedFS
certificate. backup.endpointCA closes that, so adopt the shape mariadb and
postgres already use to run their round-trips un-gated:

- examples: resolve the S3 endpoint from S3_ENDPOINT when set (BucketInfo
  advertises the external ingress URL, an unroutable placeholder in CI), copy the
  seaweedfs CA (auto-discovered from its cert-manager Certificate when the
  fullname default is absent) into a per-release Secret, and reference it from
  backup.endpointCA on both the source and the to-copy target. NAMESPACE now
  defaults to tenant-root, matching the other backup demos.
- suite: drop the gate, run in tenant-root — an isolated tenant's Cilium egress
  allowlist blocks its Pods from reaching tenant-root's seaweedfs — and pre-clean
  with cleanup.sh, because tenant-root is shared and a leftover Succeeded
  BackupJob would satisfy the harness's wait without a backup running.

cleanup.sh also drops the copied CA Secret; the platform's own is never touched.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 2687e095f4 feat(clickhouse): let the backup sidecar trust a private S3 CA
The clickhouse-backup sidecar had no TLS/CA surface at all: bucket coordinates
and credentials arrive as S3_* env, and nothing let it verify an endpoint whose
certificate is signed by a private CA. Cozystack's own in-cluster SeaweedFS is
exactly that case — it serves :8333 behind the self-signed "SeaweedFS CA" — so a
ClickHouse release could only ever back up to a publicly-trusted endpoint, while
MariaDB (storage.s3.tls.caSecretKeyRef) and Postgres (barman endpointCA) have had
a CA field all along.

Add backup.endpointCA {name, key}. When set, the chart mounts that Secret into
the sidecar and points SSL_CERT_DIR at it. Empty (the default) renders no env, no
mount and no volume, so existing releases are untouched.

SSL_CERT_DIR rather than AWS_CA_BUNDLE, measured against
altinity/clickhouse-backup:2.7.4 with a self-signed endpoint and public AWS S3 in
turn: Go reads SSL_CERT_DIR in addition to the system bundle, so both verify;
AWS_CA_BUNDLE replaces the SDK trust pool, so the private endpoint verifies and
public S3 starts failing `x509: certificate signed by unknown authority`. That
would silently break a release that also talks to a public endpoint.
S3_DISABLE_CERT_VERIFICATION was rejected outright — it drops verification
instead of trusting one more CA.

Covered by helm-unittest: the unset default, the mount/env/volume wiring, a
custom key still projected as ca.crt, composition with useSystemBucket, and that
backup.enabled=false renders nothing.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 541060c508 test(e2e): assert clickhouse-backup sidecar auth-secret linkage
Extend verify-backup-sidecar to assert the sidecar's API_USERNAME/API_PASSWORD
env resolve valueFrom the backup-api-auth Secret's username/password keys, not
just that the Secret and the :7171 port exist independently. A chart change
that repointed or dropped those secretKeyRefs while keeping both would slip
through and surface only as 401s at BackupJob time — the silent break this
suite exists to catch.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 dbfd56f695 test(e2e): guard clickhouse-backup TIA mapping + tighten suite
- Add a select-e2e_test.bats case asserting a packages/apps/clickhouse diff
  selects BOTH the clickhouse and clickhouse-backup suites (mirrors the
  kubernetes-application multi-suite test). Fails without the src_to_suites
  override that makes the backup contracts suite run on ClickHouse chart PRs.
- Correct the backup-api-auth Secret assertion's prose to match what it proves
  (present with both keys, not "populated").
- Capture the to-copy restore target's Pod logs in the round-trip catch so a
  failure isolated to step 07 keeps its diagnostics.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 4be4594161 test(e2e): map clickhouse-backup suite in select-install.sh
hack/select-install.sh --validate (run in the unit-test job) requires every
Chainsaw suite dir to resolve to a PackageSource via suite_to_source(); the
new clickhouse-backup suite has no <suite>-application fallback, so map it to
cozystack.clickhouse-application, in lockstep with the select-e2e.sh override.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrey KolkovandClaude Opus 4.8 d8e003f4cb test(e2e): add clickhouse altinity backup chainsaw suite
Port the intended ClickHouse Altinity backup/restore e2e coverage from
cozystack#2600 (never-merged hack/e2e-apps/backup-clickhouse.bats) to the
current Chainsaw approach, reworked on origin/main where the Altinity
backup feature and examples/backups/clickhouse already landed.

Two Tests, etcd-suite shaped:
- clickhouse-backup-1-contracts (CI): a backup-enabled ClickHouse, asserting
  the chart<->Altinity-strategy contract against the running cluster — the
  clickhouse-backup sidecar on the ch-backup-api :7171 port and the populated
  <release>-backup-api-auth Secret the strategy Pod authenticates with. Needs
  no reachable object store.
- clickhouse-backup-2-roundtrip: the full Bucket -> BackupJob -> Backup ->
  in-place + to-copy RestoreJob round-trip, driving
  examples/backups/clickhouse/run-all.sh. Gated out of CI
  (CLICKHOUSE_E2E_S3_ROUNDTRIP=1) because kind's self-signed SeaweedFS S3
  endpoint cannot be TLS-validated by the in-Pod sidecar.

Wire clickhouse-application -> "clickhouse clickhouse-backup" in
hack/select-e2e.sh so ClickHouse chart edits exercise the backup suite too.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
myasnikovdaniil 10554f9dce [cozystack-scheduler] Publish SchedulingClass demo (#2252)
## What this PR does

Publish the scripts used for running the Cozystack scheduler demo at the
community meetup on 19.03.2026.

### Release note

```release-note
[cozystack-scheduler] Add demo scripts to the cozystack-scheduler system
packages.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a complete set of interactive Bash demo scripts covering
scheduling classes: creating scheduling classes and tenants, colocating
pods on a single node, spreading replicas evenly, enforcing
one-pod-per-node anti-affinity, and pinning workloads to a specific
node.
* Included guided scaling steps plus cleanup and teardown scripts to
remove demo resources.

* **Documentation**
* Introduced a shared demo helper with colored, step-by-step terminal
logging and Kubernetes polling to wait for pods and Redis readiness,
improving clarity and consistency across demos.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-30 14:20:18 +05:00
Andrei Kvapil e16df56019 chore(codeowners): scope CODEOWNERS to per-area reviewers (#3228)
## What this PR does

Replaces the single catch-all owner line in `.github/CODEOWNERS` with
area-scoped reviewer ownership, so each part of the tree routes to the
people who actually own it (per the Contributor Ladder), instead of
requiring the whole roster on every change.

Highlights:
- Architects (`@kvaps @lllamnyp`) are the default owner and own the API
surface (`/api/`) and the VPC app.
- CI / automation → `@myasnikovdaniil @sircthulhu`; backups → `@androndo
@lllamnyp`; virtualization (kubevirt, vm-instance/disk) → `@scooby87
@lllamnyp @kvaps @myasnikovdaniil`.
- `cozystack-api` / controller / lineage webhook → `@IvanHunters @kvaps
@lllamnyp`.
- General packages / apps / CLI ownership → `@lexfrei @IvanHunters
@myasnikovdaniil` (+ `@scooby87` for apps), with `kubernetes` and
`extra` scoped separately.
- Flux → `@kingdonb`; GPU (hami, gpu-operator, GPU dashboards) →
`@lexfrei`; metrics-server and seaweedfs → `@nbykov0`.
- Governance / licensing files → `@tym83 @kvaps`.

Ordering follows "last match wins" (broad → specific). The API review
gate block is kept last so it cannot be overridden. This is
review-routing only — no runtime/functional change — and only binds once
"Require review from Code Owners" is enabled in branch protection.

This file is what the CNCF Incubation review reads as "code ownership",
so it is paired with #3462, which documents the same people in
`MAINTAINERS.md` and introduces the Reviewers table the ladder already
defined. Merging one without the other leaves the two describing
different projects, which is the gap both PRs exist to close. Three of
the people named here — `@kingdonb`, `@matthieu-robin`,
`@mattia-eleuteri` — hold only read access today, and a CODEOWNERS entry
for someone without write is silently ignored by GitHub, so the org-side
permission changes have to land alongside.

### Screenshots

N/A — governance/config change, no user-facing or UI impact.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walked the trigger map against the diff: no row matches. The change
touches only `.github/CODEOWNERS` — no package, no `values.schema.json`,
no `ApplicationDefinition`, no platform variant, no release asset, no
node prerequisite. Every path named here was checked against current
`main`; all 65 exist.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated CODEOWNERS from a single catch-all to an ordered, path-scoped
reviewer mapping.
* Clarified “last matching pattern wins” behavior and ensured the API
review gate rules remain last to avoid overrides.
* Expanded explicit ownership coverage across dashboards/examples,
packages and Go components, apps, platform/infrastructure modules,
CI/automation, and governance/licensing documentation (including
SECURITY).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-30 11:04:03 +02:00
Timofei Larkin 337bb4933e [cozystack-scheduler] Publish SchedulingClass demo
## What this PR does

Publish the scripts used for running the Cozystack scheduler demo at the
community meetup on 19.03.2026.

### Release note

```release-note
[cozystack-scheduler] Add demo scripts to the cozystack-scheduler system
packages.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-30 11:03:21 +03:00
Myasnikov Daniil 507336dba0 fix(ci): report "E2E Tests" green on a release promotion PR
Under main's rc-gate model a promote PR carries `release` and deliberately not
`full-e2e`, so the `e2e` job skips: the immutable rc whose digests the tree pins
was already verified, which is the cost the rc gate exists to remove. While the
required gate was a job, that skip satisfied branch protection. Now that it is a
commit status, e2e-report saw PLAN_RESULT=success, CODE=true (a promotion
rewrites ~38 non-docs image refs) and E2E_RESULT=skipped, fell to the last
branch and posted failure — so every release would have needed a full suite
re-run or an admin override of the very gate this PR strengthens.

Give e2e-report the carve-out the `e2e` job already has: release without
full-e2e reports success and says why. Adding full-e2e still forces a real run
and a real verdict.

Close the second route to the same red as well. `gh pr create --label` attaches
the label just after opening, so the `labeled` event cancels the `opened` run
through cancel-in-progress; under always() that dying run still posted failure
with PLAN_RESULT=cancelled, and the `labeled` run replacing it is skipped by the
allow-list guard, so nothing flipped the status back. !cancelled() makes a
cancelled run report nothing, which is the same fail-closed posture used
elsewhere here: not posting leaves the required status "Expected", which blocks.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-30 11:54:52 +05:00
Andrei Kvapil 0f0204a6c2 fix(e2e): disable guest fsync on ephemeral CI VM disks (#3455)
## What this PR does

The e2e QEMU guests are launched in `hack/e2e-prepare-cluster.bats` with
three `-drive … if=virtio,format=raw` lines that carry no `cache=`
option, so QEMU falls back to its `cache=writeback` default, which
honors every guest fsync against the host-backed raw disk files.

The management-cluster Talos etcd — which holds every operator's
leader-election lease — fsyncs on each Raft commit. During the VM-heavy
suites, containerd is simultaneously unpacking dozens of images and CDI
is importing multi-GB VM disks onto the same host-backed raw files. The
resulting fsync storm serializes etcd's writes into multi-second stalls,
so the management apiserver returns `context deadline exceeded` on lease
PUTs, and several operators (capk, etcd-operator, fdb-operator,
cilium-operator, kamaji) lose leadership at the same moment. Whichever
chainsaw suite is mid-flight then fails — a different one each run
(vmcluster, kubernetes-latest, vmdisk), which is the fingerprint of
shared-resource contention rather than a bug in any single suite.

This adds `cache=unsafe` to each guest drive. That makes guest flushes
no-ops against the host page cache, which removes the fsync-gated etcd
stalls at the root of the flakiness. It is safe here precisely because
these CI VMs are ephemeral and destroyed at the end of every run — a
crash before writeback loses only throwaway state — and it must never be
used for real data. The change is minimal and uniform across all three
guest drives; no non-guest drive is touched (there are none in this
file).

Runner sizing alone (#3251) did not resolve this, because more CPU and
memory do not change the I/O path: the guest fsyncs still propagate
through `cache=writeback` to the host, and etcd still stalls behind the
same fsync storm. This addresses the I/O path directly.

`git log -S 'cache='` on this file is empty, so a cache mode has never
previously been set here.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
actual diff. The only entries that reference
`hack/e2e-prepare-cluster.bats` (ansible-cozystack and ccp) are scoped
to node prerequisites — kernel modules, containerd settings, the LVM
`global_filter`, and the cluster domain. This change touches none of
those: it adds a QEMU host-side hypervisor flag (`cache=unsafe`) to the
CI VM `-drive` options, which is invisible to the guest OS and to any
real node. No downstream repository is affected.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(e2e): boot e2e QEMU guests with cache=unsafe so ephemeral CI VM fsyncs no longer stall the management-cluster etcd, removing the cross-operator leader-election flakiness that failed a different chainsaw suite each run
```

---

This PR was prepared with the assistance of an AI agent (Claude). A
human maintainer reviewed the diff and the reasoning before submission
and remains responsible for the change.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Updated end-to-end virtual machine startup testing to use optimized
QEMU disk caching settings for improved reliability and performance.
* Improved end-to-end Kubernetes cluster readiness checks under load by
extending the deadline for reaching “at least 2 nodes Ready,” and
aligned related messaging and failure diagnostics accordingly.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-29 19:32:45 +02:00
Andrei Kvapil 3dc0f56477 feat(kube-ovn): build the chart and image in-tree, retiring the wrapper repo (#3460)
## What this PR does

Brings the `kubeovn-chart` wrapper into the tree, so
`cozystack/kubeovn-chart` can be archived.

Until now `packages/system/kubeovn` pulled a chart tarball from the
latest `cozystack/kubeovn-chart` tag at `make update` time, and that
repository also built and pushed the `kubeovn` image. The wrapper
existed only to cache the upstream chart; with the current caching
approach that reason is gone, and the split cost a second repository, a
second release step, and a package whose provenance a reader cannot
follow without leaving the tree.

The package is now shaped like `packages/system/cilium`:

- `update` pulls the upstream `kubeovn/kube-ovn` chart directly and
applies `patches/mtu.diff`, then syncs the version into the Dockerfile.
The MTU patch already carried monorepo-relative paths, so it applies
unchanged.
- `image` builds `images/kubeovn` (carrying the existing
`fix-keepvmip-portgroup` patch) and pins the digest into `values.yaml`,
using the shared `image-tags` / `cache-args` helpers.

It also carries `cozystack/kubeovn-chart#5`, which never got a tag:
`OVN_IPSEC_KEY_DIR` is pinned under `/var/lib`. The kube-ovn default
sits in `/etc/origin`, which a Talos node cannot create, so enabling
IPsec left the node without ESP security associations.

### The nightly host rewrite had to be fixed first

Moving the image build here activates a gap recorded in
`docs/agents/image-refs.md`. The nightly host rewrite is a literal
`<src-registry>/` substring replace, and kubeovn keeps its host in
`global.registry.address` with the bare repository in a sibling key — no
trailing slash, no match. While the image came from a separate
repository the baked reference carried the public host and the ownership
filter skipped it; once CI builds the image here it stamps the
build-registry host into that key, and the result is an image mirrored
to the public registry while the published tree keeps pointing at the
private build registry.

The second commit adds an end-of-line-anchored expression that fires
only when the source registry is the complete scalar value, so a
contiguous reference — which continues past the host — is still handled
by the first expression and the two can never both fire on one
reference. `keycloak-operator` splits at a different boundary and stays
unfixed; that is documented, and dormant while no CI path rebuilds it.

Verified: `hack/nightly-mirror_test.bats` passes with the new case and
the case fails without the fix; `promote-retag` and
`promote-rewrite-tags` suites unaffected; `helm template` and `helm
unittest` on the package pass. The pre-existing `helm lint` complaint
about a missing `kube-ovn` dependency reproduces on `main` and is left
alone.

### Follow-ups this PR does not do

- `cozystack/kubeovn-chart` should be archived once this merges, with a
pointer here. Its open PR #5 is carried by this change and needs no
separate landing.
- `cozystack/website` links the Kube-OVN license at
`cozystack/kubeovn-chart` in `operations/configuration/licenses.md` and
`guides/platform-stack/_index.md` (all versions, including `next`).
Archived repositories stay readable so nothing breaks, but the link
should move upstream or in-tree. **No follow-up has been opened yet** —
flagging rather than ticking the box.

### Screenshots

N/A — build and packaging change, no user-facing or UI impact.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walked the trigger map against the diff. The website row matches — not
through the app-list or platform-component triggers, but because two of
its pages link the repository being archived. Left unticked because no
follow-up exists there yet; a human should decide whether the license
link moves to `kubeovn/kube-ovn` or to this package. No other row
matches: the change touches no `packages/apps/` or `packages/extra/`
package, no `values.schema.json`, no `ApplicationDefinition`, no
platform variant, and no node prerequisite.

### Release note

```release-note
feat(kube-ovn): the Kube-OVN chart and image are now built in this repository instead of cozystack/kubeovn-chart. OVN_IPSEC_KEY_DIR now defaults to /var/lib/ovs_ipsec_keys, so enabling IPsec on Talos nodes no longer leaves them without ESP security associations.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Updated Kube-OVN image build to use upstream chart metadata and keep
registry/version alignment during `update`.
  - Added conditional MTU support for the OVN CNI DS via chart values.
- Introduced `OVN_IPSEC_KEY_DIR` to standardize the OVN IPsec keys
directory.

- **Bug Fixes**
- Preserved VM networking port-group assignments during live migration
when an alive virt-launcher sibling pod exists.
- Improved nightly mirror host rewriting for split and standalone
registry host formats.

- **Tests**
- Added coverage for the end-of-line “whole scalar value” host rewrite
case.

- **Documentation**
- Refreshed “Known gaps” guidance for nightly mirror host rewriting
behavior and limitations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-29 18:19:41 +02:00
Myasnikov Daniil 6728bd4245 fix(ci): don't clobber a fork PR's green "E2E Tests" on a label event
Same defect as the e2e-report fix, on the fork path. main's `labeled`
pull_request trigger produces a "Pull Request" run whose `plan` guard
discards every label but full-e2e, so every job skips and nothing is built.
workflow_run still fires for it, and `resolve` then reddened the required
status down either branch: a non-success conclusion fails closed, and a
`success` conclusion falls through to the missing-pr-patch check, which also
fails closed. Labelling an already-green open fork PR therefore turned its
E2E Tests red until the next push.

Detect that shape directly — the triggering run's `Plan build` job concluded
`skipped` — and return without posting anything. Statuses are per-SHA and an
unposted required status stays "Expected", which blocks the merge, so
declining to post can only preserve a verdict some real run reached, never
invent one. A fork that edits the workflow to force this path leaves its head
SHA with no status at all, i.e. not mergeable. The job already holds the
actions:read scope the job lookup needs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 18:35:14 +05:00
Myasnikov Daniil 8b79a0f20c fix(ci): don't clobber a green "E2E Tests" status on a discarded label event
main gained a `labeled` pull_request trigger whose `plan` guard discards
every label event except full-e2e. e2e-report carried no such guard, so
labelling an already-green open PR started a run in which `plan` was
skipped, e2e-report still ran (always()), read PLAN_RESULT != 'success' and
posted "E2E Tests" = failure — overwriting the passing required status and
making a merge-ready PR unmergeable until the next push.

Give e2e-report the same `labeled` guard as `plan`, so it runs if and only if
`plan` runs. Skipping posts nothing, which is fail-closed: an unposted
required status stays "Expected" and blocks the merge, so the guard can only
ever preserve a verdict, never invent one.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:57:17 +05:00
Myasnikov Daniil 2b53ac32a1 ci(e2e): run the fork e2e job in the workspace, not /tmp
e2e-fork.yaml's e2e job was written against the pre-78fc3f917 in-tree job,
which copied the checkout to /tmp/$SANDBOX_NAME, ran every step from there
and removed it on teardown. main has since dropped that workspace: e2e is a
single job on an ephemeral runner, so there is no cross-job rendezvous and
the checkout is never reused.

Keeping it here would also reintroduce the unguarded
`rm -rf /tmp/$SANDBOX_NAME` that commit deliberately deleted rather than
guarded — when an upstream step fails the unconditional sandbox-id write is
skipped, SANDBOX_NAME stays empty, and the `if: always()` removal expands
to `rm -rf /tmp/`.

Drop the Prepare/Remove workspace steps and the seven `cd
/tmp/$SANDBOX_NAME` prefixes, and point the chainsaw-report and cozyreport
upload paths at _out/. /tmp/changed.txt stays — it is a scratch file, not a
workspace copy, and the in-tree lane writes it the same way.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:48:43 +05:00
Myasnikov Daniil 60e910b863 docs(ci): correct the select-e2e.sh escalation claim in e2e-fork (#3262)
Both comments stated that select-e2e.sh escalates to the full suite on
any unrecognised path, so fork TIA "can only narrow". That is not what
it does: it escalates for an unrecognised path INSIDE packages/, but a
path it does not recognise at all (hack/*.mk, for one) selects nothing,
the Chainsaw step is skipped, and the required status still concludes
success.

Found by coderabbitai in review. The behaviour is pre-existing and
identical for same-repo PRs through pull-requests.yaml, so it is tracked
separately in #3392 rather than fixed on the fork lane alone — which
would leave forks stricter than same-repo for the same diff. Correct the
comments here so the guarantee is not overstated in the meantime.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov Daniil 705e76f5db fix(ci): fail closed on renamed and truncated fork PR file lists (#3262)
Address review feedback from coderabbitai on
.github/workflows/e2e-fork.yaml: the docs-only decision read only
`filename`, so a rename reported the NEW path alone. A PR renaming
packages/<app>/values.yaml -> docs/whatever.md therefore looked
docs-only, concluded "E2E Tests" green, and never ran the suite —
while in fact removing a code path. Both sides of a rename must be
under docs/ for a PR to count as docs-only.

`pulls.listFiles` is also capped at 3000 files, and a truncated list
could omit the only non-docs path. Fail closed at the cap instead of
ruling docs-only on a list known to be incomplete.

Feed both sides of a rename to TIA as well: the old path is a removal
the selector must see. Extra paths can only widen the selection, so
this cannot cause a suite to be skipped.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov Daniil 4f9f599cce test(hack): cover the default (non-export) build path (#3262)
Address review feedback from IvanHunters on hack/common-envs_test.bats:
every existing case set OCI_EXPORT_DIR, so the OFF path — the one every
same-repo PR, main and release build still takes — had no regression
test at all. A refactor of the export wiring could stop pushing entirely,
or strip the release tags, and the suite would stay green.

Add a case asserting that with OCI_EXPORT_DIR unset buildx still pushes
(--push=1) and writes no archive, image-tags expands all three release
tags, and talos' skopeo copies still target docker:// with the
`[ -z "$(OCI_EXPORT_DIR)" ]` release gates inert.

Mutation-checked: making the PUSH/LOAD override unconditional (so export
mode leaks into the default path) fails this case.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov Daniil 914d75f06f test(hack): count --tag occurrences, not matching lines (#3262)
Address review feedback from IvanHunters on hack/common-envs_test.bats:70:
`grep -c` counts matching LINES, not occurrences. The image-tags macro
expands onto a single line, so every "exactly one tag survives under
OCI_EXPORT_DIR" assertion returned 1 no matter how many --tag flags were
actually emitted.

Proven vacuous by mutation: removing the OCI_EXPORT_DIR guard from
image-tags restores three --tag flags on that one line and the suite
stayed green. Switching to `grep -o … | wc -l` counts occurrences, and
the same mutation now fails the suite as it should.

Applied to the --output type=oci,dest= assertions too, which count
occurrences the same way and had the same blind spot.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov DaniilandClaude Opus 4.8 65cd5f629c fix(ci): gate release tags/pushes off under OCI_EXPORT_DIR (defense-in-depth)
Belt-and-suspenders for the B2-class trap lexfrei flagged as latent in
image-tags. PUBLISH_* are 0 on fork PRs, so today the export path never emits a
versioned/floating tag — but if it ever did:

- image-tags: multiple --tag under `--output type=oci` write >1 manifest into
  the archive's index.json, and `skopeo copy oci-archive:<f>` then refuses it
  ("more than one image in oci"). Suppress the versioned/floating tags when
  OCI_EXPORT_DIR is set (macro-level $(if …), so `make -n` prints only the
  taken branch).
- packages/core/talos: gate its versioned/floating `skopeo copy docker://…`
  release copies on an empty OCI_EXPORT_DIR too.

The release (non-export) path is unchanged — versioned + :latest still emitted.
Covered by a new common-envs_test.bats case that forces PUBLISH_*=1 under
export.

Reported by gemini-code-assist on #3262.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:28 +05:00
Myasnikov DaniilandClaude Opus 4.8 1a1f3696ee fix(ci): fix the nullglob-defeated OCI-archive guard in the fork publish
`count=$(ls _out/oci/*.oci.tar | wc -l)` runs under `shopt -s nullglob`, so
when nothing matches the glob expands to zero words and `ls` lists the current
directory instead — count is always >=1 and the "at least one archive" guard
never fires. A zero-archive build then falls through to `flux push` and defers
the real failure to an obscure image-pull error deep in the ~180-minute e2e
run. Glob into an array (empty under nullglob) and test its length; reuse it
for the push loop.

Reported by coderabbitai on #3262.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 3b70fda870 fix(ci): make fork OCI_EXPORT_DIR absolute so archives land where the upload reads
The fork build sets OCI_EXPORT_DIR to a relative `_out/oci`, but each image is
built with `make -C <pkg> image`, so buildx resolves the relative `dest=`
against the package directory — the archive landed in <pkg>/_out/oci/ while the
`if-no-files-found: error` upload reads repo-root `_out/oci/*.oci.tar`. Every
non-docs fork build would have failed at that upload (never caught, since
same-repo PRs leave OCI_EXPORT_DIR empty). Root it at github.workspace so the
dest resolves to repo-root _out/oci regardless of the sub-make cwd.

Reported by gemini-code-assist on #3262 (its abspath suggestion resolves against
the package dir under `make -C`, so this fixes it at the workflow level instead).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 0f6e99eff8 fix(ci): publish the "E2E Tests" gate as a commit status, not a check-run
Route the required "E2E Tests" merge gate through a GitHub commit status
posted with the default GITHUB_TOKEN (+ statuses:write), replacing the
COZYSTACK_CI app check-run. The app path needed the app granted checks:write
(it isn't — checks.create 403s "Resource not accessible by integration"), and
the earlier default-token check-run floated under a random check-suite
("PR size label"). A commit status has neither problem: branch protection
matches it by context name, it carries no check-suite grouping, and it is safe
with the default token because a fork's pull_request token is read-only and
cannot post one — only same-repo runs (e2e-report, guarded !fork) and the
privileged workflow_run in e2e-fork.yaml can set it.

The security property is unchanged: a never-posted required status stays
"Expected" and blocks the merge, so a skipped job can't smuggle a green
(#3257). Removes all COZYSTACK_CI app-token machinery from e2e-fork.yaml —
`resolve` posts the status directly, no `check_run_id` threading, no `open`
step.

Branch protection must require the "E2E Tests" status context (no GitHub App
and no app-id pin needed).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 03b519dc75 fix(ci): fork e2e — Chainsaw+TIA run and image-push hardening
The fork e2e path in e2e-fork.yaml still ran the pre-#2826 BATS app loop
(`hack/e2e-apps/*.bats` + `test-apps-%`), which the merged Chainsaw
migration deleted — so the fork run was already broken. Rewrite it to the
same Chainsaw + Test-Impact-Analysis path as the in-tree job, and clear the
remaining review follow-ups on the publish job.

- Run E2E via a single `make test-chainsaw` invocation (no per-app retry —
  e2e-testing.md §1). Suite selection and the full-e2e override come from
  `resolve`'s TRUSTED base-repo listFiles/labels (new `changed_b64` /
  `full_e2e` outputs), never a git diff of the fork-controlled merge ref, so
  TIA cannot be shaped into skipping the suite (keeps the #3257 property).
  Collect + upload the Chainsaw JUnit report like the in-tree job.
- Restrict fork image pushes to a trusted allowlist derived from the base
  parent (HEAD^1) Makefiles, so a fork cannot mint an arbitrary repo path
  under the CI registry by renaming an image. The charset guard stays as
  defence in depth.
- `skopeo copy --all` so a multi-platform build's index digest (the one
  baked into pr.patch) is preserved.
- Pin the flux CLI (FLUX_VERSION) in the privileged publish job instead of
  install.sh's rolling latest.

Docs: e2e-testing.md §10 records that the fork path now runs Chainsaw + TIA
and that fork suite selection must stay sourced from trusted data.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 9cddb532bf fix(ci): publish "E2E Tests" check-run as the COZYSTACK_CI app
The synthetic "E2E Tests" required check-run was created with the default
GITHUB_TOKEN (the "GitHub Actions" app). A token-created check-run has no
owning job, so GitHub files it under an arbitrary GitHub Actions check-suite
for the head SHA — in practice the earliest one, the PR-size labeler — so
"E2E Tests" rendered under "PR size label".

Publish it as the COZYSTACK_CI GitHub App instead (create-github-app-token,
scoped to permission-checks: write) so the check lands in the app own
dedicated suite. This is also stricter: only the CI app can satisfy the gate.

- pull-requests.yaml (e2e-report): mint the app token, pass it to checks.create.
- e2e-fork.yaml: split resolve into a verdict decision on the default
  workflow_run token (keeps the actions:read / pull-requests:read lookups) and
  an "open" step that publishes the check as the app; report re-mints the token
  so checks.update runs as the same app that opened the pending run.
- docs/agents/e2e-testing.md: document the change and the branch-protection
  requirement.

Requires main branch protection to require "E2E Tests" from the COZYSTACK_CI
app id (not app_id 15368).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:27 +05:00
Myasnikov DaniilandClaude Opus 4.8 06d734cf53 docs(agents): document the fork-PR e2e split and check-run gate (#3257)
docs/agents/e2e-testing.md is the mandated reference for E2E CI changes but
described a single pull-requests.yaml pipeline. Document the split (same-repo
e2e in pull-requests.yaml; fork e2e in the privileged e2e-fork.yaml via
workflow_run), that the required "E2E Tests" context is a synthetic check-run
rather than a job (so renaming the e2e job must preserve it), that fork builds
export OCI archives instead of pushing, and the repo prerequisite Actions →
"Require approval for all outside collaborators".

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 056ff92e16 test(hack): cover OCI_EXPORT_DIR export mode in common-envs (#3257)
Add hack/common-envs_test.bats (runs under make bats-unit-tests). For a
representative slice of the root build: matrix — including the two recipes
that previously slipped through, capi-providers-cpprovider and core/talos —
it asserts that under OCI_EXPORT_DIR each built image emits exactly one
--output type=oci and no ungated docker:// push, and that capi emits a single
--tag. This would have caught the talos-push and capi-two-tag regressions.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 eba55976a0 fix(ci): decide fork docs-only from the PR file list and harden publish (#3257)
resolve inferred "docs-only" from the absence of a pr-patch artifact, but
artifact presence is decided by pull-requests.yaml running from the fork's
merge commit — a fork could suppress it (code=false, or if:false on finalize)
and conclude the required E2E Tests check green with e2e never run, re-opening
the very bypass this feature closes.

Derive docs-only from trusted base-repo data (pulls.listFiles): green only
when every changed file is under docs/; any non-docs file makes e2e required,
and a missing pr-patch then fails the check closed. Resolve the PR open-only
(no closed/merged fallback).

Also harden the privileged jobs:
- drop `packages: write` from publish (it authenticates to OCIR, not GHCR);
- assert at least one OCI archive before pushing, and validate each archive
  basename against a strict charset before using it as a registry repo;
- install flux from a downloaded script rather than piping curl to sudo bash;
- assert `git rev-parse HEAD^2` equals the resolved head SHA in publish and
  e2e, so a moved merge ref cannot operate on a different tree than the check.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 68cdeb8541 fix(ci): emit a single tag for capi-providers under OCI export (#3257)
capi-providers-cpprovider passes two --tag flags; under --output type=oci
buildx writes both refs into the archive index, and `skopeo copy
oci-archive:<f> docker://…` in the privileged push loop then fails with
"more than one image in oci, choose an image".

Gate the secondary :$(IMAGE_TAG)-$(TAG) tag off when OCI_EXPORT_DIR is set so
the exported archive holds a single manifest.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 ab14922495 fix(ci): export talos image as an OCI archive on fork builds (#3257)
packages/core/talos `image-talos` pushed to the registry via `skopeo copy
docker://…` unconditionally — it is not a buildx call, so neither
OCI_EXPORT_DIR nor PUSH reached it. On a fork PR the registry login is
skipped, so this was an anonymous push that died with `denied`, collapsing
build-talos → finalize and the whole chain.

Under OCI_EXPORT_DIR, copy the installer to oci-archive:<dir>/talos.oci.tar
instead of pushing; keep the docker:// push otherwise. A make-level $(if …)
is used (not a shell if/else) so `make -n` prints only the taken branch.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 076658e70f fix(ci): avoid zizmor github-env finding on fork e2e sandbox name
The `Set sandbox ID` step wrote a computed value to $GITHUB_ENV, which the
zizmor github-env audit flags as high severity (write to GITHUB_ENV may allow
code execution). Set SANDBOX_NAME as a job-level env var instead — no
$GITHUB_ENV write, so the audit does not apply. Uses github.run_id, which is
unique per run; in workflow_run context GITHUB_REF is the default branch and
cannot disambiguate concurrent fork runs sharing a self-hosted host.

Refs: #3257

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 76c8f890fa fix(ci): export OCI archives for buildx calls that bypass image-tags (#3257)
Fork PRs build images to per-image OCI archives instead of pushing,
gated on OCI_EXPORT_DIR. The `--output type=oci` was appended only inside
the `image-tags` macro, so `packages/system/capi-providers-cpprovider`,
whose `image:` recipe uses a hardcoded `--tag` and never calls
`image-tags`, ran buildx with no exporter under export mode (PUSH/LOAD
forced to 0). No archive was written, and the fork build's
`if-no-files-found: error` upload then failed the whole build. Its
`--cache-to type=inline` also needs an image exporter.

- hack/common-envs.mk: extract the OCI-output snippet from `image-tags`
  into a reusable `oci-output <image-name>` macro so any buildx call can
  opt in; `image-tags` now ends with `$(call oci-output,$(1))` (no-op for
  the ~33 conforming packages). Move `comma :=` above its first user.
- packages/system/capi-providers-cpprovider/Makefile: add
  `$(call oci-output,cluster-api-control-plane-provider-kamaji)` (archive
  base name equals the OCIR repo the privileged push keys on) and drop the
  inline cache under export (`--cache-to type=inline` gated off).
- .github/workflows/pull-requests.yaml: wrap the `E2E Tests` check-run
  creation in try/catch. On error, `core.setFailed` so the required merge
  gate fails loudly and fail-closed instead of silently succeeding.

Repo-wide audit of every `docker buildx build` reachable from the root
`build:` matrix confirms capi was the only recipe bypassing both
`image-tags` and `oci-output`.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-29 17:47:25 +05:00