Commit Graph
539 Commits
Author SHA1 Message Date
Aleksei Sviridkin bd504e0b4e ci(e2e): run the isp-hosted install daily against the nightly
The lane installs the latest GHCR nightly on a kind cluster once a day,
after the nightly has published. It takes no pull request trigger: a
run would install the nightly's images, not the pull request's, and
report on code the pull request did not change.

The unit test holds the lane's wiring in place, since a lane that
quietly installed another variant would test something other than
its name says. The sandbox headroom check reads any workflow touching
packages/core/testing as one that boots the QEMU guests; this lane
starts only the sandbox container, so it is exempted by name, after
the job match, and is sized like the others if it ever starts calling
the guest targets.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-03 01:42:38 +03:00
Aleksei Sviridkin bf03c84711 chore(deps): move to the Kubernetes 1.37 libraries without the apimachinery fork (#4473)
## What this PR does

This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, and drops the
`k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and
#4472; those were a stack of the same commits.

The four commits are meant to be read one by one, and each builds and
passes the tests on its own:

1. **Plain watches stop getting the initial-events-end bookmark.** The
apiserver defaults `sendInitialEvents=true` onto any watch with an empty
or `0` resourceVersion and leaves bookmarks off, and our aggregated
storages took that flag alone as a WatchList request. From
`k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api
panics on that bookmark, so `kubectl get -w` on any cozystack-api
resource would die after the bump. The marker now needs both flags, the
same condition the watch cache uses. There are tests per storage and one
through the real handler chain.
2. **The apimachinery fork goes.** Every apps kind shares the
`Application` Go type, and stock `Scheme.ConvertToVersion` picks the
first kind registered for it. The fork patched that; the apps group now
encodes through a serializer that keeps the kind the object carries. A
handler-chain test covers create, get, list, update, watch, apply and
table for two kinds and fails without it.
3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the
plugin is on by default and refuses every write until it has synced
`mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and
which clusters before 1.36 do not serve. Without this, every create,
update and delete would fail after the bump. It is also re-applied in
`Complete`, since `--disable-admission-plugins` replaces the list.
4. **The bump itself**, with the regenerated clients, openapi and
strategy CRDs.

Commits 2 and 4 are where the review effort goes; 1 and 3 are small.

Known changes in behaviour:

- Without the fork, the row objects of a Table requested with
`includeObject=Object` carry the first registered app kind. kubectl asks
for that only with `--sort-by`, and it sorts on fields, not on the kind.
Nothing at this layer can fix it, because the handler converts through
`APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks
and ValidatingAdmissionPolicy still see the right kind.
- The alpha PodSpec field `workloadRef` became `schedulingGroup`
upstream. Once the new strategy CRDs are applied, the apiserver prunes
`workloadRef` from stored strategies it serves, and their next write
erases it. It only ever worked behind an alpha feature gate.

The branch is rebased onto the Renovate rollup (#4447), with `go mod
tidy` and codegen rerun on top of it.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application API responses now preserve the requested resource kind
across operations, including lists and watches.
* Backup strategy resources support additional Kubernetes pod settings,
including probe protocols, volume ownership options, and scheduling
groups.
* Backup strategy schemas include updated Kubernetes pod fields and
descriptions.
* **Bug Fixes**
* Initial-event watch bookmarks are sent only when requested; plain
watches continue without an ending bookmark.
* Mutating Admission Policy is disabled by default, including when
admission plugin options are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 21:45:51 +03:00
Aleksei Sviridkin 6df5d076b1 ci(workflows): pin flux, yq and helm-unittest installs
Every flux install fetched fluxcd.io/install.sh and ran it as root, so
whatever that host served at the moment executed on runners that later
hold registry credentials. Use the upstream setup action instead, pinned
by commit and given an explicit version: it checks the release tarball
against that release's checksums file and runs no script fetched at run
time.

nightly and the fork e2e lane still took yq from the floating latest
release. They now use the pinned, checksum-verified install the
promotion steps already have.

The helm-unittest plugin embeds its own Helm and was installed at
whatever version was newest, so a plugin release could change chart
test results on a PR that touched no chart. Pin it, and replace a
different version a persistent runner may carry.

A contract test scans every workflow for these three tools, so an
unpinned flux, yq or helm-unittest install copied from an older step
fails the unit lane.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 11:02:18 +03:00
Aleksei Sviridkin 2d0c359f4c test(e2e): stop bringing up the monitoring instance nothing looks at (#4023)
## What this PR does

This PR turns off monitoring instance of root tenant in e2e install and
removes waits on it. Install patched root tenant with `monitoring: true`
and then waited up to 15m each on vmalert and vmalertmanager,
`vlclusters/generic`, shortterm and longterm vmclusters, `grafana-db`
CNPG cluster and grafana deployment, but nothing in the run looks at any
of it.

No chainsaw suite touches tenant-root monitoring objects. Only one that
mentions monitoring is `foundationdb`, it asserts `WorkloadMonitor`
whose CRD comes from `cozystack-controller` and exists without
monitoring stack. Monitoring OIDC wiring is covered by
`packages/system/monitoring/tests/oidc_test.yaml` since #4019 removed
orphaned BATS suites. Kubernetes suites run with
`monitoringAgents.enabled: false` and tracing is off by default
(`tracingStorages` is empty), so they don't depend on this instance
either.

Those waits were not free. Monitoring stack competes with rest of
install for node capacity, and removed block own comment said slow
victorialogs start can fail install on PR that never touched monitoring
- that's why it was moved to 15m budget.

### What stays

Monitoring package stays installed, because `tenant-application` has
`dependsOn cozystack.monitoring-application` and without the package
platform release is never resolved. Only tenant-root instance is off.

Container lane `logsStorages` override for victorialogs also stays. It
does nothing while monitoring is off, but keeps a run that turns
monitoring back on from creating immutable replicated PVC on lane that
has only `local`. Its comment, paragraph in `docs/agents/e2e-testing.md`
and comment on Install step env in `pull-requests.yaml` now say that
instead of calling root monitoring install-time workload.

### What is lost

`curl` through `root-ingress-controller` with `Host:
grafana.example.org`, it was the only check that root ingress actually
routes to backend. Grafana was the target just because it was installed.
Moving it to seaweedfs host needs expected response checked on live
stand first, so it's tracked in #4615.

### Savings

Not measured. The happy path never spent the 15m budgets, so what this
returns is real bring-up time of the stack plus tail risk, and only CI
runs can show how much it is.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```
2026-10-02 09:58:52 +03:00
Aleksei Sviridkin 30689f8109 fix(ci): map the recurring unmapped scopes and add area/dx
Over the full PR history a set of scopes recurring three or more
times still fell through to area/uncategorized. Most of them have an
obvious home and were only missing a row. Subsystem names such as
`networking` or `storage` are now accepted as the scope of their own
area, since contributors reach for those as often as for a package
name.

`dx`, `hack` and `cozypkg` describe developer tooling, which no
existing area covers: area/build is image builds, area/testing is
tests. They get a new area/dx rather than being filed under either.

The contract test also pins that every map target exists in
labels.yml, since the label sync workflow creates only what that file
defines, and that no key is written twice, since a repeated key
silently overrides the earlier one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:37:22 +03:00
Aleksei Sviridkin a8c3eb5260 fix(ci): strip every leading backport prefix from a PR title
A backport of a backport is titled with two `[Backport ...]` prefixes.
Only the first was stripped, so the second was parsed as a bracket
scope: the real scope and the kind were lost, and the area/release
added by the strip suppressed the area/uncategorized fallback, so the
PR looked labelled correctly.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:21:44 +03:00
Aleksei Sviridkin cd6476ccb7 fix(ci): look up the labeler's type and scope maps by own key only
Both maps are plain object literals, so a title typed or scoped
`constructor`, `toString` or `__proto__` resolved to an
Object.prototype member. That value is truthy, landed in the label
set and killed the step on `l.startsWith`, leaving the PR with no
labels at all.

The new contract test runs the workflow's own script under node with
stubbed context, so it exercises the code as shipped rather than a
copy of it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:21:39 +03:00
Aleksei Sviridkin 5eee22fe28 ci(workflows): cache Go modules and fall back to direct in Go jobs
The unit and controller test job has no module cache and runs on an
ephemeral runner, so every run fetches every module from
proxy.golang.org, and one transient stream reset fails the job.

It now restores the module cache keyed on every go.sum in the tree and
prefetches the modules of every tracked go.mod through a bounded retry,
hack/go-mod-download.sh. The cache is written from main by the build
cache warmer under the same key, because a cache saved by a PR run is
visible to that PR only, while one saved on the default branch is
readable from every PR.

Every job that runs Go on the runner sets
GOPROXY=https://proxy.golang.org|direct so any proxy error falls back
to the origin. The release jobs get only that, not a cache.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
Aleksei Sviridkin 312409ecd2 test(e2e): fail the suite whose pods outlive its cleanup (#4492)
## What this PR does

A Chainsaw suite could pass while the workloads it created kept running
after its own cleanup, and nothing noticed until a later suite failed.
On one E2E run the brokers of four Kafka releases from suites that had
passed and deleted their applications (#4488), plus a TenantControlPlane
sitting on its finalizer with two apiserver pods (#4489), pushed pod
memory requests to the node limit. Harbor then failed on `Insufficient
memory` two suites later, under the wrong name.

This makes the suite that leaked fail, and name what it left behind.

- Chainsaw v0.2.15 has no hook that runs after a test's cleanup: its
`Configuration` carries `error.catch`, which runs only on failure. So
`make test-chainsaw` now runs `hack/e2e-chainsaw-run.sh`, which invokes
`chainsaw test` once per suite directory and checks the cluster between
invocations with `hack/e2e-chainsaw/_lib/leak-guard.sh`.
- Attribution: a pod is charged to a suite when all three hold. It is
not terminal. Its UID was not in the pod list taken before the suite.
The root of its ownerReferences chain was created at or after the suite
started. A platform pod replaced mid-suite by a rollout, an eviction or
a crash traces back to an object the install created, so it is not
charged. A pre-existing pod adopted by a controller recreated mid-suite
is excluded by the UID baseline. A leak is charged once, to its own
suite.
- The suite start is read off an apiserver, from the `creationTimestamp`
of a server-side dry-run create, rather than off the sandbox. With
several control-plane nodes the reading and a root's timestamp can come
from different apiservers, so the rule assumes their clocks agree more
closely than the gap between the reading and the suite's first object;
no skew margin is added. Source for that behaviour: the generic registry
`Store.create` stamps it through `rest.FillObjectMetaSystemFields`, then
`DryRunnableStorage.Create` copies it back without persisting
(`staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go`
and `dryrun.go` at Kubernetes v1.33.0).
- Budget: charged pods get 300s to disappear, the default
`timeouts.cleanup` in `.chainsaw.yaml`; a unit test keeps the two equal.
A suite that overrides its own cleanup timeout (kafka sets 8m) is
covered the same way, because the check runs only after `chainsaw test`
has returned, which is after every cleanup in the suite has finished
under its own timeout; the 300s start from there. The leaks this targets
sat about 110 minutes (Kafka brokers) and 30+ minutes (the
TenantControlPlane) past their suites. Every green suite logs
`leak-guard: suite=<dir> clear after Ns`, so the number can be checked
against later runs.
- Past the budget the run prints `LEAKED-PODS suite=<dir> budget=300s
count=N`, then one line per pod with its phase, deletion state, root and
`app.kubernetes.io/instance`. It raises a `leaked-pods` error annotation
and adds a failed `leak-guard` test case to `chainsaw-report.xml`. The
run continues through the remaining suites and exits non-zero at the
end, as the single invocation did on a red test.
- A suite that already failed on its own is checked once without the
wait. Its leftovers are reported as a `::warning` with `LEAKED-PODS
suite=<dir> checked-once=chainsaw-failed`, since some may still be
tearing down.
- Each pod gets one `kubectl wait` inside the remaining budget, and
nothing is retried. A timeout or a spent budget gives the verdict from
the last good pod list, also when the re-list after it fails, so a leak
fails the suite. A wait that fails only after running for its whole
`--timeout` counts as a timeout whatever it printed: kubectl maps an
expired context to "timed out waiting for the condition" only inside the
watch, so a deadline that passes during its initial lookup surfaces as
"context deadline exceeded" (`wait.go` and `delete.go` in kubectl
v1.33.0). A spent budget is caught before calling kubectl and never
passed as `--timeout=0`, which means "check once".
- Any other error on a wait or a re-list, before the budget is spent and
before any timeout, stops the waiting and is reported as `LEAKED-PODS
suite=<dir> count=N inconclusive="<error>"`. That is a warning, not a
leak failure, so an apiserver blip cannot become a leak red.
- A failed first reading after a suite, including a pod list that cannot
be parsed, leaves the suite unchecked. It fails with a `leak-guard`
annotation and a report case saying the cluster could not be read, not
that it leaked. Errors that are not leaks carry the annotation title
`leak-guard`, not `leaked-pods`.
- Known leaks are listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt`
as `<suite-dir> <issue-ref>`: `kubernetes-latest` and
`kubernetes-previous` against #4489 (both suites create a
TenantControlPlane through `_lib/run-kubernetes.sh`). `kafka` and
`kafka-metadata` were listed against #4488 until #4490 fixed it; those
entries are gone. An allowlisted suite prints its leak as a `::warning`
and does not fail. An entry that names a directory that is not a suite
stops the run before the first suite, so a rename cannot keep an
exemption alive. The PR fixing each issue deletes its lines.

Suite selection is unchanged. The runner runs the named directories when
there are any, and otherwise every directory directly under
`hack/e2e-chainsaw/`, each handed to Chainsaw's own discovery; it
refuses a tree with a test at its top level, where that split would stop
being exact. I verified this with Chainsaw v0.2.15's own
`discovery.DiscoverTests`, built from source at that tag. For the full
tree, for `kafka redis harbor` and for `kubernetes-latest`, the union of
the per-suite discoveries equals what a single `chainsaw test [suites]`
discovers: 57, 7 and 2 tests, identical lists. 57 matches the test count
in an earlier full run's report. A unit test pins that the runner passes
chainsaw the suites and flags it was given, in order. The order changes
too: suites used to run in Chainsaw's own order and now run one
directory at a time in alphabetical order, which moves the kubernetes
suites after the kafka ones. Before #4490 fixed #4488 this starved them
(see the E2E criterion below). With the fix merged the order has no
effect, and a future leak that lands before a memory-heavy suite is
named at the suite that leaked. One behaviour does change: a selected
name that is not a directory used to fail the single invocation before
any test ran; now only that suite fails and the others run.

`chainsaw-report.xml` keeps its path and shape: one `<testsuites>` root
with the summed counts, one `<testsuite>` per suite directory. Its
consumers are the `Collect chainsaw report` / `Upload chainsaw report`
steps in `pull-requests.yaml`, `e2e-fork.yaml`, `nightly.yaml` and
`e2e-tag.yaml`, which `docker cp` the file and upload it as an artifact.
Nothing in the repository parses it. The merge is unit-tested on two
per-suite reports in Chainsaw's real shape, one of them failed, and the
result is checked to parse as XML.

Tests: `hack/chainsaw-leak-guard_test.bats` (31 tests, in `make
unit-tests`; named without the `e2e-` prefix, which the unit lane
excludes) covers the attribution rule, the budget, the allowlist, the
merge and the runner, all against stub `kubectl` and `chainsaw`. Every
rule was mutation-checked: removing the guard call, having the Makefile
bypass the runner, inverting the allowlist match, flipping or tightening
the root-timestamp comparison, removing the UID baseline, removing the
terminal-phase filter, skipping the budget wait, changing the budget,
lowering or raising the depth cap, accepting a stale allowlist entry,
reading an unreadable pod list as empty, dropping the leak case from the
report, skipping directories in the selection, treating any failed owner
read as a missing owner, dropping the separate report for a suite that
already failed, dropping the not-run reason from the report, reading any
wait error as a timeout, removing the inconclusive branch, letting a
failed re-list after a timeout turn a leak inconclusive, ignoring a
failed re-list, reporting an unreadable cluster as a leak, reading an
unparsable pod list as clean, reporting a kubectl warning instead of the
error, passing the report name without its extension, wording a failed
re-list as a failed wait, dropping an allowlist's last line when it has
no newline, dropping the check that `--set-string` carries a value, and
reading a wait that ran its whole timeout but printed a context error as
inconclusive. Each one turns its named test red, and restoring turns it
green.

E2E criterion, registered before the run: in the `Run E2E tests` log of
this PR, every suite that ran prints `leak-guard: suite=<dir> clear
after`, a `LEAKED-PODS suite=<dir>` line carrying `allowlisted=`, or one
carrying `checked-once=` (a suite that failed on its own) or
`inconclusive=` (waiting stopped on a failed read), and no line matches
`LEAKED-PODS suite=.* budget=`. With #4490 merged, `kafka` and
`kafka-metadata` are no longer allowlisted and must print `leak-guard:
suite=kafka clear after` and `leak-guard: suite=kafka-metadata clear
after`, which is the live check that #4490 fixed #4488. The first run of
this PR (before #4490, run 36223896793) is the positive control: it
printed `LEAKED-PODS suite=kafka count=15` and `suite=kafka-metadata
count=8`, allowlisted, and every other suite cleared in 0-37s. That run
also went red on both kubernetes suites, because those 23 leaked brokers
left no memory for the second tenant worker VM (`0/3 nodes are
available: 3 Insufficient memory`), which the order change described
above made possible. The guard has not yet run against a real cluster,
so the first run of this PR should carry the `full-e2e` label; its
`clear after` lines are the first measured teardown times, and any
unexpected `LEAKED-PODS` line is either a real leak or a false positive
to understand before merge.

Follow-ups, known and out of scope here:

- The allowlist exempts a whole suite, so a new leak in an allowlisted
suite stays hidden until the issue's fix removes the line.
- Attribution is per suite directory, not per Chainsaw `Test`; the leak
lines carry the release instance, which usually names the test.
- Only pods are checked. A leaked PVC, LoadBalancer IP or bucket holds
no node memory and is not seen.
- The 300s budget rests on Chainsaw's cleanup timeout, not on a measured
pod-gone latency; the `clear after` lines are where that measurement
will come from.
- A platform object recreated mid-suite (for example a HelmRelease
uninstall/reinstall remediation, or a force-replace on an immutable
field) has a root newer than the suite start, and its pods would be
charged to whichever suite was running as a leak; the first CI runs are
where to look for this.
- The per-pod wait recomputes the remaining budget before each pod, and
no unit test pins that.
- `kubernetes-previous` is allowlisted for #4489 although the stuck
finalizer was seen only in `kubernetes-latest`; both suites create a
TenantControlPlane the same way, and the exemption hides any new leak
there until #4489 is fixed.
- Each clean suite now waits for its own pods to finish terminating
before the next starts, where the tail used to overlap the next suite.
The first full run's `clear after` lines say what that costs against the
E2E job's 215-minute cap.
- After a timeout followed by a failed re-list, the leak lines come from
the list taken before the wait, so a pod whose own wait had already
succeeded is still named and the count can be too high; the verdict is
unaffected.
- Likewise, when every wait succeeded and the re-list then fails, the
inconclusive report still lists the pods whose deletion was just
confirmed. In the boundary case where the budget runs out during the
last successful wait and the re-list then fails, the result is
inconclusive rather than a verdict from the last good list, because the
guard only treats the budget as spent when it finds it exhausted at a
check.
- A mistyped `COZY_LEAK_ALLOWLIST` path passes allowlist validation; it
fails closed, since no suite is then exempt.
- An allowlisted suite is not waited on, so `kubernetes-latest` and
`kubernetes-previous` will print `LEAKED-PODS ... allowlisted=` on most
runs even for pods still in normal teardown; waiting out the budget
first and applying the allowlist only to what remains would stop that
noise.
- A slow teardown in each of several suites can add up to 300s apiece,
against the E2E job's 215-minute cap. On the first full run of this PR
(run 36223896793) the `clear after` values of the suites that were
waited on summed to 210s; the last full run before this change took 158
of the 215 minutes.
- A pod whose owner's resource type is no longer served gets "the server
doesn't have a resource type" rather than NotFound, so its suite is
reported unchecked instead of leaked; it still fails.
- `kubectl wait --for=delete` waits for deletion, while the guard skips
pods in `Succeeded` or `Failed`; a pod that finishes but is never
deleted (a Job under a leaked root) spends the rest of the budget before
the suite shows `clear`. The verdict is right; only job time is lost.
- The unit tests do not pin the one-line empty report Chainsaw writes
for a directory with no tests (`_lib`), nor the branch that catches a
spent budget before calling `kubectl wait`; both are verified against
the Chainsaw and kubectl sources rather than by a test.
- The runner's per-suite report directory from `mktemp -d` is not
removed; the sandbox is discarded after the job.
- Each non-suite directory under `hack/e2e-chainsaw/` (today `_lib`)
costs one empty Chainsaw invocation, because the runner leaves the
decision of what is a test to Chainsaw.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds files under `hack/`, changes the `test-chainsaw` target,
and rewords one comment and the failure line in the four E2E workflows.
The trigger map's `hack/` entry is for cozystack/ccp, whose skills
anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`,
none of which change here. A GitHub code search for `chainsaw` across
the cozystack organisation finds hits only in this repository.

### Release note

```release-note
test(e2e): the E2E app suite now fails the suite whose pods outlive its own cleanup by more than 300s, naming each leftover pod and its owner; suites with a filed leak are listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt and only warn.
```
2026-10-01 14:03:13 +03:00
Aleksei Sviridkin 4d7f9e485a docs(ci): say why the overlay's ancestry tie goes to main (#4469)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This fixes the comment on the overlay's ancestry tie, which described
the wrong reason for the tie. No behaviour changes.

The comment said a line fresh off main is the same as main. That holds
only for the graph. `cozystack-packages:main` is built from main's tip,
so once main moves past the cut, a stack on such a line gets newer
images than the line. The comment now says so. It also says why the tie
still goes to main: a branch forked from main before the cut ties the
same way, and that case is far more common.

This follows up on a review note in #4460 and is stacked on it.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
docs(ci): explain why the base-branch overlay sends an ancestry tie to main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified how artifact selection works for stacked pull requests on
release lines without line-specific commits, including why `main` is
preferred in a tie and how that may affect image freshness.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:02:54 +03:00
Aleksei Sviridkin d8ec8f0af1 fix(ci): run every bats unit file after a failure and cover build-*.sh in the cross-compile check (#4633)
## What this PR does

Two fixes to the tooling of the unit lane.

A local run of the bats unit tests stopped at the first red file and
never ran the files after it, while the output looked like a finished
suite. Each bats file is its own make target, and without `--keep-going`
make stops scheduling targets after the first failure. CI passes `-k`,
so it was not affected. The target now runs the files in a
`--keep-going` sub-make. That sub-make still exits non-zero if any file
failed, and it shares the caller's `-j` slots and `--output-sync`. The
new `hack/bats-unit-keep-going.bats` drives the real Makefile against
two fixture files. It checks that both run and the target fails when the
first one fails, that the target exits 0 when both pass, and that the
sub-make still finds the root Makefile when `MAKEFILES` names another
file. I split the exact-line check on this target in
`hack/bats-runner-coverage.bats` in two to match the new recipe. A
workflow comment and a test name that still called each file a
prerequisite now call it a target.

The cross-compile check never looked at the multus-cni builder. It
treated only a script named exactly `*build.sh` as a compile step, and
multus runs upstream `./hack/build-go.sh`. The pattern now allows a
suffix after `build` but no `/`, and each case has a fixture test. Among
the Dockerfiles the check scans under `packages/`, this matches exactly
one new line, the multus stage, and that stage passes.

Closes #3692 for the make-level truncation. The in-file half, where
`hack/cozytest.sh` ends a file at its first failing test, is tracked in
#4620.

Closes #4500

Things I left out:

- The multus stage passes because a `curl ... TARGETARCH` step sits in
the same stage. The check does not see that `build-go.sh` reads
`TARGETPLATFORM`. The test header already documents this per-stage
limit.
- The sub-make reads only the root Makefile, so an extra `make -f
other.mk` given to the parent no longer reaches the bats targets.
Command-line variables and `MAKEFILES` still do. Nothing in the repo
calls it that way.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): `make bats-unit-tests` runs every bats file after a failing one and still fails at the end, and the image cross-compile check now covers build scripts such as `build-go.sh`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Unit tests now continue running across individual test-file failures
while still reporting an overall failure if any file fails.
* Build-stage detection now recognizes build scripts with names such as
`build-go.sh`.
* **Tests**
  * Added coverage for unit-test results and build-script detection.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 13:17:13 +03:00
Aleksei Sviridkin 71a943277f build(make): keep running bats unit files after one fails
Each bats file is its own make target, so a plain `make bats-unit-tests`
stopped scheduling files at the first red one and never ran the rest,
while the output looked like a finished suite. CI passes -k and was not
affected; a local run was. The target now runs the files in a
--keep-going sub-make, which still exits non-zero when any file fails
and shares the caller's job slots and output sync.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 02:09:49 +03:00
Myasnikov Daniil 3111fa7802 chore(ci): add area/openbao to the label set and the scope mapping
Every openbao PR lands on area/uncategorized, because the labeler has no
scope entry for it and there would be no label to apply if it did. #4168
and #4177 are both sitting on the fallback right now.

labels.yml sets the bar for a new area at three open issues on the topic,
and openbao is at three: #2787, #2483 and #2793. It also follows the
shape already used for a single shipped component, next to area/kamaji,
area/keycloak and area/cilium.

Maps the plain `openbao` scope and `openbao-central` alongside it, since
the platform instance lives in its own package and gets its own scope.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-01 00:24:20 +03:00
Myasnikov Daniil 57bf10acae test(e2e): stop bringing up the monitoring instance nothing looks at
The install turned monitoring on in the root Tenant and then waited up
to 15m each on vmalert and vmalertmanager, vlclusters/generic, the
shortterm and longterm vmclusters, the grafana-db CNPG cluster and the
Grafana Deployment. No suite inspects any of it. The one Chainsaw suite
that mentions monitoring, foundationdb, asserts a WorkloadMonitor, whose
CRD ships with cozystack-controller and exists without the monitoring
stack. The OIDC wiring of the monitoring chart is covered by its
helm-unittest suite rather than by the e2e run.

Those waits were not idle. The stack competes with the rest of the
install for node capacity, and their own comment recorded that a slow
VictoriaLogs start could fail the install on a pull request that never
touched monitoring. The run was paying bring-up time and flake risk for
a component it never looked at.

The monitoring package stays installed: tenant-application declares
dependsOn cozystack.monitoring-application, so dropping the package
would leave the platform release unresolved. Only the tenant-root
instance is turned off.

The container lane's VictoriaLogs storage override stays. It does
nothing while monitoring is off, and it keeps a run that turns
monitoring back on from creating a replicated PVC on a lane that only
has local storage.

One check goes with the Grafana waits: the curl through
root-ingress-controller at grafana.example.org, the only assertion that
the root ingress routes to a backend. Grafana was its target because it
happened to be installed, and pointing it at a seaweedfs host needs the
expected response checked on a live stand first.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-30 22:51:38 +05:00
Myasnikov Daniil 941101b65e ci: drop unused packages:write from the pull-request workflow
The build, build-talos and finalize jobs each requested `packages: write`,
but none of them writes to GHCR. They authenticate to OCIR with
OCIR_USER/OCIR_TOKEN and push to REGISTRY, which the workflow sets to
OCIR. The only GHCR login in the file is in verify-release-candidate,
and that job holds `packages: read`.

GHCR pushes elsewhere in CI authenticate with GITHUB_TOKEN plus
packages:write rather than a stored secret, so on those paths the scope
is the whole credential. Granting it on the pull_request lane hands every
same-repo PR run a token with GHCR write authority that the lane has no
use for. Whether a package honours that write depends on its Actions
access setting, which lives outside this repo, so this is least-privilege
hygiene rather than a fix for a demonstrated exploit.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-30 12:42:50 +05:00
Marian Koreniuk 7500a9fbae test(kubernetes): add the Proxmox e2e suite, parked disabled
Upstream CI has no hypervisor, so this suite cannot run there; it ships
`.disabled` and is run by a lane that owns a Proxmox stand.

Named for what the chart actually creates. The resource definition prefixes the
release, so a Kubernetes CR called k8s-e2e produces
ProxmoxCluster/kubernetes-k8s-e2e in the tenant's own namespace, and the admin
kubeconfig Secret is kubernetes-k8s-e2e-admin-kubeconfig.

Stand-specific values are variables rather than hardcoded. Every one of them is
a bare ${COZY_PVE_*} that run-proxmox.sh expands with envsubst, supplying the
defaults itself — GNU envsubst leaves a ${VAR:-default} in the text exactly as
it is, set or not, and that reaches the apiserver as a string where the schema
wants an integer — and refusing a fixture with a variable still in it. The
StorageClass and the PVC probe read the same COZY_PROXMOX_STORAGE, so the volume
lands where the probe looks. The runner reads the tenant kubeconfig's .conf key,
the address a runner outside the management cluster can reach.
The stand's resource pool comes in through COZY_PVE_POOL, empty by default.
hack/proxmox-e2e-fixture_test.bats pins the shape of the fixture and the render.

The PVC probe reads the storage again after the claim is deleted: a deleted
claim whose disk is still there is the leak the pre-delete hook's Step 1b
exists to prevent. keep_tenant is honoured by the suite's own teardown, not
only by the workflow's. The workflow carries no fallback for the stand's
variables: an unset one fails the job before a key is sent anywhere. The
schedule runs only where vars.PROXMOX_NIGHTLY is true: the job targets a
self-hosted runner with the `proxmox` label, and a scheduled run on a repository
without one would sit in the queue until GitHub cancels it.

The suite also asserts the half of the deployment that lives in the management
cluster — the CSI controller and the cloud-controller-manager — because that is
the property the pair is designed around and a tenant-only assertion would pass
with the token in the wrong place.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-09-29 21:04:06 +02:00
Aleksei Sviridkin e126158404 fix(ci): keep the controller Go tests running when a unit test fails first (#4233)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The `checks` job in `.github/workflows/pull-requests.yaml` runs `make
unit-tests test-controllers -j4 --output-sync=target` as a single step.
`unit-tests` and `test-controllers` are independent goals on that
command line, and without `-k` GNU Make stops handing out new work the
instant any earlier-processed goal's prerequisite fails — a goal that
had not yet been scheduled never starts. So whenever a BATS file inside
`unit-tests` fails before `test-controllers` gets a job slot, the entire
Go controller suite (`go test ./internal/...`) silently never runs, and
the PR page shows only the unit-test failure with no signal at all about
the controllers. This adds `-k` so already-schedulable goals keep
running after an earlier one fails; the job still exits non-zero and
still fails whenever anything failed, `-k` only stops make from
abandoning independent work.

This addresses the second half of
https://github.com/cozystack/cozystack/issues/3776. The first half of
that report — whether the job's comment overclaims that it gates the
merge, since `Unit & controller tests` is not currently in `main`'s
required status checks — needs a maintainer's call between two mutually
exclusive fixes (weaken the comment, or add the job to branch
protection), and is deliberately left untouched here; the issue itself
frames the two as independent and separately useful.

Note: `hack/unit-test-parallelism.bats`, which pins the literal
invocation string in a test, was updated to match.

### Screenshots

N/A — CI workflow and test-script change only, no UI.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map: this PR only touches CI scheduling flags
inside this repo's own `.github/workflows/pull-requests.yaml` and a
`hack/*.bats` meta-test that asserts against that same file. Nothing in
the trigger map names either file, and the change has no effect on any
package, API, chart, or contract another repository consumes.

### Validation

Reproduced the exact scheduling gap in an isolated Makefile mirroring
this repo's real dependency shape (a `unit-tests`-like goal depending on
several independent prerequisites, one of which fails immediately, plus
a sibling goal with no dependency on it), under the same GNU Make 4.4.1
`-j4 --output-sync=target` invocation used here. Without `-k` the
sibling goal never ran, across 5 repeated trials; with `-k` added it ran
to completion every time across 3 repeated trials, and the overall exit
code stayed non-zero either way. `bats hack/unit-test-parallelism.bats`
passes locally against the real files. I also ran `make unit-tests
test-controllers -j4 -k --output-sync=target` against this repo
directly: local disk space (a limit of my sandbox, unrelated to this
change) made `go test ./internal/...` fail to compile with "no space
left on device", so I could not confirm a clean end-to-end pass of
`test-controllers` itself in this environment — but the run still
exercised the fix: after that build failure, unrelated `hack/*.bats`
prerequisites kept being scheduled and ran to completion instead of the
whole invocation aborting, which is exactly the behavior `-k` adds over
the prior command line. I could not run this repo's actual CI runners to
confirm disk is not similarly constrained there; nothing in the workflow
or runner spec suggests it would be.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ci): the `Unit & controller tests` PR check no longer silently skips the Go controller test suite when a Helm-unittest or BATS unit test fails first; both suites now always run and report.
```

---
Part of #3776

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Updated automated test execution so unit and controller test suites
continue running when one test target fails, while the job still reports
failures.
  * Updated workflow validation to reflect the revised test command.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 20:00:54 +03:00
Aleksei Sviridkin afe23a2c3f ci(release): pin flux where the release pushes the packages artifact
prepare-release now runs flux push artifact to publish the rc's packages
artifact, and it installed whatever flux was latest, while build-amd64
and the candidate checks pin one version. It installs that version now.
The arm64 leg and the nightly never run flux, so they no longer install
it. The rc-e2e comment gave a stable tag as the reason for its
!cancelled(), but the job only runs for an rc; it now says it follows
the rule every job below build-arm64 has.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 3f6de5eac0 ci(release): keep every installer tarball out of the release assets
The release job now builds the Talos installer for both arches, so
_out/assets holds an arm64 tarball next to the amd64 one. Both are the
image the build pushed, not something upload-assets publishes, and the
artifact only excluded the amd64 one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 51a2d859ac docs(release): describe what the release job builds and what blocks an rc
The amd64 release job builds the Talos installer and matchbox for both
arches, so the pages that said CI never builds both in one job, and
that build-amd64 builds amd64 only, were wrong. Every rc stitch also
skips matchbox and the e2e sandbox, and the nightly report lists
matchbox and the refs an in-tree rebuild stamps over; the page said
every such skip and line fails the next rc. Name the ones that do not.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 12:36:56 +03:00
Aleksei Sviridkin 71144d4fd6 ci(release): pin the matchbox rebuild in the release job
tags.yaml only runs on tags, so no pull request exercises it. The
matchbox rebuild is the only thing that makes its ref an index, and
losing the step, moving it past the tree capture or letting it switch
the default builder would surface as a failed gate on the next rc. The
comments that called the talos images amd64 only now say they are
built for both arches in the amd64 job.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 11:52:08 +03:00
Aleksei Sviridkin 6ef23da6e7 ci(release): publish matchbox as an amd64 and arm64 index
matchbox already carries the kernel and initramfs of both arches, but
the release built it for amd64 only: the arm64 leg skips the talos
package, so there is no arm64 half to stitch, and the gate let it
through on an allowlist entry. Its Dockerfile only copies files, so the
amd64 job now rebuilds it for both platforms on a docker-container
builder and the stamped ref pins the index. The allowlist entry goes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 11:29:04 +03:00
Aleksei Sviridkin c740031453 Merge remote-tracking branch 'origin/main' into ci/build-arm64-leg
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 11:20:27 +03:00
Pujitha Paladugu 65b02bf5ac fix(ci): keep the controller Go tests running when a unit test fails first
The `checks` job runs `make unit-tests test-controllers -j4` as one
step. Without -k, GNU Make stops scheduling new work as soon as any
prerequisite fails, so when a BATS file in unit-tests fails first, the
independent test-controllers goal never starts and the PR page shows no
signal about the Go controller suite at all.

With -k make keeps running every goal it still can, and the job stays
red on any failure.

Report: https://github.com/cozystack/cozystack/issues/3776

Assisted-by: LLM
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
2026-09-28 12:26:47 -07:00
Aleksei Sviridkin 8c1266649d Merge remote-tracking branch 'origin/feat/harbor-multiarch' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 19:49:55 +03:00
Aleksei Sviridkin 2f08832392 Merge remote-tracking branch 'origin/feat/talos-arm64' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 19:49:46 +03:00
Aleksei Sviridkin 6d759cc73e feat(build): build package images for amd64 and arm64 (#4498)
## What this PR does

Package images can now be built for amd64 and arm64 in one `make image`,
and a published build is multi-arch unless `PLATFORM` says otherwise.
This is the build half of arm64 support from #1961 and #903. CI still
publishes amd64 only until it gets a native arm64 leg, which comes in a
separate PR.

Builder stages now run on the build platform and compile for the target
one, so Go never runs under emulation. Under emulation it is slow and it
crashes: the migration-controller Dockerfile already records a "bad
pointer in Go heap" from exactly that. On top of this, `PLATFORM`
defaults to `linux/amd64,linux/arm64`. A build on an arm64 workstation
can't publish an arm64-only digest any more. That happened once already,
the cilium pin was arm64-only until 3f36a1b45 rebuilt it by hand.

Two images were only right because they were emulated. On a native
builder they would ship the wrong architecture. token-proxy ran `go
build` with no GOARCH, and kube-ovn calls upstream `make build-go`,
which hardcodes `GOARCH=amd64`. Both are fixed now. The new bats test
wants every compiling stage on `$BUILDPLATFORM` and actually using
`TARGETARCH`. A bare `ARG TARGETARCH` does not count, this is how
token-proxy got through.

Some things stay amd64. talos and testing pin it with `override`,
because matchbox serves amd64 Talos assets and the sandbox runs
`qemu-system-x86_64`. `LOAD=1` builds for the host only, since the
classic docker image store can't load an index. Every CI build job pins
`PLATFORM: linux/amd64`, so CI output does not change.

This changes local builds. A `make image` on the default docker driver
with the classic image store now fails with "Multi-platform build is not
supported for the docker driver", where before it built for the host.
Use a `docker-container` builder (`BUILDER=<name>`), or pass
`PLATFORM=linux/amd64` (or `LOAD=1`) to build one architecture.

I checked it on an arm64 host with a docker-container builder.
cozystack-api, dashboard, kubeovn, platform-migrations, cilium and
capi-providers-cpprovider build and push as amd64+arm64 indexes.
Binaries pulled from each platform are x86-64 and aarch64:
cozystack-api, token-proxy, the kamaji provider manager, cilium-agent
and all four kube-ovn binaries. piraeus-server builds for both
platforms, but the push to my test registry failed on registry 500s. The
capi-providers-cpprovider inline cache works with a multi-platform push.
Not covered is running the arm64 images on an arm64 cluster, CI has
none.

This is stacked on #4485 and targets its branch, so the diff here is
only the two commits on top.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#711
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(build): package images build for linux/amd64 and linux/arm64 by default, with Go builders cross-compiling on the build platform; a local `make image` now needs a docker-container builder, or PLATFORM=linux/amd64 / LOAD=1 for a single architecture. CI still publishes amd64.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Build Improvements**
* Container image builds now target both amd64 and arm64 by default when
building without loading images locally.
* CI and release builds are pinned to amd64; Talos and testing images
remain amd64-only.
* Cross-platform builds now compile on the builder’s native platform
while targeting the requested architecture.
* **Documentation**
* Clarified platform defaults and architecture requirements for local
and CI builds.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:43:47 +03:00
Aleksei Sviridkin 11991b9d05 feat(harbor): build the Harbor images for amd64 and arm64
Upstream publishes the Harbor v2.15 component images for amd64 only,
so every Harbor pod crash-loops with "exec format error" on an arm64
node. The multi-arch build upstream exists on main but is in no
release yet, so the package now rebuilds the pinned v2.15.1 release
itself: the Go components cross-compile on the build platform, the
registry and the Trivy adapter build from the tags upstream's Makefile
pins, the Trivy binary is fetched per architecture against its release
checksum, and the runtime images keep upstream's photon base, users
and entrypoints. The build fails when those registry and Trivy pins,
the swagger generator or the golang and node build images no longer
match upstream's Makefile at the pinned release, so a version bump
cannot pair a new core with an old registry, scanner or toolchain.

The portal compiles to static JS, so it lives in its own Dockerfile
listed as architecture-neutral in the cross-compile check, and takes
the release pin from the main one so both files build the same
commit.

Because values.yaml now pins the images, `make update` pins the chart
release that ships the same Harbor version and fails when its
appVersion differs, instead of pulling the newest chart onto old
images. Renovate stops proposing tag bumps for the rebuild's build
bases, which follow upstream's toolchain for that release.

The committed refs stay on the upstream digests until CI stamps the
rebuilt ones.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 02:11:46 +03:00
Aleksei Sviridkin 82aa8a048d ci(release): fail an rc on any single-arch image its e2e pulled
The release gate reads only refs the tree pins by digest, so an image
a vendored chart renders from its defaults, or one an operator
launches, can still be amd64 only. The rc e2e now reads what the
three Talos nodes actually pulled from their image stores, each by the
digest the pull resolved to, and runs verify-multiarch on that list
before the sandbox is torn down. The image store, not the running
pods, because the tests delete what they create.

collect-images.sh writes that list only when every node answered, so
a node that could not be read fails the audit instead of passing the
images only it held. A pre-release fails on the audit; a stable tag's
run only warns, as it ships its rc's images unchanged.

The audit runs in the tree of the tag under test, so a tag cut before
it existed has neither the script nor the image list. That run warns
and skips instead of failing, so such an rc can still be re-validated
from main; a tree that has the script and no list still fails.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 01:32:37 +03:00
Aleksei Sviridkin e8d4f757e2 ci: build main for arm64 nightly and report its single-arch refs
The arm64 images are built only for a pre-release, so a change that
breaks one would first surface as a failed rc. A daily build of main
on the arm64 pool surfaces it the next morning instead, and is the
one writer of the arm64 build cache that the rc leg only reads.

Before building, it prints hack/verify-multiarch.sh's report on
main's tree, the list of refs the next rc's gate would fail on unless
the stitch turns them into indexes. Nothing consumes its images and
it is not a required check.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 01:32:36 +03:00
Aleksei Sviridkin 67fa638f2a ci(release): publish every pre-release image as an amd64 and arm64 index
A release was amd64 only: prepare-release built on an x86 runner and
nothing built arm64. A pre-release now builds its images twice, side
by side: build-amd64 on x86, build-arm64 on a native arm64 runner
under <tag>-arm64, reading the arm64 build cache. Neither waits for
the other, so the rc takes as long as the slower build rather than
the sum of both.

prepare-release runs once both have succeeded. It applies the tree
build-amd64 stamped, stitches each image and its arm64 twin into one
index under the tags build-amd64 logged, gates on every pinned image
being multi-arch, republishes the packages artifact and the installer
chart pinned on the index digests, and only then commits, creates and
publishes the release and pushes the staging branch. The stamped
tree, the tag log and the assets reach it as artifacts that must
exist, so a lost hand-off fails loudly. The assets are built in
build-amd64, reusing the Talos images make build produced, while the
arm64 build is still running.

An rc whose arm64 or amd64 build fails is not published at all rather
than published amd64 only. A stable tag skips the arm64 build: it
never builds, and promotion copies the rc's indexes with every
platform. build-amd64 keeps the release-exists no-op and the
rejection of a hand-pushed stable tag, and prepare-release no-ops
with it. A skipped ancestor makes the implicit success() false for
every job below it, so each job below prepare-release states
!cancelled() itself; without it the stable-tag changelog backstop
would never run. The arm64 build makes the same no-op when the tag's
release already exists, so re-running a prepared rc rebuilds nothing.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 01:32:36 +03:00
Aleksei Sviridkin 1bc7effadb fix(ci): let the fork e2e lane check out the pull request again
actions/checkout v4.4.0 refuses to check out a fork's pull request ref
from a workflow_run workflow unless the step opts in. The Renovate
roll-up that moved e2e-fork.yaml to v4.4.0 made the publish job fail
at checkout for every fork PR, so the required "E2E Tests" status now
goes red on all of them regardless of the change.

The lane was built to take fork code this way: publish runs no
fork-authored script and e2e holds no push token. Opting in restores
the behaviour it had on v4.3.1, and a unit test fails if either
pull-request checkout loses the opt-in.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 20:37:41 +03:00
Aleksei Sviridkin 6bb563f123 feat(release): publish the arm64 Talos assets
arm64 users can only take the Talos ISO, disk images, kernel and
initramfs from the Image Factory today, because a release carries the
amd64 ones only. The tag build now builds the assets for both arches
and uploads the arm64 set next to the amd64 one under the same naming.

It also pushes the talos installer image as a two-arch index, because
a node installed from the arm64 assets pulls its installer from that
image. The rest of the build stays amd64.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-09-26 17:53:13 +03:00
Aleksei Sviridkin 2df6015414 ci(talos): keep the Talos builds in CI amd64
The talos package now builds for every arch in PLATFORM, and the jobs
that build it without setting PLATFORM would pick up arm64 as well:
the PR Talos build, the nightly disk and the release assets. None of
them has a consumer for arm64 output, since e2e boots amd64 and the
release publishes the amd64 assets only, so each one pins amd64 the
way the image builds already do.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 17:44:19 +03:00
Aleksei Sviridkin 3dde6503ec refactor(api): keep each app's kind without the apimachinery fork
Every apps kind is registered on the one Application Go type, and the
stock Scheme.ConvertToVersion picks the first kind registered for a
type, so without help a Redis comes back from the API as whichever app
kind was registered first. So far the fix lived in a fork of
k8s.io/apimachinery that patches ConvertToVersion. Upstream has not
taken the change, and the fork has to be rebased onto every Kubernetes
minor before the libraries can move.

The apps group now encodes through a serializer that picks the kind the
object carries, along the lines upstream suggested instead of changing
Scheme. The replace directive goes, and with it the Renovate rule that
kept the fork's stale release tags from being proposed as updates. The
serializer relies on the storage stamping the kind on every object it
returns, which it does. A test drives create, get, list, update, watch,
apply and table through the real handler chain for two kinds and fails
without the serializer on upstream apimachinery.

One path converts through the Scheme rather than the serializer and
goes back to the first registered kind: the row objects of a Table
requested with includeObject=Object, which kubectl asks for only with
--sort-by. Admission webhooks and ValidatingAdmissionPolicy get the
request kind set explicitly.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 16:08:24 +03:00
Aleksei Sviridkin b6a30bd504 docs(e2e): describe the per-suite runner and the pod-leak guard
The agent guide, the suite README and a workflow comment said the E2E
step is a single chainsaw invocation; it is now one per suite, and a
suite whose pods outlive its cleanup fails. Say how a pod is charged
to a suite, what the budget is, and how the allowlist is kept honest.
The workflows' failure line now points at LEAKED-PODS as well, since a
run that failed only on a leak has no assertion diff to point at.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:19:49 +03:00
Aleksei Sviridkin 084ac47237 feat(build): build published images for amd64 and arm64 by default
PLATFORM defaulted to empty, so buildx built for the host alone. On an
arm64 workstation `make image` published an arm64-only digest, which
amd64 nodes then fail to pull; the cilium pin shipped exactly that
until it was rebuilt by hand. A build now targets linux/amd64 and
linux/arm64 unless PLATFORM is set, and the digest a package stamps is
the index digest.

LOAD=1 keeps building for the host only. The classic docker image store
cannot load a multi-platform index, and loading is the documented local
workflow.

This changes local pushes: a `make image` on the default docker driver
without the containerd image store now fails with "Multi-platform build
is not supported for the docker driver", where it used to build for
the host. Such a build needs a docker-container builder (BUILDER), or
LOAD=1 or an explicit PLATFORM for a single architecture.

talos and testing pin linux/amd64 with `override`, placed before the
include because common-envs.mk expands BUILDX_ARGS immediately. The
Talos assets are amd64: matchbox bakes the kernel and initramfs into
its image, and the e2e sandbox boots the disk under qemu-system-x86_64.
An arm64 variant of either would still serve or boot amd64 files.

CI pins linux/amd64 on every image build. Nothing there sets up QEMU,
fork builds export a single-platform archive, and tags.yaml builds on
the default docker driver, so the default would break or emulate those
jobs. Native arm64 builds belong in a leg of their own.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 15:14:12 +03:00
IvanHunters 95e68bad86 ci(workflows): use a literal match in the flux version guards
The guard ran `grep -qx "flux: v${FLUX_VERSION}"`, where the dots in the
version are regex metacharacters, so a client printing `flux: v2X8X6`
would satisfy it. Use `grep -Fqx` for a fixed-string match in the build
toolchain guards this change added.

Also correct the step comments that still read "install if absent
(idempotent)": the step now installs whenever the pinned version is not
the one present, not only when flux is missing.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:58 +03:00
IvanHunters 8c0f91ff00 ci(workflows): pin flux version in remaining install-latest steps
The nightly toolchain steps still fetched flux through the installer's
floating "latest". Pin them to FLUX_VERSION 2.8.6 with the same
version-gated guard so nightly's flux push artifact step uses a fixed,
reproducible CLI rather than whatever latest resolves to.

Also switch e2e-fork's already-pinned step from a presence check to the
same version gate, so a runner with a pre-cached older flux is corrected
rather than left as is, and fix that step's comment to point at the flux
distro cozystack actually ships
(internal/fluxinstall/manifests/fluxcd.yaml, v2.8.0).

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:47 +03:00
IvanHunters edbc4f1b3e ci(workflows): pin flux version in build toolchain steps
The "Set up build toolchain" steps installed flux with a bare
`curl https://fluxcd.io/install.sh | sudo bash`, letting the installer
resolve its floating "latest". latest had advanced to v2.9.5, and a
matrix job on PR #4429 failed when that release's checksums transiently
4xx'd during install.

Pin FLUX_VERSION to 2.8.6 and guard on the installed client version, the
same shape the "Set up promotion toolchain" steps already use, so the
flux CLI is fixed and reproducible across CI instead of following latest.
This does not remove the checksum host's transient errors; it removes the
uncontrolled version bump. Related to the yq floating-latest pin in #3640.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-26 14:53:47 +03:00
Aleksei Sviridkin 2c2c0301f7 chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447)
## What this PR does

This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.

Before the bumps:

- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.

Held back on purpose, with the reasons in the commit messages:

- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.

govulncheck reports nothing reachable in any of the five modules,
standard library included.

Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.

Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.

Closes #4487.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added a restore resource policy option for Velero strategies.
  * Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
  * Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 10:17:29 +04:00
Aleksei Sviridkin cce3387f8d docs(ci): say why the overlay's ancestry tie goes to main
The comment claimed a line fresh off main is the same as main, but the
cozystack-packages:main artifact is built from main's tip, not from
the cut commit. Once main moves past the cut, a stack on such a line
gets newer images than the line's, until the stack holds a commit of
the line's own.

State that limit, and why the tie still goes to main: a branch forked
from main before a cut ties the same way and is far more common.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-25 01:19:52 +03:00
Aleksei Sviridkin 9c5c5f9d4e fix(ci): overlay the right line's refs for a PR on a feature branch
The base-branch overlay pulled cozystack-packages:<base_ref>. A stacked
PR's base is a feature branch such as fix/foo: the slash is not valid
in an OCI tag and no artifact is published for it, so the pull failed,
the step only warned, and every package the PR did not rebuild ran its
committed release ref. With a committed ref older than its chart, that
broke E2E (Grafana plugin 404).

main and release-X.Y still read their own artifact. Any other base
reads the artifact of whichever of main and the release lines it has
the fewest commits on top of. Always picking main would put main's
images under a release line's charts for a stack rooted on that line,
the cross-generation mix that fails install.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 23:54:28 +03:00
renovate[bot] a7dfbe6f8c chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:04 +03:00
Aleksei Sviridkin 509bd9362f chore(renovate): stop proposing tags of the apimachinery fork
The replace for k8s.io/apimachinery points at a pseudo-version of the
cozystack/apimachinery fork that carries the ConvertToVersion patch.
The fork's tags mirror upstream, so Renovate offers one as an update
that would silently drop the patch and downgrade the module.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:02 +03:00
tym83 e24ffa3bce docs(governance): nominate @themoriarti as Reviewer for the Proxmox integration
Marian has driven the Proxmox integration since the first Proxmox bundle
and is now landing the Cluster API infrastructure provider, CCM and CSI.
Giving him code-owner approval on those packages lets Proxmox changes be
reviewed by the person who knows them best, without granting approval
rights over the rest of the tree.

Assisted-by: LLM
Signed-off-by: tym83 <6355522@gmail.com>
2026-09-24 16:17:39 +05:00
Timofei Larkin 06b41da030 feat(apps): declare the Kubernetes and Kafka name caps in their charts
Both caps are facts about what each chart renders, and nothing outside
the chart can derive them. Every Kubernetes worker pool is a separate
KubernetesNodes release named kubernetes-nodes-<name>-<pool>, so a
cluster name has to leave room for the default md0 pool inside the
53-character Helm release name limit: 32 characters. A Kafka KRaft
controller pod is named kafka-<name>-c-<hash>-<id> with an 8-character
hash, and that hostname has to fit the 63-character DNS-1123 label
limit: 44 characters.

Each chart now states its cap with a @name annotation, which
cozyvalues-gen v1.7.0 publishes as x-cozystack-name in
values.schema.json and hack/update-crd.sh carries into the
ApplicationDefinition. The pre-commit job moves to that release, the
first that understands the annotation.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-09-23 10:59:51 +03:00
Alexey Artamonov 42a27567ce fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.

Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.

Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.

Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.

Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.

Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.

Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.

Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.

A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.

Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-15 23:34:52 +03:00
Myasnikov Daniil 975ea6acb7 ci(workflows): gate Talos builds on their inputs
Classify the rename-blind changed-path list against direct Talos
sources and shared build inputs. This lets same-repository pull
requests skip the heavy build while removals, renames, and build
configuration changes still select it. Fork builds remain
unconditional.

Preserve the matchbox digest produced by a successful Talos build when
finalize overlays base-branch image references. When the job is
skipped, bootbox continues to receive the current base reference.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-09-10 17:48:22 +05:00
Andrei Kvapil 774f788016 ci(agents): check commit trailers on every pull request (#4103)
## What this PR does

`docs/agents/contributing.md` makes a few properties of a commit message
blocking at review: the attribution trailer takes the value `LLM` and
nothing else and appears once, no message carries a session trailer or a
link to a transcript, no line is a generation byline naming the tool.
Nothing checked any of it, so the only reader was a human scrolling `git
log`. That reader is expensive and misses things. One open PR carried
eight commits naming a model in the trailer, another four, all found by
hand after the work was done, each one costing a review round and a
rebase.

`hack/check-commit-trailers.sh` decides all of it from the text, and the
`Commit trailers` job in `.github/workflows/pre-commit.yml` runs it over
the commits between the merge base and the head.

Over all 6062 commits on `main` it rejects 1300, every one a true
positive and no prose false-fires: 1289 attribution trailers naming a
model, 24 `Generated with <tool>` bylines, 15 commits carrying two
trailers, two `claude.ai/code/session_...` links.

### Where it lives

Next to the other per-PR text lint in `pre-commit.yml`. The unit-test
job in `pull-requests.yaml` was the other candidate and is wrong for
this: it is gated on the diff touching code, so a docs-only PR skips it,
and a trailer is wrong or right whatever files the commit changed.

The range is `merge-base..head`, recomputed every run, so a rebase or a
force-push is read as it now is. It never reaches the history on `main`,
roughly a quarter of which carries the old form.

### Decisions worth arguing with

It fails rather than warns. A warning in a job nobody watches guards
nothing.

It is not a required check, and that part is deliberate. Branch
protection on `main` requires `pre-commit` and `E2E Tests`, and `Commit
trailers` is a new context, so a PR can still merge with it red. 77 of
the 200 open PRs carry at least one commit this rejects, so requiring it
today turns those red at their next push. Flipping it is a
branch-protection change once the open set has drained, and somebody
else's call.

Fork PRs are checked like any other. The job runs on `pull_request` with
a read-only token and reads commit messages as data.

A merge commit with no trailer of its own passes. Nothing is required of
a commit, only what is written is judged, so a merge commit that does
carry a bad trailer fails like the rest.

Backport branches are exempt. The backport action cherry-picks merged
commits verbatim, so those messages are history and not the author's to
rewrite. The exemption matches the generated name
`backport-<pr>-to-<branch>`, which is the only signal a `pull_request`
event carries, since the label sits on the original PR. So it is a
convention, not something enforced.

The checker is read from the base branch instead of the checked-out PR.
A `pull_request` event takes the workflow definition from the merge of
base and head, so the job shows up on every open PR the moment it lands,
including the ones whose head predates the script, where reading it from
the head is exit 127 and a red check saying nothing about trailers. It
also keeps the checker out of reach of the PR being measured, though the
job around it is not.

Two shapes are left to the reviewer. A model named in `Co-authored-by:`
needs a list of model names to tell it from a person, and such a list
either misses next month's model or fires on somebody's colleague; most
of the 771 such lines on `main` name a model, so it would be deciding a
lot of PRs. A `Generated-by:` trailer has never appeared here at all.
The byline rule matches a capital `G` for a related reason:
case-insensitively, one line in 25 on `main` is an ordinary sentence
that wrapped onto "generated with cozyvalues-gen".

### Known costs

`hack/check-commit-trailers.sh` matches `hack/[^/]+\.sh$` in
`select-e2e.sh`'s `full_suite_pattern`, so this PR and any later one
touching the script escalates to the full e2e suite. That pattern's own
comment scopes it to the e2e harness, which this script is not, and
`inert_config_pattern` already carries `hack/[^/]+\.bats$`. Escalation
is fail-safe so it is cost rather than breakage, and reclassifying it
wants its own review.

The step drops the explicit `git fetch origin "$BASE_REF"` that
`api-review-gate.yaml` keeps before its own `merge-base`. `fetch-depth:
0` already fetches `+refs/heads/*:refs/remotes/origin/*`, and if that
were ever wrong the failure is a loud `merge-base` error under `set -euo
pipefail`, not a silent pass.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. The two entries that could fire
are `cozystack/ccp` and `cozystack/community`, both keyed on a change to
the commit convention itself. The convention text is untouched here,
only its enforcement is new, so both copies stay accurate. `ccp`'s
`package-bump` skill already writes `Assisted-by: LLM`, which this check
accepts.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(agents): a pull request now fails when one of its commits carries an attribution trailer whose value is not `LLM`, a second attribution trailer, a session trailer or transcript link, or a `Generated with <tool>` byline. The check reads only the commits the pull request adds; backport branches are exempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Pull requests now include automated checks for commit attribution
trailers, session links, and generation bylines.
  * Backport branches are excluded from this validation.

* **Documentation**
* Added guidance describing the commit-message checks, accepted
attribution formats, exceptions, and how to run the validation locally.

* **Tests**
* Added coverage for valid and invalid commit-message formats, range
handling, merge commits, and exempted content.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 23:02:10 +02:00