Commit Graph
7777 Commits
Author SHA1 Message Date
Andrei Kvapil df157da618 ci(workflows): open the E2E breakpoint for maintainer-authored PRs (#3158)
## What this PR does

Decouples opening the E2E SSH breakpoint from the `debug` label.

Today, the only way to open a breakpoint on a failed feature PR is to
add the `debug` label. That same label also flips every job's `runs-on`
onto self-hosted runners, so you cannot get a breakpoint on the normal
ephemeral runner — debugging always drags the whole pipeline onto
self-hosted infrastructure.

This adds a maintainer-authorship path to the breakpoint's `if`
condition: a failed PR whose author is an `OWNER`, `MEMBER`, or
`COLLABORATOR` now opens the breakpoint automatically, with no label
required and without moving the jobs onto self-hosted runners. The
breakpoint simply holds the default `oracle-vm` ephemeral runner the job
already runs on.

The existing `debug` and `release` label paths are unchanged, and the
`authorized-users` list still controls who may actually SSH into an open
breakpoint (second defense layer). Workflow triggers are untouched —
this is purely an `if`-condition change.

### Release note

```release-note
ci(workflows): open the E2E SSH breakpoint automatically on failed maintainer-authored PRs, without requiring the `debug` label or moving the jobs onto self-hosted runners
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Updated E2E failure debugging so breakpoints can now open for pull
requests authored by repository maintainers, not just when specific
labels are applied.
* Clarified the conditions and runner behavior for breakpoint access in
the workflow documentation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-05 14:29:45 +02:00
IvanHunters a537bb647b fix(kubeovn-webhook): harden certificate reloader
Address review of the reload path:
- Observe the cert file mtime before reading the key pair, so a Secret
  swap racing the read is retried on the next handshake instead of being
  cached under a newer mtime and missed.
- Record the load attempt regardless of outcome, so a persistently
  unreadable file is retried only once its mtime advances, not on every
  handshake.
- Detect change with mtime inequality instead of a strictly-newer
  comparison, catching equal-mtime replacements and backward clock steps.
- Log stat failures instead of swallowing them, so a broken mount is not
  invisible until expiry.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 15:24:17 +03:00
IvanHunters 0612a538a3 fix(e2e): bound the Talos-capture failure-branch forensic kubectl calls
The describe/get-events/get-certificaterequests calls that run when the
reader Certificate or the diagnostics Pod misses its readiness wait were
unbounded, yet those branches fire precisely when the management apiserver is
wedged. A stalled forensic call there can burn minutes of the 50m step budget
before the pre-existing (a2)/(c) diagnostics and the EXIT-trap tenant
snapshot run. Add --request-timeout=30s to them, matching the happy-path
capture calls.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 15:13:56 +03:00
IvanHunters 791d99d8e5 fix(kubeovn-webhook): widen serving cert renewBefore to 720h
renewBefore: 24h on a one-year certificate leaves a one-day window for
cert-manager to rotate the Secret and for the webhook to pick it up.
Widen it to 720h (30 days) so renewal happens well ahead of expiry.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 15:04:43 +03:00
IvanHunters 44d9f534a9 fix(kubeovn-webhook): reload serving certificate on cert-manager renewal
The webhook loaded its TLS key pair once at startup and served it as a
static tls.Config.Certificates entry for the lifetime of the process. A
cert-manager renewal of the backing Secret was never picked up by a
running pod, so once the certificate expired (about a year after install)
the pod kept presenting the expired certificate. The kube-apiserver's TLS
call to the webhook then failed and, because the MutatingWebhookConfiguration
uses failurePolicy: Fail, every pod creation in tenant namespaces was
rejected (including virt-launcher pods, blocking all VMIs).

Serve the certificate through a reloading tls.Config.GetCertificate
callback that re-reads the key pair when the mounted files change and
keeps serving the last good certificate on a failed reload. Add a
regression test that fails against the static implementation and passes
once the certificate is reloaded.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 15:04:35 +03:00
Aleksei Sviridkin 4ca64a99f4 ci: re-trigger checks after label events cancelled the real run
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 14:58:16 +03:00
IvanHunters cfbfd770b8 fix(e2e): bound the tenant Talos capture kubectl calls
The in-guest diagnostics chain runs only when the sandbox is already
degraded enough to miss the 18m node-join deadline (the documented
management-etcd fsync-storm regime), yet its kubectl legs were unbounded
while every talosctl leg is wrapped in timeout. A stalled apiserver call in
the (b) capture could consume the remaining step budget before (a2), (c) and
the EXIT-trap tenant crust-gather run, so a diagnostics feature could reduce
total diagnostic yield under the exact conditions it targets.

Add --request-timeout=30s to the raw Certificate/Pod apply, the three
reader-Secret reads and the worker VMI list, and route the Certificate Ready
wait through kubectl_wait_retry so a single transient etcd-leader flap
retries instead of aborting the whole capture. Update the bats assertion for
the reworded wait call. Also correct a stale comment: the chainsaw step
timeout is 50m, not 40m.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 14:47:49 +03:00
IvanHunters 5e3c286e20 fix(e2e): read files with cat in the Talos diagnostics bats helper
assert_file_contains used $(<file), a bash-only construct. cozytest.sh is
run under /bin/sh (dash on the Ubuntu CI runner via Makefile), where
$(<file) expands to an empty string instead of the file contents, so every
assertion after it failed and reddened the unit-tests job, which in turn
skips the e2e gate and leaves the diagnostics path unexercised. Read the
file with $(cat ...), which is POSIX and behaves identically under bash and
dash.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 14:19:50 +03:00
Aleksei Sviridkin cd57ee6678 test(multus): anchor the master-CNI assertion to the top-level key
The check that multus is pointed at the cilium conflist matched its
JSON fragment anywhere in the rendered daemon config.  The same text
nested inside any other key satisfies it, so a config whose effective
multusMasterCNI names a different conflist passes.  Multus uses that
value as written when it is non-empty -- it delegates to whatever is
named, so a wrong-but-existing name silently routes through the other
CNI instead of cilium, which is the state this assertion exists to
prevent.  Auto-discovery happens only when the value is empty, and a
name that does not exist fails manager startup rather than falling
back.

Anchor it to the top level, where the template writes its keys.  That
pins the position rather than parsing the document; deciding which key
is effective needs a JSON parser, which helm-unittest does not have.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 14:19:47 +03:00
Aleksei Sviridkin 7739d1e294 feat(multus)!: stage reference CNI plugins from the multus image
Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (bridge, macvlan, ipvlan, ...),
and nothing guarantees a node has them in the directory Multus
searches.  On k3s they live under /var/lib/rancher/k3s/data/cni while
Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails
there with `failed to find plugin "bridge" in path [/opt/cni/bin]`.

Bundle 14 of the release's plugins into the multus image and add an
install-cni-plugins init container that installs them into the host
/opt/cni/bin, mirroring the existing install-multus-binary container.
loopback, dhcp, tap and dummy are left out; the Dockerfile records why.

Gate it on a new chart value, stageCniPlugins, resolved per bundle and
on for both.  Talos bundles only bridge, firewall, flannel, host-local,
loopback and portmap, so the other ten are missing there exactly as on
generic Linux; the four that overlap are rewritten with the same
upstream version, v1.9.1 on both sides, rather than an older one.  A
marker in the image Dockerfile records the Talos release that pin was
compared against, and a test holds it to the release the platform
ships.  multus therefore moves out of system.common-packages into the
two variant branches, the way linstor already does for talos.enabled.
networking.stageCniPlugins overrides the variant default in either
direction.

Where staging is on the install is unconditional: a plugin is replaced
on every pod recreation, not only when the destination is missing.  A
destination that is already a directory is reported and left alone,
since replacing it is not something this can do safely.  Turning the
value off afterwards does not undo it; nothing here removes or
restores a plugin.

The container is not privileged and takes no mount propagation.  It
runs in the spc_t SELinux domain, without which an enforcing node
refuses the write.  Each plugin is installed under a temporary name and
swapped in with a rename: /opt/cni/bin is shared with cilium and
kube-ovn.  The plugin tarball is checksummed before extraction.

A plugin that cannot be installed is reported on stderr and to
/dev/termination-log, and the container still exits 0: Multus stays the
node's primary CNI and nothing removes its config, so an init container
that never completes would leave the shim with no daemon behind it.

The manifest change lives in customize-deployment.patch so it survives
make update.

BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI
plugins in /opt/cni/bin on existing nodes, Talos included -- four of the
names it writes are ones the Talos node image also carries.  Opt out
before upgrading: declining afterwards does not restore what was
replaced.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 14:19:47 +03:00
Aleksei Sviridkin 517dd11d1a fix(tests): wait for operator rollout instead of condition=Available
condition=Available answers whether the Deployment sits within the
slack its rollout strategy allows, not whether the operator came up.
That is the wrong question for an install gate, and it stays wrong
whatever the strategy is: any maxUnavailable equal to replicas drives
the minimum available replica count to 0 and the condition then reports
True with no running pod. Measured against an operator stuck in
ImagePullBackOff, the old check passed in 156 ms, and the install went
on to fail two minutes later on the CRD wait, which pointed the
investigation at the wrong component.

rollout status reads updated and available replicas directly and only
succeeds once the updated pod is available, which holds under any
rollout strategy. Dump the deployment, pods, operator logs and events
on failure so the cause is legible where it is detected.

Drop --wait from the helm step. Helm 3 decides a Deployment is ready
with the same replicas - maxUnavailable arithmetic, so on the current
strategy it is satisfied at zero ready pods and gates nothing; were the
strategy tightened it would instead fail the install on its own 2m
budget before this gate runs, hiding the diagnostics behind a bare Helm
timeout. That arithmetic is specific to Helm 3, since Helm 4 reworked
--wait into a wait strategy with a different default, so treat the
reason as version-bound. The conclusion is not: one gate that reports
what it saw is better than two that can disagree about whose budget
expired first, whichever major is installed. Hooks are waited for with
or without the flag. The 5m budget here covers scheduling and a cold
pull of an image that is not in the prepull set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 13:13:26 +03:00
cozystack-ci[bot] eeaab0f21c docs: add changelog for v1.6.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-08-05 10:04:43 +00:00
Aleksei Sviridkin 80a2090444 fix(kubernetes): retry the tenant CNI install instead of uninstalling it
The tenant cilium HelmRelease carried no install strategy, so Flux
applied its default, RemediateOnFailure, which uninstalls the release
between attempts. Cilium is the tenant cluster's only CNI, so every
cycle takes it away and gives it back when the next install applies the
manifests again, and retries: -1 lets that repeat without end.

What sets a cycle off is an install that outruns its budget, which can
happen while the tenant still has no registered node, since the operator
Deployment cannot become Available until one exists. Remediation neither
causes that delay nor is fixed by this change; what it adds is a
repeated teardown of the CNI on top of a delay that resolves itself once
the node arrives. This removes the amplifier, not the cause.

RetryOnFailure keeps the applied manifests in place and retries a failed
install as an upgrade, so the rollout completes on its own once a worker
registers. Both actions carry the strategy because the retry of a failed
install runs as an upgrade, which would otherwise fall back to upgrade
remediation. It is the strategy and retry interval cozystack-api and
cozystack-operator already set on every HelmRelease they generate,
except that they leave remediation nil where this release keeps it. The
other tenant addons the chart renders still carry the remediation-only
shape; this one is where the teardown removes the cluster's only CNI
rather than churning a single component.

The cost is that a genuine upgrade failure no longer rolls back either,
it retries on the failed manifests instead. That is forced, since the
controller cannot tell a retried install from a real upgrade, and what
it replaces was an unbounded rollback-and-retry flap rather than a
stable safety net.

The remediation blocks stay, since they cover the branches no strategy
reaches: an absent release, which consults the install block, and an
out-of-sync one, which consults upgrade.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-05 13:04:16 +03:00
Myasnikov Daniil 60e44b24f1 fix(e2e): make tenant Talos diagnostics reliable
Fix admission, capture setup, timeout headroom, and portability.

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-05 14:52:31 +05:00
Myasnikov Daniil c8ec60cfe3 test(e2e): remove optional search tool dependencies
Use Bash and awk assertions in the Talos diagnostics regression tests so
they run in the standard E2E sandbox without optional search utilities.

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-05 14:20:33 +05:00
Myasnikov Daniil 80152b1a07 fix(e2e): capture Talos logs for tenant worker join failures
Tenant Kubernetes worker join failures currently stop at host-side VMI
and CSR state because the worker-only topology does not materialize a
tenant talosconfig.

Mint a short-lived os:reader client certificate and run talosctl from a
hardened same-namespace Pod after the Ready deadline. Preserve bounded
dmesg and kubelet captures for every worker without masking the original
test failure.

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-05 13:18:20 +05:00
IvanHunters 8b006391c2 test(backupstrategy-controller): mirror all six cozy-default routes in the gate fixture
The gate reads the BackupClass as the manifest of what must exist, but the Go
fixture listed only CNPG, Etcd and the two Velero routes while the shipped
backupclass-default.yaml also routes MariaDB and Altinity. Add both to the
fixture, and to the RESTMapper and list-kinds derived from it, so the
expected-object set the tests exercise equals the real route set.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 10:09:04 +03:00
IvanHunters bf152ba864 fix(backupstrategy-controller): resolve the default BSL through the RESTMapper
The BSL existence check used a hardcoded GVR and the dynamic client, whose
Get returns a plain 404 (IsNotFound) for an unserved API group. The
IsNoMatchError branch meant to skip an absent Velero API was therefore dead
code, and if Velero were uninstalled after bootstrap while bslEnabled=true
the BSL would be counted missing and force a Helm upgrade forever. Route the
lookup through the RESTMapper like the strategy loop, so an absent velero.io
API is a NoMatch we skip. A regression test fails without the change.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 09:55:12 +03:00
IvanHunters aa3accd128 fix(backupstrategy-controller): stop the default-objects gate looping when velero.bslEnabled=false
The cozy-default BackupClass routes VMInstance/VMDisk to the Velero
Strategy CRs unconditionally, but the chart gates those CRs on
velero.bslEnabled. With the BSL disabled they never render, so the gate
counted them as permanently missing and forced a Helm upgrade every
MinForceInterval forever, climbing force_reconciles_total and pinning the
missing gauge. Skip Velero-kind strategyRefs when the BSL is disabled
(empty VeleroNamespace), the same flag that gates their render. The
regression test now omits the manually-created Velero objects so it fails
without the skip. Correct the values.yaml comment that wrongly claimed the
Strategy CRs still ship in this mode.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 09:52:18 +03:00
IvanHunters affdd70e9d fix(flux-shard-operator): inherit probe timeout, correct comments
Review follow-ups:
- Do not force the cloned startupProbe TimeoutSeconds to 1: keep the value
  inherited from the liveness handler, otherwise a future flux-aio with a
  larger liveness timeout would get a stricter startup probe and could
  recreate the crashloop. Covered by a regression guard (liveness timeout 5s
  must be inherited).
- Soften the proxy-strip rationale to the A/B-confirmed mechanism (blocking
  external HTTPS call at startup) and note the remote spec.kubeConfig
  exception that does not apply to cozystack in-cluster guest apiservers.
- Correct the test fixture comment: the proxy env is operator/patch-level, not
  installer-injected (the installer injects only KUBERNETES_SERVICE_*).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-04 23:22:59 +03:00
IvanHunters d44c87d939 fix(flux-shard-operator): add startupProbe to cloned helm-controller
The cloned shard Deployment inherits only a liveness probe (~30s window,
no startupProbe), so a controller still syncing caches on startup is killed
before it serves /healthz and never recovers. Derive a startupProbe from the
liveness handler with a generous budget so a slow start is not fatal, while
liveness still catches a wedged running pod.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-04 23:08:39 +03:00
IvanHunters c475faae0f fix(flux-shard-operator): strip proxy env from cloned helm-controller
A sharded helm-controller inherits HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the
flux-aio helm-controller container. A standalone shard needs no external
egress (source-controller fetches artifacts), so behind an unreachable
corporate proxy it makes a blocking startup HTTPS call that never completes,
the manager never serves /healthz, and the liveness probe crashloops the pod
forever, freezing every HelmRelease carrying a sharding key. flux-aio itself
survives only because it starts once and never restarts.

Drop the proxy env (and the now-pointless NO_PROXY) when cloning, alongside
the existing localhost/KubePrism sanitisation, so the shard talks to the
in-cluster apiserver directly.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-04 23:07:10 +03:00
IvanHunters c010745e8e Merge branch 'main' into feat/kubernetes-nodes-phase2b
Bring the worker-pool split branch up to date with main (111 commits),
picking up the node-join and CI fixes it was missing: the 18m tenant
node-join deadline, guest-fsync-off on ephemeral CI disks (#3455),
kubevirt-csi client rate-limit raise (#3428), the Cilium ingress-IP
race guard (#3430), and etcd-operator v0.5.4.

Conflict resolution: main's #3535 (render the talos-reconcile Job for
the default md0 group) modified packages/apps/kubernetes/templates/talos/
talos-reconcile-job.yaml, which this branch deletes because the worker
split moves that Job into the kubernetes-nodes chart. Kept the parent
template deleted and dropped the parent-scoped test
talos_reconcile_nodegroups_test.yaml: the empty-nodeGroups md0-default
gap it guards cannot occur in the per-pool child chart, and migration 54
materialises the implicit md0 into an explicit child HelmRelease.

targetVersion resolves to 55 (migration 54 on top of main's 53).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-04 21:52:36 +03:00
myasnikovdaniil ee26a50f0e fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix (#3510)
## What this PR does

Raises the CloudNativePG operator image **and its CRDs together** to
1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 /
cloudnative-pg#9981). Symptom: after a simultaneous `resources` + `size`
change on a single-instance Postgres, the operator deletes the sole
primary Pod, classifies the PVC as `resizing`, and never recreates the
Pod — the cluster wedges with zero instances and the disk FS resize
never completes.

Upstream publishes a CloudNativePG chart only per minor (`.0`/`.1`), so
there is no 1.28.2 chart on the 1.28 line. Following the pattern of
#3526 / #3528, `patches/cloudnative-pg-1.28.2.patch` (applied by `make
update`) raises the vendored chart's `appVersion` and CRDs from 1.28.1
to 1.28.2 in lockstep. The operator image follows `appVersion`, so there
is no `image.tag` pin (keeping operator and CRDs aligned and avoiding
#3479).

Verified #3479-safe: there are no CRD `status` field changes between
1.28.1 and 1.28.2 — only the `extensions` spec grows. Backportable to
release-1.6 as-is (same 0.27.1 base chart).

### Release note

```release-note
fix(postgres-operator): align CloudNativePG operator and CRDs to 1.28.2, fixing the PVC resize-deadlock that wedged single-instance PostgreSQL clusters after a simultaneous resources+size change
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added support for PostgreSQL extension configuration in catalog and
cluster schemas.
  - Made extension image settings optional where appropriate.

- **Bug Fixes**
- Updated CloudNativePG to version 1.28.2, including the PVC resize
deadlock fix.
  - Kept the operator image and CRDs aligned at the same version.

- **Documentation**
  - Clarified versioning and chart update guidance.

- **Tests**
  - Added checks confirming operator image and CRD version alignment.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-04 20:09:19 +05:00
Alexey Artamonov bcb36262ef fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix
Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster.

Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following #3526/#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified #3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-08-04 17:31:05 +03:00
myasnikovdaniil 7cef85f988 feat(release): cutting an rc freezes the line into release-X.Y (#3399)
## What this changes

Cutting the first `vX.Y.0-rc.N` now creates `release-X.Y` at the tagged
commit and closes the line. Every later cut for that line must be
dispatched from the branch; a dispatch from `main` is refused.

Before this, `release-X.Y` was created only when the stable promote PR
merged. So `rc.2` and `rc.3` were cut from `main`'s tip and silently
absorbed every feature merged since `rc.1` — the shipped release could
contain code that no rc had ever validated. `main` now reopens for the
next minor immediately, and fixes reach the release by cherry-pick.

## Commits

| Commit | What |
| --- | --- |
| `feat(release)` | `cut-prerelease.yaml`: create `release-X.Y` after
the tag push; refuse a frozen line from `main` |
| `ci(backport)` | `backport.yaml`: target the newest existing
`release-X.Y` branch |
| `docs(release)` | Rewrite the RC/Regular/Patch sections, diagrams, and
workflow reference |

## Behaviour changes worth a close look

**The promote PR base flips from `main` to `release-X.Y` on its own.**
No code change causes this — `promote-rc.yaml` already prefers
`release-X.Y` whenever it exists, and after the freeze it always does.
Consequence: the digest-vendored `Prepare release vX.Y.0` commit stays
on the release line and no longer lands on `main`. This matches how
patch releases already work, but it is a real change for `.0` releases
and is easy to miss in review.

**`finalize`'s `Ensure maintenance branch` becomes a no-op.** The merge
commit is already the branch tip, so the fast-forward has nothing to do.
No change was needed there.

**Backport targeting changes during a freeze window.** `backport.yaml`
derived its target from `getLatestRelease`, which returns the newest
published *stable*. Since `release-X.Y` now exists before `vX.Y.0`
ships, that name lagged by a full line for the whole freeze window — a
`backport` on a fix for the release being stabilised would have landed
on the previous line and missed the release it was written for. It now
enumerates real `release-X.Y` branches and takes the newest. Outside a
freeze window the result is unchanged.

This also drops a latent bug: `previous` was computed as `min - 1`,
which names a branch that does not exist whenever a minor is skipped.
Sorting is now numeric, so `release-1.10` correctly outranks
`release-1.9`.

## A note on image pins

An earlier revision of this PR also replaced the committed
`ghcr.io/cozystack/cozystack/*` digests on `main` with an unpullable
sentinel tag, on the theory that they are a build leftover that nothing
restores — and that the freeze makes this worse, since the promote PR no
longer merges into `main` to refresh them.

That was wrong and has been dropped. Per #3143, PR CI rebuilds only the
packages a diff touches (`hack/build-matrix.sh` "emits only the units
whose package dir changed") and the installer bundles the whole
digest-patched tree, so **every untouched package deploys from its
committed pin**. A sentinel would have made those unpullable across E2E.

The staleness problem is real, though, and #3143 already tracks it as a
systemic gap. Evidence gathered while investigating is posted there
rather than acted on here.

## Testing

- Full `hack/helm-unit-tests.sh` green
- Branch-selection logic tested against the live 424-branch list,
including double-digit minors (`1.10 > 1.9`) and skipped minors; outside
a freeze window the result is unchanged (`release-1.5` / `release-1.4`)
- `zizmor` clean on both workflows
- No `values.yaml` or `images/*.tag` file is touched by this PR

## Release note

```release-note
Cutting a release candidate now creates the `release-X.Y` branch and freezes the release line. Features merged to `main` after the first rc no longer ship in that release; fixes must be cherry-picked or backported onto `release-X.Y`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Release branches are now created and frozen when the first release
candidate is published.
* Later pre-releases and patches use the frozen release branch while
`main` advances independently.
* Release-line package artifacts can be generated manually, including
for recovery.
* Backport targets are selected from existing release branches, with
clearer failures when a prior line is unavailable.
* **Documentation**
* Updated release, patch-release, workflow, promotion, finalization, and
cleanup guidance to reflect the new freeze model.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-04 18:53:22 +05:00
myasnikovdaniil a7149d11a3 chore(etcd-operator): bump etcd-operator to v0.5.4 (#3529)
## What this PR does

Bumps the cozystack etcd-operator packages from **v0.5.3 to v0.5.4**.

`v0.5.4` is a controller bug-fix release — no API, RBAC or values
changes:

- fix(controllers): derive `--initial-cluster-state` from phase, not
from the seed
([cozystack/etcd-operator#355](https://github.com/cozystack/etcd-operator/pull/355))
- fix(controllers): stop exempting the bootstrap seed from self-heal
([cozystack/etcd-operator#354](https://github.com/cozystack/etcd-operator/pull/354))
- fix(controllers): extend crash-loop self-heal to memory members
([cozystack/etcd-operator#352](https://github.com/cozystack/etcd-operator/pull/352))
- fix(controllers): switch the PDB from `maxUnavailable` to
`minAvailable`
([cozystack/etcd-operator#351](https://github.com/cozystack/etcd-operator/pull/351))

Changes in this repo:

- `packages/system/etcd-operator/Chart.yaml` — `appVersion: v0.5.3 →
v0.5.4`. The manager image tag defaults to `.Chart.AppVersion`
(`values.yaml` keeps `tag: ""`), so this reimages the controller to
`ghcr.io/cozystack/etcd-operator:v0.5.4`.
- `packages/system/etcd-operator/Makefile`,
`packages/system/etcd-operator-crds/Makefile` — `ETCD_OPERATOR_REF:
v0.5.3 → v0.5.4`.
- `packages/system/etcd-operator-crds/templates/etcdmembers.yaml` —
re-vendored at v0.5.4 via `make update`. Description-only change to the
`/scale` `replicas`/`selector` field docs, tracking the PDB
`minAvailable` fix. `etcdclusters` and `etcdsnapshots` are
byte-identical to v0.5.3.
- `templates/rbac.yaml` intentionally left as-is:
`manager-role-rules.yaml` is byte-identical between v0.5.3 and v0.5.4.

**Upgrade path (PDB switch, cozystack/etcd-operator#351).** This is the
one change that rewrites live objects: the operator moves each
EtcdCluster's PodDisruptionBudget from `maxUnavailable` to
`minAvailable`. Setting both fields is invalid, but upstream
`reconcilePDB` handles the migration — it treats a surviving
pre-migration `maxUnavailable` as divergence and explicitly clears it
(`MaxUnavailable = nil`) before writing `MinAvailable`, so existing
clusters are reconciled cleanly on upgrade rather than wedging their
PDB.

Verified locally: `helm lint` and `helm template` pass for both
packages; rendered manager image resolves to
`ghcr.io/cozystack/etcd-operator:v0.5.4`. These `packages/system/*`
packages have no `generate:` target and no `values.schema.json`, so
there is nothing for `make generate` to regenerate.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file-by-file
against the diff:

- The diff touches only `packages/system/etcd-operator*` — no
`packages/apps/**` or `packages/extra/**` add/rename/remove, no
`packages/core/platform` or `installer` values, no Talos bump, no
asset-name or dev-tooling change → **website / ansible-cozystack** not
reached.
- The CRD edit is description-only, inside etcd-operator's own
`etcd-operator.cozystack.io` CRDs — not the provider's hand-typed
`Package`/`Plan`/`RestoreJob` types, and no new managed app →
**terraform-provider-cozystack** not reached.
- No `hack/` change, no `packages/system/<name>-rd/cozyrds/**`, no
`ApplicationDefinition` CRD / `chartRef.kind` enum change, no
`cozyhr`/`package.mk` contract change, no telemetry-metric or
proxy-label rename → **ccp / talm / cozyhr / cozy-proxy /
telemetry-server / examples / external-apps-example** not reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(etcd-operator): bump etcd-operator to v0.5.4 (controller bug-fixes: PDB switched to minAvailable, crash-loop self-heal extended to memory members, --initial-cluster-state derived from phase, seed no longer exempt from self-heal)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Updates**
  - Updated the etcd operator to version v0.5.4.
- Clarified scale-related resource descriptions, including replica
counts, selectors, and disruption budget behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-04 18:44:00 +05:00
myasnikovdaniil 2f8c953cf5 fix(kubernetes): render the talos-reconcile Job for the default md0 group (#3535)
## What this PR does

Fixes #3504.

The chart has two loops over worker node groups and they read different
sources. `templates/cluster.yaml` iterates the effective set through the
`kubernetes.nodeGroups` helper, whose else-branch supplies the built-in
`md0` group when the user declares none — that is how #2936 made `md0`
removable without a Helm values merge re-adding it.
`templates/talos/talos-reconcile-job.yaml` iterated the raw
`.Values.nodeGroups` map instead, which is empty in exactly that case.

The consequence is a dangling reference. The chart cannot render the
`TalosConfigTemplate` itself — it needs the apiserver Service ClusterIP,
the Talos CA and the Kubernetes CA, none of which exist while Helm is
executing the template — so `cluster.yaml` deliberately stopped
rendering it and the Job became its only producer, while the
`MachineDeployment` still names it in
`spec.template.spec.bootstrap.configRef`. On a cluster with the default
`nodeGroups`, `MachineDeployment/<release>-md0` renders and zero Jobs
render, so `TalosConfigTemplate/<release>-md0` is never created and CAPI
blocks every Machine that ever joins the group. The same Job also
patches `KamajiControlPlane.spec.network.certSANs` with the live Service
ClusterIP, so that is skipped too.

Nothing fails at install time, because the built-in group carries
`minReplicas: 0`. The failure waits for the first scale-up, which is the
documented path rather than a corner: `values.yaml` tells operators that
enabling the ingress-nginx addon on a cluster with default `nodeGroups`
means waiting for the cluster-autoscaler to bring `md0` up in response
to the controller Pods becoming Pending, and the autoscaler is deployed
unconditionally whenever the tenant has an etcd DataStore.

The fix is to range over the helper so the Job set tracks the
MachineDeployment set exactly. A cluster that declares its own groups
renders byte-identically — which is why the pinned content-hash fixtures
in `tests/talos_templates_test.yaml` are untouched — and `md0` stays
removable, because the fix defers to the helper rather than merging
`md0` in unconditionally. On upgrade it is additive: the Job appears
where there was none, applies the `TalosConfigTemplate` the
MachineDeployment already expects, and the content-hash name suffix
means Helm creates a fresh Job rather than attempting to patch an
immutable one.

### How this survived

Every helm-unittest fixture and every e2e suite declares `md0`
explicitly. `tests/nodegroups_default_test.yaml` does render the
empty-`nodeGroups` case, but lists only `templates/cluster.yaml`, so it
never looked at a Job. The two OIDC chainsaw lanes do install with an
empty map, but assert that the HelmRelease exists rather than that it
becomes Ready, so a worker that never boots is invisible to them.
Offline `helm template` cannot reach this path at all: `cluster.yaml`'s
instanceType validator needs a live `lookup` for `u1.medium`.

`tests/talos_reconcile_nodegroups_test.yaml` closes the gap and pins
both halves of the helper contract — an empty map yields exactly one
`md0` Job carrying `GROUP_NAME=md0`, and a two-group map yields exactly
two Jobs with no `md0` among them, so a future fix that merges `md0` in
unconditionally fails here. Reverting the template change fails the
first case on the document count, so it guards the behaviour rather than
restating it.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff: it is one Helm template loop and one new helm-unittest suite
inside `packages/apps/kubernetes`. No package is added, renamed or
removed; no `values.yaml`, `values.schema.json`,
`ApplicationDefinition`, version enum or default changes; nothing under
`hack/`, no namespace, variant, label, annotation or metric renamed. The
behaviour now matches what `values.yaml` already documents, so no
reference page drifts.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(kubernetes): a tenant Kubernetes cluster created without an explicit `nodeGroups` map now gets a talos-reconcile Job for the built-in `md0` node group. Previously that Job rendered only for explicitly declared groups, so the default `md0` MachineDeployment referenced a TalosConfigTemplate nothing ever created: every worker the cluster-autoscaler added to `md0` stayed stuck waiting for its bootstrap config, and the KamajiControlPlane certSANs patch the same Job performs never ran. Clusters that declare their own node groups were unaffected.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Ensured the built-in `md0` node group receives its Talos
reconciliation Job when no node groups are configured.
* Prevented duplicate reconciliation Jobs when custom node groups are
specified.
* Preserved correct group-specific settings and control-plane
configuration in generated Jobs.

* **Tests**
* Added regression coverage for default and custom node-group scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-04 15:19:54 +05:00
Myasnikov Daniil 83fce03839 test(release): scope the freeze pins to their steps and cover the guard
Three assertions extracted a step block, checked it was non-empty, and
then ran their grep over the whole workflow. An unrelated step carrying
the same condition would keep them green after the protected step lost
its own guard, which is precisely the slip they exist to catch. Each now
greps its own block, and `id: freeze` moved into the freeze step's test
where it belongs.

The force-detection greps looked for `+refs/heads/` and `+refs/tags/`,
which is not how git spells a forced refspec: the `+` leads the refspec,
so `git push origin "+HEAD:refs/heads/$BRANCH"` force-moves the branch
and passed both checks. Match a leading `+` on any push refspec instead.

Adds a test for the new build-release dispatch guard: the pattern
anchored at both ends, the ref_type comparison and not merely its env
wiring, a hard failure rather than a skip, and the guard ordered before
the checkout.

Every pin mutation-tested — forcing either push with a `+` refspec,
parking the freeze condition on another step, dropping `id: freeze`,
removing or unanchoring the dispatch guard, demoting it to exit 0, and
relocating it after the checkout all turn the suite red.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-04 14:23:51 +05:00
Myasnikov Daniil dfb1a41f28 ci(release): refuse a build-release dispatch from a non-line ref
The workflow_dispatch trigger added for the freeze takes any ref, and the
"Run workflow" ref selector preselects the default branch — which is
exactly where this workflow's own recovery advice, and cut-prerelease's
warning, send an operator. IMAGE_TAG is github.ref_name, so a dispatch
from main spends a two-hour `make build` republishing
cozystack-packages:main and every main image tag while build-main.yaml
may be writing the same ones: the 409 tag-collision class #2711 fixed.

Nothing is ever mislabeled, since the content really is that ref, so the
cost is a wasted runner rather than a wrong artifact. Guard it anyway: it
is five lines, and the mis-click is easy.

Anchored at both ends so the per-release staging branches (release-1.6.1,
release-1.6.0-rc.4) do not qualify — those are built by tags.yaml — and
gated on ref_type so a tag named release-1.6 cannot pass the pattern on
its own. Runs before the checkout and the OCIR login, so a refused
dispatch never touches the credentials. Fails rather than skips: a
skipped job reads in the run list like one that produced the artifact.

The push trigger was already filtered to release-X.Y; this closes the
dispatch path to the same set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-04 14:23:51 +05:00
Myasnikov Daniil e8675dd321 docs(release): reword the changelog-backstop comments for the freeze model
Three comments still described the changelog reaching main by default,
with a release-X.Y-targeted promote PR as a patch-release corner case.
After the freeze that is inverted: promote-rc.yaml bases the promote PR
on release-X.Y whenever the branch exists, and cutting vX.Y.0-rc.1
creates it, so every release merges its changelog onto the maintenance
line and none reach main on the way. The tags.yaml backstop is now the
normal route, and its self-skip on `exists` is the exception.

These comments are the first thing anyone debugging a missing changelog
reads, so pointing at the wrong model of which path is normal costs real
time. Reworded in place: the generate-changelog job header, the
three-paths comment above check_changelog, its inline at_tag note,
finalize's explanation of why update-releasenotes.yaml is not trusted
with the release body, and the contract test that pins the porting
behaviour. Prose only, no behaviour change.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-04 14:23:32 +05:00
Andrey Kolkov 22c1df8b94 test(etcd-operator): pin deployment unittest to v0.5.4
The manager image tag derives from .Chart.AppVersion, so bumping appVersion
to v0.5.4 without updating the two image literals in the deployment unittest
broke the suite (helm-unittest gates the PR via `make unit-tests`). Update
both asserts (manager container image and the agent OPERATOR_IMAGE env) to
ghcr.io/cozystack/etcd-operator:v0.5.4. Suite is green again (18/18).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-04 13:09:25 +04:00
Andrey Kolkov 19f969b2f8 chore(etcd-operator): bump etcd-operator to v0.5.4
Bump the cozystack etcd-operator packages from v0.5.3 to v0.5.4. The
release is a controller bug-fix set with no API, RBAC or values changes:

- fix(controllers): derive --initial-cluster-state from phase, not from
  the seed (cozystack/etcd-operator#355)
- fix(controllers): stop exempting the bootstrap seed from self-heal
  (cozystack/etcd-operator#354)
- fix(controllers): extend crash-loop self-heal to memory members
  (cozystack/etcd-operator#352)
- fix(controllers): switch the PDB from maxUnavailable to minAvailable
  (cozystack/etcd-operator#351)

Adaptations:
- etcd-operator/Chart.yaml: appVersion v0.5.3 -> v0.5.4 (the manager image
  tag defaults to .Chart.AppVersion, so this reimages the controller).
- etcd-operator/Makefile, etcd-operator-crds/Makefile: ETCD_OPERATOR_REF
  v0.5.3 -> v0.5.4.
- etcd-operator-crds/templates/etcdmembers.yaml: re-vendored at v0.5.4 via
  `make update`; description-only change to the /scale replicas/selector
  fields tracking the PDB minAvailable fix. etcdclusters/etcdsnapshots
  unchanged.
- templates/rbac.yaml left as-is: manager-role-rules.yaml is byte-identical
  between v0.5.3 and v0.5.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-04 12:41:43 +04:00
Myasnikov Daniil 4e9c5ce8ca fix(kubernetes): render the talos-reconcile Job for the default md0 group
The Job in templates/talos/talos-reconcile-job.yaml is the only
producer of the TalosConfigTemplate that each worker
MachineDeployment names in spec.template.spec.bootstrap.configRef;
cluster.yaml renders the reference but deliberately not the object.
Its loop read .Values.nodeGroups directly, while the
MachineDeployment loop reads the effective set through the
kubernetes.nodeGroups helper.

For a cluster that supplies no nodeGroups the two disagree: the
helper's else-branch emits the built-in md0 group, so the md0
MachineDeployment renders, but the raw map is empty and no Job
renders at all. Every Machine the cluster-autoscaler adds to md0
then blocks indefinitely on a TalosConfigTemplate that nothing will
ever create, and the KamajiControlPlane spec.network.certSANs patch
the same Job performs is skipped too.

Range over the helper so the Job set tracks the MachineDeployment
set exactly. No change for a cluster that declares its own groups,
and md0 stays removable: the helper's if-branch keeps a
user-supplied map authoritative.

The gap survived because every helm-unittest fixture and e2e suite
declares md0 explicitly. tests/nodegroups_default_test.yaml does
render the empty-nodeGroups case but lists only
templates/cluster.yaml, so it never looked at the Job. The new
suite pins both halves of the helper contract and fails without
this fix.

Fixes #3504

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-04 13:40:36 +05:00
IvanHunters 97eadc661d docs(kubernetes): document pool-before-parent delete ordering in breaking changes
Deleting a cluster now requires deleting its KubernetesNodes pools before the
parent Kubernetes resource, so each pool's pre-delete unpin hook runs while
the tenant apiserver and CAPI CRDs still exist. This ordering lived only in
the child kubernetes-nodes NOTES.txt; surface it in the parent chart's
operator-facing breaking-change list, which is where someone deleting a
cluster looks. Unlike the pre-split monolith (pools died with the parent
release), pools are now separate resources with their own lifecycle.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 23:06:13 +03:00
IvanHunters 951b98b81c docs(kubernetes-nodes): note the unpin Job also consumes images.kubectl
The images.kubectl override now feeds both the talos-reconcile Job and the
pre-delete unpin Job, but its @field description named only the former.
Update it and regenerate the derived schema, README, Go types and the
kubernetes-nodes-rd cozyrd via cozyvalues-gen + update-crd.sh.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 23:05:18 +03:00
IvanHunters 6ccd1ed8c4 fix(kubernetes-nodes): tolerate a missing KMT CRD in the pre-delete unpin hook
The unpin hook listed KubevirtMachineTemplates under set -eu with no
missing-CRD carve-out, unlike migration 54's list_kmts. During a
management-cluster CAPI teardown or an already-Terminating namespace the CRD
can be gone; the list then exits non-zero, the Job fails, and after
backoffLimit the HelmRelease finalizer wedges cluster deletion on a resource
type that no longer exists.

Distinguish a missing CRD (skip: nothing to unpin) from a real API/RBAC
failure (still exit 1) by kubectl's two deterministic RESTMapper messages,
mirroring list_kmts. stderr is folded into the captured output only because
the strict <BASE>-<6hex> grep cannot match a warning line.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 23:04:09 +03:00
IvanHunters e969d71379 chore(platform): use bash shebang for migration 54
Migration 54 uses set -o pipefail, a bashism. The migrations image ships
bash (alpine:3.24 + apk add bash) and sibling migrations 45/47/50/51 already
use #!/bin/bash for the same reason. Align 54 so the interpreter matches the
options it sets and cannot silently no-op pipefail if /bin/sh were ever a
non-bash shell.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 23:02:36 +03:00
IvanHunters cb56f140ad fix(platform): pin keep on a foreign-owned worker object before skipping it
The foreign-owner skip branch warned and returned without applying
helm.sh/resource-policy=keep, on the premise that a foreign release-name
means the parent will not prune the object. That premise is unsound: Helm
computes the upgrade prune set from the release's own stored old manifest
minus the new manifest and gates deletion only on resource-policy=keep, not
on the live meta.helm.sh/release-name. The worker object is in the parent's
old pre-split manifest, so the control-plane-only re-render would prune it
and cascade-destroy the tenant's worker VMs.

Pin keep before returning, exactly as every sibling skip branch does, while
still refusing to rewrite the foreign release-name (which could corrupt that
release). Also correct the printed remedy: migration 54 stamps 55
unconditionally and never re-runs, so recovery is manual, not a re-run.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 22:19:35 +03:00
IvanHunters 4f323bfce5 fix(platform): pin live worker pools when migration-54 sees non-object nodeGroups
The top-level non-object nodeGroups branch warned and skipped the cluster
without pinning any objects, unlike every sibling skip branch (RFC-1123
invalid key, 53-char overflow, per-group non-object) which pins first. If a
cluster's pools were rendered under an earlier good revision and its
nodeGroups was later mangled out-of-band into an array/string/number, the
worker objects stay live but un-pinned; the parent's control-plane-only
re-render then prunes them, destroying the tenant's running worker VMs.

The declared pool keys are unreadable in this shape, so discover the live
pools from their KubevirtMachineTemplates (kubernetes-<cluster>-<group>-<hash>)
and pin each via pin_pool_objects before skipping adoption. Fail-closed
through list_kmts so a transient read never masquerades as no-pools.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 21:52:40 +03:00
IvanHunters 2d73043518 fix(platform): scope migration-54 foreign-owner refusal to one object
A worker object whose meta.helm.sh/release-name is neither the parent nor
the child release made adopt_one exit 1. Under set -e that aborts the whole
per-cluster pre-upgrade hook, so run-migrations.sh never stamps the version
and one tenant's out-of-band-edited annotation deadlocks the platform
upgrade for every tenant, with no path out.

This is the same hostile-stored-shape class the non-object, overflow and
RFC-1123-invalid branches were already reshaped to warn-and-skip rather than
deadlock the fleet. A foreign owner also means the parent release no longer
owns the object, so its control-plane-only re-render will not prune it:
skipping is prune-safe. Refuse to adopt (rewriting a foreign release-name
could corrupt that release) but scope the blast radius to the single object:
warn, skip it, document the operator remedy, and let the run complete.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 21:51:28 +03:00
Myasnikov Daniil e532844007 test(release): pin the freeze and backport-target contracts
Neither cut-prerelease.yaml nor backport.yaml had any test coverage, and
neither can be exercised by a PR lane: the first runs only on
workflow_dispatch and its whole purpose is a one-way write-once side
effect, the second only on a merged main-targeted PR. Every invariant
here is one that would otherwise first report a slip at a real release.

Pinned, following promote-gate-contract.bats so a guard demoted to a
comment cannot satisfy a pin:

  - the freeze is gated to kind == 'rc' && patch == '0'. Widening either
    half re-opens a frozen line with no error surfacing.
  - neither the branch create nor the tag push is forced, in any spelling
    including a `+refs/` refspec. A force-push on the branch would drag
    in everything merged since the freeze.
  - an existing release-X.Y is left untouched rather than reused.
  - the refuse gate is gated on a main dispatch and runs BEFORE the tag
    push, so a mistaken dispatch cannot burn a write-once tag name; the
    freeze runs after it, at the commit that was actually tagged.
  - backport targets come from enumerated branches, with no
    getLatestRelease and no min-1 arithmetic, and `previous` is the
    second-newest existing line.
  - the comparator sorts numerically descending, which is the only reason
    release-1.10 outranks release-1.9. A slip to the default
    lexicographic sort aims every backport at a stale line silently.
  - freezing a line dispatches its first build-release run.

backport.yaml's logic is a github-script block whose comments name
getLatestRelease and min-1, so the assertions over it strip `//`
comments; the YAML `#` filter alone would let a comment satisfy a pin.

The Makefile discovers hack/*.bats, so this runs under `make unit-tests`
with no wiring.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:38:41 +05:00
Myasnikov Daniil 5ed6b2204a ci(release): build the packages artifact when a freeze creates a line
build-release.yaml publishes `cozystack-packages:<line>` on push to a
release line, and pull-requests.yaml's overlay reads it so a release-line
PR tests its own line's binaries instead of main's (#3471, #3437).

It cannot fire for the push that creates the line. The freeze points
release-X.Y at a commit that is already on main, so the push carries no
new commits, and GitHub does not run a workflow whose paths/paths-ignore
filter finds no changed files ("If there are no files changed, the
workflow will not run"). The line would therefore have no artifact until
its first cherry-pick merged, and in that window the overlay finds
nothing to pull and leaves every package on its committed ref — at
freeze time the previous release's, which is exactly the
cross-generation mix #3437 fixed. The old flow had no such window: it
branched at the promote merge commit, which carried real commits and its
own release's refs.

Add workflow_dispatch to build-release.yaml and have the freeze step
dispatch it for the branch it just created, so the artifact exists from
the moment the line does. The dispatch is non-fatal: without it the
overlay no-ops and early cherry-pick PRs test their committed refs,
which is where they were before #3471. The tag is pushed and the line is
frozen by that point, so failing there would misreport both.

The trigger also gives a line build a re-run button, which previously
needed an empty commit pushed to the line.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:38:26 +05:00
Myasnikov Daniil 24e9c5396c docs(release): resync the release doc with the freeze model
The freeze changed three things the prose still described the old way.

The backport table claimed `backport-previous` resolves to
`release-X.(Y-1)` and that resolution runs through `getLatestRelease`.
Neither is true any more: the job enumerates the real `release-X.Y`
branches, sorts them numerically descending, and takes the newest and
second-newest existing lines. The `Y-1` derivation is the min-1 bug this
branch dropped. Line 155 already carried the correct description, so the
file contradicted itself.

The freeze window's effect on backport targets lived only in a code
comment. During it `backport` aims at the line being stabilised and
`backport-previous` at the last published stable, so the line one step
further back has no label for the duration. Document the trade-off where
the labels are described.

The changelog backstop and Phase 6 both framed a promote PR targeting
`release-X.Y` as "a patch release". After the freeze every promote PR
targets the release line, minor and patch alike, which makes the
backstop the only path a `.0` changelog takes to `main`.

Also note that re-cutting a line after an unusable rc.1 means deleting
`release-X.Y` by hand — the refuse step blocks every other route, by
design.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:35:01 +05:00
Myasnikov Daniil 66ed32c6f9 ci(release): print the recovery command if the freeze push fails
Address review feedback from lexfrei on .github/workflows/cut-prerelease.yaml:293:
the tag push and this branch push are independent, so a transient failure
here leaves the line tagged but not frozen, and the write-once tag guard
refuses a same-tag re-dispatch — recovery is manual.

Fail with the exact push command rather than leaving the operator to
reconstruct it. Deliberately not an atomic two-ref push: the tag push
relies on being anchored to the branch tip for GitHub to populate the
push event's base_ref, which tags.yaml requires, and whether a multi-ref
atomic push preserves that is only observable on a live release cut.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:32:00 +05:00
Myasnikov Daniil a59ad88eb5 ci(release): distinguish absent from broken in the freeze branch probe
Address review feedback from lexfrei on .github/workflows/cut-prerelease.yaml:286:
the release-X.Y existence probe piped ls-remote straight into cut, so a
transport or auth failure produced empty output and read as "branch not
found", steering into the create path. The write-once tag guard, the
stale-tip guard, and the refuse step above all make this distinction;
match them here.

Behaviour is unchanged on the happy path — pipefail already failed the
step on a real error, and the non-forced push below fails closed on a
conflict. This makes the intent explicit rather than incidental.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil b55e822eaf docs(release): align the patch-release diagrams with the freeze model
Address review feedback from coderabbitai on docs/release.md:150: the
patch-release gitGraphs still branched release-1.2 off main after the
v1.2.0 commit, depicting the branch as created at the stable release.
Under the rc freeze the line forks at vX.Y.0-rc.1 and the release commit
never lands on main, as the surrounding prose already states.

Fork release-1.2 at the rc.1 commit and place 'Release v1.2.0' on the
release line in all three patch-section diagrams.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil 3f15229795 docs(release): fix the Backports link fragment
Address review feedback from lexfrei and coderabbitai on docs/release.md:39:
the anchor pointed at #backporting, but the heading is '## Backports'
(slug #backports), so the link resolved nowhere and tripped markdownlint
MD051.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil e5428caaa6 docs(release): document the rc freeze
The Release Candidates section said the opposite of current behaviour —
"new features and changes can still be added before the regular release"
— so it is rewritten around the freeze, along with the Regular Release
step list, both gitGraph diagrams, and the workflow reference sections.

The promote PR now targets release-X.Y rather than main, so the diagrams
show main forking away at the freeze and never receiving the release
commit. Patch Releases collapses to "same as above, the branch already
exists", since after the freeze both flows are identical.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00
Myasnikov Daniil 6ff5b98d56 ci(backport): target the newest existing release line
Determine the backport targets by enumerating real release-X.Y branches
instead of deriving them from getLatestRelease.

getLatestRelease returns the newest published stable. Now that cutting
an rc creates release-X.Y before vX.Y.0 exists, that name lagged by a
full line for the whole freeze window: a `backport` on a fix for the
release being stabilised would have landed on the previous line and
missed the release it was written for. The freeze window is when
backports matter most, since the frozen branch is the only way in.

Also drops the contiguous-minor assumption — `previous` was computed as
min-1, naming a branch that does not exist whenever a minor is skipped.
Sorting is numeric, so release-1.10 correctly outranks release-1.9.

Outside a freeze window the result is unchanged: with release-1.0..1.5
present it still resolves to release-1.5 / release-1.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-03 21:31:59 +05:00