The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
## What this PR does
Completes #2937. Replaces the single serial `Build` job — one ephemeral
VM running `make build` (~26 images one after another) that wedged on
the shared embedded buildkit under concurrent multi-PR load — with a
parallel, diff-scoped matrix.
**New job graph (regular PR path):**
```
plan changed files -> hack/build-matrix.sh -> scoped matrix + docs gate
checks make unit-tests + test-controllers (gates finalize -> e2e)
build one package per job, own docker-container buildx,
uploads a per-package digest patch fragment [parallel]
build-talos talos image + nocloud disk (its own leg; e2e always needs the disk)
finalize download+apply all fragments, then build the installer LAST
(its flux-push bundles the whole digest-patched tree),
emit the unified pr.patch
e2e consumes talos-image + pr.patch unchanged
```
**Why finalize is special:** `packages/core/installer`'s `image` target
runs `flux push artifact --path=packages`, bundling the *entire*
packages tree into the OCI artifact the operator pulls — so it must run
after every other package's digest edit is applied, never as a peer
matrix job. `talos` is likewise pulled out of the matrix (its image and
nocloud disk share heavy `_out/assets` and build once).
**Gating:** `finalize` (and therefore `e2e`, which needs it) depends on
`checks`, so a red unit/controller test skips the expensive install+e2e
path instead of running it for ~1h — the gating the old in-build-job
tests gave, without re-coupling the cheap parallel builds.
**Change-scoping:** `hack/build-matrix.sh` parses the root Makefile
`build:` target (single source of truth) and emits only the packages a
PR touches — most PRs build ~0–1 images — escalating to the full set on
a shared-dependency change (`cozy-lib`, the build macros, `go.mod`, the
build workflows). It fans `seaweedfs` out to `objectstorage-controller`
(cross-package sidecar-tag dependency). 16 bats cases cover the
selector.
**SBOM:** `SBOM=1` wires buildx `--sbom` through `common-envs.mk` (off
by default). It stays off in the matrix until an OCIR
referrers/attestation probe confirms the registry accepts attestations.
**Decommission:** the cache warmer (`build-main.yaml`) moves to the
ephemeral pool, so no build path depends on the dedicated self-hosted
runner. The runner remains referenced only by the e2e `debug` breakpoint
path; fully de-registering it is a separate infra step.
### Validation
- Local: `hack/build-matrix_test.bats` 16/16; `actionlint` clean (custom
runner labels registered in `.github/actionlint.yaml`); `make -n`
confirms `--sbom` is off by default and on with `SBOM=1`.
- CI: this PR edits `pull-requests.yaml`, so `build-matrix.sh` forces a
full matrix — the PR exercises the new pipeline on itself. First run was
green through `plan → 25 parallel builds → build-talos → finalize →
install → 21/22 app tests`; the only red was the pre-existing
`bucket`/BucketAccess flake (install and the other 21 suites passed on
the matrix-built tree), which validates the decomposition end-to-end.
### Follow-ups (tracked on #2937)
- Probe OCIR referrers/attestation support; enable `SBOM=1` in the
matrix if supported.
- Infra: de-register the self-hosted build runner (the e2e `debug`
breakpoint path is its only remaining user).
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Refined CI to a staged, artifact-based pipeline with gated checks,
per-package parallel builds, and centralized patch assembly before
building final images
* Improved ephemeral runner performance and reliability (runner
selection, longer timeout, build toolchain setup)
* Added build-matrix auto-selection with targeted fan-out rules and
“full rebuild” triggers
* Updated CI lint configuration to recognize ephemeral runner labels
* Introduced an optional SBOM toggle for build artifacts
* **Documentation**
* Updated e2e testing docs to better explain Test-Impact Analysis scope
and gating semantics
* **Tests**
* Added Bats tests validating build-matrix selection and edge cases
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pin the PR #2872 B1 fix: when the management cluster exposes only remote
LINSTOR StorageClasses whose names sort after "kubevirt" and none is named
the configured storageClass (default "replicated"), the tenant default must
be chosen among the propagated classes and never the legacy "kubevirt" alias.
helm-unittest cannot reach this branch (no live cluster -> lookup returns
empty -> the storageClasses map collapses to the "replicated" fallback), so
it is exercised in e2e via a single server-side dry-run render of the
kubernetes chart against the live management cluster: two remote LINSTOR
classes are added and "replicated" is removed for the one render, then the
rendered -csi HelmRelease storageClasses map is asserted to carry exactly one
default among the propagated classes, never the "kubevirt" alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- csi_test.yaml: assert the fallback storageClasses map yields a default
"replicated" plus the non-default "kubevirt" alias, and that the
alias tracks the storageClass selector.
- kubevirt-csi-node: add a Makefile test target (otherwise the package
is silently skipped by the unit-test runner) and storageclasses_test
covering per-entry rendering, the single-default invariant, and the
absence of any enforcement allowList in driver-config.
- run-kubernetes.sh: verify in e2e that "replicated" propagates as the
default kubevirt-CSI class, the "kubevirt" alias is retained, and the
node-local "local" infra class is filtered out.
Refs: #2094
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Resolve the .github/workflows/pull-requests.yaml conflict: keep this branch's
per-package build matrix and carry main's `git diff --binary` patch fix
(#e566992) onto every fragment and pr.patch capture, so provider bundles that
package image targets regenerate (components*.gz) survive fragment reassembly
in finalize and reach the patch e2e applies.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
> **Supersedes #2610.** Same work, rebased cleanly onto current `main`
so it now sits on top of #2867 (the `kube-controller-manager` VAP
type-checker fix). That fix is what was killing the base install in
#2610's E2E *before* the `kubernetes` app test ever ran, leaving the
Talos worker-bootstrap rewrite verified only by code reading. #2610's
67-commit branch is squashed here into 8 logical commits with original
authorship preserved (@kvaps and @IvanHunters as authors); the tree is
**byte-identical** to #2610's head — no code changes, just the rebase
onto `main` (to pick up #2867) plus history cleanup. The detailed review
history lives on #2610.
Phase 1 of the Kubernetes-app split design (see cozystack/community#8):
replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by
`cluster-api-bootstrap-provider-talos` (CABPT) and a
`clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane
pod. Existing tenants keep working — old machines roll out and get
replaced by Talos workers without manual intervention.
Highlights:
- Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the
existing kubeadm one in `capi-providers-bootstrap`.
- Bump the Kamaji control-plane provider with an upstream-bound patch
that exposes `KamajiControlPlane.spec.network.additionalServicePorts`,
used to publish trustd (50001/TCP) on the apiserver service.
- Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and
stable random Talos secrets (`token`, `clusterId`, `clusterSecret`,
`bootstrapToken`) using the helm lookup-and-reuse pattern.
- Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the
tenant `kube-system` via a Helm post-install/upgrade Job.
- Render a `TalosConfigTemplate` (worker machineconfig, `generateType:
none`) and switch the `MachineDeployment.bootstrap` reference from
`KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets
and the Kamaji apiserver service being ready.
- Boot workers from the Talos openstack image via a CDI `DataVolume`
(`source.http.url` → `factory.talos.dev`), expose the system disk as
virtio-blk with `blockSize.custom: logical=512, physical=4096` so
4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's
O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk
(Talos lays out EPHEMERAL itself).
- Pin pod/service CIDRs in the worker machineconfig to the tenant ranges
(`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap
diagnostic against the host pod CIDR.
- Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the
Kamaji apiserver Service ClusterIP (looked up at render time), so
bootstrap survives the chicken-and-egg moment where tenant DNS does not
exist yet and the host CoreDNS does not serve the tenant zone.
End-to-end on a dev cluster: a fresh tenant brings up a Talos worker,
kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving
by the talos-csr-signer sidecar), Cilium initialises with hostNetwork
against the apiserver ClusterIP, CoreDNS comes up, and the node
transitions to `Ready`.
### Release note
```release-note
feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
* Added Talos worker configuration for `talos.version` and
`talos.schematicID`
* Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`)
for worker remediation tuning
* Added image overrides for `kubectl` and `talosCsrSigner`
* Exposed additional tenant control-plane Service ports support
**Breaking Changes**
* Removed Kubernetes v1.30; supported versions start at v1.31
* Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with
consolidated Talos system-disk semantics
**Removals**
* Removed Ubuntu container-disk images and related containerd patch
mechanisms
* Air-gapped tenant workers temporarily unsupported during the Talos
transition
**Documentation**
* Updated chart docs, parameters, and GitOps upgrade guidance for
version bumps
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolve conflict in .github/workflows/pull-requests.yaml: keep the new
per-package build matrix and fold in main's docker.io->mirror.gcr.io
base-image routing (#3042). Apply buildkitd-config: hack/buildkitd.toml
to all three new build legs (build, build-talos, finalize) so the mirror
fix covers the whole decomposed pipeline, not just the conflicted leg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
On E2E test failure, capture the host→pod CNI data-plane state for each
NotReady pod (scheduled, with a podIP, not terminal) from that pod's
node, into the crust-gather snapshot dir so it ships in the uploaded
report. New helper `hack/e2e-capture-dataplane.sh`, invoked from the
shared on-failure hook in `hack/cozytest.sh` right after the existing
crust-gather collect.
Per affected pod, from the pod's node:
- **host netns** (via the kube-ovn `cni-server`, hostNetwork): `ip route
get <podIP>`, `ip neigh`, `ip rule`, `ip addr show ovn0`, and the kernel
conntrack entries for the podIP (`conntrack -L`, falling back to
`/proc/net/nf_conntrack`).
- **cilium-agent**: `endpoint list`, `bpf ct list global` filtered to
the podIP, a bounded `monitor --type drop`, and `hubble observe
--verdict DROPPED` when the CLI is present.
- **OVS/OVN**: `ovs-ofctl dump-flows br-int`, the OVN `Port_Binding` /
`Logical_Switch_Port` for the pod, and a bounded `ovn-sbctl lflow-list`.
## Why
The install intermittently fails on a host→local-pod data-plane
transient: kubelet on a node hits a local pod's readiness/startup probe
with `connection refused` for several minutes while overlay pod→pod
traffic works, then it self-heals. This stems from the cilium+kube-ovn
chaining config (forced `enable-host-legacy-routing`, CNI
`InstallEndpointRoute:false`), which delegates host→local-pod routing to
kube-ovn/ovn0. Crucially, under `enable-host-legacy-routing` the
host→local-pod path traverses the kernel netfilter stack, not cilium BPF
— so the authoritative evidence is the kernel conntrack table plus the
host route/rule/neighbor state, captured here from the node's host
netns.
The crust-gather snapshot records Kubernetes object state but not the
node's L3 forwarding state, so the mechanism cannot be root-caused after
the fact. This adds exactly that evidence so the next recurrence is
dispositive.
## Diagnostic-only
This changes nothing about install or test behavior — it only collects
more evidence on an already-failed test. It runs after the existing
crust-gather collect and only when a test has already failed, never
mutates the cluster (read-only `get`/`exec`), and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
`|| true`, the whole run is wrapped in a `timeout -k 30 600` wall-clock
backstop, node-global captures are deduped per node, and the pod count
is capped (`MAX_PODS=12`, env-overridable). It no-ops cleanly when no
pods are affected or when a tool/pod is absent. No new retries, no
`EXIT`/`RETURN` traps — consistent with `docs/agents/e2e-testing.md`.
## Validation
`sh -n`, `bash -n`, and `shellcheck` are clean on both touched files.
The affected-pod selection filter and the no-cluster no-op path were
smoke-tested. Not exercised against a live cluster — container/label
names and the kube-ovn LSP naming (`<pod>.<ns>`) are taken from the
`packages/system/{cilium,kubeovn}` charts and the existing
`hack/e2e-cilium-endpoint-leak-healer.sh`; all captures are defensively
`|| true` and time-boxed so absent tooling degrades to a labeled note
rather than a failure.
## What
Raise the `kafka` and `zookeeper` components in the Kafka e2e test from
the `nano` preset (250m / 128Mi) to `c1.small` (1 CPU / 1Gi), the chart
default.
## Why
128Mi cannot hold the Strimzi Kafka 3.9.1 and ZooKeeper JVMs. The
brokers OOMKill on startup (exit 137) and enter CrashLoopBackOff, so the
pods never reach Ready, the `Kafka` CR and its HelmRelease never go
Ready, and the 5m readiness wait times out. It presents as a flake
rather than a hard failure because 128Mi sits right at the JVM-start
threshold — occasionally a broker becomes Ready before its RSS crosses
the cgroup limit.
`c1.small` (1Gi) comfortably fits both JVMs and matches the chart
default, so the test exercises the documented default sizing. The e2e
sandbox VMs are 8 CPU / 24Gi, so 1Gi per pod (4Gi total) across the four
pods (kafka-0/1, zookeeper-0/1) is well within budget.
## Scope
Test-only; values changed, test structure untouched. MariaDB's e2e test
also uses `nano` but was deliberately left as-is: it runs MariaDB (C) +
a Go exporter with no JVM, its chart default is already `t1.nano`
(128Mi), and it shares the `nano` preset with the passing
postgres/redis/qdrant tests — not the same OOM class.
## What
Add a bounded escalation tier to the E2E cilium endpoint-leak healer:
when a pod stays wedged on `FailedCreatePodSandBox` "IP ipv4:<X> is
already in use" for the same IP across several heal cycles, restart the
cilium-agent on that node. Keep the existing per-endpoint remedies as
the first tier.
## Why
The leak (cilium/cilium#38313 class) sometimes lives purely in the
cilium-agent's in-memory endpoint state. The existing remedies —
`cilium-dbg endpoint disconnect` and delete-rescheduling the wedged pod
— cannot reach it: IPAM keeps re-handing the SAME released IP, so the
recreated pod re-wedges on it, and on some variants the stale entry is
not even in the queryable endpoint registry, so disconnect is a no-op.
Both are inescapable loops that fail the install. The only thing that
clears an in-memory leak is restarting the agent, which drops the node's
endpoint registry and rebuilds it from CRDs/IPAM without the stale
entry.
## How
- Track recurrence per `(node, ip)`. After `ESCALATE_AFTER` (default 3)
heal cycles of a futile first-tier remedy on the same IP, restart that
node's cilium-agent pod.
- Bounded by `MAX_ESCALATIONS_PER_NODE` (default 2); after the cap, log
loudly and surface the failure instead of restarting again. State is
per-node and resets after each restart so one restart covers every
leaked IP on the node.
- A genuine duplicate-IP (endpoint backs a live owner) is refused before
counting, so an agent restart can never mask a real product bug.
- RBAC unchanged: the existing cluster-wide `pods` `get/list/delete`
already covers deleting a cilium-agent pod in `cozy-cilium`.
E2E-test-only resilience; no product code change. Remove alongside the
rest of the healer once a fixed Cilium ships.
## Problem
On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.
## Fix
Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.
## Why
The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.
## Test
- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.
* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
**Problem**
The `foundationdb` e2e test fails intermittently with a `.status`
jsonpath poll timeout even though the FoundationDB cluster is fully
healthy and reconciled. The `generations.reconciled`,
`hasListenIPsForAllPods`, `health.fullReplication` and `health.healthy`
gates polled their status fields with a 60s ceiling, while the
strictly-earlier `health.available` gate already allowed 300s. The
operator stamps `generations.reconciled` only after every process holds
its role for `minimumUptimeSecondsForBounce` (60s, set in
`packages/apps/foundationdb/templates/cluster.yaml`), which makes
`reconciled` — and the health fields gated on it — structurally the last
status fields to converge. The short ceiling closed the gate seconds
before the field landed; a snapshot taken a few seconds after a failure
shows the live object fully reconciled (`generations.reconciled: 1`,
`health.healthy: true`).
**Fix**
Raise those four late-converging gates to 300s, matching the existing
`health.available` budget. The waits stay event-driven (`until …; do
sleep …; done`) — only the ceilings widen, no new retries or fixed
sleeps. A comment records why `reconciled` converges last so the budget
ordering stays legible to future readers. The early-converging 60s gates
(`configured`, `connectionString`, `databaseConfiguration.*`,
`desiredProcessGroups`) and the spec-field gates are left untouched,
since they settle before `health.available` and 60s is already adequate.
**Why not just rerun**
Rerunning masks a deterministic ordering bug: the latest-converging
field carried the tightest budget. The cluster was healthy on every
"failure" — the test was racing its own success criteria, so the failure
recurs until the budget ordering is corrected. This change touches only
the test harness; the platform behaviour it asserts is unchanged.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Increased waiting time in one end-to-end FoundationDB setup test so
cluster status checks have more time to stabilize.
* Reduced flaky failures by allowing key health and reconciliation
fields to converge before the test continues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem
The `Bootstrap Talos cluster` E2E step fails intermittently with `❌ Test
failed: Bootstrap Talos cluster (exit 1)` even though the cluster is
healthy. The exit comes from the etcd-members convergence gate, which is
bounded at `timeout 180`.
A 3-node etcd converges via Talos's serialized learner promotion — etcd
admits one learner at a time, so the third member is only promoted after
the second has fully caught up. On a loaded ephemeral runner this
deterministic path can legitimately approach ~3 minutes of wall clock
and photo-finish against the 180s ceiling, timing out a cluster that has
in fact converged (every member reaches `Health check successful`; no
kernel panic / OOM / boot failure / network-unreachable in dmesg).
Observed: a run where the third member reached etcd health at +178.6s
against the 180s budget, failing ~1.4s short.
## Fix
Widen the etcd-members convergence budget from 180s to 300s. The wait is
already event-driven (`until talosctl etcd members …; do sleep 1; done`)
— only the ceiling was too tight. No retry is added and the
dmesg-on-failure diagnostic is preserved.
300s matches the repo's other in-line budgets for deterministic
convergence (HelmRelease-ready 5m, harbor/NFS 10m, install 15m).
## Why not just rerun
This is not environmental noise: the cluster converges correctly every
time, it is only timed out. Rerunning wastes a full ~1h E2E cycle to
re-roll the dice on whether the runner shaves the last ~2s. Correcting
the mistuned budget removes the photo-finish.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Extended the cluster bootstrap wait time for control plane health
checks, reducing false failures during slow startup.
* Improved readiness handling so cluster setup is more resilient before
moving on to later initialization steps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The on-failure dataplane diagnostic only captured host->pod CNI state for
NotReady pods. A recurrent flake presents differently: a tenant Service
type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready
and Running, so there is no NotReady pod and the hook captured nothing --
leaving the host->cross-node->bridged-worker-VMI->nodePort path
uncharacterised, which is exactly where root-causing has stalled.
Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer
services with an ingress IP, derives the EndpointSlice backend, the
nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork
MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB
whose every probe fails gets the heavy capture, and that probe doubles as
the traffic for the tcpdumps. For a failing LB it captures, on both the
announcer and endpoint nodes, the host cilium LB map, kernel
conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on
the geneve tunnel and the backend tap -- localising the failing hop to
host-cilium programming vs kube-ovn delivery. In-guest VMI capture is
documented as a deferred stretch that needs tenant-cluster access.
The existing NotReady-pod capture is unchanged. The new section is
best-effort and time-boxed like the rest of the script, and a unit test
pins the pure enumeration, announcer-node, and capture-or-skip logic against
mock input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Every migration that stamps the `cozystack-version` ConfigMap must carry
the `platform.cozystack.io/no-delete=true` label so the
`cozystack-no-delete-guardrail` ValidatingAdmissionPolicy guards it. The
chart template renders the labeled ConfigMap only on first install, so
each upgrade-time stamp has to re-emit the label by hand — migration 43
had to inline-copy migration 42's heredoc to avoid a label-less apply by
the same field manager stripping it. The bug class was patched
per-migration, not eliminated.
This extracts the stamp into a shared, sourced helper
(`render_cozystack_version_manifest` + `stamp_cozystack_version`) under
`images/migrations/migrations/lib/`, and routes migrations 42, 43, 44
and the `run-migrations.sh` bootstrap path through it. The labeled
manifest now has a single definition, so future migrations cannot drift
back to a label-less apply.
The upgrade path is behavior-preserving: the rendered manifest is
byte-identical to the previous heredocs (pinned by a golden unit test).
The bootstrap path in `run-migrations.sh` previously created the
ConfigMap without the label — it now carries it like every
migration-driven stamp.
Adds `hack/cozystack-version-stamp.bats` (8 tests) covering the helper:
label presence, ConfigMap name, quoted-string version type, namespace
default and override, required-argument guard, and the byte-exact golden
manifest.
Closes#2781
### Release note
```release-note
fix(platform): always stamp the platform.cozystack.io/no-delete label on the cozystack-version ConfigMap, including the migration bootstrap path, so deletion protection cannot be silently dropped
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Tests**
* Expanded validation to confirm the version-stamp manifest is parseable
YAML, includes the required no-delete label, targets the correct
`cozystack-version` ConfigMap, and quotes the version value correctly.
* Added coverage for default and overridden namespaces, argument
validation failures, and a golden “byte-for-byte” render check.
* Added a guardrail test to prevent bypassing the shared stamping logic
by direct `kubectl` ConfigMap stamping.
* **Refactor**
* Centralized `cozystack-version` rendering and stamping into a shared
helper and updated migrations and the migrations runner to use it
consistently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch
diverged landed a competing single-VM build restructure (#2939) and
test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build
supersedes #2939:
- .github/workflows/pull-requests.yaml: keep #2983's
plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer
resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact
interface (pr-patch + talos-image) is identical on both sides, so e2e depends
on finalize/build-talos with no input rewiring. main's top-level permissions
and SHA-pinned action refs applied across all jobs.
- All other files merged cleanly (only pull-requests.yaml conflicted).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Fixes two failure modes in the COSI control plane built from upstream
container-object-storage-interface v0.2.2, both carried as image
patches.
**BucketClaim never converges to ready (#3032).** The controller does
not watch Bucket objects, and the generic controller drops no-op resync
deltas, so a BucketClaim whose backend Bucket flips to ready *after* the
first reconcile stayed `bucketReady=false` indefinitely — BucketAccess
was never granted and the consuming app (bucket, harbor) got no
credentials. The existing re-read of the live Bucket only ran during a
reconcile that nothing re-triggered. The controller now returns an error
while the Bucket is not ready, forcing a rate-limited requeue that
re-reads the Bucket until its readiness propagates, so the claim
converges deterministically instead of racing the async sidecar.
**Slow provisioner failover on pod recreate (part of #2429).** The
controller and the per-driver sidecar run single-replica, so leader
election adds no availability — only failover latency. The upstream 150s
lease made a recreated provisioner pod stall reconciliation for up to
~150s. The lease is lowered to the standard Kubernetes values
(15s/10s/2s), so a recreated pod resumes in ~15s. The stale unix-socket
CrashLoop in #2429 was already resolved by the seaweedfs-cosi-driver
v0.3.1 self-heal; the silent "Bucket ready with no backend bucket"
finding on that issue is a separate driver bug and is left tracked
there.
Coverage: a unit test in the patched controller pins the
requeue/propagation contract (fails without the requeue), and the
bucket/harbor e2e suites now assert `bucketReady=true` on a tight bound
instead of tolerating the lag, with scoped diagnostics on the failure
path.
Closes#3032
Refs #2429
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved object storage bucket readiness propagation so bucket claims
now track the backend bucket’s actual `bucketReady` status and requeue
when readiness isn’t yet observed.
* Reduced and made readiness waits more reliable in end-to-end checks,
with explicit match conditions and clearer debug output (including claim
YAML, backend bucket list, and controller pod status).
* Shortened leader-election timing to speed up recovery after pod
restarts.
* **Chores**
* Updated automated PR labeling to map the object storage controller
scope to the storage area label.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The matrix selector's full_rebuild_pattern omitted the root Go module dirs
(api/, cmd/, internal/, pkg/). Eight matrix image Dockerfiles COPY those dirs,
so a PR changing only root Go source emitted an empty matrix, skipped the build
job, and let e2e install stale digests — the Go change was never exercised. Add
the four dirs to the pattern and pin it with bats cases for api/- and pkg/-only
diffs forcing the full matrix.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The kafka e2e test pinned both the kafka and zookeeper components to the
legacy "nano" preset (250m CPU / 128Mi memory). 128Mi cannot hold the
Strimzi Kafka 3.9.1 and ZooKeeper JVMs: the brokers OOMKill on startup
(exit 137) and CrashLoopBackOff, so the pods never reach Ready, the Kafka
CR and its HelmRelease never go Ready, and the 5m readiness wait times
out. It surfaced as a flake because 128Mi sits right at the JVM-start
threshold — occasionally a broker passes readiness before its RSS crosses
the cgroup limit.
Raise both components to "c1.small" (1 CPU / 1Gi), the chart default,
which comfortably fits the JVMs. The e2e sandbox VMs are 8 CPU / 24Gi, so
1Gi across the four pods is well within budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint
disconnect, or delete-reschedule the wedged pod) cannot clear a leak
that lives purely in the cilium-agent's in-memory state: when IPAM keeps
re-handing the SAME released IP, a delete-rescheduled pod re-wedges on
it, and on some variants the stale entry isn't even in the queryable
endpoint registry so disconnect no-ops. Both produce an inescapable
delete -> re-wedge / no-op loop that fails the install on
FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313).
Add a bounded Tier-2 escalation: track recurrence per (node,ip); once
the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that
node's cilium-agent (delete its pod; the DaemonSet rebuilds it from
CRDs/IPAM without the stale entry), capped per node so it can never
loop-restart agents. A genuine duplicate-IP (live owner) is refused
before counting, so an agent restart never masks a real product bug.
Tier-1 stays the first remedy; escalation fires only after it proves
futile on the same IP.
RBAC needs no new verb: the existing cluster-wide pods get/list/delete
already covers deleting a cilium-agent pod in cozy-cilium.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.
The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.
Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Pull Request Build job authenticates only to the push registry (OCIR),
so every base image (golang, ubuntu, alpine, nginx, ...) is pulled from
Docker Hub anonymously. Those pulls intermittently fail at
auth.docker.io/token under Docker Hub's anonymous-pull rate limit, leaving
the base image unresolved and failing make build.
Build on a docker-container buildx builder whose buildkitd routes docker.io
through mirror.gcr.io, Google's public pull-through cache for Docker Hub: no
auth, far higher limits, transparent fall-through to Docker Hub on a miss.
This reuses the builder pattern already proven in build-main.yaml (same
make build, same DOCKER_CONFIG auth forwarding); only the committed
hack/buildkitd.toml mirror config is new.
Fail-safe: if mirror.gcr.io is unreachable, BuildKit falls back to
registry-1.docker.io and the build still succeeds, so this strictly reduces
build failures without adding a new hard dependency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.
Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The reconciled, hasListenIPsForAllPods, fullReplication and healthy gates
polled status fields with a 60s ceiling, while the strictly-earlier
health.available gate had 300s. The operator stamps generations.reconciled
only after every process holds its role for minimumUptimeSecondsForBounce
(60s), making reconciled and the health fields that depend on it the last
status fields to converge. The short ceiling closed the gate seconds before
the field landed on a healthy, fully-reconciled cluster, causing intermittent
jsonpath poll timeouts.
Raise those four gates to 300s to match the health.available budget. The waits
stay event-driven; only the ceilings widen.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The "Bootstrap Talos cluster" step times out the etcd-members health gate
at 180s. A 3-node etcd converges via Talos's serialized learner promotion
(one learner admitted at a time), so on a loaded ephemeral runner this
deterministic path can approach ~3m wall clock and photo-finish against the
180s ceiling — failing a cluster that is in fact healthy. The wait is
already event-driven; only the ceiling was too tight. Widen it to 300s.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The bucket suite waited on '{.status.bucketReady}' without an expected value;
a bare jsonpath match is satisfied by the literal "false", so the wait passed
against an unready claim. Both suites also allowed 300s/600s, long enough to
tolerate the propagation race rather than catch it, so they stayed green
whether or not the BucketClaim ever converged.
Assert bucketReady=true on a 120s bound. Convergence is now controller-driven
(the controller requeues the claim until the backend Bucket's readiness
propagates) and completes within tens of seconds; a claim frozen by the
propagation race never converges and now fails the wait fast instead of being
masked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The helper header and run-migrations.sh claimed every migration-driven
stamp routes through the helper, but migrations 1-41 still stamp the
cozystack-version ConfigMap label-less and are backfilled by migration 42.
Scope the comments to go-forward stamps (migration 42 onward and the
bootstrap path), and add a unittest asserting no migration numbered >= 42
nor run-migrations.sh stamps cozystack-version without the helper, so the
no-drift invariant is enforced rather than aspirational.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Mirror render_cozystack_version_manifest's argument guard in
stamp_cozystack_version so a missing version aborts before the kubectl
pipe. Without it, a render failure in a shell without pipefail would
leave the pipeline exit status to kubectl, which would apply empty input.
Add a unittest covering the guard.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extract the labeled cozystack-version ConfigMap stamp into a sourced
shell helper so migrations cannot drift back to a label-less apply that
strips the platform.cozystack.io/no-delete label.
render_cozystack_version_manifest emits the manifest on stdout (pure,
unit-testable); stamp_cozystack_version renders and applies it. The
namespace defaults to cozy-system and honors $NAMESPACE.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Rebuilds the e2e test sandbox on current upstream to clear the published
advisories (containerd, helm, go-git, moby/spdystream, Go stdlib)
carried by the stale tool binaries and base layers.
- base: `ubuntu:22.04` → `24.04`; `netcat` → `netcat-openbsd` (on 24.04
`netcat` is a candidate-less virtual package, so the concrete provider
must be named)
- kubectl: `1.33.2` → `1.33.13` — stays on the 1.33 line that the e2e
control plane pins at `v1.33.12` (`hack/e2e-prepare-cluster.bats`), so
client/server skew is unchanged
- talosctl: `1.10.4` → `1.13.5` — aligns the client with the v1.13.x
Talos nodes the sandbox provisions
- helm: `3.18.3` → `3.21.2` (latest helm 3) — the e2e flow drives the v3
CLI, so it stays on the 3.x line rather than moving to the breaking 4.x
- yq: `4.44.3` → `4.53.3`
- `values.yaml`: re-pinned to the rebuilt multi-arch digest
(`linux/amd64` + `linux/arm64`)
`crust-gather` and `cozyhr` already track their latest releases; `flux`
and `mc` install the current release at build time. The built image's
tool versions were verified in place (kubectl `v1.33.13`, talosctl
`v1.13.5`, helm `v3.21.2`, yq `v4.53.3`, flux `2.8.8`, ubuntu `24.04`).
Closes#2891
### Release note
```release-note
chore(testing): refresh e2e-sandbox tools and base (ubuntu 24.04, kubectl 1.33.13, talosctl 1.13.5, helm 3.21.2)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the e2e sandbox image to a new fixed digest for more
consistent test runs.
* Refreshed the sandbox build environment (Ubuntu 24.04) and upgraded
tooling (Kubernetes, Talos, Helm, and yq) to newer stable versions.
* Improved OS package setup by switching from `netcat` to
`netcat-openbsd` and consolidating apt cleanup to reduce image size.
* **Documentation**
* Updated e2e testing guidance: removed the recommendation to pin the
management cluster Kubernetes version, and added guidance to
snapshot/restore cluster state between test groups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add helm-unittest suites for the new Talos templates: MHC maxUnhealthy int-vs-percent rendering, the Talos<->Kubernetes support-matrix guard, the talos templates, and the ported kubelet-reservation math; restore the gpu=on node-label assertions. Remove the obsolete containerd-config-path test and fixtures. Raise tenant control-plane wait timeouts in the e2e runner for Talos PKI + sidecar boot.
Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The full-matrix PR run validated the decomposition functionally (green
install + 21/22 app tests on the matrix-built tree), stronger evidence
than the file-set parity diff would give — and it avoids running the
serial make build the parity job exists to compare against. SBOM stays
capability-only (common-envs.mk) until an OCIR referrers probe is run
when SBOM is actually pursued.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
talosctl 1.13.5 now defaults to Kubernetes 1.36.2, so the comment's premise
— that talosctl's bundled default predates the KCM VAP nil-pointer fix — no
longer holds (1.36.2 already carries that fix). The reason e2e pins v1.33.12
is now that 1.36 is outside cozystack's supported range; reword to lead with
that and keep the KCM VAP fix as the secondary property v1.33.12 carries.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Six further references to the alpine/k8s tools image were still pinned at
1.33.4 across the installer, the cozystack-api post-upgrade hook, the vm-disk
and vm-instance resize hooks, the vertical-pod-autoscaler CRD-install override,
and the e2e cilium leak healer. Move them all to 1.36.2 so the whole repo runs
a single current tools-image version (the two digest-pinned refs get the
matching multi-arch digest sha256:44ef4942...). Same stale-stdlib remediation
as the seaweedfs bump in this PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replaces the single serial Build job (one ephemeral VM running `make
build`, ~26 images one after another, wedging on the shared buildkit
under concurrent load — #2937) with a fan-out:
plan changed files + hack/build-matrix.sh -> scoped matrix
checks unit + controller Go tests (decoupled from e2e)
build one package per job, own docker-container buildx,
uploads a per-package digest patch fragment
build-talos talos image + nocloud disk (its own leg; e2e needs it)
finalize apply all fragments, then build the installer LAST (its
flux-push bundles the whole patched tree) and emit the
unified pr.patch e2e consumes unchanged
build-matrix.sh scopes the matrix to the packages a PR touches (full set
on shared-dependency changes) and fans seaweedfs out to
objectstorage-controller. common-envs.mk gains an SBOM=1 flag wiring
buildx --sbom, kept OFF until the OCIR referrers probe confirms support.
.github/actionlint.yaml registers the custom ephemeral runner labels.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/build-matrix.sh emits the CI build matrix (the list of package dirs
to build) from the root Makefile `build:` target — the single source of
truth — so the matrix can't drift from `make build`. It supports
diff-driven scoping: only units whose package dir changed are emitted,
unless a shared-dependency path (cozy-lib, the build macros, go.mod, the
build workflows) is touched, in which case the full matrix is emitted.
talos and installer are excluded from the parallel matrix: talos shares
heavy _out/assets between its image and the nocloud disk (built once in a
dedicated leg), and installer's `flux push artifact --path=packages`
bundles the whole digest-patched tree, so it must run in finalize after
all other fragments merge.
hack/build-matrix_test.bats covers full/scoped/empty selection, the
talos+installer exclusion, shared-dep escalation, and JSON/Makefile parity.
Refs #2937
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on
every release tag as a non-blocking alarm — its own header notes "a
failure here is an alarm, not a release blocker." It duplicates coverage
the release-cut PR already provides: the "Prepare release" commit bakes
image digests into `packages/core/` (and other packages), which TIA
escalates to the full suite, so the release PR exercises the whole suite
against the commit the tag will point at.
It has also never been green since it was added (2026-05-02): every tag
run failed at the build step. Even after #2973 fixed the registry login
today, the next run still failed (Build Talos image). Removing a
perpetually-red, non-gating duplicate removes noise without losing
coverage.
Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s
full-suite pattern, repoints its unit test to `pull-requests.yaml`, and
updates the e2e-testing doc.
Refs #2397
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The per-test failure snapshots (host in cozytest.sh, tenant in run-kubernetes.sh)
ran `crust-gather collect` unbounded. On a contended/degraded cluster the collect
can hang indefinitely — observed wedging the E2E install step for ~2h35m after a
test failure until the job-level cancel, with no output after the capture line.
`|| true` only catches a bad exit code, never a hang.
Wrap both invocations in `timeout -k 30 300`: a 5-minute cap (hard SIGKILL 30s
later if the collect ignores SIGTERM) bounds the hang; a partial snapshot still
beats a multi-hour wedge, and timeout's exit 124 is already swallowed by `|| true`.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The leak-healer only refused to disconnect an endpoint whose owning pod was
Running, so a Pending owner that already held a live IPAM allocation
(status.podIP is populated once the sandbox is created, before the pod goes
Running) could have its endpoint evicted mid-startup. Refuse for any
non-terminal phase (!= Succeeded && != Failed); only a terminal owner is
safely past needing its endpoint.
Addresses non-blocking follow-up 4 on #2948.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubernetes-{latest,previous} intermittently failed their node-Ready wait
with the tenant API "connection refused" on localhost, while the cluster was
in fact healthy (host snapshot: both CAPI Machines NodeHealthy=True, TCP
Ready 2/2). Root cause: `kubectl port-forward service/kubernetes-<name>`
pins to ONE of the two Kamaji control-plane pods; when that pod's
kube-apiserver container restarted once (observed restartCount=1), the
forward broke and did not recover, so the 12m wait ran against a dead socket
and the failure-path describe/snapshot also got nothing.
Expose the tenant API with a test-scoped LoadBalancer Service selecting the
Kamaji control-plane pods instead. It load-balances across ALL ready apiserver
endpoints, so a single pod's apiserver restart is routed around transparently;
the host runs MetalLB on the sandbox /24 (pool 192.168.123.200-250) so the IP
is directly routable from the test (the in-tenant LB test already curls such
an address). The kubeconfig points at the LB IP with insecure-skip-tls-verify
(the MetalLB IP isn't in the apiserver cert SANs; e2e only) and the CA dropped.
Bonus: the LB endpoint is stable until teardown, so the on-failure tenant
crust-gather snapshot can finally reach the tenant (the port-forward was dead
by snapshot time, which is why every prior tenant snapshot came back empty).
Removes the port-forward supervisor + pidfile machinery. Verified against the
real hack/cozytest.sh under dash (CI's /bin/sh) in ubuntu:22.04: no pipeline
deadlock, the in-subshell snapshot trap fires and captures the tenant, and the
LB Service is cleaned up on both the success path and cozy_cleanup.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The healer's reserved/infra-endpoint remedy and its disconnect-failure
fallback both run `kubectl delete pod` to reschedule the wedged pod onto a
free IP, but the ClusterRole granted pods only get/list — so the delete was
forbidden and the pod stayed wedged. Add delete to the pods rule.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
crust-gather can only reach a tenant Kamaji cluster through the test's
localhost port-forward (it connects directly to the kubeconfig server URL —
there is no apiserver-proxy mode — and the in-cluster URL is unreachable from
the runner). cozytest.sh's generic EXIT trap runs in the parent shell AFTER
the failing test subshell has exited and its port-forward is reaped, so a
collect there hits a dead socket and crust-gather bails before writing
anything (every tenant snapshot so far came back empty).
Capture the tenant from an EXIT trap registered INSIDE run_kubernetes_test
instead: it fires during THAT subshell's exit, while the supervised
port-forward is still alive. Verified end-to-end against the real
hack/cozytest.sh under dash (CI's /bin/sh) in an ubuntu:22.04 container —
the in-subshell trap fires on failure with the PF alive, the tenant snapshot
lands in the same snapshots/<test>/ dir as the host snapshot, the generic
cozy_cleanup still runs, and there is no pipeline deadlock.
Drop the now-dead tenant loop from cozytest.sh's generic trap (it could only
ever see a reaped PF); it keeps the host snapshot.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A single `kubectl port-forward` to the tenant Kamaji API silently dies on
an SPDY/connection drop and never reconnects. The tenant apiserver itself
stays healthy (control-plane pods observed at 0 restarts while localhost
refused connections), so this is a client-side port-forward death, not a
control-plane outage. Once it dies every later tenant kubectl op wedges:
a long `kubectl wait` burns its full timeout against the dead socket
(observed: nfs-test-pod hit its 10m deadline) and the failure-path
`describe`/`get events` print 'connection refused', destroying the
diagnostics too.
Run the port-forward under a respawn loop so a drop self-heals within ~1s;
`kubectl wait` retries across the gap and the in-test failure dumps work
again. Track the supervisor PID in a pidfile so cozy_cleanup() can stop it
on a failed run (the success path kills it directly), and reap the child
port-forward with pkill in both paths.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The endpoint-leak healer evicts the stale endpoint holding a disputed IP
via 'cilium-dbg endpoint disconnect'. But when the IP is held by a
reserved/infra endpoint with no k8s pod identity — e.g. reserved:ingress,
the per-node Cilium Ingress endpoint — that call is rejected with
'endpoint may not be associated reserved labels', so the healer spins
forever and the wedged pod (observed: cozy-backup-controller) never
recovers, failing the run.
Such infra endpoints legitimately own the IP; the duplicate is an IPAM
race that also handed it to a pod. Evicting infra is both impossible and
wrong. Instead reschedule the wedged pod onto a free IP by deleting it
(verified live: the recreated pod lands on a clean address while the
ingress endpoint keeps the disputed one). Also fall back to deleting the
wedged pod if a disconnect of a presumed pod endpoint fails for any
reason, so a pod is never left wedged.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 19a9063ae0)
mattia-eleuteri