## What this PR does
The e2e runner `hack/cozytest.sh` is not real bats — it converts `@test`
blocks to shell functions and recognises a `cozy_cleanup()` hook, but
never calls `setup()`/`teardown()` and never opens fd 3. So
`etcd.bats`'s `setup()` never ran (etcd leaked its HelmRelease and data
PVCs into the shared `tenant-test` namespace) and `dump_diagnostics`
wrote to an unopened fd 3, silently dropping failure diagnostics. The
teardown's `kubectl wait --for=delete` lines also passed
`--ignore-not-found`, which `wait` does not accept.
This moves teardown into an `etcd_drain()` helper, run three ways: a
pre-clean at the start of the first scenario, inline at the end of the
last scenario (under `set -e`, so a teardown that does not settle fails
the suite), and as `cozy_cleanup` (the best-effort safety net cozytest
runs at suite exit and on the first failing test). The drain reclaims
what a plain CR/HR delete leaves behind:
- the StatefulSet's retained `data-etcd-*` PVCs (not owned by the Etcd
CR) — a later install otherwise reuses the same-named volumes, boots the
previous member identity, never forms a fresh quorum, and the chart's
`helm --wait` fails; and
- the cluster-scoped Kamaji `DataStore`, whose leaked finalizer wedges
the HR uninstall — cleared each pass until the Etcd CR, HR, DataStore,
and PVCs are all gone, bounded by a 4-minute deadline that fails the
drain on timeout.
Remnants are probed by name so a transient API error is never mistaken
for a successful delete. Diagnostics now go to stdout, which cozytest
captures and prints on failure.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved end-to-end cleanup so leftover resources are removed more
reliably between runs.
* Made failure diagnostics easier to capture when tests fail.
* Added stronger cleanup handling at the start and end of the test flow
to reduce flaky suite behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Lays the foundation for migrating tenant LoadBalancer exposure off the
deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off
MetalLB onto Cilium-native announcement — the mechanism-agnostic shape
prescribed when the earlier chart-side approach (#2468) was reverted in
#2603.
It adds a new `network.cozystack.io/v1alpha1` API group and a controller
that translates intent into whichever LoadBalancer backend the cluster
admin configured:
- `ExposureClass` (cluster-scoped, admin): binds a logical class name to
a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP
address scope — the StorageClass/IngressClass pattern.
- `ServiceExposure` (namespaced, tenant intent): "expose this Service
via class X"; status reports the resolved backend and assigned IPs.
- A `serviceexposure` controller renders the backend's pool + announcer
once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium
`CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to
the namespaces that opted in. Backends are pluggable behind a small Go
interface, so adding BGP or another cloud LB is one implementation, not
an API change. The controller never mutates the target Service — the
chart still owns its shape.
Platform wiring: `publishing.exposureClass.*` renders the admin
`ExposureClass` and the host ingress publishes through it. The default
(no class set, `externalIPs`) leaves every existing cluster's behaviour
byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes
a single `ExposureClass.spec.backend` change.
This is the foundation only — Part of #2586. Converting per-app
exposure, flipping Cilium `l2announcements` on, changing the default,
and removing MetalLB are deferred to later phases.
### Release note
```release-note
feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added new `ExposureClass` and `ServiceExposure` custom resources with
support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB),
including optional L2 and address configuration.
* Added Helm wiring and templates to enable service/ingress publishing
via `ServiceExposure`, selecting a configured (or default)
`ExposureClass`.
* **Bug Fixes**
* Improved reconciliation, finalizer-based cleanup, and status updates
for readiness and assigned IP reporting, including safer handling when
backends or managed resources change.
* **Tests**
* Added end-to-end and unit test coverage for backend rendering,
readiness, and cleanup flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Migration 47's safety property is that any per-object read or patch failure
aborts the migration (exit 1) before the cozystack-version stamp advances, so
the pre-upgrade hook fails and Helm never rolls the new kubernetes chart against
an un-pinned cluster (whose live md0 MachineDeployment would otherwise be
pruned). That branch was verified by reading only.
Add hack/kubernetes-md0-migration.bats, mirroring the kubectl-mock style of the
plain-shell cozytest.sh runner: a fake kubectl on PATH simulates a non-NotFound
read failure and a patch failure, and the tests assert the migration exits
non-zero and never reaches the version stamp (no `kubectl apply` of
cozystack-version is logged). A positive-control case asserts a clean run does
reach the stamp, so the "no stamp" assertions are meaningful rather than vacuous.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
The previous approach routed a release-unique label through
qdrant.persistence.additionalLabels, which the upstream chart renders into
the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is
immutable on an existing StatefulSet, so for any already-deployed Qdrant the
API server would reject the upgrade and the <release>-system HelmRelease
(retries: -1) would retry-fail forever — permanently wedging reconciliation.
Label changes on a volumeClaimTemplate also never apply to PVCs that already
exist, so the cleanup hook could not reclaim those instances anyway.
Switch the post-delete hook to select PVCs by the deterministic StatefulSet
naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely
from the release name, so this needs no StatefulSet change (upgrade-safe) and
also reclaims PVCs from instances created before the hook existed.
- qdrant.yaml: drop the additionalLabels passthrough entirely.
- hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep.
- tests: assert the name-prefix command instead of the label passthrough.
- e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited).
BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC
and the vector data it holds.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The e2e runner (hack/cozytest.sh) is not real bats: it never invokes
setup()/teardown() and never opens fd 3. So etcd.bats's setup() never ran
and dump_diagnostics wrote to an unopened fd 3, silently aborting on
failure — and etcd leaked into the shared tenant-test namespace.
Move teardown into etcd_drain() and run it inline at the end of the last
scenario (under set -e, so a teardown that does not settle fails the
suite) and as cozy_cleanup (the best-effort hook cozytest runs at suite
exit and on the first failing test). The first scenario also drains up
front so a dirty namespace cannot taint it.
etcd_drain reclaims what a plain CR/HR delete leaves behind:
- the StatefulSet's retained data-etcd-* PVCs (not owned by the Etcd CR);
a later install otherwise reuses them, boots the old member identity,
never forms a fresh quorum, and the chart's helm --wait fails; and
- the cluster-scoped Kamaji DataStore, whose leaked finalizer wedges the
HR uninstall — cleared each pass until the Etcd CR, HR, DataStore, and
PVCs are all gone, bounded by a deadline that fails the drain on
timeout. Remnants are probed by name so a transient API error is never
mistaken for a successful delete.
Drop --ignore-not-found from the teardown waits (not a valid wait flag);
the drain polls for existence instead. Diagnostics now go to stdout,
which cozytest captures and prints on failure.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a deterministic e2e for the externalIPs backend: it creates an
ExposureClass, a Service with pinned externalIPs, and a ServiceExposure,
then asserts the controller resolves the class, renders no MetalLB/Cilium
pool, reports the assigned IPs, and cleans up via its finalizer on delete.
The externalIPs path is chosen so the test creates no backend pool — it
cannot collide with the cluster's existing LoadBalancer address pools or
depend on real L2 reachability. The metallb and cilium rendering paths are
covered by the Go unit tests.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Completes #2937. Replaces the single serial `Build` job — one ephemeral
VM running `make build` (~26 images one after another) that wedged on
the shared embedded buildkit under concurrent multi-PR load — with a
parallel, diff-scoped matrix.
**New job graph (regular PR path):**
```
plan changed files -> hack/build-matrix.sh -> scoped matrix + docs gate
checks make unit-tests + test-controllers (gates finalize -> e2e)
build one package per job, own docker-container buildx,
uploads a per-package digest patch fragment [parallel]
build-talos talos image + nocloud disk (its own leg; e2e always needs the disk)
finalize download+apply all fragments, then build the installer LAST
(its flux-push bundles the whole digest-patched tree),
emit the unified pr.patch
e2e consumes talos-image + pr.patch unchanged
```
**Why finalize is special:** `packages/core/installer`'s `image` target
runs `flux push artifact --path=packages`, bundling the *entire*
packages tree into the OCI artifact the operator pulls — so it must run
after every other package's digest edit is applied, never as a peer
matrix job. `talos` is likewise pulled out of the matrix (its image and
nocloud disk share heavy `_out/assets` and build once).
**Gating:** `finalize` (and therefore `e2e`, which needs it) depends on
`checks`, so a red unit/controller test skips the expensive install+e2e
path instead of running it for ~1h — the gating the old in-build-job
tests gave, without re-coupling the cheap parallel builds.
**Change-scoping:** `hack/build-matrix.sh` parses the root Makefile
`build:` target (single source of truth) and emits only the packages a
PR touches — most PRs build ~0–1 images — escalating to the full set on
a shared-dependency change (`cozy-lib`, the build macros, `go.mod`, the
build workflows). It fans `seaweedfs` out to `objectstorage-controller`
(cross-package sidecar-tag dependency). 16 bats cases cover the
selector.
**SBOM:** `SBOM=1` wires buildx `--sbom` through `common-envs.mk` (off
by default). It stays off in the matrix until an OCIR
referrers/attestation probe confirms the registry accepts attestations.
**Decommission:** the cache warmer (`build-main.yaml`) moves to the
ephemeral pool, so no build path depends on the dedicated self-hosted
runner. The runner remains referenced only by the e2e `debug` breakpoint
path; fully de-registering it is a separate infra step.
### Validation
- Local: `hack/build-matrix_test.bats` 16/16; `actionlint` clean (custom
runner labels registered in `.github/actionlint.yaml`); `make -n`
confirms `--sbom` is off by default and on with `SBOM=1`.
- CI: this PR edits `pull-requests.yaml`, so `build-matrix.sh` forces a
full matrix — the PR exercises the new pipeline on itself. First run was
green through `plan → 25 parallel builds → build-talos → finalize →
install → 21/22 app tests`; the only red was the pre-existing
`bucket`/BucketAccess flake (install and the other 21 suites passed on
the matrix-built tree), which validates the decomposition end-to-end.
### Follow-ups (tracked on #2937)
- Probe OCIR referrers/attestation support; enable `SBOM=1` in the
matrix if supported.
- Infra: de-register the self-hosted build runner (the e2e `debug`
breakpoint path is its only remaining user).
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Refined CI to a staged, artifact-based pipeline with gated checks,
per-package parallel builds, and centralized patch assembly before
building final images
* Improved ephemeral runner performance and reliability (runner
selection, longer timeout, build toolchain setup)
* Added build-matrix auto-selection with targeted fan-out rules and
“full rebuild” triggers
* Updated CI lint configuration to recognize ephemeral runner labels
* Introduced an optional SBOM toggle for build artifacts
* **Documentation**
* Updated e2e testing docs to better explain Test-Impact Analysis scope
and gating semantics
* **Tests**
* Added Bats tests validating build-matrix selection and edge cases
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pin the PR #2872 B1 fix: when the management cluster exposes only remote
LINSTOR StorageClasses whose names sort after "kubevirt" and none is named
the configured storageClass (default "replicated"), the tenant default must
be chosen among the propagated classes and never the legacy "kubevirt" alias.
helm-unittest cannot reach this branch (no live cluster -> lookup returns
empty -> the storageClasses map collapses to the "replicated" fallback), so
it is exercised in e2e via a single server-side dry-run render of the
kubernetes chart against the live management cluster: two remote LINSTOR
classes are added and "replicated" is removed for the one render, then the
rendered -csi HelmRelease storageClasses map is asserted to carry exactly one
default among the propagated classes, never the "kubevirt" alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- csi_test.yaml: assert the fallback storageClasses map yields a default
"replicated" plus the non-default "kubevirt" alias, and that the
alias tracks the storageClass selector.
- kubevirt-csi-node: add a Makefile test target (otherwise the package
is silently skipped by the unit-test runner) and storageclasses_test
covering per-entry rendering, the single-default invariant, and the
absence of any enforcement allowList in driver-config.
- run-kubernetes.sh: verify in e2e that "replicated" propagates as the
default kubevirt-CSI class, the "kubevirt" alias is retained, and the
node-local "local" infra class is filtered out.
Refs: #2094
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Resolve the .github/workflows/pull-requests.yaml conflict: keep this branch's
per-package build matrix and carry main's `git diff --binary` patch fix
(#e566992) onto every fragment and pr.patch capture, so provider bundles that
package image targets regenerate (components*.gz) survive fragment reassembly
in finalize and reach the patch e2e applies.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
> **Supersedes #2610.** Same work, rebased cleanly onto current `main`
so it now sits on top of #2867 (the `kube-controller-manager` VAP
type-checker fix). That fix is what was killing the base install in
#2610's E2E *before* the `kubernetes` app test ever ran, leaving the
Talos worker-bootstrap rewrite verified only by code reading. #2610's
67-commit branch is squashed here into 8 logical commits with original
authorship preserved (@kvaps and @IvanHunters as authors); the tree is
**byte-identical** to #2610's head — no code changes, just the rebase
onto `main` (to pick up #2867) plus history cleanup. The detailed review
history lives on #2610.
Phase 1 of the Kubernetes-app split design (see cozystack/community#8):
replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by
`cluster-api-bootstrap-provider-talos` (CABPT) and a
`clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane
pod. Existing tenants keep working — old machines roll out and get
replaced by Talos workers without manual intervention.
Highlights:
- Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the
existing kubeadm one in `capi-providers-bootstrap`.
- Bump the Kamaji control-plane provider with an upstream-bound patch
that exposes `KamajiControlPlane.spec.network.additionalServicePorts`,
used to publish trustd (50001/TCP) on the apiserver service.
- Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and
stable random Talos secrets (`token`, `clusterId`, `clusterSecret`,
`bootstrapToken`) using the helm lookup-and-reuse pattern.
- Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the
tenant `kube-system` via a Helm post-install/upgrade Job.
- Render a `TalosConfigTemplate` (worker machineconfig, `generateType:
none`) and switch the `MachineDeployment.bootstrap` reference from
`KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets
and the Kamaji apiserver service being ready.
- Boot workers from the Talos openstack image via a CDI `DataVolume`
(`source.http.url` → `factory.talos.dev`), expose the system disk as
virtio-blk with `blockSize.custom: logical=512, physical=4096` so
4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's
O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk
(Talos lays out EPHEMERAL itself).
- Pin pod/service CIDRs in the worker machineconfig to the tenant ranges
(`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap
diagnostic against the host pod CIDR.
- Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the
Kamaji apiserver Service ClusterIP (looked up at render time), so
bootstrap survives the chicken-and-egg moment where tenant DNS does not
exist yet and the host CoreDNS does not serve the tenant zone.
End-to-end on a dev cluster: a fresh tenant brings up a Talos worker,
kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving
by the talos-csr-signer sidecar), Cilium initialises with hostNetwork
against the apiserver ClusterIP, CoreDNS comes up, and the node
transitions to `Ready`.
### Release note
```release-note
feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
* Added Talos worker configuration for `talos.version` and
`talos.schematicID`
* Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`)
for worker remediation tuning
* Added image overrides for `kubectl` and `talosCsrSigner`
* Exposed additional tenant control-plane Service ports support
**Breaking Changes**
* Removed Kubernetes v1.30; supported versions start at v1.31
* Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with
consolidated Talos system-disk semantics
**Removals**
* Removed Ubuntu container-disk images and related containerd patch
mechanisms
* Air-gapped tenant workers temporarily unsupported during the Talos
transition
**Documentation**
* Updated chart docs, parameters, and GitOps upgrade guidance for
version bumps
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolve conflict in .github/workflows/pull-requests.yaml: keep the new
per-package build matrix and fold in main's docker.io->mirror.gcr.io
base-image routing (#3042). Apply buildkitd-config: hack/buildkitd.toml
to all three new build legs (build, build-talos, finalize) so the mirror
fix covers the whole decomposed pipeline, not just the conflicted leg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
On E2E test failure, capture the host→pod CNI data-plane state for each
NotReady pod (scheduled, with a podIP, not terminal) from that pod's
node, into the crust-gather snapshot dir so it ships in the uploaded
report. New helper `hack/e2e-capture-dataplane.sh`, invoked from the
shared on-failure hook in `hack/cozytest.sh` right after the existing
crust-gather collect.
Per affected pod, from the pod's node:
- **host netns** (via the kube-ovn `cni-server`, hostNetwork): `ip route
get <podIP>`, `ip neigh`, `ip rule`, `ip addr show ovn0`, and the kernel
conntrack entries for the podIP (`conntrack -L`, falling back to
`/proc/net/nf_conntrack`).
- **cilium-agent**: `endpoint list`, `bpf ct list global` filtered to
the podIP, a bounded `monitor --type drop`, and `hubble observe
--verdict DROPPED` when the CLI is present.
- **OVS/OVN**: `ovs-ofctl dump-flows br-int`, the OVN `Port_Binding` /
`Logical_Switch_Port` for the pod, and a bounded `ovn-sbctl lflow-list`.
## Why
The install intermittently fails on a host→local-pod data-plane
transient: kubelet on a node hits a local pod's readiness/startup probe
with `connection refused` for several minutes while overlay pod→pod
traffic works, then it self-heals. This stems from the cilium+kube-ovn
chaining config (forced `enable-host-legacy-routing`, CNI
`InstallEndpointRoute:false`), which delegates host→local-pod routing to
kube-ovn/ovn0. Crucially, under `enable-host-legacy-routing` the
host→local-pod path traverses the kernel netfilter stack, not cilium BPF
— so the authoritative evidence is the kernel conntrack table plus the
host route/rule/neighbor state, captured here from the node's host
netns.
The crust-gather snapshot records Kubernetes object state but not the
node's L3 forwarding state, so the mechanism cannot be root-caused after
the fact. This adds exactly that evidence so the next recurrence is
dispositive.
## Diagnostic-only
This changes nothing about install or test behavior — it only collects
more evidence on an already-failed test. It runs after the existing
crust-gather collect and only when a test has already failed, never
mutates the cluster (read-only `get`/`exec`), and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
`|| true`, the whole run is wrapped in a `timeout -k 30 600` wall-clock
backstop, node-global captures are deduped per node, and the pod count
is capped (`MAX_PODS=12`, env-overridable). It no-ops cleanly when no
pods are affected or when a tool/pod is absent. No new retries, no
`EXIT`/`RETURN` traps — consistent with `docs/agents/e2e-testing.md`.
## Validation
`sh -n`, `bash -n`, and `shellcheck` are clean on both touched files.
The affected-pod selection filter and the no-cluster no-op path were
smoke-tested. Not exercised against a live cluster — container/label
names and the kube-ovn LSP naming (`<pod>.<ns>`) are taken from the
`packages/system/{cilium,kubeovn}` charts and the existing
`hack/e2e-cilium-endpoint-leak-healer.sh`; all captures are defensively
`|| true` and time-boxed so absent tooling degrades to a labeled note
rather than a failure.
## What
Raise the `kafka` and `zookeeper` components in the Kafka e2e test from
the `nano` preset (250m / 128Mi) to `c1.small` (1 CPU / 1Gi), the chart
default.
## Why
128Mi cannot hold the Strimzi Kafka 3.9.1 and ZooKeeper JVMs. The
brokers OOMKill on startup (exit 137) and enter CrashLoopBackOff, so the
pods never reach Ready, the `Kafka` CR and its HelmRelease never go
Ready, and the 5m readiness wait times out. It presents as a flake
rather than a hard failure because 128Mi sits right at the JVM-start
threshold — occasionally a broker becomes Ready before its RSS crosses
the cgroup limit.
`c1.small` (1Gi) comfortably fits both JVMs and matches the chart
default, so the test exercises the documented default sizing. The e2e
sandbox VMs are 8 CPU / 24Gi, so 1Gi per pod (4Gi total) across the four
pods (kafka-0/1, zookeeper-0/1) is well within budget.
## Scope
Test-only; values changed, test structure untouched. MariaDB's e2e test
also uses `nano` but was deliberately left as-is: it runs MariaDB (C) +
a Go exporter with no JVM, its chart default is already `t1.nano`
(128Mi), and it shares the `nano` preset with the passing
postgres/redis/qdrant tests — not the same OOM class.
## What
Add a bounded escalation tier to the E2E cilium endpoint-leak healer:
when a pod stays wedged on `FailedCreatePodSandBox` "IP ipv4:<X> is
already in use" for the same IP across several heal cycles, restart the
cilium-agent on that node. Keep the existing per-endpoint remedies as
the first tier.
## Why
The leak (cilium/cilium#38313 class) sometimes lives purely in the
cilium-agent's in-memory endpoint state. The existing remedies —
`cilium-dbg endpoint disconnect` and delete-rescheduling the wedged pod
— cannot reach it: IPAM keeps re-handing the SAME released IP, so the
recreated pod re-wedges on it, and on some variants the stale entry is
not even in the queryable endpoint registry, so disconnect is a no-op.
Both are inescapable loops that fail the install. The only thing that
clears an in-memory leak is restarting the agent, which drops the node's
endpoint registry and rebuilds it from CRDs/IPAM without the stale
entry.
## How
- Track recurrence per `(node, ip)`. After `ESCALATE_AFTER` (default 3)
heal cycles of a futile first-tier remedy on the same IP, restart that
node's cilium-agent pod.
- Bounded by `MAX_ESCALATIONS_PER_NODE` (default 2); after the cap, log
loudly and surface the failure instead of restarting again. State is
per-node and resets after each restart so one restart covers every
leaked IP on the node.
- A genuine duplicate-IP (endpoint backs a live owner) is refused before
counting, so an agent restart can never mask a real product bug.
- RBAC unchanged: the existing cluster-wide `pods` `get/list/delete`
already covers deleting a cilium-agent pod in `cozy-cilium`.
E2E-test-only resilience; no product code change. Remove alongside the
rest of the healer once a fixed Cilium ships.
## Problem
On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.
## Fix
Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.
## Why
The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.
## Test
- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.
* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
**Problem**
The `foundationdb` e2e test fails intermittently with a `.status`
jsonpath poll timeout even though the FoundationDB cluster is fully
healthy and reconciled. The `generations.reconciled`,
`hasListenIPsForAllPods`, `health.fullReplication` and `health.healthy`
gates polled their status fields with a 60s ceiling, while the
strictly-earlier `health.available` gate already allowed 300s. The
operator stamps `generations.reconciled` only after every process holds
its role for `minimumUptimeSecondsForBounce` (60s, set in
`packages/apps/foundationdb/templates/cluster.yaml`), which makes
`reconciled` — and the health fields gated on it — structurally the last
status fields to converge. The short ceiling closed the gate seconds
before the field landed; a snapshot taken a few seconds after a failure
shows the live object fully reconciled (`generations.reconciled: 1`,
`health.healthy: true`).
**Fix**
Raise those four late-converging gates to 300s, matching the existing
`health.available` budget. The waits stay event-driven (`until …; do
sleep …; done`) — only the ceilings widen, no new retries or fixed
sleeps. A comment records why `reconciled` converges last so the budget
ordering stays legible to future readers. The early-converging 60s gates
(`configured`, `connectionString`, `databaseConfiguration.*`,
`desiredProcessGroups`) and the spec-field gates are left untouched,
since they settle before `health.available` and 60s is already adequate.
**Why not just rerun**
Rerunning masks a deterministic ordering bug: the latest-converging
field carried the tightest budget. The cluster was healthy on every
"failure" — the test was racing its own success criteria, so the failure
recurs until the budget ordering is corrected. This change touches only
the test harness; the platform behaviour it asserts is unchanged.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Increased waiting time in one end-to-end FoundationDB setup test so
cluster status checks have more time to stabilize.
* Reduced flaky failures by allowing key health and reconciliation
fields to converge before the test continues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem
The `Bootstrap Talos cluster` E2E step fails intermittently with `❌ Test
failed: Bootstrap Talos cluster (exit 1)` even though the cluster is
healthy. The exit comes from the etcd-members convergence gate, which is
bounded at `timeout 180`.
A 3-node etcd converges via Talos's serialized learner promotion — etcd
admits one learner at a time, so the third member is only promoted after
the second has fully caught up. On a loaded ephemeral runner this
deterministic path can legitimately approach ~3 minutes of wall clock
and photo-finish against the 180s ceiling, timing out a cluster that has
in fact converged (every member reaches `Health check successful`; no
kernel panic / OOM / boot failure / network-unreachable in dmesg).
Observed: a run where the third member reached etcd health at +178.6s
against the 180s budget, failing ~1.4s short.
## Fix
Widen the etcd-members convergence budget from 180s to 300s. The wait is
already event-driven (`until talosctl etcd members …; do sleep 1; done`)
— only the ceiling was too tight. No retry is added and the
dmesg-on-failure diagnostic is preserved.
300s matches the repo's other in-line budgets for deterministic
convergence (HelmRelease-ready 5m, harbor/NFS 10m, install 15m).
## Why not just rerun
This is not environmental noise: the cluster converges correctly every
time, it is only timed out. Rerunning wastes a full ~1h E2E cycle to
re-roll the dice on whether the runner shaves the last ~2s. Correcting
the mistuned budget removes the photo-finish.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Extended the cluster bootstrap wait time for control plane health
checks, reducing false failures during slow startup.
* Improved readiness handling so cluster setup is more resilient before
moving on to later initialization steps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Introduce a new network.cozystack.io/v1alpha1 API group with two kinds
that form the mechanism-agnostic LoadBalancer exposure model:
- ExposureClass (cluster-scoped): the admin binds a logical class to a
concrete backend (externalIPs, metallb, cilium, robotlb) and its VIP
address scope, following the StorageClass / IngressClass pattern.
- ServiceExposure (namespaced): the tenant-facing intent to expose a
Service externally via a named class, with status reporting the
resolved backend and assigned IPs.
Wire api/network into the controller-gen CRD generation so deepcopy and
the two CRD manifests are produced alongside the existing groups.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrite the SecurityGroup e2e for the membership model: assert the backing
CiliumNetworkPolicy's endpointSelector is the group's own membership label,
that attachments ride a storage-owned annotation hidden from the view, and
that fromApp/fromSG peers project to lineage- and membership-label endpoint
selectors the real cilium.io/v2 CRD accepts. A second test creates a
managed-labeled pod and asserts the securitygroup-controller stamps the
membership label on attach and clears it on detach.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The on-failure dataplane diagnostic only captured host->pod CNI state for
NotReady pods. A recurrent flake presents differently: a tenant Service
type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready
and Running, so there is no NotReady pod and the hook captured nothing --
leaving the host->cross-node->bridged-worker-VMI->nodePort path
uncharacterised, which is exactly where root-causing has stalled.
Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer
services with an ingress IP, derives the EndpointSlice backend, the
nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork
MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB
whose every probe fails gets the heavy capture, and that probe doubles as
the traffic for the tcpdumps. For a failing LB it captures, on both the
announcer and endpoint nodes, the host cilium LB map, kernel
conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on
the geneve tunnel and the backend tap -- localising the failing hop to
host-cilium programming vs kube-ovn delivery. In-guest VMI capture is
documented as a deferred stretch that needs tenant-cluster access.
The existing NotReady-pod capture is unchanged. The new section is
best-effort and time-boxed like the rest of the script, and a unit test
pins the pure enumeration, announcer-node, and capture-or-skip logic against
mock input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Every migration that stamps the `cozystack-version` ConfigMap must carry
the `platform.cozystack.io/no-delete=true` label so the
`cozystack-no-delete-guardrail` ValidatingAdmissionPolicy guards it. The
chart template renders the labeled ConfigMap only on first install, so
each upgrade-time stamp has to re-emit the label by hand — migration 43
had to inline-copy migration 42's heredoc to avoid a label-less apply by
the same field manager stripping it. The bug class was patched
per-migration, not eliminated.
This extracts the stamp into a shared, sourced helper
(`render_cozystack_version_manifest` + `stamp_cozystack_version`) under
`images/migrations/migrations/lib/`, and routes migrations 42, 43, 44
and the `run-migrations.sh` bootstrap path through it. The labeled
manifest now has a single definition, so future migrations cannot drift
back to a label-less apply.
The upgrade path is behavior-preserving: the rendered manifest is
byte-identical to the previous heredocs (pinned by a golden unit test).
The bootstrap path in `run-migrations.sh` previously created the
ConfigMap without the label — it now carries it like every
migration-driven stamp.
Adds `hack/cozystack-version-stamp.bats` (8 tests) covering the helper:
label presence, ConfigMap name, quoted-string version type, namespace
default and override, required-argument guard, and the byte-exact golden
manifest.
Closes#2781
### Release note
```release-note
fix(platform): always stamp the platform.cozystack.io/no-delete label on the cozystack-version ConfigMap, including the migration bootstrap path, so deletion protection cannot be silently dropped
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Tests**
* Expanded validation to confirm the version-stamp manifest is parseable
YAML, includes the required no-delete label, targets the correct
`cozystack-version` ConfigMap, and quotes the version value correctly.
* Added coverage for default and overridden namespaces, argument
validation failures, and a golden “byte-for-byte” render check.
* Added a guardrail test to prevent bypassing the shared stamping logic
by direct `kubectl` ConfigMap stamping.
* **Refactor**
* Centralized `cozystack-version` rendering and stamping into a shared
helper and updated migrations and the migrations runner to use it
consistently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The etcd suite still failed every run after the clean-after-each rework:
etcd-empty-backup (first to run) passed, but etcd-basic and etcd-backup-schedule
wedged at EtcdCluster WaitingForFirstQuorum until the chart's helm --wait timed
out (InstallFailed).
Root cause, from the failed run's crust-gather snapshot: the StatefulSet's
data-etcd-{0,1,2} PVCs are retained on uninstall and the drain never reclaimed
them, so each later scenario rebound the first scenario's volumes (identical PV
UIDs and a single creationTimestamp across all three scenarios) and booted the
previous etcd's data dir and member identity — never forming a fresh first
quorum. The pods were individually Ready (StatefulSet readyReplicas: 3) with all
certs valid, which is why it looked like an operator/network issue rather than
stale storage.
Delete the data PVCs (app.kubernetes.io/managed-by=etcd-operator) in
etcd_cleanup and wait for them to clear alongside the HR and DataStore, so each
scenario starts from empty storage.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The viewer (readonly) S3 credential can still upload: the seaweedfs-cosi-driver
does not yet honor accessPolicy=readonly (cozystack/cozystack#1921, blocked
upstream on kubernetes-sigs/container-object-storage-interface#244). The whole
cozystack chain is correct — Bucket.spec.users.viewer.readonly: true binds the
viewer BucketAccess to the tenant-root-readonly BucketAccessClass whose
parameters.accessPolicy is readonly — but the driver grants the identity write
access anyway (verified live on dev10). The BATS original asserted the same
denial; its retries and toothless gate hid the failure, which this migration
surfaces.
Keep exercising the path and verifying admin read/write + viewer read, but emit
a ::warning:: instead of failing when the viewer can upload, so the suite is not
blocked on an unimplemented upstream feature. TODO(#1921): restore the hard
failure once the driver enforces read-only access.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- mongodb/redis: assert status.readyReplicas instead of status.replicas, so the
StatefulSet check verifies pods are actually Ready (matches openbao/qdrant).
- kuberture: poll for the asynchronously-populated target annotation before
asserting the annotation shape (it was a one-shot test under set -e that could
race on a slow cluster), and guard the probe-Pod phase capture so the
describe/logs diagnostics still run when the Pod never materializes.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd scenarios share the singleton 'etcd' release and relied on the
next scenario's ensure-clean to drain the previous one. Chainsaw runs Tests in
name order (not file order), and v0.2.15 has no test-level finally, so the
last-run scenario leaked its etcd plus the wedged Kamaji DataStore into every
later suite on full/release runs (lexfrei review B3).
Make each scenario self-contained instead: a single run step pre-cleans any
leftover etcd, installs and asserts, then drains in a finally that always runs
(the step is always reached, so teardown fires on success and failure alike).
Extract the shared drain to _lib/etcd-cleanup.sh, removing the ~25-line block
that was copied across all three scenarios.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw port capped the harbor-test-registry BucketClaim bucketReady wait
at the default 5m, half the 600s the BATS original used. seaweedfs/COSI
provisioning can lag several minutes on a loaded runner, so the dependent
harbor-test-system HelmRelease keeps its 10m budget while the BucketClaim it
depends on was capped at 5m — a latent false-flake. Restore 10m
(docs/agents/e2e-testing.md section 6).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The openbao suite's finally deleted PVC data-openbao-test-0 while the
StatefulSet pod still mounted it; the kubernetes.io/pvc-protection finalizer
blocked removal and the delete hit its 5m timeout, failing the suite on every
run (confirmed in CI runs 28084122948 and 28025182013). Delete the OpenBAO CR
first so the HelmRelease/StatefulSet/pod tear down and release the PVC, then
remove the now-unmounted PVC — mirroring the BATS teardown order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
run_kubernetes_test applies the tenant Kubernetes CR imperatively (kubectl), so
Chainsaw's auto-cleanup — which only tracks resources it apply/create-d — never
reclaims it. On success the script's tail deleted it, but on failure nothing did:
cozy_cleanup() used to be invoked by hack/cozytest.sh's trap, and the Chainsaw
harness has no equivalent caller, so a failed kubernetes-latest/previous run
leaked the tenant cluster (tens of GiB of worker PVCs) into the shared
tenant-test namespace and starved later suites.
Add a `finally` block to both kubernetes suites that runs cozy_cleanup. `finally`
always executes, after the `catch` phase, so the ordering matches the BATS
harness: pass -> cleanup; fail -> crust-gather (tenant trap + host catch) ->
cleanup. cozy_cleanup stays best-effort (|| true) so a slow tenant teardown never
flips a passing test red.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Convert hack/e2e-apps/backup.bats.disabled to a Chainsaw test under
hack/e2e-chainsaw/backup/. It stays opt-in: the chainsaw-test.yaml.disabled
filename keeps it out of chainsaw's default discovery and out of
hack/select-e2e.sh enumeration, matching the previous disabled state.
The declarative readiness chain (Bucket -> VirtualMachine -> BackupJob) becomes
assert operations; the imperative Velero-backup + S3 verification stays a single
script step, mirroring the bucket suite. README documents the opt-in run command
and the backup-infra preconditions.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd Chainsaw scenarios share the singleton release name 'etcd' and
the cluster-scoped DataStore/<namespace> the etcd chart provisions for Kamaji.
On teardown Kamaji leaks its kamaji.clastix.io/TenantControlPlane finalizer on
that DataStore — once the backing etcd is gone the reconcile that would drop it
cannot complete — so the DataStore wedges in Terminating and the etcd
HelmRelease uninstall hangs. In CI this errored every etcd case: the first
scenario's stuck teardown blocked the next scenario's install (HelmRelease
never Ready, "DataStore status: Terminating", etcd pods refusing :2381 because
the CA issuer had no key material).
Make teardown explicit instead of relying on Chainsaw auto-cleanup (which was
waiting on the wedged uninstall): set skipDelete on each Test and have every
scenario's ensure-clean drain the previous one — delete the etcd and
force-clear the orphaned finalizer (no TenantControlPlane uses this
per-namespace test DataStore) in a loop until the HelmRelease and DataStore are
both gone. Each scenario then installs from a clean slate.
Verified on a dev cluster: all three scenarios pass back-to-back where they
previously cascaded.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch
diverged landed a competing single-VM build restructure (#2939) and
test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build
supersedes #2939:
- .github/workflows/pull-requests.yaml: keep #2983's
plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer
resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact
interface (pr-patch + talos-image) is identical on both sides, so e2e depends
on finalize/build-talos with no input rewiring. main's top-level permissions
and SHA-pinned action refs applied across all jobs.
- All other files merged cleanly (only pull-requests.yaml conflicted).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Fixes two failure modes in the COSI control plane built from upstream
container-object-storage-interface v0.2.2, both carried as image
patches.
**BucketClaim never converges to ready (#3032).** The controller does
not watch Bucket objects, and the generic controller drops no-op resync
deltas, so a BucketClaim whose backend Bucket flips to ready *after* the
first reconcile stayed `bucketReady=false` indefinitely — BucketAccess
was never granted and the consuming app (bucket, harbor) got no
credentials. The existing re-read of the live Bucket only ran during a
reconcile that nothing re-triggered. The controller now returns an error
while the Bucket is not ready, forcing a rate-limited requeue that
re-reads the Bucket until its readiness propagates, so the claim
converges deterministically instead of racing the async sidecar.
**Slow provisioner failover on pod recreate (part of #2429).** The
controller and the per-driver sidecar run single-replica, so leader
election adds no availability — only failover latency. The upstream 150s
lease made a recreated provisioner pod stall reconciliation for up to
~150s. The lease is lowered to the standard Kubernetes values
(15s/10s/2s), so a recreated pod resumes in ~15s. The stale unix-socket
CrashLoop in #2429 was already resolved by the seaweedfs-cosi-driver
v0.3.1 self-heal; the silent "Bucket ready with no backend bucket"
finding on that issue is a separate driver bug and is left tracked
there.
Coverage: a unit test in the patched controller pins the
requeue/propagation contract (fails without the requeue), and the
bucket/harbor e2e suites now assert `bucketReady=true` on a tight bound
instead of tolerating the lag, with scoped diagnostics on the failure
path.
Closes#3032
Refs #2429
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved object storage bucket readiness propagation so bucket claims
now track the backend bucket’s actual `bucketReady` status and requeue
when readiness isn’t yet observed.
* Reduced and made readiness waits more reliable in end-to-end checks,
with explicit match conditions and clearer debug output (including claim
YAML, backend bucket list, and controller pod status).
* Shortened leader-election timing to speed up recovery after pod
restarts.
* **Chores**
* Updated automated PR labeling to map the object storage controller
scope to the storage area label.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The matrix selector's full_rebuild_pattern omitted the root Go module dirs
(api/, cmd/, internal/, pkg/). Eight matrix image Dockerfiles COPY those dirs,
so a PR changing only root Go source emitted an empty matrix, skipped the build
job, and let e2e install stale digests — the Go change was never exercised. Add
the four dirs to the pattern and pin it with bats cases for api/- and pkg/-only
diffs forcing the full matrix.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The kafka e2e test pinned both the kafka and zookeeper components to the
legacy "nano" preset (250m CPU / 128Mi memory). 128Mi cannot hold the
Strimzi Kafka 3.9.1 and ZooKeeper JVMs: the brokers OOMKill on startup
(exit 137) and CrashLoopBackOff, so the pods never reach Ready, the Kafka
CR and its HelmRelease never go Ready, and the 5m readiness wait times
out. It surfaced as a flake because 128Mi sits right at the JVM-start
threshold — occasionally a broker passes readiness before its RSS crosses
the cgroup limit.
Raise both components to "c1.small" (1 CPU / 1Gi), the chart default,
which comfortably fits the JVMs. The e2e sandbox VMs are 8 CPU / 24Gi, so
1Gi across the four pods is well within budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint
disconnect, or delete-reschedule the wedged pod) cannot clear a leak
that lives purely in the cilium-agent's in-memory state: when IPAM keeps
re-handing the SAME released IP, a delete-rescheduled pod re-wedges on
it, and on some variants the stale entry isn't even in the queryable
endpoint registry so disconnect no-ops. Both produce an inescapable
delete -> re-wedge / no-op loop that fails the install on
FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313).
Add a bounded Tier-2 escalation: track recurrence per (node,ip); once
the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that
node's cilium-agent (delete its pod; the DaemonSet rebuilds it from
CRDs/IPAM without the stale entry), capped per node so it can never
loop-restart agents. A genuine duplicate-IP (live owner) is refused
before counting, so an agent restart never masks a real product bug.
Tier-1 stays the first remedy; escalation fires only after it proves
futile on the same IP.
RBAC needs no new verb: the existing cluster-wide pods get/list/delete
already covers deleting a cilium-agent pod in cozy-cilium.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.
The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.
Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Pull Request Build job authenticates only to the push registry (OCIR),
so every base image (golang, ubuntu, alpine, nginx, ...) is pulled from
Docker Hub anonymously. Those pulls intermittently fail at
auth.docker.io/token under Docker Hub's anonymous-pull rate limit, leaving
the base image unresolved and failing make build.
Build on a docker-container buildx builder whose buildkitd routes docker.io
through mirror.gcr.io, Google's public pull-through cache for Docker Hub: no
auth, far higher limits, transparent fall-through to Docker Hub on a miss.
This reuses the builder pattern already proven in build-main.yaml (same
make build, same DOCKER_CONFIG auth forwarding); only the committed
hack/buildkitd.toml mirror config is new.
Fail-safe: if mirror.gcr.io is unreachable, BuildKit falls back to
registry-1.docker.io and the build still succeeds, so this strictly reduces
build failures without adding a new hard dependency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.
Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>