## What this PR does
Every migration that stamps the `cozystack-version` ConfigMap must carry
the `platform.cozystack.io/no-delete=true` label so the
`cozystack-no-delete-guardrail` ValidatingAdmissionPolicy guards it. The
chart template renders the labeled ConfigMap only on first install, so
each upgrade-time stamp has to re-emit the label by hand — migration 43
had to inline-copy migration 42's heredoc to avoid a label-less apply by
the same field manager stripping it. The bug class was patched
per-migration, not eliminated.
This extracts the stamp into a shared, sourced helper
(`render_cozystack_version_manifest` + `stamp_cozystack_version`) under
`images/migrations/migrations/lib/`, and routes migrations 42, 43, 44
and the `run-migrations.sh` bootstrap path through it. The labeled
manifest now has a single definition, so future migrations cannot drift
back to a label-less apply.
The upgrade path is behavior-preserving: the rendered manifest is
byte-identical to the previous heredocs (pinned by a golden unit test).
The bootstrap path in `run-migrations.sh` previously created the
ConfigMap without the label — it now carries it like every
migration-driven stamp.
Adds `hack/cozystack-version-stamp.bats` (8 tests) covering the helper:
label presence, ConfigMap name, quoted-string version type, namespace
default and override, required-argument guard, and the byte-exact golden
manifest.
Closes#2781
### Release note
```release-note
fix(platform): always stamp the platform.cozystack.io/no-delete label on the cozystack-version ConfigMap, including the migration bootstrap path, so deletion protection cannot be silently dropped
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Tests**
* Expanded validation to confirm the version-stamp manifest is parseable
YAML, includes the required no-delete label, targets the correct
`cozystack-version` ConfigMap, and quotes the version value correctly.
* Added coverage for default and overridden namespaces, argument
validation failures, and a golden “byte-for-byte” render check.
* Added a guardrail test to prevent bypassing the shared stamping logic
by direct `kubectl` ConfigMap stamping.
* **Refactor**
* Centralized `cozystack-version` rendering and stamping into a shared
helper and updated migrations and the migrations runner to use it
consistently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Fixes two failure modes in the COSI control plane built from upstream
container-object-storage-interface v0.2.2, both carried as image
patches.
**BucketClaim never converges to ready (#3032).** The controller does
not watch Bucket objects, and the generic controller drops no-op resync
deltas, so a BucketClaim whose backend Bucket flips to ready *after* the
first reconcile stayed `bucketReady=false` indefinitely — BucketAccess
was never granted and the consuming app (bucket, harbor) got no
credentials. The existing re-read of the live Bucket only ran during a
reconcile that nothing re-triggered. The controller now returns an error
while the Bucket is not ready, forcing a rate-limited requeue that
re-reads the Bucket until its readiness propagates, so the claim
converges deterministically instead of racing the async sidecar.
**Slow provisioner failover on pod recreate (part of #2429).** The
controller and the per-driver sidecar run single-replica, so leader
election adds no availability — only failover latency. The upstream 150s
lease made a recreated provisioner pod stall reconciliation for up to
~150s. The lease is lowered to the standard Kubernetes values
(15s/10s/2s), so a recreated pod resumes in ~15s. The stale unix-socket
CrashLoop in #2429 was already resolved by the seaweedfs-cosi-driver
v0.3.1 self-heal; the silent "Bucket ready with no backend bucket"
finding on that issue is a separate driver bug and is left tracked
there.
Coverage: a unit test in the patched controller pins the
requeue/propagation contract (fails without the requeue), and the
bucket/harbor e2e suites now assert `bucketReady=true` on a tight bound
instead of tolerating the lag, with scoped diagnostics on the failure
path.
Closes#3032
Refs #2429
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved object storage bucket readiness propagation so bucket claims
now track the backend bucket’s actual `bucketReady` status and requeue
when readiness isn’t yet observed.
* Reduced and made readiness waits more reliable in end-to-end checks,
with explicit match conditions and clearer debug output (including claim
YAML, backend bucket list, and controller pod status).
* Shortened leader-election timing to speed up recovery after pod
restarts.
* **Chores**
* Updated automated PR labeling to map the object storage controller
scope to the storage area label.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The kafka e2e test pinned both the kafka and zookeeper components to the
legacy "nano" preset (250m CPU / 128Mi memory). 128Mi cannot hold the
Strimzi Kafka 3.9.1 and ZooKeeper JVMs: the brokers OOMKill on startup
(exit 137) and CrashLoopBackOff, so the pods never reach Ready, the Kafka
CR and its HelmRelease never go Ready, and the 5m readiness wait times
out. It surfaced as a flake because 128Mi sits right at the JVM-start
threshold — occasionally a broker passes readiness before its RSS crosses
the cgroup limit.
Raise both components to "c1.small" (1 CPU / 1Gi), the chart default,
which comfortably fits the JVMs. The e2e sandbox VMs are 8 CPU / 24Gi, so
1Gi across the four pods is well within budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint
disconnect, or delete-reschedule the wedged pod) cannot clear a leak
that lives purely in the cilium-agent's in-memory state: when IPAM keeps
re-handing the SAME released IP, a delete-rescheduled pod re-wedges on
it, and on some variants the stale entry isn't even in the queryable
endpoint registry so disconnect no-ops. Both produce an inescapable
delete -> re-wedge / no-op loop that fails the install on
FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313).
Add a bounded Tier-2 escalation: track recurrence per (node,ip); once
the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that
node's cilium-agent (delete its pod; the DaemonSet rebuilds it from
CRDs/IPAM without the stale entry), capped per node so it can never
loop-restart agents. A genuine duplicate-IP (live owner) is refused
before counting, so an agent restart never masks a real product bug.
Tier-1 stays the first remedy; escalation fires only after it proves
futile on the same IP.
RBAC needs no new verb: the existing cluster-wide pods get/list/delete
already covers deleting a cilium-agent pod in cozy-cilium.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.
The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.
Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Pull Request Build job authenticates only to the push registry (OCIR),
so every base image (golang, ubuntu, alpine, nginx, ...) is pulled from
Docker Hub anonymously. Those pulls intermittently fail at
auth.docker.io/token under Docker Hub's anonymous-pull rate limit, leaving
the base image unresolved and failing make build.
Build on a docker-container buildx builder whose buildkitd routes docker.io
through mirror.gcr.io, Google's public pull-through cache for Docker Hub: no
auth, far higher limits, transparent fall-through to Docker Hub on a miss.
This reuses the builder pattern already proven in build-main.yaml (same
make build, same DOCKER_CONFIG auth forwarding); only the committed
hack/buildkitd.toml mirror config is new.
Fail-safe: if mirror.gcr.io is unreachable, BuildKit falls back to
registry-1.docker.io and the build still succeeds, so this strictly reduces
build failures without adding a new hard dependency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.
Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The reconciled, hasListenIPsForAllPods, fullReplication and healthy gates
polled status fields with a 60s ceiling, while the strictly-earlier
health.available gate had 300s. The operator stamps generations.reconciled
only after every process holds its role for minimumUptimeSecondsForBounce
(60s), making reconciled and the health fields that depend on it the last
status fields to converge. The short ceiling closed the gate seconds before
the field landed on a healthy, fully-reconciled cluster, causing intermittent
jsonpath poll timeouts.
Raise those four gates to 300s to match the health.available budget. The waits
stay event-driven; only the ceilings widen.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The "Bootstrap Talos cluster" step times out the etcd-members health gate
at 180s. A 3-node etcd converges via Talos's serialized learner promotion
(one learner admitted at a time), so on a loaded ephemeral runner this
deterministic path can approach ~3m wall clock and photo-finish against the
180s ceiling — failing a cluster that is in fact healthy. The wait is
already event-driven; only the ceiling was too tight. Widen it to 300s.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The bucket suite waited on '{.status.bucketReady}' without an expected value;
a bare jsonpath match is satisfied by the literal "false", so the wait passed
against an unready claim. Both suites also allowed 300s/600s, long enough to
tolerate the propagation race rather than catch it, so they stayed green
whether or not the BucketClaim ever converged.
Assert bucketReady=true on a 120s bound. Convergence is now controller-driven
(the controller requeues the claim until the backend Bucket's readiness
propagates) and completes within tens of seconds; a claim frozen by the
propagation race never converges and now fails the wait fast instead of being
masked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The helper header and run-migrations.sh claimed every migration-driven
stamp routes through the helper, but migrations 1-41 still stamp the
cozystack-version ConfigMap label-less and are backfilled by migration 42.
Scope the comments to go-forward stamps (migration 42 onward and the
bootstrap path), and add a unittest asserting no migration numbered >= 42
nor run-migrations.sh stamps cozystack-version without the helper, so the
no-drift invariant is enforced rather than aspirational.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Mirror render_cozystack_version_manifest's argument guard in
stamp_cozystack_version so a missing version aborts before the kubectl
pipe. Without it, a render failure in a shell without pipefail would
leave the pipeline exit status to kubectl, which would apply empty input.
Add a unittest covering the guard.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extract the labeled cozystack-version ConfigMap stamp into a sourced
shell helper so migrations cannot drift back to a label-less apply that
strips the platform.cozystack.io/no-delete label.
render_cozystack_version_manifest emits the manifest on stdout (pure,
unit-testable); stamp_cozystack_version renders and applies it. The
namespace defaults to cozy-system and honors $NAMESPACE.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Rebuilds the e2e test sandbox on current upstream to clear the published
advisories (containerd, helm, go-git, moby/spdystream, Go stdlib)
carried by the stale tool binaries and base layers.
- base: `ubuntu:22.04` → `24.04`; `netcat` → `netcat-openbsd` (on 24.04
`netcat` is a candidate-less virtual package, so the concrete provider
must be named)
- kubectl: `1.33.2` → `1.33.13` — stays on the 1.33 line that the e2e
control plane pins at `v1.33.12` (`hack/e2e-prepare-cluster.bats`), so
client/server skew is unchanged
- talosctl: `1.10.4` → `1.13.5` — aligns the client with the v1.13.x
Talos nodes the sandbox provisions
- helm: `3.18.3` → `3.21.2` (latest helm 3) — the e2e flow drives the v3
CLI, so it stays on the 3.x line rather than moving to the breaking 4.x
- yq: `4.44.3` → `4.53.3`
- `values.yaml`: re-pinned to the rebuilt multi-arch digest
(`linux/amd64` + `linux/arm64`)
`crust-gather` and `cozyhr` already track their latest releases; `flux`
and `mc` install the current release at build time. The built image's
tool versions were verified in place (kubectl `v1.33.13`, talosctl
`v1.13.5`, helm `v3.21.2`, yq `v4.53.3`, flux `2.8.8`, ubuntu `24.04`).
Closes#2891
### Release note
```release-note
chore(testing): refresh e2e-sandbox tools and base (ubuntu 24.04, kubectl 1.33.13, talosctl 1.13.5, helm 3.21.2)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the e2e sandbox image to a new fixed digest for more
consistent test runs.
* Refreshed the sandbox build environment (Ubuntu 24.04) and upgraded
tooling (Kubernetes, Talos, Helm, and yq) to newer stable versions.
* Improved OS package setup by switching from `netcat` to
`netcat-openbsd` and consolidating apt cleanup to reduce image size.
* **Documentation**
* Updated e2e testing guidance: removed the recommendation to pin the
management cluster Kubernetes version, and added guidance to
snapshot/restore cluster state between test groups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
talosctl 1.13.5 now defaults to Kubernetes 1.36.2, so the comment's premise
— that talosctl's bundled default predates the KCM VAP nil-pointer fix — no
longer holds (1.36.2 already carries that fix). The reason e2e pins v1.33.12
is now that 1.36 is outside cozystack's supported range; reword to lead with
that and keep the KCM VAP fix as the secondary property v1.33.12 carries.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Six further references to the alpine/k8s tools image were still pinned at
1.33.4 across the installer, the cozystack-api post-upgrade hook, the vm-disk
and vm-instance resize hooks, the vertical-pod-autoscaler CRD-install override,
and the e2e cilium leak healer. Move them all to 1.36.2 so the whole repo runs
a single current tools-image version (the two digest-pinned refs get the
matching multi-arch digest sha256:44ef4942...). Same stale-stdlib remediation
as the seaweedfs bump in this PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Release E2E workflow (`release-e2e.yaml`) ran the full bats suite on
every release tag as a non-blocking alarm — its own header notes "a
failure here is an alarm, not a release blocker." It duplicates coverage
the release-cut PR already provides: the "Prepare release" commit bakes
image digests into `packages/core/` (and other packages), which TIA
escalates to the full suite, so the release PR exercises the whole suite
against the commit the tag will point at.
It has also never been green since it was added (2026-05-02): every tag
run failed at the build step. Even after #2973 fixed the registry login
today, the next run still failed (Build Talos image). Removing a
perpetually-red, non-gating duplicate removes noise without losing
coverage.
Removes the workflow, drops `release-e2e` from `select-e2e.sh`'s
full-suite pattern, repoints its unit test to `pull-requests.yaml`, and
updates the e2e-testing doc.
Refs #2397
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The per-test failure snapshots (host in cozytest.sh, tenant in run-kubernetes.sh)
ran `crust-gather collect` unbounded. On a contended/degraded cluster the collect
can hang indefinitely — observed wedging the E2E install step for ~2h35m after a
test failure until the job-level cancel, with no output after the capture line.
`|| true` only catches a bad exit code, never a hang.
Wrap both invocations in `timeout -k 30 300`: a 5-minute cap (hard SIGKILL 30s
later if the collect ignores SIGTERM) bounds the hang; a partial snapshot still
beats a multi-hour wedge, and timeout's exit 124 is already swallowed by `|| true`.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The leak-healer only refused to disconnect an endpoint whose owning pod was
Running, so a Pending owner that already held a live IPAM allocation
(status.podIP is populated once the sandbox is created, before the pod goes
Running) could have its endpoint evicted mid-startup. Refuse for any
non-terminal phase (!= Succeeded && != Failed); only a terminal owner is
safely past needing its endpoint.
Addresses non-blocking follow-up 4 on #2948.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubernetes-{latest,previous} intermittently failed their node-Ready wait
with the tenant API "connection refused" on localhost, while the cluster was
in fact healthy (host snapshot: both CAPI Machines NodeHealthy=True, TCP
Ready 2/2). Root cause: `kubectl port-forward service/kubernetes-<name>`
pins to ONE of the two Kamaji control-plane pods; when that pod's
kube-apiserver container restarted once (observed restartCount=1), the
forward broke and did not recover, so the 12m wait ran against a dead socket
and the failure-path describe/snapshot also got nothing.
Expose the tenant API with a test-scoped LoadBalancer Service selecting the
Kamaji control-plane pods instead. It load-balances across ALL ready apiserver
endpoints, so a single pod's apiserver restart is routed around transparently;
the host runs MetalLB on the sandbox /24 (pool 192.168.123.200-250) so the IP
is directly routable from the test (the in-tenant LB test already curls such
an address). The kubeconfig points at the LB IP with insecure-skip-tls-verify
(the MetalLB IP isn't in the apiserver cert SANs; e2e only) and the CA dropped.
Bonus: the LB endpoint is stable until teardown, so the on-failure tenant
crust-gather snapshot can finally reach the tenant (the port-forward was dead
by snapshot time, which is why every prior tenant snapshot came back empty).
Removes the port-forward supervisor + pidfile machinery. Verified against the
real hack/cozytest.sh under dash (CI's /bin/sh) in ubuntu:22.04: no pipeline
deadlock, the in-subshell snapshot trap fires and captures the tenant, and the
LB Service is cleaned up on both the success path and cozy_cleanup.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The healer's reserved/infra-endpoint remedy and its disconnect-failure
fallback both run `kubectl delete pod` to reschedule the wedged pod onto a
free IP, but the ClusterRole granted pods only get/list — so the delete was
forbidden and the pod stayed wedged. Add delete to the pods rule.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
crust-gather can only reach a tenant Kamaji cluster through the test's
localhost port-forward (it connects directly to the kubeconfig server URL —
there is no apiserver-proxy mode — and the in-cluster URL is unreachable from
the runner). cozytest.sh's generic EXIT trap runs in the parent shell AFTER
the failing test subshell has exited and its port-forward is reaped, so a
collect there hits a dead socket and crust-gather bails before writing
anything (every tenant snapshot so far came back empty).
Capture the tenant from an EXIT trap registered INSIDE run_kubernetes_test
instead: it fires during THAT subshell's exit, while the supervised
port-forward is still alive. Verified end-to-end against the real
hack/cozytest.sh under dash (CI's /bin/sh) in an ubuntu:22.04 container —
the in-subshell trap fires on failure with the PF alive, the tenant snapshot
lands in the same snapshots/<test>/ dir as the host snapshot, the generic
cozy_cleanup still runs, and there is no pipeline deadlock.
Drop the now-dead tenant loop from cozytest.sh's generic trap (it could only
ever see a reaped PF); it keeps the host snapshot.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A single `kubectl port-forward` to the tenant Kamaji API silently dies on
an SPDY/connection drop and never reconnects. The tenant apiserver itself
stays healthy (control-plane pods observed at 0 restarts while localhost
refused connections), so this is a client-side port-forward death, not a
control-plane outage. Once it dies every later tenant kubectl op wedges:
a long `kubectl wait` burns its full timeout against the dead socket
(observed: nfs-test-pod hit its 10m deadline) and the failure-path
`describe`/`get events` print 'connection refused', destroying the
diagnostics too.
Run the port-forward under a respawn loop so a drop self-heals within ~1s;
`kubectl wait` retries across the gap and the in-test failure dumps work
again. Track the supervisor PID in a pidfile so cozy_cleanup() can stop it
on a failed run (the success path kills it directly), and reap the child
port-forward with pkill in both paths.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The endpoint-leak healer evicts the stale endpoint holding a disputed IP
via 'cilium-dbg endpoint disconnect'. But when the IP is held by a
reserved/infra endpoint with no k8s pod identity — e.g. reserved:ingress,
the per-node Cilium Ingress endpoint — that call is rejected with
'endpoint may not be associated reserved labels', so the healer spins
forever and the wedged pod (observed: cozy-backup-controller) never
recovers, failing the run.
Such infra endpoints legitimately own the IP; the duplicate is an IPAM
race that also handed it to a pod. Evicting infra is both impossible and
wrong. Instead reschedule the wedged pod onto a free IP by deleting it
(verified live: the recreated pod lands on a clean address while the
ingress endpoint keeps the disputed one). Also fall back to deleting the
wedged pod if a disconnect of a presumed pod endpoint fails for any
reason, so a pod is never left wedged.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 19a9063ae0)
Harbor's registry BucketClaim wait timed out at 300s on an overloaded
runner even though the underlying Bucket reached bucketReady=true and the
BucketAccess was granted — only the central COSI controller's propagation
of readiness back onto the namespaced BucketClaim lagged (all HelmReleases,
harbor included, were Ready by end-of-run). Raise the wait to 600s to match
the dependent harbor-system HelmRelease budget and remove the flake.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 074501f999)
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined.
run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook.
Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The e2e runner hack/cozytest.sh converts only @test blocks into functions and runs those; it never invokes bats setup_file/teardown_file. So the healer's setup_file deploy silently never executed (confirmed: the Job was absent from the cozyreport while the same manifest deploys and runs fine on a real Talos cozystack cluster). Move the deploy into a real, first @test so cozytest.sh runs it (with visible set -x trace) before the install churn, and assert the Job landed without ever failing the suite (band-aid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubernetes-latest flaked because the worker-readiness wait was split into 8m-to-join + 3m-to-become-Ready. Under sandbox load a slow KubeVirt VM boot consumes most of the join budget, then the tenant cluster's cilium CNI needs several more minutes to make the freshly-joined nodes Ready, overflowing the fixed 3m Ready window even though the CNI converges fine (its HelmRelease reached True shortly after). Replace with one 12m deadline polling for >=2 Ready nodes, robust to wherever the time goes. Same single-deadline shape as the LINSTOR post-install fix (#2928).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconfiguring the Etcd CR triggers a HelmRelease upgrade that renders the EtcdBackupSchedule; hr/etcd and etcdcluster can report ready on the prior revision before the new resource materializes, so the immediate get raced to NotFound. Poll for it (timeout 120s), mirroring the CronJob wait already used later in the same test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The e2e cluster bootstrap let `talosctl gen config` pick its bundled
default Kubernetes version (v1.33.1 for Talos v1.13.0). That version
predates the fix for the kube-controller-manager ValidatingAdmissionPolicy
status type-checker nil-pointer panic on `additionalProperties: true`
schemas (kubernetes/kubernetes#135155, backported to release-1.33 in
#136958, first released in v1.33.10). cozystack-api's aggregated Tenant
schema publishes `additionalProperties: true`, so on the older default
KCM crash-loops cluster-wide during install and fails E2E.
Pin the e2e management cluster to v1.33.12 (latest 1.33 patch) so it runs
a Kubernetes that contains the fix. This complements the schema-side fix
in #2867 and is independent of it.
Refs: #2863
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).
The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.
The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:
- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.
Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).
The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.
Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.
Closes#2764
### Release note
```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.
* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
* Clarified Talos native pod example wording.
* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Replaces the `hack/check-readiness.sh` diagnostic with a Go command at
`cmd/check-readiness`, integrated into the build the same way as
`cozypkg`:
- `make check-readiness` builds `_out/bin/check-readiness`, with
`Version`
injected via `-ldflags`.
- The tool shells out to `kubectl` and reports only **non-ready**
cozystack /
Flux / Kubernetes resources, parsed from status conditions (not column
heuristics). Flags: `--wait`/`--timeout`, `--watch`, `--core`,
`--parallel`,
`--verbose`, `-n`/`-l` scoping, `--no-color`, `--version`.
Adds a black-box golden test suite under `test/check-readiness/`:
- A mock `kubectl` serves canned fixtures; the suite diffs
stdout/stderr/exit
against golden files across 30 scenarios (arg parsing,
condition/phase/suspend
handling, CRD-not-installed, kubectl arg propagation, wait/timeout, edge
cases) plus a PTY test for color output.
- `make test-check-readiness` runs it; wired into `make unit-tests`.
Supersedes #2740, which added the same feature set to the bash script.
### Release note
```release-note
feat(hack): replace check-readiness.sh with a `check-readiness` Go command (build with `make check-readiness`); reports non-ready cozystack/Flux/Kubernetes resources with --wait/--watch/--core/--parallel/--verbose and namespace/label scoping.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added `check-readiness` command for verifying Kubernetes resource
readiness status across namespaces
* Supports multiple execution modes: one-shot, wait with timeout, and
watch with configurable intervals
* Includes namespace filtering, label selectors, parallel resource
fetching, and color output control
* **Tests**
* Added comprehensive test suite validating command behavior across
various scenarios
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2755?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Aleksei Sviridkin
