mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 10:37:21 +09:00
ci/cozystack-api-pin-securitygroup
506
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
a40be5e212 |
fix(e2e): widen tenant-root monitoring bring-up waits to 15m under load (#3160)
## What this PR does The `Configure Tenant and wait for applications` install test brings up the entire root-tenant monitoring stack and waits for each component to become ready/operational. Those waits span the VictoriaMetrics operator resources (`vmalert`/`vmalertmanager`, `vlclusters/generic`, `vmcluster/shortterm`+`longterm`) and the Grafana pair (`grafana-db` CNPG cluster, `grafana-deployment`). Each converges only once its underlying workloads schedule and turn Ready, and during platform bring-up they contend for the same node resources as the rest of the install — so their convergence is load-sensitive. Only `vmalert`/`vmalertmanager` used a 15m budget; the other four used 10m, so this block carried a 10m/15m split even though all its resources contend for the same node capacity during install. Under install-time load (multiple concurrent e2e sandboxes on one runner) the monitoring bring-up is slow across the whole block rather than on a single resource, so the shorter budgets risk failing the whole install on PRs whose branches never touched monitoring — and per-resource bumps are whack-a-mole: raising one budget just moves the risk to the next component in the block. On a calm sandbox each of these reaches ready/operational in under a second, which is what distinguishes the two cases — slow-not-stuck convergence under contention, not a product regression. This lifts the whole tenant-root monitoring wait-set to a single uniform 15m budget and wraps each wait in an on-timeout diagnostic block that dumps the resource YAML (status conditions / `updateStatus`) plus tenant-root pod state. A genuinely stuck resource still fails — now at 15m, and with legible diagnostics instead of a bare `timed out` line — so this does not weaken the gate's ability to catch real breakage. The extra budget costs near-zero wall-time on the happy path and stays well inside the E2E job's 180m cap. The remaining upstream angle is the real driver: how long the VictoriaMetrics/VictoriaLogs stack and the Grafana bring-up take to become operational under resource contention during install. This change addresses the e2e robustness side; reducing that bring-up cost is a separate, product-side improvement. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved and standardized readiness checks during tenant setup so application bring-up waits behave more consistently. * Enhanced failure handling to provide clearer diagnostics (including resource details and pod status) when readiness checks time out. * **Tests** * Standardized longer readiness wait timeouts for multiple application components to reduce flakiness during e2e runs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
cd68d5efdc |
fix(e2e): drain tenant VMs/PVCs between kubernetes tests to fix the node-join flake (#3159)
## What this PR does Fixes the dominant flake in the tenant-Kubernetes e2e suite, where a run fails on worker-node-join under sandbox memory/storage pressure. The two heavy Kamaji tenant tests — `kubernetes-latest` then `kubernetes-previous` — run back-to-back in one CI sandbox, each provisioning a tenant cluster with KubeVirt worker VMs. The inter-test cleanup hook (`cozy_cleanup` in `hack/e2e-apps/run-kubernetes.sh`) deletes the tenant `Kubernetes` CR and waits only for the CR to disappear. That wait finalizes as soon as the CR's finalizers clear, which merely *triggers* teardown of the CAPK worker VMs and their DataVolume-backed disk PVCs — it does not await it. The virt-launcher pods keep their guest RAM reserved until the VMIs are gone, so the next tenant test's worker VMs begin scheduling against a sandbox the previous tenant has not yet vacated. Under the constrained sandbox this starves memory, a worker VM misses the node-join budget, and the MachineHealthCheck cannot remediate when both workers are unhealthy — so the run flakes on worker-node-join. This adds a bounded drain barrier to `cozy_cleanup`: after deleting the CR it now blocks until the tenant's VirtualMachines, VirtualMachineInstances and disk PVCs have actually drained from the `tenant-test` namespace, so the next test starts on a freed sandbox. The wait is bounded (5m) and best-effort: cozytest already runs `cozy_cleanup` wrapped in `|| true`, and the waiter returns loudly (warning plus a leftover dump) on timeout, so a stuck teardown can never hang the job past the deadline — it leaves the sandbox no worse than before. The probe is namespace-scoped: `tenant-test` is provisioned with etcd/monitoring/seaweedfs disabled so it carries no baseline PVCs, and the e2e apps run sequentially each cleaning up after themselves, so the only VMs/VMIs/PVCs present belong to the tenant being torn down. The pure drain exit-condition (`cozy_tenant_drained`) is factored out and unit-tested in `hack/run-kubernetes-drain_test.bats` (runs under `make unit-tests`), including the invariant that a failed probe's `err` sentinel is never misread as a completed drain. ### Screenshots Not applicable — e2e harness change, no UI. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes test cleanup so tenant resources are fully drained before the next test starts, reducing leftover VMs, VM instances, and PVCs. * Added more reliable timeout handling and clearer diagnostics when teardown or node readiness checks fail, making flaky test issues easier to identify. * **Tests** * Added coverage for tenant-drain detection to verify empty, whitespace-only, multi-resource, and error scenarios behave as expected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
09f9d0bdec |
fix(e2e): widen Grafana bring-up waits to 15m under load
The Configure-Tenant install test completes the tenant-root monitoring bring-up by waiting for the grafana-db CNPG cluster and the grafana-deployment Deployment. Both used a 10m budget while the VictoriaMetrics waits above them use 15m. They complete the same monitoring bring-up and contend for the same node resources during install, so under install-time load (concurrent e2e sandboxes on one runner) either can be slow, and the 10m/15m split within the same contend-for-resources bring-up risks failing the whole install on a PR that never touched monitoring. Raise both Grafana waits to the same uniform 15m budget as the vm-operator waits and wrap them in an on-timeout diagnostic block that dumps the resource YAML plus tenant-root pod state, so a genuinely stuck resource still fails at 15m with legible diagnostics instead of a bare "timed out" line. The extra budget costs near-zero wall-time on the happy path and stays inside the E2E job budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1ffbb30d25 |
fix(e2e): widen VictoriaMetrics operational waits to 15m under load
vlclusters/generic and vmcluster wait for updateStatus=operational with a 10m budget, while their vmalert/vmalertmanager sibling in the same block already uses 15m. All three are vm-operator-managed monitoring resources that only become operational once their workloads schedule and turn Ready, and during platform bring-up they contend for node resources with the rest of the install. On a calm sandbox each converges in under a second, but under install-time load (concurrent e2e sandboxes on one runner) monitoring bring-up is slow, and a 10m/15m split inside a block whose resources all contend for the same node capacity risks failing the whole install on a PR that never touched monitoring. Unify vlclusters and vmcluster on the sibling's 15m budget so the whole block shares one headroom, and dump resource YAML plus pod state on timeout across all three waits so a genuine stuck-not-slow regression stays legible instead of a bare "timed out" line. The extra budget costs near-zero wall-time on the happy path and stays inside the E2E job budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9c18260f95 |
fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What The e2e install pipeline still flakes on a loaded runner, failing at a different point each run. Root-caused via repeated runs and a live SSH breakpoint; the platform itself is fine (the components do converge), the failures are slowness + self-inflicted churn under resource pressure. ### Fixes 1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory limit was `110Mi`. The updater watches every pod in the cluster, so on a full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the HelmRelease "early due to stalled resources", cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raised to `512Mi` (request `200Mi`). 2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA `VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA recommender(s) during install and adding pod churn on an already busy cluster. Switched to `Initial`, matching the existing monitoring/etcd `VPA->Initial` fixes. 3. **Install wait budgets** — components reach readiness but overshoot tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`. (The crust-gather failure-snapshot timeout that kept whole jobs hanging to the 3h cancel is already on `main`.) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased several installation and readiness wait times to better handle slower environment startup and reduce premature failures. * Updated vertical autoscaling behavior to avoid unnecessary pod restarts during deployment. * Raised resource limits for the autoscaling updater to improve reliability under load. * **Tests** * Aligned automated checks with the new autoscaling behavior and longer readiness expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
ec42c52967 |
fix(e2e): skip cilium-agent restart for the reserved:ingress IP variant (#3142)
## What this PR does The Cilium leak healer (`hack/e2e-cilium-endpoint-leak-healer.sh`, the interim CI band-aid for the Cilium endpoint-leak class) escalates an unhealed leaked-IP reuse to a Tier-2 `cilium-agent` restart. For one variant that restart is both futile and harmful. When the disputed IP is the node's per-node Cilium ingress IP (`CiliumNode.spec.ingress.ipv4`), it is held by the `reserved:ingress` endpoint — allocated from the node pod CIDR whenever Gateway API / Envoy is enabled (`enable-envoy-config`, `daemon/infraendpoints/infra_ip_allocation.go`). Restarting the agent cannot clear it: the agent re-derives the same ingress IP from the CiliumNode spec on every boot. Worse, the restart can re-trigger the race that caused the wedge in the first place — it rebuilds the host-scope (`kubernetes` IPAM) allocation bitmap empty (`pkg/ipam/hostscope.go`; `RestoreFinished()` is a no-op), then re-reserves the ingress IP via a path decoupled from pod-endpoint restore, so a CNI ADD served in that window can be handed the ingress IP again. That is the infra-IP-vs-pod-IP double-allocation behind the "IP already in use" sandbox rejection. This is an upstream Cilium bug in host-scope IPAM with Gateway API; no released Cilium fixes it as of v1.19.5 (the deployed version). The complete fix belongs upstream — the agent must reserve `spec.ingress.ipv4` in the host-scope allocation bitmap before the CNI server serves allocations, or reconcile it with the restored `reserved:ingress` endpoint, or source it from a dedicated range. That is tracked upstream in cilium/cilium#46799. This PR makes the CI healer correct for that variant: it detects an exact match of the wedged IP against the node's `CiliumNode.spec.ingress.ipv4` and skips the Tier-2 restart — surfacing the case and rescheduling the wedged pod best-effort instead. The match is gated so an empty ingress IP (CiliumNode absent or RBAC denied) never matches, so a false positive can never suppress a restart that would clear a real in-memory leak. The change adds a `ciliumnodes` read to the healer RBAC and a unit test pinning the exact-match-and-non-empty contract. ### Screenshots N/A — CI test-infrastructure change, no UI impact. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced endpoint leak healer to skip Tier-2 agent restarts when the disputed IP exactly matches a node’s reserved per-node Cilium ingress IP. * Added lightweight unit tests for the new IP-matching behavior. * **Bug Fixes** * Prevents unnecessary remediation when the disputed IP is empty or doesn’t exactly match the node ingress IP. * When a match is detected, the healer deletes the affected wedged pod as a best-effort reschedule step. * **Chores** * Updated healer permissions to read Cilium node ingress information cluster-wide. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d54e3e4b58 |
test(e2e): capture worker VM/CSR state on tenant node-join failure
When a tenant Kubernetes cluster hits the 12m node-Ready deadline with fewer than two nodes joined, the failure path dumped only `describe nodes` and `get hr`. That cannot tell apart the two node-join failure sub-modes: a worker VM that never booted (virt-launcher Pending or OOMKilled) versus a VM that booted fine but whose kubelet never registered a Node. The tenant cilium-operator HelmRelease reports "InProgress" identically in both, so the current dump is not enough to diagnose the failure or design a fix. Dump scoped, guarded diagnostics before failing fast: - worker VM/VMI/virt-launcher state on the management cluster, which is what actually splits the two sub-modes; - tenant kubelet CSRs plus the talos-csr-signer sidecar log, which reveal a boot-but-no-Node stall. In-guest Talos/kubelet capture is deliberately left out: talosctl needs a tenant-cluster talosconfig the runner does not have (the tenant workers use a separate Talos CA from the management sandbox), so the gap is documented in the failure path rather than shipped as a call that would fail mTLS and capture nothing. Every capture is guarded with `|| true` so a capture failure never masks the real exit. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei KvapilandClaude
|
433203c500 |
test(e2e): widen install waits for a loaded runner
The e2e install still flakes on a busy runner where components reach readiness but overshoot tight wait budgets: - keycloak HR chain (keycloak-db CNPG -> keycloak StatefulSet -> keycloak-configure) reaches Ready ~15-18m in: 10m -> 20m. - tenant aggregate wait (hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root): the tenant stack only starts creating pods ~9-10m in and then needs several more minutes: 10m -> 20m. - per-app tenant readiness waits (ingress, etcd, VictoriaLogs/Metrics clusters, grafana-db, grafana): 5m -> 10m. Diagnosed via a live SSH breakpoint: the HRs all do reach Ready, just outside the old budgets. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
fc862528fd |
fix(e2e): drain tenant VMs/PVCs between kubernetes tests
The inter-test cleanup hook deletes the tenant Kubernetes CR and waits only for the CR to disappear. That wait finalizes as soon as the CR's finalizers clear, which merely TRIGGERS teardown of the CAPK worker VMs and their DataVolume-backed disk PVCs -- it does not await it. The virt-launcher pods keep their guest RAM reserved until the VMIs are gone, so the next back-to-back tenant test's worker VMs begin scheduling against a sandbox the previous tenant has not yet vacated. Under the constrained CI sandbox this starves memory, a worker VM misses the node-join budget, and the run flakes on worker-node-join (the MachineHealthCheck cannot remediate when both workers are unhealthy). Block cozy_cleanup until the tenant VMs, VMIs and disk PVCs have actually drained from tenant-test before it returns, so the next test starts on a freed sandbox. The wait is bounded (5m) and best-effort: cozytest runs cozy_cleanup wrapped in `|| true` and the waiter returns loudly on timeout, so a stuck teardown can never hang the job longer than the deadline. The probe is namespace-scoped -- tenant-test is provisioned with etcd/monitoring/seaweedfs disabled so it carries no baseline PVCs, and e2e apps run sequentially each cleaning up after themselves, so the only VMs/VMIs/PVCs present belong to the tenant being torn down. The pure drain exit-condition (cozy_tenant_drained) is unit-tested in hack/run-kubernetes-drain_test.bats, including the invariant that a failed probe's "err" sentinel is never misread as a completed drain. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
20e4eab281 |
test: make negative bats assertions fail under cozytest (#3132)
## What this PR does
Every `*.bats` file in the repo runs through `hack/cozytest.sh`, not
real bats — the e2e files via `packages/core/testing/Makefile` and the
`hack/*.bats` unit tests via `make bats-unit-tests`. cozytest runs each
`@test` under `set -e`, and POSIX errexit is suppressed for a pipeline
beginning with `!`. So a bare `! cmd` must-not-happen assertion never
fails the test: when the forbidden condition actually occurs (the
command succeeds), the negated pipeline returns non-zero but errexit
ignores it and the test passes anyway. Ten such assertions across six
files were silently vacuous.
This converts each to `if cmd; then echo "<reason>"; false; fi`. The
`if` condition is exempt from errexit (so the probe itself never aborts
the test), and the `false` in the taken branch is a plain command under
`set -e`, so it aborts and fails the test with a legible message when
the forbidden condition holds. This is the must-not-happen inverse of
the `cmd || { echo …; false; }` pattern already used in the suite (e.g.
harbor.bats, per docs/agents/e2e-testing.md). Original redirections and
quoting are preserved, so happy-path behavior is unchanged — only the
failure path now actually fails.
Converted assertions:
- `bucket`: a read-only user must not be able to upload (access-control
regression guard)
- `gateway`: a child tenant must not have its own gateway HelmRelease
(inheritance)
- `kuberture`: external-dns public/internal hostname isolation
- `e2e-install-cozystack`: tenant-name validation must reject rather
than warn; the version ConfigMap must be gone after delete
- `build-matrix` (unit test): talos/installer must be excluded from the
parallel matrix
- `capture-dataplane` (unit test): kube-dns and pending rows must be
filtered out
A separate in-flight etcd test change owns two more such assertions in
`etcd.bats`; they are left out of this PR to avoid a conflicting edit.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Strengthened several end-to-end/Bats assertions to fail more
explicitly when expected cluster conditions aren’t met.
* Improved matrix validation to reliably confirm excluded components are
absent.
* Updated dataplane load balancer filtering, bucket readonly upload, and
gateway inheritance checks to avoid false/ambiguous pass cases.
* Tightened DNS hostname isolation probes and cozystack installer
validations (tenant name dashes, deletion protection) with clearer
“expected vs found” failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
12fe3256fd |
test: make negative bats assertions fail under cozytest
These bats files are all run by hack/cozytest.sh, which executes each @test under `set -e`. POSIX errexit is suppressed for a pipeline that begins with `!`, so a bare `! cmd` must-not-happen assertion never fails the test: when the forbidden condition actually occurs (cmd succeeds) the negated pipeline returns non-zero but errexit ignores it and the test passes anyway. These assertions were silently vacuous. Replace each with `if cmd; then echo "<reason>"; false; fi`, the idiom already used and documented elsewhere in these files — the if-condition is exempt from errexit and the `false` in the taken branch fails the test loudly. The forbidden condition now fails the test as intended: - bucket: read-only user must not be able to upload (access control) - gateway: child tenant must not have its own gateway HelmRelease - kuberture: external-dns public/internal hostname isolation - e2e-install: tenant-name validation must reject (not warn); the version ConfigMap must be gone after delete - build-matrix: talos/installer excluded from the parallel matrix - capture-dataplane: kube-dns and pending rows are filtered out Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bf07b8c42b |
test: fix dead bats setup/teardown hooks under cozytest
cozytest.sh — the runner for every bats file here — is not real bats: it never invokes setup()/teardown(). So these hooks were dead. Cleanup ran only via each test's success-path inline delete, and a failing test leaked its resources into the shared tenant-test namespace (and, for kuberture, cluster-scoped RBAC that persists across sandbox re-runs). Wire the cleanup to cozy_cleanup(), the hook cozytest does run at suite exit and on the first failing test: - external-dns: dead teardown() -> cozy_cleanup() deleting all ExternalDNS CRs. ExternalDNS is a stateless Deployment with the inmemory provider, so a plain delete fully reclaims it — no drain needed. - kuberture: dead teardown() -> cozy_cleanup() routed through the existing _cleanup(), so the probe Pods, SA, and cluster-scoped ClusterRole and ClusterRoleBinding are reclaimed on the failure path too. build-matrix_test: drop the dead setup() entirely — it only cd'd to the repo root, which cozytest already runs from, so the relative hack/build-matrix.sh calls resolve without it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4e53f5e448 |
fix(e2e): skip cilium-agent restart for the reserved:ingress IP variant
The cilium leak healer escalates an unhealed leaked-IP reuse to a Tier-2 cilium-agent restart. For one variant that restart is both futile and harmful: when the disputed IP is the node's per-node Cilium ingress IP (CiliumNode.spec.ingress.ipv4), held by the reserved:ingress endpoint (allocated from the node pod CIDR whenever Gateway API / Envoy is on), restarting the agent cannot clear it -- the agent re-derives the same ingress IP from the CiliumNode spec on every boot -- and can re-trigger the very race that caused the wedge: the restart rebuilds the host-scope (kubernetes) IPAM bitmap empty, then re-reserves the ingress IP via a path decoupled from pod-endpoint restore, so a CNI ADD in that window can be handed the ingress IP again. This is an upstream cilium bug in host-scope IPAM with Gateway API; no released cilium fixes it as of v1.19.5. Detect that variant by an exact match of the wedged IP against the node's CiliumNode.spec.ingress.ipv4 and skip the Tier-2 restart -- surface the case and reschedule the wedged pod best-effort instead. The match is gated so an empty ingress IP (CiliumNode absent or RBAC denied) never matches, so a false positive can never suppress a restart that would clear a real in-memory leak. Add a CiliumNode read to the healer RBAC and a unit test pinning the exact-match-and-non-empty contract. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
3333b325a8 |
fix(ci): preserve a PR's own edits to non-rebuilt package image refs
The current-main overlay skipped only rebuilt units (plus talos/installer), so for a package the PR edited but did not rebuild — an upstream image bump in a non-build-unit package such as keycloak or cert-manager — the committed ref differed from the main artifact on image-reference lines only, the drift check passed, and the overlay reverted the PR's edit back to current-main. e2e and the installer artifact then ran against the stale image. Pass the PR's edited package dirs (derived from the plan job's git diff) into the overlay skip set so a PR's own image-ref edits win over the artifact. Add tests pinning the preserve behaviour and covering the --image= arg form and the file-absent-in-PR-tree branch. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
52f7d05786 |
ci: overlay current-main images for packages a PR didn't rebuild
Conditional PR builds (the per-package matrix in pull-requests.yaml) only rebuild packages whose dir changed. Every other package falls back to the image refs committed in the tree — the last release's digests (currently v1.5.0), which go stale as main advances. So e2e and the installer's packages artifact ran current-main configs against last-release images for everything outside the PR's build matrix, and a PR installed via the cozy-installer chart was an incoherent mix. This is the systemic gap in issue #3143 (its concrete bite: the merged objectstorage-controller requeue fix #3034 never ran in e2e, flaking the harbor BucketClaim check). In finalize, pull the cozystack-packages:main OCI artifact that build-main already publishes (the whole current-main packages tree, refs digest-pinned to the images build-main pushed) and walk every ref-bearing file in it (values.yaml + images/*.tag), overlaying the repo's copy. This is driven by the artifact, not the build matrix, so it covers the ENTIRE first-party tree — apps/system/core AND extra/, library/ — and a merged fix to any first-party image (e.g. the objectstorage-sidecar ref under extra/seaweedfs) takes effect. For a file build-main did not rebuild the artifact copy is byte-identical and is skipped. Skipped explicitly: units the PR itself rebuilt (their pr-<N>-<sha> refs win), packages/core/talos and packages/core/installer (owned by their dedicated build-talos / finalize-installer jobs), and vendored charts/ subtrees. The copy is self-validating: a file is overlaid only when every differing line is image-reference-bearing (a full @sha256: ref, or a split image/repository/registry/tag/digest key, or a --…-image= arg); otherwise it keeps its committed ref. A missing artifact is a no-op (prior behaviour) and the step is best-effort so it never blocks a PR. No build-main change is needed: cozystack-packages:main is already published by `make build`. Runs before the installer build so the flux-pushed packages artifact and the pr.patch e2e applies both reflect the overlay. Result: a PR is tested and installed as current-main + its own changes, coherent across config and image, instead of current-main configs running on last-release images. Covered by hack/overlay-main-images_test.bats (run by `make unit-tests` in the checks job): single-line and split-form ref overlay, extra/* coverage, skip-rebuilt, talos/installer skip, vendored-charts prune, drift self-validation, and missing-artifact no-op. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
0892c5a38c |
fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does After deleting a `Qdrant` app CR, the StatefulSet data PVC (`qdrant-storage-<release>-0`) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default. The vendored upstream Qdrant chart does **not** support `persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml` never renders that field), so the OpenBAO-style value passthrough is not possible here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the MariaDB hook) that deletes the release's PVCs by label. The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by every Qdrant in a namespace), so to select PVCs safely per release this also passes `persistence.additionalLabels.app.kubernetes.io/instance: <release>` through the HelmRelease — giving the data PVC a release-unique label the hook selects on. The cleanup Job is hardened: runs as non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped to `persistentvolumeclaims`. A real cleanup failure is logged (not silently swallowed) but never blocks namespace teardown. > [!WARNING] > **Deleting a Qdrant app now permanently removes its data PVC and the vector data it holds.** > This is intentional and fixes the storage leak; the hook only fires on full > release deletion (not on scale-down). Back up before deleting. Fixes #3059. ### Release note ```release-note fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a post-delete cleanup hook for Qdrant to remove leftover StatefulSet data PVCs after deletion. * Included hardened Kubernetes settings for the cleanup job to improve runtime safety. * **Tests** * Added Helm chart tests that validate the post-delete cleanup job and its RBAC permissions. * Enhanced end-to-end verification to confirm the data PVC is fully removed after deleting a Qdrant instance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
bf14a73b39 |
feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922)
## What this PR does
Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.
The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.
The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.
Implements the attachment model from #1614.
### Release note
```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
* Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Алексей Артамонов
|
ae7c232409 |
fix(qdrant): anchor PVC ordinal so cleanup never deletes a sibling's data
The post-delete hook selected PVCs with an unanchored name prefix (qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting release "qdrant-test" matched not only its own PVC (qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC (qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting the sibling's data (or wedging it in Terminating via pvc-protection). Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own data PVCs match. A release name that is a prefix of a sibling no longer collides. - cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$". - tests: assert the anchored selector and add a notMatchRegex guard against regressing to the unanchored prefix. - e2e: gate the post-delete wait on a NotFound result specifically, so a transient API/auth error is not misread as "PVC reclaimed". Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Aleksei Sviridkin
|
665e7e4609 |
test(e2e): make etcd.bats teardown run and settle under cozytest (#3131)
## What this PR does The e2e runner `hack/cozytest.sh` is not real bats — it converts `@test` blocks to shell functions and recognises a `cozy_cleanup()` hook, but never calls `setup()`/`teardown()` and never opens fd 3. So `etcd.bats`'s `setup()` never ran (etcd leaked its HelmRelease and data PVCs into the shared `tenant-test` namespace) and `dump_diagnostics` wrote to an unopened fd 3, silently dropping failure diagnostics. The teardown's `kubectl wait --for=delete` lines also passed `--ignore-not-found`, which `wait` does not accept. This moves teardown into an `etcd_drain()` helper, run three ways: a pre-clean at the start of the first scenario, inline at the end of the last scenario (under `set -e`, so a teardown that does not settle fails the suite), and as `cozy_cleanup` (the best-effort safety net cozytest runs at suite exit and on the first failing test). The drain reclaims what a plain CR/HR delete leaves behind: - the StatefulSet's retained `data-etcd-*` PVCs (not owned by the Etcd CR) — a later install otherwise reuses the same-named volumes, boots the previous member identity, never forms a fresh quorum, and the chart's `helm --wait` fails; and - the cluster-scoped Kamaji `DataStore`, whose leaked finalizer wedges the HR uninstall — cleared each pass until the Etcd CR, HR, DataStore, and PVCs are all gone, bounded by a 4-minute deadline that fails the drain on timeout. Remnants are probed by name so a transient API error is never mistaken for a successful delete. Diagnostics now go to stdout, which cozytest captures and prints on failure. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved end-to-end cleanup so leftover resources are removed more reliably between runs. * Made failure diagnostics easier to capture when tests fail. * Added stronger cleanup handling at the start and end of the test flow to reduce flaky suite behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
31c15b282f |
feat(network): mechanism-agnostic LoadBalancer exposure (ExposureClass + ServiceExposure) (#3081)
## What this PR does Lays the foundation for migrating tenant LoadBalancer exposure off the deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off MetalLB onto Cilium-native announcement — the mechanism-agnostic shape prescribed when the earlier chart-side approach (#2468) was reverted in #2603. It adds a new `network.cozystack.io/v1alpha1` API group and a controller that translates intent into whichever LoadBalancer backend the cluster admin configured: - `ExposureClass` (cluster-scoped, admin): binds a logical class name to a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP address scope — the StorageClass/IngressClass pattern. - `ServiceExposure` (namespaced, tenant intent): "expose this Service via class X"; status reports the resolved backend and assigned IPs. - A `serviceexposure` controller renders the backend's pool + announcer once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium `CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to the namespaces that opted in. Backends are pluggable behind a small Go interface, so adding BGP or another cloud LB is one implementation, not an API change. The controller never mutates the target Service — the chart still owns its shape. Platform wiring: `publishing.exposureClass.*` renders the admin `ExposureClass` and the host ingress publishes through it. The default (no class set, `externalIPs`) leaves every existing cluster's behaviour byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes a single `ExposureClass.spec.backend` change. This is the foundation only — Part of #2586. Converting per-app exposure, flipping Cilium `l2announcements` on, changing the default, and removing MetalLB are deferred to later phases. ### Release note ```release-note feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added new `ExposureClass` and `ServiceExposure` custom resources with support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB), including optional L2 and address configuration. * Added Helm wiring and templates to enable service/ingress publishing via `ServiceExposure`, selecting a configured (or default) `ExposureClass`. * **Bug Fixes** * Improved reconciliation, finalizer-based cleanup, and status updates for readiness and assigned IP reporting, including safer handling when backends or managed resources change. * **Tests** * Added end-to-end and unit test coverage for backend rendering, readiness, and cleanup flows. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
4c5a8c830c |
test(platform): cover migration 47 fail-closed contract
Migration 47's safety property is that any per-object read or patch failure aborts the migration (exit 1) before the cozystack-version stamp advances, so the pre-upgrade hook fails and Helm never rolls the new kubernetes chart against an un-pinned cluster (whose live md0 MachineDeployment would otherwise be pruned). That branch was verified by reading only. Add hack/kubernetes-md0-migration.bats, mirroring the kubectl-mock style of the plain-shell cozytest.sh runner: a fake kubectl on PATH simulates a non-NotFound read failure and a patch failure, and the tests assert the migration exits non-zero and never reaches the version stamp (no `kubectl apply` of cozystack-version is logged). A positive-control case asserts a clean run does reach the stamp, so the "no stamp" assertions are meaningful rather than vacuous. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
mattia-eleuteri
|
7e3c5f99db |
fix(kubernetes): apply default md0 node group only when nodeGroups is empty
The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
|
||
|
Алексей Артамонов
|
2587458bd4 |
fix(qdrant)!: reclaim data PVC by name prefix, not volumeClaimTemplate label
The previous approach routed a release-unique label through qdrant.persistence.additionalLabels, which the upstream chart renders into the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is immutable on an existing StatefulSet, so for any already-deployed Qdrant the API server would reject the upgrade and the <release>-system HelmRelease (retries: -1) would retry-fail forever — permanently wedging reconciliation. Label changes on a volumeClaimTemplate also never apply to PVCs that already exist, so the cleanup hook could not reclaim those instances anyway. Switch the post-delete hook to select PVCs by the deterministic StatefulSet naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely from the release name, so this needs no StatefulSet change (upgrade-safe) and also reclaims PVCs from instances created before the hook existed. - qdrant.yaml: drop the additionalLabels passthrough entirely. - hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep. - tests: assert the name-prefix command instead of the label passthrough. - e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited). BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC and the vector data it holds. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Aleksei Sviridkin
|
fdbb9cc55a |
test(e2e): make etcd.bats teardown run and settle under cozytest
The e2e runner (hack/cozytest.sh) is not real bats: it never invokes setup()/teardown() and never opens fd 3. So etcd.bats's setup() never ran and dump_diagnostics wrote to an unopened fd 3, silently aborting on failure — and etcd leaked into the shared tenant-test namespace. Move teardown into etcd_drain() and run it inline at the end of the last scenario (under set -e, so a teardown that does not settle fails the suite) and as cozy_cleanup (the best-effort hook cozytest runs at suite exit and on the first failing test). The first scenario also drains up front so a dirty namespace cannot taint it. etcd_drain reclaims what a plain CR/HR delete leaves behind: - the StatefulSet's retained data-etcd-* PVCs (not owned by the Etcd CR); a later install otherwise reuses them, boots the old member identity, never forms a fresh quorum, and the chart's helm --wait fails; and - the cluster-scoped Kamaji DataStore, whose leaked finalizer wedges the HR uninstall — cleared each pass until the Etcd CR, HR, DataStore, and PVCs are all gone, bounded by a deadline that fails the drain on timeout. Remnants are probed by name so a transient API error is never mistaken for a successful delete. Drop --ignore-not-found from the teardown waits (not a valid wait flag); the drain polls for existence instead. Diagnostics now go to stdout, which cozytest captures and prints on failure. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
511f2a3c1c |
test(e2e): smoke-test the ServiceExposure controller
Add a deterministic e2e for the externalIPs backend: it creates an ExposureClass, a Service with pinned externalIPs, and a ServiceExposure, then asserts the controller resolves the class, renders no MetalLB/Cilium pool, reports the assigned IPs, and cleans up via its finalizer on delete. The externalIPs path is chosen so the test creates no backend pool — it cannot collide with the cluster's existing LoadBalancer address pools or depend on real L2 reachability. The metallb and cilium rendering paths are covered by the Go unit tests. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
aec16212ee |
refactor(build): parallel matrix image builds on ephemeral runners (#2983)
## What this PR does Completes #2937. Replaces the single serial `Build` job — one ephemeral VM running `make build` (~26 images one after another) that wedged on the shared embedded buildkit under concurrent multi-PR load — with a parallel, diff-scoped matrix. **New job graph (regular PR path):** ``` plan changed files -> hack/build-matrix.sh -> scoped matrix + docs gate checks make unit-tests + test-controllers (gates finalize -> e2e) build one package per job, own docker-container buildx, uploads a per-package digest patch fragment [parallel] build-talos talos image + nocloud disk (its own leg; e2e always needs the disk) finalize download+apply all fragments, then build the installer LAST (its flux-push bundles the whole digest-patched tree), emit the unified pr.patch e2e consumes talos-image + pr.patch unchanged ``` **Why finalize is special:** `packages/core/installer`'s `image` target runs `flux push artifact --path=packages`, bundling the *entire* packages tree into the OCI artifact the operator pulls — so it must run after every other package's digest edit is applied, never as a peer matrix job. `talos` is likewise pulled out of the matrix (its image and nocloud disk share heavy `_out/assets` and build once). **Gating:** `finalize` (and therefore `e2e`, which needs it) depends on `checks`, so a red unit/controller test skips the expensive install+e2e path instead of running it for ~1h — the gating the old in-build-job tests gave, without re-coupling the cheap parallel builds. **Change-scoping:** `hack/build-matrix.sh` parses the root Makefile `build:` target (single source of truth) and emits only the packages a PR touches — most PRs build ~0–1 images — escalating to the full set on a shared-dependency change (`cozy-lib`, the build macros, `go.mod`, the build workflows). It fans `seaweedfs` out to `objectstorage-controller` (cross-package sidecar-tag dependency). 16 bats cases cover the selector. **SBOM:** `SBOM=1` wires buildx `--sbom` through `common-envs.mk` (off by default). It stays off in the matrix until an OCIR referrers/attestation probe confirms the registry accepts attestations. **Decommission:** the cache warmer (`build-main.yaml`) moves to the ephemeral pool, so no build path depends on the dedicated self-hosted runner. The runner remains referenced only by the e2e `debug` breakpoint path; fully de-registering it is a separate infra step. ### Validation - Local: `hack/build-matrix_test.bats` 16/16; `actionlint` clean (custom runner labels registered in `.github/actionlint.yaml`); `make -n` confirms `--sbom` is off by default and on with `SBOM=1`. - CI: this PR edits `pull-requests.yaml`, so `build-matrix.sh` forces a full matrix — the PR exercises the new pipeline on itself. First run was green through `plan → 25 parallel builds → build-talos → finalize → install → 21/22 app tests`; the only red was the pre-existing `bucket`/BucketAccess flake (install and the other 21 suites passed on the matrix-built tree), which validates the decomposition end-to-end. ### Follow-ups (tracked on #2937) - Probe OCIR referrers/attestation support; enable `SBOM=1` in the matrix if supported. - Infra: de-register the self-hosted build runner (the e2e `debug` breakpoint path is its only remaining user). ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Refined CI to a staged, artifact-based pipeline with gated checks, per-package parallel builds, and centralized patch assembly before building final images * Improved ephemeral runner performance and reliability (runner selection, longer timeout, build toolchain setup) * Added build-matrix auto-selection with targeted fan-out rules and “full rebuild” triggers * Updated CI lint configuration to recognize ephemeral runner labels * Introduced an optional SBOM toggle for build artifacts * **Documentation** * Updated e2e testing docs to better explain Test-Impact Analysis scope and gating semantics * **Tests** * Added Bats tests validating build-matrix selection and edge cases <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
da084ef51c |
test(kubernetes): cover fallback-default StorageClass selection (#2872 B1)
Pin the PR #2872 B1 fix: when the management cluster exposes only remote LINSTOR StorageClasses whose names sort after "kubevirt" and none is named the configured storageClass (default "replicated"), the tenant default must be chosen among the propagated classes and never the legacy "kubevirt" alias. helm-unittest cannot reach this branch (no live cluster -> lookup returns empty -> the storageClasses map collapses to the "replicated" fallback), so it is exercised in e2e via a single server-side dry-run render of the kubernetes chart against the live management cluster: two remote LINSTOR classes are added and "replicated" is removed for the one render, then the rendered -csi HelmRelease storageClasses map is asserted to carry exactly one default among the propagated classes, never the "kubevirt" alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
6a1b56f2b5 |
test(kubernetes): cover LINSTOR StorageClass propagation
- csi_test.yaml: assert the fallback storageClasses map yields a default "replicated" plus the non-default "kubevirt" alias, and that the alias tracks the storageClass selector. - kubevirt-csi-node: add a Makefile test target (otherwise the package is silently skipped by the unit-test runner) and storageclasses_test covering per-entry rendering, the single-default invariant, and the absence of any enforcement allowList in driver-config. - run-kubernetes.sh: verify in e2e that "replicated" propagates as the default kubevirt-CSI class, the "kubevirt" alias is retained, and the node-local "local" infra class is filtered out. Refs: #2094 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
0b9f5a1005 |
Merge remote-tracking branch 'origin/main' into refactor/build-matrix-2937
Resolve the .github/workflows/pull-requests.yaml conflict: keep this branch's per-package build matrix and carry main's `git diff --binary` patch fix (#e566992) onto every fragment and pr.patch capture, so provider bundles that package image targets regenerate (components*.gz) survive fragment reassembly in finalize and reach the patch e2e applies. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
1df8c7c557 |
feat(kubernetes): bootstrap tenant workers with Talos via CABPT (#2931)
## What this PR does > **Supersedes #2610.** Same work, rebased cleanly onto current `main` so it now sits on top of #2867 (the `kube-controller-manager` VAP type-checker fix). That fix is what was killing the base install in #2610's E2E *before* the `kubernetes` app test ever ran, leaving the Talos worker-bootstrap rewrite verified only by code reading. #2610's 67-commit branch is squashed here into 8 logical commits with original authorship preserved (@kvaps and @IvanHunters as authors); the tree is **byte-identical** to #2610's head — no code changes, just the rebase onto `main` (to pick up #2867) plus history cleanup. The detailed review history lives on #2610. Phase 1 of the Kubernetes-app split design (see cozystack/community#8): replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by `cluster-api-bootstrap-provider-talos` (CABPT) and a `clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane pod. Existing tenants keep working — old machines roll out and get replaced by Talos workers without manual intervention. Highlights: - Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the existing kubeadm one in `capi-providers-bootstrap`. - Bump the Kamaji control-plane provider with an upstream-bound patch that exposes `KamajiControlPlane.spec.network.additionalServicePorts`, used to publish trustd (50001/TCP) on the apiserver service. - Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and stable random Talos secrets (`token`, `clusterId`, `clusterSecret`, `bootstrapToken`) using the helm lookup-and-reuse pattern. - Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the tenant `kube-system` via a Helm post-install/upgrade Job. - Render a `TalosConfigTemplate` (worker machineconfig, `generateType: none`) and switch the `MachineDeployment.bootstrap` reference from `KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets and the Kamaji apiserver service being ready. - Boot workers from the Talos openstack image via a CDI `DataVolume` (`source.http.url` → `factory.talos.dev`), expose the system disk as virtio-blk with `blockSize.custom: logical=512, physical=4096` so 4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk (Talos lays out EPHEMERAL itself). - Pin pod/service CIDRs in the worker machineconfig to the tenant ranges (`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap diagnostic against the host pod CIDR. - Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the Kamaji apiserver Service ClusterIP (looked up at render time), so bootstrap survives the chicken-and-egg moment where tenant DNS does not exist yet and the host CoreDNS does not serve the tenant zone. End-to-end on a dev cluster: a fresh tenant brings up a Talos worker, kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving by the talos-csr-signer sidecar), Cilium initialises with hostNetwork against the apiserver ClusterIP, CoreDNS comes up, and the node transitions to `Ready`. ### Release note ```release-note feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes **New Features** * Added Talos worker configuration for `talos.version` and `talos.schematicID` * Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`) for worker remediation tuning * Added image overrides for `kubectl` and `talosCsrSigner` * Exposed additional tenant control-plane Service ports support **Breaking Changes** * Removed Kubernetes v1.30; supported versions start at v1.31 * Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with consolidated Talos system-disk semantics **Removals** * Removed Ubuntu container-disk images and related containerd patch mechanisms * Air-gapped tenant workers temporarily unsupported during the Talos transition **Documentation** * Updated chart docs, parameters, and GitOps upgrade guidance for version bumps <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
d978305e33 |
Merge origin/main into refactor/build-matrix-2937
Resolve conflict in .github/workflows/pull-requests.yaml: keep the new per-package build matrix and fold in main's docker.io->mirror.gcr.io base-image routing (#3042). Apply buildkitd-config: hack/buildkitd.toml to all three new build legs (build, build-talos, finalize) so the mirror fix covers the whole decomposed pipeline, not just the conflicted leg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
f933918b3f | Merge branch 'main' into ray/build-dockerhub-mirror | ||
|
Aleksei Sviridkin
|
5f9ac28418 |
test(e2e): capture host→pod CNI data-plane on failure (#3047)
## What
On E2E test failure, capture the host→pod CNI data-plane state for each
NotReady pod (scheduled, with a podIP, not terminal) from that pod's
node, into the crust-gather snapshot dir so it ships in the uploaded
report. New helper `hack/e2e-capture-dataplane.sh`, invoked from the
shared on-failure hook in `hack/cozytest.sh` right after the existing
crust-gather collect.
Per affected pod, from the pod's node:
- **host netns** (via the kube-ovn `cni-server`, hostNetwork): `ip route
get <podIP>`, `ip neigh`, `ip rule`, `ip addr show ovn0`, and the kernel
conntrack entries for the podIP (`conntrack -L`, falling back to
`/proc/net/nf_conntrack`).
- **cilium-agent**: `endpoint list`, `bpf ct list global` filtered to
the podIP, a bounded `monitor --type drop`, and `hubble observe
--verdict DROPPED` when the CLI is present.
- **OVS/OVN**: `ovs-ofctl dump-flows br-int`, the OVN `Port_Binding` /
`Logical_Switch_Port` for the pod, and a bounded `ovn-sbctl lflow-list`.
## Why
The install intermittently fails on a host→local-pod data-plane
transient: kubelet on a node hits a local pod's readiness/startup probe
with `connection refused` for several minutes while overlay pod→pod
traffic works, then it self-heals. This stems from the cilium+kube-ovn
chaining config (forced `enable-host-legacy-routing`, CNI
`InstallEndpointRoute:false`), which delegates host→local-pod routing to
kube-ovn/ovn0. Crucially, under `enable-host-legacy-routing` the
host→local-pod path traverses the kernel netfilter stack, not cilium BPF
— so the authoritative evidence is the kernel conntrack table plus the
host route/rule/neighbor state, captured here from the node's host
netns.
The crust-gather snapshot records Kubernetes object state but not the
node's L3 forwarding state, so the mechanism cannot be root-caused after
the fact. This adds exactly that evidence so the next recurrence is
dispositive.
## Diagnostic-only
This changes nothing about install or test behavior — it only collects
more evidence on an already-failed test. It runs after the existing
crust-gather collect and only when a test has already failed, never
mutates the cluster (read-only `get`/`exec`), and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
`|| true`, the whole run is wrapped in a `timeout -k 30 600` wall-clock
backstop, node-global captures are deduped per node, and the pod count
is capped (`MAX_PODS=12`, env-overridable). It no-ops cleanly when no
pods are affected or when a tool/pod is absent. No new retries, no
`EXIT`/`RETURN` traps — consistent with `docs/agents/e2e-testing.md`.
## Validation
`sh -n`, `bash -n`, and `shellcheck` are clean on both touched files.
The affected-pod selection filter and the no-cluster no-op path were
smoke-tested. Not exercised against a live cluster — container/label
names and the kube-ovn LSP naming (`<pod>.<ns>`) are taken from the
`packages/system/{cilium,kubeovn}` charts and the existing
`hack/e2e-cilium-endpoint-leak-healer.sh`; all captures are defensively
`|| true` and time-boxed so absent tooling degrades to a labeled note
rather than a failure.
|
||
|
IvanHunters
|
41aed829e6 |
test(e2e): raise kafka/zookeeper preset off the 128Mi OOM threshold (#3052)
## What Raise the `kafka` and `zookeeper` components in the Kafka e2e test from the `nano` preset (250m / 128Mi) to `c1.small` (1 CPU / 1Gi), the chart default. ## Why 128Mi cannot hold the Strimzi Kafka 3.9.1 and ZooKeeper JVMs. The brokers OOMKill on startup (exit 137) and enter CrashLoopBackOff, so the pods never reach Ready, the `Kafka` CR and its HelmRelease never go Ready, and the 5m readiness wait times out. It presents as a flake rather than a hard failure because 128Mi sits right at the JVM-start threshold — occasionally a broker becomes Ready before its RSS crosses the cgroup limit. `c1.small` (1Gi) comfortably fits both JVMs and matches the chart default, so the test exercises the documented default sizing. The e2e sandbox VMs are 8 CPU / 24Gi, so 1Gi per pod (4Gi total) across the four pods (kafka-0/1, zookeeper-0/1) is well within budget. ## Scope Test-only; values changed, test structure untouched. MariaDB's e2e test also uses `nano` but was deliberately left as-is: it runs MariaDB (C) + a Go exporter with no JVM, its chart default is already `t1.nano` (128Mi), and it shares the `nano` preset with the passing postgres/redis/qdrant tests — not the same OOM class. |
||
|
IvanHunters
|
42eafd4381 |
test(e2e): escalate cilium leak healer to agent restart on leaked-IP reuse (#3051)
## What Add a bounded escalation tier to the E2E cilium endpoint-leak healer: when a pod stays wedged on `FailedCreatePodSandBox` "IP ipv4:<X> is already in use" for the same IP across several heal cycles, restart the cilium-agent on that node. Keep the existing per-endpoint remedies as the first tier. ## Why The leak (cilium/cilium#38313 class) sometimes lives purely in the cilium-agent's in-memory endpoint state. The existing remedies — `cilium-dbg endpoint disconnect` and delete-rescheduling the wedged pod — cannot reach it: IPAM keeps re-handing the SAME released IP, so the recreated pod re-wedges on it, and on some variants the stale entry is not even in the queryable endpoint registry, so disconnect is a no-op. Both are inescapable loops that fail the install. The only thing that clears an in-memory leak is restarting the agent, which drops the node's endpoint registry and rebuilds it from CRDs/IPAM without the stale entry. ## How - Track recurrence per `(node, ip)`. After `ESCALATE_AFTER` (default 3) heal cycles of a futile first-tier remedy on the same IP, restart that node's cilium-agent pod. - Bounded by `MAX_ESCALATIONS_PER_NODE` (default 2); after the cap, log loudly and surface the failure instead of restarting again. State is per-node and resets after each restart so one restart covers every leaked IP on the node. - A genuine duplicate-IP (endpoint backs a live owner) is refused before counting, so an agent restart can never mask a real product bug. - RBAC unchanged: the existing cluster-wide `pods` `get/list/delete` already covers deleting a cilium-agent pod in `cozy-cilium`. E2E-test-only resilience; no product code change. Remove alongside the rest of the healer once a fixed Cilium ships. |
||
|
IvanHunters
|
e0597e3d99 |
fix(platform): make *-rd sources depend on cozystack-engine (#3041)
## Problem On an unlucky Flux reconcile order, a `*-application` PackageSource that ships a `system/*-rd` component can install before the `applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD is installed by the `application-definition-crd` component of `cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first, it hard-fails with `no matches for kind "ApplicationDefinition" in version "cozystack.io/v1alpha1"`, and that cascades into a broad install timeout — surfacing even on changes that have nothing to do with ordering, because it is a latent race. ## Fix Add `- cozystack.cozystack-engine` to the `dependsOn` of every `*-application` source that ships a `system/*-rd` component (29 sources). `gateway-application` already carried this edge — it was added earlier for the TenantGateway CRD, the same bug class, but only for that one source. A new invariant test fails if any source with a `system/*-rd` component is missing the edge. ## Why The ordering was previously implicit, relying on Flux happening to reconcile `cozystack-engine` before the `*-application` sources. Making the edge explicit removes the race deterministically. No dependency cycle is introduced: `cozystack-engine` depends only on `networking` and `cert-manager` (plus keycloak in its oidc variant), none of which reach back to any `*-application`. ## Test - New generic guard `hack/check-applicationdefinition-crd-ordering.bats` (auto-discovered by `make bats-unit-tests`, runs in CI via `make unit-tests`): parses every source and fails if any PackageSource variant with a `system/*-rd` component lacks the `cozystack.cozystack-engine` edge — so future sources are covered, not just today's set. - `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the source edits do not disturb existing assertions. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved dependency ordering across several platform packages to ensure related components install in the correct sequence. * Prevented end-to-end test selection from being overly broad when only ordering-related dependencies change. * **Tests** * Added checks to catch missing dependency requirements and to verify test selection behavior stays accurate. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
1e8d905a2d |
test(foundationdb): widen late-converging status wait budgets (#3039)
## What this PR does **Problem** The `foundationdb` e2e test fails intermittently with a `.status` jsonpath poll timeout even though the FoundationDB cluster is fully healthy and reconciled. The `generations.reconciled`, `hasListenIPsForAllPods`, `health.fullReplication` and `health.healthy` gates polled their status fields with a 60s ceiling, while the strictly-earlier `health.available` gate already allowed 300s. The operator stamps `generations.reconciled` only after every process holds its role for `minimumUptimeSecondsForBounce` (60s, set in `packages/apps/foundationdb/templates/cluster.yaml`), which makes `reconciled` — and the health fields gated on it — structurally the last status fields to converge. The short ceiling closed the gate seconds before the field landed; a snapshot taken a few seconds after a failure shows the live object fully reconciled (`generations.reconciled: 1`, `health.healthy: true`). **Fix** Raise those four late-converging gates to 300s, matching the existing `health.available` budget. The waits stay event-driven (`until …; do sleep …; done`) — only the ceilings widen, no new retries or fixed sleeps. A comment records why `reconciled` converges last so the budget ordering stays legible to future readers. The early-converging 60s gates (`configured`, `connectionString`, `databaseConfiguration.*`, `desiredProcessGroups`) and the spec-field gates are left untouched, since they settle before `health.available` and 60s is already adequate. **Why not just rerun** Rerunning masks a deterministic ordering bug: the latest-converging field carried the tightest budget. The cluster was healthy on every "failure" — the test was racing its own success criteria, so the failure recurs until the budget ordering is corrected. This change touches only the test harness; the platform behaviour it asserts is unchanged. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased waiting time in one end-to-end FoundationDB setup test so cluster status checks have more time to stabilize. * Reduced flaky failures by allowing key health and reconciliation fields to converge before the test continues. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
144f592e1b |
test(e2e): widen etcd convergence budget to 5m in Talos bootstrap (#3036)
## Problem
The `Bootstrap Talos cluster` E2E step fails intermittently with `❌ Test
failed: Bootstrap Talos cluster (exit 1)` even though the cluster is
healthy. The exit comes from the etcd-members convergence gate, which is
bounded at `timeout 180`.
A 3-node etcd converges via Talos's serialized learner promotion — etcd
admits one learner at a time, so the third member is only promoted after
the second has fully caught up. On a loaded ephemeral runner this
deterministic path can legitimately approach ~3 minutes of wall clock
and photo-finish against the 180s ceiling, timing out a cluster that has
in fact converged (every member reaches `Health check successful`; no
kernel panic / OOM / boot failure / network-unreachable in dmesg).
Observed: a run where the third member reached etcd health at +178.6s
against the 180s budget, failing ~1.4s short.
## Fix
Widen the etcd-members convergence budget from 180s to 300s. The wait is
already event-driven (`until talosctl etcd members …; do sleep 1; done`)
— only the ceiling was too tight. No retry is added and the
dmesg-on-failure diagnostic is preserved.
300s matches the repo's other in-line budgets for deterministic
convergence (HelmRelease-ready 5m, harbor/NFS 10m, install 15m).
## Why not just rerun
This is not environmental noise: the cluster converges correctly every
time, it is only timed out. Rerunning wastes a full ~1h E2E cycle to
re-roll the dice on whether the runner shaves the last ~2s. Correcting
the mistuned budget removes the photo-finish.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Extended the cluster bootstrap wait time for control plane health
checks, reducing false failures during slow startup.
* Improved readiness handling so cluster setup is more resilient before
moving on to later initialization steps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
13f1f9ece1 |
feat(api): add network.cozystack.io ExposureClass and ServiceExposure
Introduce a new network.cozystack.io/v1alpha1 API group with two kinds that form the mechanism-agnostic LoadBalancer exposure model: - ExposureClass (cluster-scoped): the admin binds a logical class to a concrete backend (externalIPs, metallb, cilium, robotlb) and its VIP address scope, following the StorageClass / IngressClass pattern. - ServiceExposure (namespaced): the tenant-facing intent to expose a Service externally via a named class, with status reporting the resolved backend and assigned IPs. Wire api/network into the controller-gen CRD generation so deepcopy and the two CRD manifests are produced alongside the existing groups. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
98e60d03eb |
test(e2e): exercise the membership SecurityGroup model end-to-end
Rewrite the SecurityGroup e2e for the membership model: assert the backing CiliumNetworkPolicy's endpointSelector is the group's own membership label, that attachments ride a storage-owned annotation hidden from the view, and that fromApp/fromSG peers project to lineage- and membership-label endpoint selectors the real cilium.io/v2 CRD accepts. A second test creates a managed-labeled pod and asserts the securitygroup-controller stamps the membership label on attach and clears it on detach. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
3ece61ba04 |
test(e2e): make the LoadBalancer announcer detection failover-robust
The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
631cdaa069 |
test(e2e): capture the LoadBalancer datapath when an LB IP is unreachable
The on-failure dataplane diagnostic only captured host->pod CNI state for NotReady pods. A recurrent flake presents differently: a tenant Service type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready and Running, so there is no NotReady pod and the hook captured nothing -- leaving the host->cross-node->bridged-worker-VMI->nodePort path uncharacterised, which is exactly where root-causing has stalled. Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer services with an ingress IP, derives the EndpointSlice backend, the nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB whose every probe fails gets the heavy capture, and that probe doubles as the traffic for the tcpdumps. For a failing LB it captures, on both the announcer and endpoint nodes, the host cilium LB map, kernel conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on the geneve tunnel and the backend tap -- localising the failing hop to host-cilium programming vs kube-ovn delivery. In-guest VMI capture is documented as a deferred stretch that needs tenant-cluster access. The existing NotReady-pod capture is unchanged. The new section is best-effort and time-boxed like the rest of the script, and a unit test pins the pure enumeration, announcer-node, and capture-or-skip logic against mock input. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7214efc8e5 |
refactor(platform): factor the cozystack-version stamp into a shared migration helper (#2980)
## What this PR does Every migration that stamps the `cozystack-version` ConfigMap must carry the `platform.cozystack.io/no-delete=true` label so the `cozystack-no-delete-guardrail` ValidatingAdmissionPolicy guards it. The chart template renders the labeled ConfigMap only on first install, so each upgrade-time stamp has to re-emit the label by hand — migration 43 had to inline-copy migration 42's heredoc to avoid a label-less apply by the same field manager stripping it. The bug class was patched per-migration, not eliminated. This extracts the stamp into a shared, sourced helper (`render_cozystack_version_manifest` + `stamp_cozystack_version`) under `images/migrations/migrations/lib/`, and routes migrations 42, 43, 44 and the `run-migrations.sh` bootstrap path through it. The labeled manifest now has a single definition, so future migrations cannot drift back to a label-less apply. The upgrade path is behavior-preserving: the rendered manifest is byte-identical to the previous heredocs (pinned by a golden unit test). The bootstrap path in `run-migrations.sh` previously created the ConfigMap without the label — it now carries it like every migration-driven stamp. Adds `hack/cozystack-version-stamp.bats` (8 tests) covering the helper: label presence, ConfigMap name, quoted-string version type, namespace default and override, required-argument guard, and the byte-exact golden manifest. Closes #2781 ### Release note ```release-note fix(platform): always stamp the platform.cozystack.io/no-delete label on the cozystack-version ConfigMap, including the migration bootstrap path, so deletion protection cannot be silently dropped ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Tests** * Expanded validation to confirm the version-stamp manifest is parseable YAML, includes the required no-delete label, targets the correct `cozystack-version` ConfigMap, and quotes the version value correctly. * Added coverage for default and overridden namespaces, argument validation failures, and a golden “byte-for-byte” render check. * Added a guardrail test to prevent bypassing the shared stamping logic by direct `kubectl` ConfigMap stamping. * **Refactor** * Centralized `cozystack-version` rendering and stamping into a shared helper and updated migrations and the migrations runner to use it consistently. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
fa49dbaa33 |
Merge origin/main into refactor/build-matrix-2937
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch diverged landed a competing single-VM build restructure (#2939) and test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build supersedes #2939: - .github/workflows/pull-requests.yaml: keep #2983's plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact interface (pr-patch + talos-image) is identical on both sides, so e2e depends on finalize/build-talos with no input rewiring. main's top-level permissions and SHA-pinned action refs applied across all jobs. - All other files merged cleanly (only pull-requests.yaml conflicted). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
0f1cc548b5 |
fix(objectstorage-controller): converge BucketClaim readiness and speed up COSI provisioner failover (#3034)
## What this PR does Fixes two failure modes in the COSI control plane built from upstream container-object-storage-interface v0.2.2, both carried as image patches. **BucketClaim never converges to ready (#3032).** The controller does not watch Bucket objects, and the generic controller drops no-op resync deltas, so a BucketClaim whose backend Bucket flips to ready *after* the first reconcile stayed `bucketReady=false` indefinitely — BucketAccess was never granted and the consuming app (bucket, harbor) got no credentials. The existing re-read of the live Bucket only ran during a reconcile that nothing re-triggered. The controller now returns an error while the Bucket is not ready, forcing a rate-limited requeue that re-reads the Bucket until its readiness propagates, so the claim converges deterministically instead of racing the async sidecar. **Slow provisioner failover on pod recreate (part of #2429).** The controller and the per-driver sidecar run single-replica, so leader election adds no availability — only failover latency. The upstream 150s lease made a recreated provisioner pod stall reconciliation for up to ~150s. The lease is lowered to the standard Kubernetes values (15s/10s/2s), so a recreated pod resumes in ~15s. The stale unix-socket CrashLoop in #2429 was already resolved by the seaweedfs-cosi-driver v0.3.1 self-heal; the silent "Bucket ready with no backend bucket" finding on that issue is a separate driver bug and is left tracked there. Coverage: a unit test in the patched controller pins the requeue/propagation contract (fails without the requeue), and the bucket/harbor e2e suites now assert `bucketReady=true` on a tight bound instead of tolerating the lag, with scoped diagnostics on the failure path. Closes #3032 Refs #2429 ### Screenshots N/A — no UI changes. ### Release note ```release-note fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved object storage bucket readiness propagation so bucket claims now track the backend bucket’s actual `bucketReady` status and requeue when readiness isn’t yet observed. * Reduced and made readiness waits more reliable in end-to-end checks, with explicit match conditions and clearer debug output (including claim YAML, backend bucket list, and controller pod status). * Shortened leader-election timing to speed up recovery after pod restarts. * **Chores** * Updated automated PR labeling to map the object storage controller scope to the storage area label. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
a036cfd1e9 |
ci(tests): remove redundant Release E2E workflow (#2978)
## What this PR does Removes the `Release E2E` workflow (`.github/workflows/release-e2e.yaml`), which ran the full bats suite on every release tag. **Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm — its own header states *"a failure here is an alarm, not a release blocker."* The coverage it provided is already delivered by the release-cut PR: the "Prepare release" commit bakes image digests into `packages/core/` (installer, platform, testing) and other packages, and `select-e2e.sh`'s full-suite pattern escalates any `packages/core/` change to the **full** suite. So the release PR already runs the whole suite against the commit the tag will point at. **Why now.** The workflow has never been green since it was added (2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed at the build step. Even after #2973 fixed the registry login earlier today, the next run still failed at *Build Talos image*. A perpetually-red, non-gating duplicate is pure noise. **Changes** - Delete `.github/workflows/release-e2e.yaml`. - Drop `release-e2e` from the full-suite pattern in `hack/select-e2e.sh`. - Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats` to `pull-requests.yaml` (the remaining E2E workflow), preserving "workflow edit → full suite" coverage. All 11 `select-e2e_test.bats` cases pass. - Update `docs/agents/e2e-testing.md` to describe the release-cut PR as where the full suite runs. Refs #2397. ### Release note ```release-note ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Removed automated release end-to-end testing workflow * Updated E2E test selection logic to reflect testing infrastructure changes * **Documentation** * Updated E2E testing procedures documentation to reflect current testing approach <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
ba6763af19 |
fix(build): force full image rebuild on root Go source changes
The matrix selector's full_rebuild_pattern omitted the root Go module dirs (api/, cmd/, internal/, pkg/). Eight matrix image Dockerfiles COPY those dirs, so a PR changing only root Go source emitted an empty matrix, skipped the build job, and let e2e install stale digests — the Go change was never exercised. Add the four dirs to the pattern and pin it with bats cases for api/- and pkg/-only diffs forcing the full matrix. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
de9c81ce6e |
test(e2e): raise kafka/zookeeper preset off the 128Mi OOM threshold
The kafka e2e test pinned both the kafka and zookeeper components to the legacy "nano" preset (250m CPU / 128Mi memory). 128Mi cannot hold the Strimzi Kafka 3.9.1 and ZooKeeper JVMs: the brokers OOMKill on startup (exit 137) and CrashLoopBackOff, so the pods never reach Ready, the Kafka CR and its HelmRelease never go Ready, and the 5m readiness wait times out. It surfaced as a flake because 128Mi sits right at the JVM-start threshold — occasionally a broker passes readiness before its RSS crosses the cgroup limit. Raise both components to "c1.small" (1 CPU / 1Gi), the chart default, which comfortably fits the JVMs. The e2e sandbox VMs are 8 CPU / 24Gi, so 1Gi across the four pods is well within budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2dc9c83a71 |
test(e2e): escalate cilium leak healer to agent restart on leaked-IP reuse
The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint disconnect, or delete-reschedule the wedged pod) cannot clear a leak that lives purely in the cilium-agent's in-memory state: when IPAM keeps re-handing the SAME released IP, a delete-rescheduled pod re-wedges on it, and on some variants the stale entry isn't even in the queryable endpoint registry so disconnect no-ops. Both produce an inescapable delete -> re-wedge / no-op loop that fails the install on FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313). Add a bounded Tier-2 escalation: track recurrence per (node,ip); once the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that node's cilium-agent (delete its pod; the DaemonSet rebuilds it from CRDs/IPAM without the stale entry), capped per node so it can never loop-restart agents. A genuine duplicate-IP (live owner) is refused before counting, so an agent restart never masks a real product bug. Tier-1 stays the first remedy; escalation fires only after it proves futile on the same IP. RBAC needs no new verb: the existing cluster-wide pods get/list/delete already covers deleting a cilium-agent pod in cozy-cilium. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
594c887916 |
test(e2e): capture OVN flow-programming datums in dataplane diagnostic
Extend the host->pod data-plane capture with the three datums that make an OVN incremental-processing lag dispositive on the next recurrence, all diagnostic-only and gated on the existing on-failure path: - ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output, if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded tail) -- shows whether ovn-installed was set before the LSP local-delivery flow was programmed. - per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS interface, looked up by iface-id <pod>.<ns>. - ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1 fallback) -- shows whether ovn-controller was CPU-starved. Node-global captures land in node-<node>.txt, the per-interface flag in the per-pod file. Every command stays timeboxed and || true, no behavior or exit-code change. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |