mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-07 04:23:50 +09:00
ebecbd71c9812b73bdd5bb26d15307ded768784d
554
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
1a0680b778 |
test(e2e): allow tenant CDI importers to reach the image cache
Tenant namespaces run under a default-deny Cilium egress installed by the tenant chart, which lets a worker's CDI importer reach the outside world and kube-dns but nothing in kube-system. The in-sandbox image mirror is a kube-system Service, so without an explicit allow the importer's connect to its ClusterIP is dropped and the worker disk import never starts. Ship a tightly-scoped CiliumClusterwideNetworkPolicy in the mirror manifest that lets importer Pods in tenant-test egress to the mirror Pod in kube-system. Cilium unions allow rules, so this only adds a hole; scoping it to tenant-test (already under default-deny egress) means it never flips another namespace's posture, and the mirror Pod needs no ingress rule because it is selected by no ingress policy. The policy is kept out of the pre-install apply because that runs before Cilium's CRDs exist; the resolver in hack/e2e-apps/talos-image-cache.sh applies it at point-of-use once Cilium is up. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f5ec885d8d |
test(e2e): gate the talos image cache on real tenant reachability
The in-sandbox Talos image mirror was pointed at by tenant Kubernetes CRs whenever its Deployment was Available, but "Available" only proves the seed finished and the server answers on localhost. It does not prove a worker's CDI importer can reach the Service. Tenant namespaces run under a default-deny Cilium egress that permits the outside world and kube-dns but not an arbitrary kube-system Service, so the importer resolves the name yet its TCP connect to the ClusterIP is silently dropped. The disk import then never starts, no tenant node joins within the 12-minute deadline, and the kubernetes-* tests fail deterministically -- worse than the intermittent public-factory flake the mirror was meant to relieve. Replace the localhost range-probe with one that faces the exact same network path as a worker importer: a throwaway Pod in the tenant namespace, labelled cdi.kubevirt.io=importer, that fetches the seeded image from the Service ClusterIP with a Range request and expects a 206. Point tenants at the mirror only when that end-to-end check passes; otherwise fall back to the public factory, restoring the guarantee that the mirror can only help, never make CI worse. The resolver also installs the importer egress allow at point-of-use, because Cilium's CRDs do not exist yet when the mirror manifest is first applied before install. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
336f873b07 |
fix(e2e): serve the Talos image cache with a range-capable HTTP server
The talos-image-cache serve container ran `busybox httpd`, but the pinned alpine/k8s image ships no `httpd` applet and no standalone httpd, so the container exited 127 and crash-looped and the Deployment never became Available. The resolver then burned its full 12-minute rollout wait before falling back to the public factory — a fixed latency tax on the first kubernetes-* test while the flake the mirror targets stayed live. Serve the seeded image with a small range-capable server on the image's bundled python3 instead (stock `python3 -m http.server` ignores Range headers and answers 200, not 206). It needs no extra package pull, so the mirror comes up deterministically and keeps CDI's fast range import. The seed curl now also aborts and retries a stalled transfer (--speed-time/--speed-limit), and the resolver verifies a 206 range response before pointing tenants at the mirror. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f60fe2270b |
Merge remote-tracking branch 'origin/main' into feat/kubernetes-talos-image-factory-override
Signed-off-by: Aleksei Sviridkin <f@lex.la> # Conflicts: # packages/system/kubernetes-rd/cozyrds/kubernetes.yaml |
||
|
scooby87
|
6c69c0055e |
fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091) (#3094)
## What this PR does
When the `monitoring` tenant module is disabled, the
VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and
the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left
orphaned in the tenant namespace.
This adds a `post-delete` cleanup hook
(`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that:
- deletes the VM/VL storage PVCs scoped to this release via `-l
apps.cozystack.io/application.name={{ .Release.Name }}-system`. That
label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of
the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator
copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the
selector matches and stays scoped to this monitoring release.
`managed-by=vm-operator` is intentionally **not** used — vm-operator
does not reliably stamp it onto the storage PVCs;
- deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name
(cert-manager TLS secrets carry only the cluster-wide
`controller.cert-manager.io/fao=true` label, so a label sweep would be
unsafe — they are removed by name).
CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally
**not** targeted — CNPG cleans those up itself, and they do not carry
`apps.cozystack.io/application.name`, so the selector leaves them
untouched.
**Upgrade path:** the claim-template label only lands on PVCs created
after the change, so **migration 50**
(`packages/core/platform/images/migrations/migrations/50`) backfills
`apps.cozystack.io/application.name` onto the pre-existing VM/VL storage
PVCs of every cozystack-managed monitoring cluster, so the cleanup
reclaims storage on already-deployed clusters too.
A `test` Makefile target and helm-unittest suites are added for both the
cleanup hook and the claim-template label. The cleanup Job is hardened:
non-root, read-only root fs, dropped capabilities, `seccompProfile:
RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`,
`backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC
get/list/delete and the two named secrets. Failures are logged but never
block teardown.
> [!WARNING]
> **Disabling the monitoring module now permanently deletes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they
hold.** This is intentional and fixes the storage leak. Back up before
disabling if the data matters.
Fixes #3091.
### Release note
```release-note
fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Helm **post-delete** monitoring cleanup job to best-effort
remove managed PVCs and two specific TLS secrets.
* Included dedicated cleanup RBAC and hardened pod security settings for
the cleanup job.
* **Tests**
* Added Helm chart tests validating hook weight/policies, job settings,
exact deletion commands, and negative cases.
* Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
a28ef9ba13 |
test(e2e): make the hairpin DNS check tolerate a tenant node recycle (#3241)
## What this PR does Makes the ouroboros hairpin DNS assertion in the `kubernetes-latest` e2e deterministic against a tenant worker-node recycle. This is a widespread pre-existing flake: it fails PRs that touch nothing virt-related and passes on re-run. ### Root cause (mechanism, from a failed run's own event stream) The assertion resolves `hairpin-*.example.invalid` from inside the tenant cluster using a single `--restart=Never` netshoot Pod, then waits for that Pod to reach `Succeeded`. The tenant crust-gather snapshot of a failed run shows the pod never ran the DNS loop at all: - `09:00:02` scheduler: `Successfully assigned default/dnscheck to ...-md0-vqcs7-hhfxx` - `09:00:06` kubelet on `hhfxx`: `Pulling image "nicolaka/netshoot:v0.13"` (no subsequent `Pulled`/`Started`) - node `hhfxx` then disappears from the tenant node list; on the management cluster its Machine is gone and a replacement Machine is created at `09:02:01` (`Provisioning`), while the other worker (created `08:43:39`) survives - `09:02:47` `taint-eviction-controller`: `Cancelling deletion of Pod default/dnscheck` The tenant's CAPI `MachineHealthCheck` remediates a node whose `Ready` goes `Unknown` for just 30s by deleting its Machine, KubeVirt VM and Node and provisioning a replacement. On the memory-constrained single-node CI sandbox a worker VM losing its kubelet heartbeat briefly is routine. A bare Pod bound to the deleted node is removed by the node controller and, with `restartPolicy: Never`, is never recreated — so the probe silently vanishes while its (large) image is still pulling and it has never left `Pending`. The single-shot wait then times out and reports `dnscheck pod did not reach Succeeded phase (last seen: <empty>)`, which is indistinguishable from a real DNS-resolution failure. ### The fix Wrap the probe create-and-wait in a bounded outer deadline (7m) that recreates the Pod when it is confirmed `NotFound` — the node-recycle signature — so the check is retried on a surviving node instead of failing the whole test. An empty phase is treated as ambiguous (a transient API blip is not misread as a lost Pod); only a definitive `NotFound` triggers a recreate. A Pod that actually runs to a non-`Succeeded` terminal state, or the deadline elapsing with no successful run, still fails the test. The in-Pod DNS resolution loop that the assertion exists to exercise is unchanged, so the reconciliation coverage is preserved. This fixes the mechanism at the right layer: the check tolerates a transient node recycle of a throwaway diagnostic pod rather than reporting it as a DNS regression. It is deliberately not a timeout bump — the pod was actively deleted, not merely slow, so more time alone would not help. ### Out of scope The `kubernetes-previous` failure seen in the same run is a different mechanism: both tenant worker VMs stayed `Provisioning` for 12m (DataVolume/CDI storage-import stall on the management cluster), so no tenant node ever joined and the run fails at the existing node-join diagnostic block, well before this assertion. Both smell like the same underlying single-node-sandbox resource/storage pressure, but that is an infrastructure-capacity concern, not a harness race this change can fix. Left for a separate follow-up. ### Screenshots N/A — CI/test-harness change, no UI impact. ### Release note ```release-note test(e2e): recreate the tenant hairpin DNS probe pod on a bounded deadline so a worker-node recycle mid-check is retried instead of reported as a spurious DNS failure ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability of the Kubernetes end-to-end DNS check. * The DNS validation now retries more safely during infrastructure-related pod restarts or node churn. * Failures now surface faster with clearer diagnostic output when the check does not pass. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
ff037c04d3 |
test(monitoring): reproduce the real vmselect cache PVC name in the backfill test
/branch-review flagged that the migration comment and its bats test used the simplified vmselect cache PVC name cache-vmselect-<cr>-<ordinal>, while the real name is vmselect-cachedir-vmselect-<cr>-<ordinal>. The anchored regex matches both (it suffix-matches <component>-<cr>-<ordinal>), so behaviour was correct, but the test did not exercise the production name. Feed the realistic name in the fake kubectl so the regex is validated against what vm-operator actually provisions, and reword the migration comment to describe the suffix match generically instead of asserting a specific claim-template prefix. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Алексей Артамонов
|
09c422562a |
test(monitoring): unit-test the VM/VL PVC backfill migration behaviour
helm-unittest only inspects rendered manifests, not the operator's runtime PVC labels, so it cannot prove migration 50 actually relabels the right PVCs — the exact no-op/mis-scope class of bug this PR iterated on. Add a bats test (fake kubectl on PATH, mirroring hack/kubernetes-md0-migration.bats) that pins the runtime behaviour: it relabels the monitoring cluster's vmstorage/vmselect/ vlstorage PVCs with application.name=monitoring-system, never touches the CNPG db PVCs or a user's own VMCluster, and stays a safe version-stamping no-op when the VM/VL CRDs are absent. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
IvanHunters
|
0c7b598304 |
fix(apps/kubernetes): honour nodeGroup minReplicas in MachineDeployment.spec.replicas (#3233)
Refs #3231 (parts 1 + 2 addressed here; part 3 — intra-file `cozy_wait_linstor_pool_free` gate + orphan `default-chk-clickhouse-test-keeper-cluster1-0-2-0` PVC cleanup — deferred to a separate follow-up PR against `hack/e2e-apps/run-kubernetes.sh`). **Do not auto-close the issue on merge** — the hardening work is a distinct concern and belongs on its own tracker entry. ## Root cause chain (recap from the issue) `packages/apps/kubernetes/templates/cluster.yaml` hardcoded `MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's `minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}` bats set `minReplicas: 0` and delete their `Kubernetes` CR without waiting for readiness — but the chart still spun up 2 × 20 GiB `replicated` (DRBD) worker `DataVolume`s each. Their create/delete churn overlapped `kubernetes-previous`, which then saw slow LINSTOR provisioning (my cozyreport from run 28843794670 shows a ~6-minute gap between the `Kubernetes` CR being created and the CDI `ExternalProvisioning` event actually firing for the worker root disk); the tenant node registered late; Cilium's CNI init missed the 12-minute node-join deadline; and the test failed with `cni plugin not initialized`. ## Changes Two edits, applied together — the chart edit is a no-op without the test edit, and vice-versa (dropping `nodeGroups` falls back to the chart helper's default `md0`, which itself has `minReplicas: 0`, so both pieces are needed for zero workers to actually render). ### 1. `packages/apps/kubernetes/templates/cluster.yaml` ```diff spec: clusterName: {{ $.Release.Name }} - replicas: 2 ``` The field is **omitted entirely** so the cluster-autoscaler owns it alone. CAPI's MachineDeployment defaulting webhook seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create (documented at https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling — *"if it's a new MachineDeployment or MachineSet, use min size"*), and Server-Side-Apply field-manager ownership then keeps chart upgrades from overwriting whatever value the autoscaler patched in the meantime. Emitting the field from the chart at all — even with the semantically-correct `{{ $group.minReplicas }}` — would keep the helm-controller ↔ cluster-autoscaler field-ownership fight alive: helm-controller applies with SSA + force-conflicts (https://fluxcd.io/flux/components/helm/helmreleases/), so every routine `helm upgrade` re-asserts the chart's chosen value and drains whatever the autoscaler had patched in the meantime. ### 2. `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats` Drop the explicit `nodeGroups:` block and the paired `storageClass:` from the render-side CR bodies. The tests only assert on the KamajiControlPlane and the cozy-realm Keycloak objects — worker plumbing was never part of their coverage. After the chart fix, the chart helper's default `md0` (with `minReplicas: 0`) renders a MachineDeployment with `spec.replicas: 0` — the resource is still emitted for API-surface completeness but never creates a KubevirtMachine, never provisions a DataVolume, and never touches DRBD. ## Verification ### helm-unittest ``` Charts: 1 passed, 1 total Test Suites: 16 passed, 16 total Tests: 171 passed, 171 total ``` No suite asserted on `MachineDeployment.spec.replicas` directly, so nothing needed updating. ### helm template — verify the omit With `tests/values/common.yaml` (which has `minReplicas: 0`), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "0"` annotation and **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 0` on Create — no worker DataVolumes provisioned until scale-out. With a user CR that sets `minReplicas: 2` (the shape the `kubernetes-latest` / `kubernetes-previous` bats CRs use), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "2"` annotation and again **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 2` on Create — byte-identical topology to the pre-PR chart's fresh-install state (2 workers), but without the recurring drain-on-upgrade cycle. ### Render-side render walk `helm template` of the OIDC bats CR bodies (after this PR): 1 MachineDeployment with **no chart-managed `spec.replicas`** and `min-size: "0"` on the autoscaler annotation. CAPI seeds `replicas: 0` on Create, 0 KubevirtMachines emitted, 0 DataVolume provisioning triggered. Test assertions on KamajiControlPlane / KeycloakClient / KeycloakClientScope / AuthenticationConfiguration Secret are unchanged. ## Upgrade impact Existing customers stop losing workers on `helm upgrade`. Prior to this PR the chart re-asserted `MachineDeployment.spec.replicas: 2` on every Helm action, so a customer running with the autoscaler at, say, `replicas: 8` would see 6 workers drained on every routine Cozystack platform bump and then wait for the autoscaler to scale back up in response to unschedulable Pods. With `spec.replicas` no longer emitted by the chart, SSA field-manager ownership preserves the autoscaler's value across upgrades unconditionally. Fresh installs with `nodeGroups: {}` (the schema default) transition from "2 workers by default" to "0 workers until scale-out". The chart helper's default md0 carries `minReplicas: 0`, which is what CAPI's defaulting webhook seeds `spec.replicas` from. The `values.yaml` / README / helper docstring / regenerated Go type doc + openAPISchema call this out explicitly; the release-note repeats it below. See finding 5 in the paired self-review for the specific `addons.ingressNginx.enabled: true` interaction operators need to be aware of on fresh installs. ## Test plan - [x] `helm unittest packages/apps/kubernetes/` — 171/171 passing. - [x] `helm template` with `tests/values/common.yaml` (minReplicas: 0) — MD.replicas: 0. - [x] `helm template` with a `minReplicas: 2` override — MD.replicas: 2 (byte-identical to pre-PR). - [ ] E2E: `kubernetes-latest` unchanged (spawns its two workers as before), `kubernetes-oidc-{system,customconfig}` render zero workers, `kubernetes-previous` no longer flakes on LINSTOR pool pressure originating from the earlier OIDC suite. ## Notes - Part 3 of #3231 (intra-file `cozy_wait_linstor_pool_free` gate between kube sub-tests and the orphan clickhouse keeper PVC cleanup) is **not** in this PR — worth a separate targeted change to `hack/e2e-apps/run-kubernetes.sh`, and it's a hardening-only improvement rather than a root-cause fix. Happy to send it as a follow-up if wanted. ## Release note ```release-note fix(apps/kubernetes): stop the chart from managing `MachineDeployment.spec.replicas`; the cluster-autoscaler owns the field alone. CAPI's defaulting webhook now seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create, and SSA field-manager ownership keeps subsequent `helm upgrade` actions from overwriting whatever value the autoscaler patched. Two behavioural changes worth calling out: - Existing customers: `helm upgrade` no longer drains workers on every platform bump — the autoscaler-held `spec.replicas` is preserved unconditionally. Prior to this change the chart hardcoded `replicas: 2`, which drained back to 2 on every reconcile action. - Fresh installs with `nodeGroups: {}` (the schema default) no longer come up with two workers by default. The chart helper's default md0 carries `minReplicas: 0`, and CAPI's defaulting webhook seeds `spec.replicas` from that. Operators enabling `addons.ingressNginx.enabled: true` on such a CR must either supply an explicit nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or accept that the cluster-autoscaler will bring the default md0 up in response to the ingress-nginx controller Pods becoming Pending on install. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Rendered `MachineDeployment` manifests no longer set `spec.replicas`, avoiding chart upgrades overwriting autoscaler-managed sizing. * **Documentation** * Clarified `nodeGroups: {}` behavior: a default `md0` is emitted with `minReplicas: 0`/`ingress-nginx` role, and workers are provisioned only after an autoscaler-triggering unschedulable Pod or manual scaling; updated related storage and ingress guidance. * **Tests** * Strengthened default node group rendering assertions and adjusted OIDC e2e manifests to reduce CI provisioning churn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
47e7d224d8 |
test(e2e): serve Talos worker OS image from an in-sandbox cache
The kubernetes-* e2e tests fail intermittently and run 18-22 min each because every tenant worker VM imports its Talos OS disk directly from the public factory.talos.dev: that endpoint has no byte-range support (forcing CDI into a full copy-to-scratch) and its egress stream-resets / stalls mid-transfer from the CI runner, so a worker DataVolume import hangs past the chart's 12-minute node-join deadline and the test fails with zero tenant nodes joined (observed on run 28924455578, and on the unrelated branch in run 28929095889 - i.e. not specific to any one PR). Deploy a best-effort in-sandbox mirror (hack/e2e-talos-image-cache.yaml) during install: an initContainer fetches the image once with an aggressive curl retry loop, then busybox httpd serves it locally with range support. Tenant Kubernetes CRs point spec.talos.imageFactoryURL at its Service via a shared helper (hack/e2e-apps/talos-image-cache.sh) that gates on the mirror being Available and falls back to the public factory otherwise, so the mirror can only help, never make CI worse. Deployed early so the seed overlaps the Cozystack install; readiness is resolved once and cached across bats files. Uses the overridable chart value added in the previous commit. Refs #3231 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
699e429187 |
test(e2e): make the tenant hairpin DNS check tolerate a node recycle
The ouroboros hairpin assertion resolves a name from inside the tenant cluster via a single `--restart=Never` netshoot Pod and waits for it to reach Succeeded. On the memory-constrained single-node sandbox a tenant worker node is routinely recycled mid-test: its kubelet heartbeat lapses, the tenant's CAPI MachineHealthCheck marks Ready=Unknown after 30s and deletes the Machine, its KubeVirt VM and the Node, then provisions a replacement. A bare Pod bound to the deleted node is removed by the node controller and, having restartPolicy Never, is never recreated -- so the probe silently vanishes, typically while its large image is still pulling and it has never left Pending. The single-shot wait then times out and reports "dnscheck pod did not reach Succeeded phase (last seen: <empty>)", which is indistinguishable from a real DNS failure and flakes PRs that do not touch virt at all. Keep the DNS resolution asserted exactly once and fail-fast: a probe that runs to phase Failed is the reconciliation regression this test exists to catch and fails the test immediately. Recreate only the probe Pod, and only for the pure-infrastructure node recycle -- when the Pod vanishes and the node it ran on is confirmed gone or no longer Ready, retry on a surviving node within a bounded deadline. A probe that disappears while its node is still Ready is an unexpected deletion and fails loud rather than being retried, so no pod-churn regression is masked, and a transient API error while polling is not mistaken for a deleted Pod. The in-Pod dig loop that the assertion exercises is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
IvanHunters
|
6ac52a108a |
docs(e2e): correct OIDC bats comments to match post-fix render behaviour
Both nodeGroups-omitted comments overstated the outcome as "zero
MachineDeployments render", but the chart helper (kubernetes.nodeGroups)
emits a default md0 group whenever the caller's nodeGroups map is
empty, so exactly one MachineDeployment renders. What is actually
zero is the number of KubevirtMachines / worker DataVolumes, because
the chart no longer manages spec.replicas and CAPI's defaulting
webhook seeds it to 0 from the autoscaler min-size annotation.
kubernetes-oidc-system.bats additionally described the bug in the
present tense ("chart hardcodes spec.replicas: 2 ... see #3231"),
which contradicts the sibling customconfig comment and would read
as an outstanding defect post-merge — rewrite it around the post-fix
rationale to match customconfig's shape.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
myasnikovdaniil
|
cece5c23b5 |
feat(release): immutable tags & rc→stable promotion (#2677) (#3017)
## What this PR does Implements the immutable-tag + rc-promotion flow from #2677. A **stable release becomes a renamed release-candidate**: the bytes shipped as `vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as `vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt — it is *promoted* by retagging the rc's existing images. Layered onto the build matrix from #2937/#2983 (this PR is **stacked on `refactor/build-matrix-2937`** and must merge after it). ## The five force-retag sites, removed | Site | Before | After | |---|---|---| | `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` | write-once (create-if-absent, fail if it would move) | | `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` | compare-before-force (no-op if unchanged; staging branch only) | | `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` | write-once at the PR merge commit (force impossible by construction) | | `pull-requests-release.yaml` maintenance branch | `updateRef force:true` | fast-forward-only | | `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow deleted** — stable only via explicit promote | ## Version decoupling (the enabler) The operator baked its version into the image at build time, so an rc image self-reported the rc string — blocking retag-promotion. It now reads `COZYSTACK_VERSION` from the environment (threaded via `cozystackOperator.platformVersion` → Deployment env, stamped by `make manifests`), falling back to the build-time value when unset. The same image bits can report any release name. The only runtime reader is the telemetry metric `cozy_cluster_info{cozystack_version=...}`. ## Promotion flow `promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`): 1. Validate the rc release exists and the stable tag does not. 2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from `packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to `:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`. 3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable version (the `@sha256` wins regardless), restamp the version-stamped assets (operator manifests, cozypkg, openapi; the heavy Talos assets are copied verbatim from the rc draft), open a `release-X.Y.Z` PR. 4. Merging the PR reuses the existing release-PR e2e and `pull-requests-release.yaml` finalize to cut the **write-once** stable tag at the merge commit and publish the release. Squash is disallowed (the tag needs a real merge commit). `nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by `NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line (dry-run by default). ## Validation **Local end-to-end — the core claim is proven, not asserted.** The whole design rests on "copy-by-digest to a new tag preserves the digest, so stable == the e2e-tested rc, bit-for-bit." This was exercised against a real registry (`ttl.sh`) with real `skopeo`: - Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp tree exercising all three `values.yaml` digest shapes (single `image:` string, split `repository`/`tag`/`digest` map, and the `platformSourceRef` OCI artifact), then ran the actual `hack/promote-retag.sh v1.4.0`. - **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the identical digest** (`sha256:fd8d9aa6…`), across all 17 platform manifests. The script's own `skopeo inspect` post-check passed, and an independent re-inspection confirmed it. Local validation caught (and this PR fixes) two bugs that only surface against a real registry — never in static lint: 1. `skopeo copy --multi-arch all --all` — mutually exclusive flags, fatal error; every retag would have failed on CI. Now `--multi-arch all`. 2. The same digest was retagged twice when an image appeared in two value shapes; the ref set is now deduped on the canonical `repo@digest`. **Other local checks:** `go test ./pkg/version` + `go build`/`vet`; `helm template` renders `COZYSTACK_VERSION` in all 3 operator variants (omitted when unset); `make manifests` stamps the version into the install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint` clean and `act -l` resolves the job graph on all workflows; the rc-tag parser and nightly version-math unit-tested (accept/reject + minor-vs-patch bumps). **Still CI-only (cannot be exercised offline):** the full workflow runtime — github-script API calls, OCIR auth, the rc-release / staging-branch preconditions, the nightly `base_ref` push mechanics, and self-hosted runners. The *logic* inside the steps is unit-tested; the *orchestration* is not. Treat it as unproven until the workflows run. ## Out of scope / required follow-up (needs repo admin — not doable from a PR) - Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by the CI app, no delete/update; "limit branches/tags updated in a single push" = 1. - Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo variables. - Registry-side pruning of `*-nightly.*` image tags (no OCIR delete parity yet — tracked TODO in `retention.yaml`). ## Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added nightly publishing (mirror-by-digest, disk build, e2e validation) and nightly retention pruning. * Added RC-to-stable promotion via digest retagging (no rebuild). * Operator/installer and console now expose version/platform metadata, including runtime override via `COZYSTACK_VERSION`. * **Bug Fixes** * Enforced write-once tag behavior and fast-forward-only maintenance updates across release workflows. * Made nightly mirroring/selection and retention pruning more selective and safer. * **Tests** * Added Bats coverage for nightly mirroring and RC retagging, plus new Helm/unit checks. * **Documentation** * Updated the release model to center RC promotion and tag immutability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
6fd98dc510 |
fix(apps/kubernetes): honour nodeGroup minReplicas + drop nodeGroups from render-side OIDC tests
Closes #3231. `packages/apps/kubernetes/templates/cluster.yaml` hardcoded `MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's `minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}` bats — which set `minReplicas: 0` and delete their `Kubernetes` CR without waiting for readiness — still provisioned two 20 GiB `replicated` (DRBD) worker `DataVolume`s each. Their create/delete churn overlapped the very next test, `kubernetes-previous`, and put transient pressure on a LINSTOR satellite's ZFS pool: worker root disks landed late, tenant nodes registered late, Cilium's CNI init missed the 12-minute node-join deadline, and the test failed with `cni plugin not initialized`. Two changes, applied together — the chart fix is a no-op without the test fix, and the test fix is a no-op without the chart fix, because dropping `nodeGroups` falls back to the chart helper's default `md0` (with `minReplicas: 0`). **Chart**: `spec.replicas: {{ $group.minReplicas }}` so the field matches the autoscaler annotations already emitted just above. `minReplicas: 0` groups now genuinely scale from zero and do not silently provision workers; groups with `minReplicas >= 2` (the kubernetes-latest / kubernetes-previous bats CRs, the tests/values/ common.yaml default, the values.yaml default that gets templated into the helper's `md0`) render byte-identical to before. **Render-side OIDC tests**: drop the explicit `nodeGroups:` block and `storageClass:` — falling back to the chart helper's default `md0` (now with an effective `replicas: 0` after the chart change) means the MachineDeployment renders but never creates a KubevirtMachine and no DataVolume is provisioned. The tests only assert on the KamajiControlPlane and the cozy-realm Keycloak objects, so removing worker plumbing does not narrow their coverage. Upgrade impact: any existing customer CR that already carries `nodeGroups.md0.minReplicas: 0` (rare — the default in tests/values and the chart helper both use it, but real user CRs almost always have `minReplicas >= 1`) will see the MD's `spec.replicas` reconcile from `2` down to `0` on the next chart apply. If the cluster- autoscaler is enabled it will scale back up as soon as an unschedulable pod appears; if it is not, the group intentionally becomes empty (which is what `minReplicas: 0` requests literally). This matches the semantics the autoscaler annotations were already advertising — the hardcoded `2` was contradicting them. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Timofei Larkin
|
7b65cf2afd |
feat(platform): migrate to etcd-operator v1alpha2 (#2859)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for the newer etcd snapshot workflow (BackupClass/BackupJob/RestoreJob) with in-place restore. * Updated etcd chart/operator packaging and migration flow for the v1alpha2 operator release. * **Bug Fixes** * Switched readiness/condition handling to **Available**. * Improved waits to cover all etcd member pods during restore and scenario steps. * **Documentation** * Refreshed etcd backup/restore docs and example scenarios to match the v1alpha2 APIs and parameters. * **Tests** * Updated e2e coverage to validate v1alpha2 operator runtime contracts and replaced legacy backup-block tests with a snapshot round-trip test. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Fable 5
|
d252fc93a0 |
ci(release): make rc promotion transactional; defer registry writes to finalize
Promotion mutated the registry at dispatch time — retagging ~37 images to vX.Y.Z (+:latest), publishing the stable cozy-installer chart — before the promote PR's e2e ran and before any merge. An abandoned promotion then left stable-named bytes for a version that never shipped, and the (correct) image write-once check wedged any later re-promotion to the same version. Move every irreversible registry side effect into pull-requests-release.yaml finalize, which runs only after the promote PR merged (= after its e2e passed), making promotion atomic. promote-rc.yaml now only stages the release-X.Y.Z branch, drafts the stable release, and opens the PR; an abandoned promotion leaves nothing but an overwritable draft + branch. Folded into the same restructure: - Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag in finalize, at the stable tag's commit. It was gated in tags.yaml on `release_exists==false && is_stable`, unsatisfiable under promotion (the draft always pre-exists), so promoted stables never got their submodule tag — a regression for Go consumers. Finalize is where the stable tag is born, so it cuts the matching submodule tag write-once (stable only). - Gate :latest by max-semver. promote-retag.sh now moves :latest only when MOVE_LATEST=1 (default off); finalize passes it the same make_latest decision the release gets, so a patch on an older line moves neither the release's `latest` nor the images' :latest. Added MOVE_LATEST bats coverage. - Stamp platformVersion into the published stable chart's default values (uncommitted, at package time) so `helm --install --version X.Y.Z` reports the stable version in telemetry instead of the rc version baked into the operator image. Not committed to the branch — that would leak platformVersion=vX.Y.Z onto main and mis-stamp the next rc. - promote-rc: serialize all promotions (static concurrency group, not per-rc); label the PR full-e2e so the full suite runs; scope the rc->stable sed to the depth-2 values.yaml glob plus the kubernetes .tag files (the build's own write set); tolerate a leftover draft so re-promotion is not wedged. - finalize: assert the PR author is cozystack-ci[bot], and run on an ephemeral oracle-vm runner (needs skopeo/helm). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
4f2530f9eb |
fix(release): scope the nightly-mirror host rewrite to the collected depth
collect_refs scans "$TREE"/*/*/values.yaml (depth-2), but the host rewrite used `find "$TREE" -name values.yaml` (any depth). A values.yaml nested deeper than group/pkg would then have its image host rewritten to GHCR without its image ever being mirrored there, leaving a dangling ref. Rewrite over the same depth-2 glob so the rewritten set is exactly the mirrored set (this also matches the existing --dry-run message). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
d0740ed8c7 |
test(etcd): gate S3 backup round-trip behind ETCD_E2E_S3_ROUNDTRIP
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
011d6d1639 |
fix FUs
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
f2866edcf0 |
review fixes
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Timofei Larkin
|
5f76ac0d76 |
test(etcd): capture backup Job Pod logs, fail fast on terminal phase
The backup round-trip test could wedge for 20 min and then report a bare EtcdSnapshot phase=Started with no root cause: dump_diagnostics dumped the (healthy) EtcdCluster and the operator log, but the snapshot agent runs in its own Job Pod, so the etcd-read / S3-upload error lives only in that Pod's log and in the BackupJob/EtcdSnapshot status. - dump_diagnostics now dumps the whole backup chain: BackupJob/Backup and EtcdSnapshot describe, each snapshot Job's describe + Pod logs (selected by the operator's etcd-operator.cozystack.io/cluster label), and recent namespace events (the primary signal when a hung Pod logs nothing). - wait_for_field gains an optional terminal-failure value; steps 04/05 pass Failed so the BackupJob/RestoreJob waits return immediately on a terminal phase instead of polling to the timeout — which also keeps the snapshot Job Pod within its 600s TTL so its log is still readable at dump time. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Timofei Larkin
|
bd1c8d1070 |
test(etcd): gate contracts test on readyMembers=3 not Available
The v1alpha2 operator sets Available=True early (reason ClusterDiscovered) as soon as the seed member forms the cluster, then scales the remaining members up one at a time. Gating the runtime-contracts test on Available raced the member-Pod count assertion and saw only 1/3 Pods, failing the run. Gate on status.readyMembers=3 instead — the count of members whose EtcdMember Ready condition is True (and the /scale status path), which only reaches 3 once every member Pod is Running and serving. Bump the wait timeout to 300s to cover sequential one-at-a-time rollout. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Andrey Kolkov
|
80ad7048e0 |
test(etcd): select member Pods by strict operator labels in e2e reads
Address review feedback from coderabbitai on hack/e2e-apps/etcd.bats: read the container name, metrics port, and Pod IP with the strict member selector (name+instance+managed-by) plus a Running-phase filter instead of the broad app.kubernetes.io/name=etcd, so .items[0] cannot resolve to a non-member or not-yet-running Pod. Also assert a non-empty Pod IP before the metrics probe. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
087a8c4be9 |
chore(etcd-operator): bump to v0.5.2
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
cc5d688e15 |
feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Ivan Okhotnikov
|
57fc2cd566 |
fix(e2e): retry kubectl wait on transient management-etcd errors
kubernetes-previous.bats intermittently fails on `kubectl wait deploy --for=condition=available` with `Error from server: etcdserver: leader changed` even when the target Deployment is milliseconds away from becoming Ready. Observed in run 28772259934 on head `5f0bec622`: three of the four deployments already reported `condition met`, the fourth (the Kamaji tenant CP Deployment) was ~200ms from Ready per the crust-gather snapshot captured 20s later (`readyReplicas: 2`), and the wait exited non-zero on the FIRST etcd-side error it saw on the watch channel. Cause: the e2e sandbox management cluster is a 3-node kind on Talos VMs with a 3-instance etcd HA cluster. By the time kubernetes-previous starts (after kubernetes-latest, kubernetes-oidc-customconfig, kubernetes-oidc-system, plus the storageclass-fallback probe), accumulated CDI+DRBD IO pressure and lingering "still attached to node" PVCs from prior tests push etcd fsync latency high enough to trip heartbeat timeouts on the leader. The election that follows resolves in seconds, but `kubectl wait` sees the server-side error mid-watch and exits — clients would normally reconnect, but `wait` treats a watch-channel error as terminal. Wrap the four management-cluster `kubectl wait` calls in run_kubernetes_test with a retry helper that: - Retries only on a curated allowlist of server-side transients (`etcdserver: leader changed`, `etcdserver: request timed out`, `the server was unable to return a response in the time allotted`). - Never swallows legitimate errors (NotFound, --timeout expired, permission denied, etc.). - Caps at 3 attempts with a 5s backoff so a persistently-broken etcd still surfaces loudly. Untouched: - `kubectl wait namespace tenant-test --timeout=20s` at line 308 — short-lived, hits mgmt but the object always exists ~instantly. - `kubectl wait deployment ... --kubeconfig tenantkubeconfig-...` at line 509 — waits on the TENANT apiserver's own etcd, a different failure domain (single-instance kine or CNPG-backed via Kamaji, not the 3-instance mgmt etcd that flapped). - Other bats files with mgmt-cluster `kubectl wait` — out of scope for this PR; if the same flake reappears there a follow-up can lift the helper into a shared library. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
5f0bec6222 |
test(e2e): guard oidc-system grep against leading-dash pattern
The audience-list assertion used \`grep -qF "- tenant-test-..."\`. Because the pattern begins with a dash, grep treats the leading \`-\` as an option flag, aborting the test with \`grep: invalid option -- ' '\` before the actual match ever runs. Insert an explicit \`--\` end-of-options separator so the YAML list-item pattern is passed as a pattern rather than a flag. Custom-config counterpart is unaffected — its grep patterns do not start with a dash. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
b637989fb0 |
test(e2e): port OIDC bats to cozytest lifecycle conventions
cozytest.sh (the e2e runner) is not real bats — it never invokes
setup()/teardown(). The OIDC files were structured around those hooks
to seed TEST_NAME and cleanup the tenant CR between tests, so with
cozytest strict-mode wrapping (\`set -eu\`), the first \${TEST_NAME}
expansion inside an @test aborted with 'parameter not set' before the
apply could even run.
Rework both files to the convention documented in external-dns.bats:
seed TEST_NAME at file scope (no per-test uniqueness needed — the
tenant-test namespace is single-suite), keep cleanup_kr() as a shared
helper, add a file-level cozy_cleanup() wrapper (the hook cozytest
actually calls on suite exit / first-failure), and move the pre-test
CR delete inline at the top of the first @test that creates the CR.
Subsequent @tests only read state and rely on the CR persisting until
cozy_cleanup.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
5325468c96 |
fix(apps/kubernetes,e2e): address bot-review findings on Phase 1 rewrite
Three items from the gemini-code-assist and coderabbitai passes that
survived the Phase-1 redesign:
- `oidc-rbac-job.yaml` (gemini HIGH): pre-delete cleanup uses
`kubectl --request-timeout=30s` in addition to `--timeout=30s`
on the delete verb, so an unreachable tenant apiserver cannot
stall the pre-delete hook on the HTTP round-trip itself. The
`|| echo WARNING` fallback still keeps a truly unreachable
cluster from blocking `helm uninstall`.
- `oidc-keycloak.yaml` + `_helpers.tpl` (gemini LOW): switch
`_cluster.{oidc-enabled,root-host}` reads to `dig` with an empty
map default so a missing `_cluster` key nil-safely emits an
empty string and hits the platform-OIDC guard instead of a
template panic on `.Values._cluster`.
- `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`
(coderabbit MAJOR): add an existence-poll `timeout ... until
kubectl get secret ... ; do sleep ...; done` before the
`kubectl wait secret --for=jsonpath` line. Without it a runner
that reaches the wait before the HR has rendered the Secret
gets an immediate `NotFound` and the test flakes.
Adds one helm-unittest case covering the nil-safe `_cluster` path.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
0247a17d33 |
test(e2e): add CustomConfig bats coverage; align filenames with design
Self-review fix: the design proposal explicitly lists TWO bats files: `kubernetes-oidc-system.bats` and `kubernetes-oidc-customconfig.bats`. The first revision shipped only one combined file; this commit splits them. The CustomConfig bats covers what System cannot: the operator-supplied AuthenticationConfiguration is plumbed onto the KamajiControlPlane, the resulting Secret carries the BYO issuer (and emphatically NOT a `cozy` issuer), nothing lands in the `cozy` realm, and no OIDC kubeconfig Secret is pre-rendered (the operator distributes their own). The System bats header is updated to call out the deferred parts (in-tenant CRBs / view-vs-admin RBAC behavior / browser-flow login) that need a full Kamaji bringup, which is already covered for the cluster path by `kubernetes-latest.bats`. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
d13e2c586f |
fix(apps/kubernetes): align OIDC mount + path with the design spec
Self-review fix: the design proposal specifies `name: authentication-config` and `mountPath: /etc/kubernetes/authentication-config`, but the previous revision had renamed both to `cozy-oidc-authn-config` and `/etc/kubernetes/cozy-oidc` to dodge a collision with the volume name used in `apiserver_authentication_test.yaml`. That test was demonstrating the GENERIC user-supplied `extra*` passthrough — not the OIDC use case — so rename the test fixture instead and restore the design-spec names on the chart side. The reservation guard against user-supplied volumes reusing `authentication-config` only fires when `spec.oidc.mode` is not None, so clusters with OIDC off retain the previous freedom to name their own volumes anything. Renames in `apiserver_authentication_test.yaml`: the user-supplied volume name `authentication-config` becomes `tenant-byo-mount`; the Secret name `tenant-authentication-config` and the apiserver flag `--authentication-config=` are untouched. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
48fb4fe520 |
fix(apps/kubernetes): release-own the OIDC SA/Role/RoleBinding + pre-render the kubeconfig Secret
Self-review fixes for four linked lifecycle issues:
- The post-install Job created `<release>-oidc-kubeconfig` via
`kubectl apply` from inside the script; the resulting Secret had
no Helm release labels and was orphaned on `helm uninstall`. Now
a release-owned placeholder Secret is rendered up front in
`oidc-kubeconfig-secret.yaml` and the Job patches its `data`
field at runtime (the tenant CA still has to come from the
Kamaji-issued admin kubeconfig, which appears asynchronously).
- The ServiceAccount/Role/RoleBinding for the bootstrap Job were
hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
so they leaked on `helm uninstall`. They are now regular release
resources (no hook annotations) and are reaped by `helm uninstall`
alongside everything else. Both the post-install bootstrap Job
and the pre-delete cleanup Job find a working SA.
- The previous design tried to add `hook-succeeded` to the SA, but
that would delete the SA right after the bootstrap hook
succeeded — leaving the pre-delete cleanup Job with no SA to
schedule. Release-owned resources sidestep this race entirely.
- The Role had `verbs: ["create"]` on `secrets` without
`resourceNames` (Kubernetes RBAC cannot scope `create` by name).
With the placeholder Secret pre-rendered the Job only needs
`get`/`patch` on the specific name — the broad `create` rule is
removed.
Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.
Adds three helm-unittest cases for the placeholder Secret across all
three modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
daf8009c5e |
test(e2e): add Phase-1 OIDC render-side bats coverage
Validate on a live cluster that:
- the new `spec.oidc.*` fields are admitted by the Kubernetes CRD;
- cozystack-api accepts the updated schema and renders a HelmRelease;
- the KamajiControlPlane gains `--authentication-config` and the
chart-owned `cozy-oidc-authn-config` mount and volume;
- the AuthenticationConfiguration Secret carries the cozy realm
issuer URL and the per-cluster audience;
- per-cluster `KeycloakClient` + audience `KeycloakClientScope`
land in the cozy realm (with the EDP operator present).
The full end-to-end RBAC flow (Job → CRBs inside the tenant cluster)
needs a Kamaji bringup, which kubernetes-{latest,previous}.bats
already cover for the cluster path; the OIDC browser flow is
explicitly out of scope per the design proposal. This file covers
only what the render-side regression budget can pay for.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
5db84e7c9c |
fix(hack): refuse to move an existing stable image tag
Address review feedback from coderabbitai on hack/promote-retag.sh: the loop
copied each rc digest onto ${repo}:${STABLE} unconditionally, so a partial
prior run or a manual push could mutate already-released stable bytes before
the GitHub release/tag guards catch it. Inspect the destination first — no-op
when it already points at the rc digest (idempotent re-run), fail when it
points elsewhere. :latest stays mutable.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
4777273dbd |
feat(release): add nightly OCIR->GHCR image mirror script
hack/nightly-mirror.sh copies the latest main build's cozystack-owned component images from the CI build registry (OCIR) to the public release registry (GHCR) by digest — no rebuild — then rewrites the baked package tree's image hosts so the GHCR install closure is self-contained. It reuses promote-retag.sh's three-shape ref enumeration, filters to cozystack-owned refs, and deliberately excludes the cozystack-packages artifact (rebuilt downstream from the rewritten tree). Dry-run bats test under the cozytest harness guards the ref selection, the cozystack-packages exclusion, third-party filtering, and the host rewrite plan. shellcheck clean. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
a49ee8c42e |
fix(release): pin promote-retag dry-run test to the default REGISTRY
The CI workflow exports REGISTRY=<OCIR build registry> for every job (.github/workflows/pull-requests.yaml). The dry-run test inherited it, so promote-retag.sh filtered the tree's digest refs by the OCIR registry — but the committed release tree vendors them under ghcr.io/cozystack/ cozystack (the script default), so the selector matched nothing and aborted with 'No cozystack-owned digest-pinned image refs found', failing the Unit & controller tests job. Run the script with 'env -u REGISTRY' so it uses its default registry — the one the tree's refs and this test's assertions actually use — and add a slim self-diagnosing fallback that dumps the script's stderr and the yq build on any unexpected non-zero exit. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
20c0806325 |
test(release): port promote-retag bats to the cozytest harness
hack/promote-retag_test.bats was written for upstream bats (run, $status, $output, setup(), skip), but CI runs hack/*.bats through hack/cozytest.sh, a pure-shell runner that has none of those. The first 'run' therefore died with 'cozytest.sh: run: not found', failing the Unit & controller tests job. Rewrite both cases in the cozytest idiom already used by select-e2e_test.bats and check-host-runtime.bats: redirect to temp files, capture the exit code with '|| rc=$?' so the harness's set -e does not abort the expected-failure case, and assert with plain shell. The selector logic under test is unchanged. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
31ea4d4552 |
fix(release): retag only cozystack-owned images on promotion
collect_refs scraped every @sha256 ref from the package values.yaml, including third-party images (docker.io/clastix/kubectl, ghcr.io/kvaps/...), bare upstream tags (kube-ovn/keycloak/kilo) and a "--migrate-image=..." arg string. The first skopeo copy to a registry CI cannot push to aborted the whole promotion under set -e, so stable promotion could not complete. Filter the canonical refs to those under $REGISTRY (default ghcr.io/cozystack/cozystack, matching common-envs.mk; overridable for forks). Also require skopeo only when not --dry-run, so the selection is testable without it. Add hack/promote-retag_test.bats: a --dry-run over the real packages tree must emit only cozystack-owned refs (zero third-party), and a REGISTRY override that matches nothing must fail rather than promote the wrong set. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
020e76a842 |
fix(hack): verify mikefarah yq in promote-retag
Address review feedback from gemini-code-assist on hack/promote-retag.sh:27: command -v yq only confirms some yq is on PATH; the script's queries use mikefarah syntax, so reject python-yq and other variants by checking the version string, mirroring the Makefile build-deps check. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
328394a873 |
fix(hack): make promote-retag ref_repo port-safe
Address review feedback from gemini-code-assist on hack/promote-retag.sh:48: ref_repo now strips the :tag from the last path component only, so a registry host carrying a :port (e.g. localhost:5000/cozystack/operator) on a no-tag ref is preserved instead of truncated to the host. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
9336641b95 |
ci(release): promote rc to stable by retag, no rebuild
Add the rc->stable promotion path. A stable release is a renamed release-candidate: the bytes shipped as vX.Y.Z are bit-for-bit the bytes built and e2e-tested as vX.Y.Z-rc.N. - hack/promote-retag.sh: reads the digest-pinned image refs the rc baked into packages/*/*/values.yaml (single-string, split repository/tag/digest, and the cozystack-packages OCI artifact) and `skopeo copy`s each — by digest — to the stable tag and :latest. Copying by digest guarantees the stable image is identical to the rc; the result is verified with `skopeo inspect`. Validated offline with `sh -n` and the yq extraction against the current tree; the skopeo path is CI-validated only. - promote-rc.yaml: workflow_dispatch(rc_tag). Validates the rc exists and the stable tag does not, retags via the script, rewrites the cosmetic -rc.N substring in the vendored tags to the stable version (the @sha256 wins regardless), restamps the version-stamped assets (operator manifests, cozypkg, openapi; the heavy Talos assets are copied from the rc draft), and opens a release-X.Y.Z PR. Merging it reuses the existing release-PR e2e + pull-requests-release.yaml finalize to cut the write-once stable tag. Squash is disallowed on the PR (the stable tag needs a real merge commit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
35025cf5f0 |
test(e2e): capture LINSTOR satellite ErrorReports in cozyreport (#3209)
## What this PR does Extends `hack/cozyreport.sh` linstor module to copy `/var/log/linstor-satellite/ErrorReport-*.log` off every `linstor-satellite` pod into `linstor/error-reports/<node>.tgz` in the cozyreport archive. ### Why The linstor-Satellite pod stdout log references its error reports only by ID: ``` [DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to create zfsvolume [Report number 6A4A394E-E4EA0-000000] ``` The report body (actual `zfs create -V …` stderr, DRBD adjust output, storage-driver stack trace, exit code) lives on the satellite pod's ephemeral filesystem at `/var/log/linstor-satellite/ErrorReport-*.log`. crust-gather only captures pod stdout, so the report body is lost when the e2e sandbox is torn down at end-of-job. Consequence for post-mortem of `ResourceExhausted` / `Not enough available nodes` CreateVolume retry loops (see e.g. PR #3044 CI history): cozyreport shows only the driver-level symptom. Root cause distinction (out-of-space vs dataset name conflict vs ZFS module error vs kernel signal) is not possible from the artefact alone, and the sandbox is unrecoverable by that point. ### Scope `hack/cozyreport.sh` only, +32 lines under the existing `linstor module` block. Guarded so: - a satellite with zero reports (healthy case) leaves nothing in the tree - a transient exec / tar failure never fails the surrounding cozyreport run - empty parent directory is pruned ### Verified on a live cluster - Label selector `app.kubernetes.io/component=linstor-satellite` matches all 3 satellite pods on the dev cluster used for verification. - Bundle for one active satellite comes out at ~60 KiB (dozens of historical reports tarred). No product-side change, no dependency on capture happening during a specific test phase; runs as part of the existing end-of-job cozyreport collection. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced offline LINSTOR diagnostics collection by adding an error-report index and bundling controller and satellite error-report logs into downloadable archives. * Generates per-node satellite bundles using stable node naming for clearer organization of collected diagnostics. * **Bug Fixes** * Avoids creating empty or invalid error-report archives and automatically cleans up unused bundles to prevent clutter in the collected output. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Ivan Okhotnikov
|
287dbca600 |
test(e2e): wait for LINSTOR pool free capacity between tenant Kubernetes tests
kubernetes-previous.bats intermittently fails with `zfs create -V ... returned with exitcode 1: cannot create '...': out of space` on a satellite even though the LINSTOR pool measured 30+ minutes later shows 100+ GiB free per satellite. Root cause proven from a satellite ErrorReport captured in the cozyreport artefact (PR #3044 run 28751310913, report id 6A4AADFD-349B2-000000): the previous tenant Kubernetes test's cozy_wait_tenant_drained gate returned as soon as its PVCs disappeared from the API, but the satellite-side ZFS `zvol destroy` for the just-deleted replicated (autoPlace=3, DRBD) 20 GiB worker disks lagged tens of seconds behind that signal as DRBD adjusted, unref counts drained, and ZFS batch-destroyed the datasets. The next tenant test's CreateVolume then landed inside that window against a pool that had not yet reclaimed the space, LINSTOR-CSI retried autoplace against its own still-being-torn-down previous placement, and worker Machine bringup stretched past the MHC nodeStartupTimeout budget with no cluster-side fault. Add a companion `cozy_wait_linstor_pool_free` helper that polls the LINSTOR controller for the smallest FreeCapacity across every ZFS satellite pool and blocks until it reaches a per-satellite threshold (default 90 GiB — covers the ~82 GiB worst-case footprint of two 20 GiB replicated worker targets across all three satellites plus two 21 GiB CDI scratch PVCs landing on the same node via the local storageClass). Called from cozy_cleanup immediately after cozy_wait_tenant_drained. Bounded and best-effort: 300 s timeout, `|| true` at the call site, warning printed with the observed pool state if the threshold is not reached in budget so the next test does not misdiagnose the pressure. The threshold is a floor for the concrete failure mode observed in the E2E suite; the surrounding kubernetes-latest -> kubernetes-previous handoff is the only run in the current suite that immediately follows a heavy tenant teardown with another tenant provisioning, so this gate is the natural place for the check. E2E-scope only, no product or chart change. The proximate observation (`zfs create out of space`) surfaced because of PR #3209 which taught hack/cozyreport.sh to bundle satellite ErrorReports; without that artefact the root cause was ambiguous between multiple plausible LINSTOR internal races. Ships alongside that PR. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
d5098d9430 |
test(e2e): capture LINSTOR satellite ErrorReports in cozyreport
The linstor-Satellite pod stdout log references its error reports
only by ID:
[DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to
create zfsvolume [Report number 6A4A394E-E4EA0-000000]
The body of that report (which carries the actual `zfs create -V …`
stderr, the DRBD adjust output, the storage-driver stack trace, and
the exit code) lives at `/var/log/linstor-satellite/ErrorReport-*.log`
on the satellite pod. crust-gather only captures pod stdout, so the
report body stays on the pod's ephemeral filesystem and is lost when
the e2e sandbox is torn down at end-of-job.
Consequence for post-mortem: a run that failed on a
`ResourceExhausted` / `Not enough available nodes` CreateVolume retry
loop shows only the driver-level symptom in cozyreport. Root cause
attribution (out-of-space, dataset name conflict, ZFS module error,
kernel signal) is not distinguishable without the report body, and
without the sandbox is unrecoverable.
Extend the linstor module in `hack/cozyreport.sh` to tar the
`ErrorReport-*.log` set off every `linstor-satellite` pod into
`linstor/error-reports/<node>.tgz`. Guarded so a satellite with zero
reports (the healthy case) leaves nothing in the tree, and a
transient exec / tar failure never fails the surrounding cozyreport
run.
Verified against a live dev cluster: label selector
`app.kubernetes.io/component=linstor-satellite` matches all three
satellite pods, and the tar bundle for one active satellite comes
out at ~60 KiB. No product-side change.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
3ab9f95625 |
test(e2e): bump sandbox tenant storage quota to 200Gi
The E2E sandbox tenant (tenant-test) provisions back-to-back tenant
Kubernetes control planes via kubernetes-latest.bats and
kubernetes-previous.bats. Each control plane brings up two worker VMs
whose 20Gi root disks are populated by CDI, and CDI's second-phase
importer allocates a scratch PVC of the same size before the qcow2
unpack. Peak per-tenant footprint is therefore ~82Gi during import
(2 × 20Gi target + 2 × 21Gi scratch), collapsing to 40Gi at rest.
kubernetes-latest teardown deletes the tenant Kubernetes CR and waits
via cozy_wait_tenant_drained for the KubeVirt VMs, VMIs and namespace
PVCs to disappear. The API-level PVC delete returns quickly, but the
LINSTOR CSI detach + DRBD resource release on the underlying nodes
lags past the 300s drain budget. During that window, the deleted
PVCs still count against ResourceQuota's requests.storage because
the storage-provisioner has not finished releasing the volume — so
kubernetes-previous starts its own CDI import with 40Gi of stale
kubernetes-latest quota still charged, and CDI's scratch PVC create
for the second worker (used=80Gi + scratch=21Gi = 101Gi) is
rejected by the tenant-quota admission check at 100Gi. The second
worker's DataVolume never leaves ImportInProgress, the tenant node
never joins, and the 12-minute node-join deadline in
run-kubernetes.sh trips kubernetes-previous.bats.
Raise the sandbox tenant-quota storage ceiling to 200Gi so the
overlap between one test's slow storage cleanup and the next test's
CDI import fits inside the quota. The failing-run diagnostic is
cozyreport artifact from run 28730757213:
cdi-deployment log:
scratch PVC API create errored: … forbidden: exceeded quota:
tenant-quota, requested: requests.storage=22763536384,
used: requests.storage=85899345920,
limited: requests.storage=100Gi
This is a test-infrastructure fix; the 100Gi default in the tenant
chart itself is unchanged. The assertion below the CR update is
lifted from 100Gi to 200Gi in lockstep.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Aleksei Sviridkin
|
d89ad70698 |
fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex
The Gateway (cozystack-gateway-hostname-policy) and Route (cozystack-route-hostname-policy) admission policies constrain tenant hostnames on the Gateway API path. Gateway API is opt-in and off by default, so in the default configuration tenant applications publish through a legacy networking.k8s.io/Ingress on the shared ingress-nginx, which had no equivalent policy. Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[] is allowed when it is within the namespace's own apex (namespace.cozystack.io/host), or when it is a concrete (non-wildcard) name entirely outside the platform root apex (_cluster.root-host); a hostname under the platform root apex but outside the namespace's own apex, and a wildcard taken through the outside-root path (which could match the platform apex itself), are rejected. The outside-root branch keeps the kubernetes app's Proxied exposeMethod working, since it routes a user-supplied external domain to a nested cluster. A rule with no host (absent or empty, an unbounded catch-all) and spec.defaultBackend are also rejected. The policy templates the platform root apex into its CEL and renders only when _cluster.root-host is set, so it is never shipped in a state that would admit every hostname. A tenant-* namespace missing its own apex label is denied. Cover the policy with helm-unittest (render, including the root-host, empty-host, and wildcard guards) and behavioral admission tests against the live apiserver (in-apex allow, external-domain allow, own-apex-wildcard allow, external-tls allow, under-root-outside-apex deny, prefix-adjacent-under-root deny, wildcard-matching-apex deny, hostless deny, empty-string-host deny, defaultBackend deny), and document it as an additional layer in the gateway security model. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Алексей Артамонов
|
f8e4222b7d |
test(mariadb): fix operator MariaDB CR name in single-replica failure dump
The failure-path dump queried mariadbs.k8s.mariadb.com with $name, but the
operator CR is named after the Helm release ({{ .Release.Name }} = mariadb-$name),
so the dump would hit NotFound instead of showing the rejected CR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
|
||
|
Алексей Артамонов
|
c7e8f4d975 |
fix(mariadb): re-guard replication so single-replica passes the operator webhook
PR #2279 made `replication.enabled: true` unconditional so the operator would always create -primary/-secondary services. But the mariadb-operator validating webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1, so single-replica MariaDB could no longer be created — the HelmRelease never became Ready. Re-guard replication on replicas>1 and teach the service-name consumers about the bare <release> service the operator provisions in both topologies: - mariadb.yaml: gate `replication` on replicas>1; restore single-replica external access via `service.type: LoadBalancer` and keep `primaryService` LoadBalancer for replicas>1 only. - backup-cronjob.yaml: route single-replica dumps to the bare service (no -secondary exists without replication). - dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus -primary/-secondary when replicas>1, matching the ApplicationDefinition. - mariadb-rd: list the bare service in services.include so single-replica installs are visible in the dashboard. Adds a single-replica e2e test that reproduces the webhook rejection and guards against regression. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Aleksei Sviridkin
|
a40be5e212 |
fix(e2e): widen tenant-root monitoring bring-up waits to 15m under load (#3160)
## What this PR does The `Configure Tenant and wait for applications` install test brings up the entire root-tenant monitoring stack and waits for each component to become ready/operational. Those waits span the VictoriaMetrics operator resources (`vmalert`/`vmalertmanager`, `vlclusters/generic`, `vmcluster/shortterm`+`longterm`) and the Grafana pair (`grafana-db` CNPG cluster, `grafana-deployment`). Each converges only once its underlying workloads schedule and turn Ready, and during platform bring-up they contend for the same node resources as the rest of the install — so their convergence is load-sensitive. Only `vmalert`/`vmalertmanager` used a 15m budget; the other four used 10m, so this block carried a 10m/15m split even though all its resources contend for the same node capacity during install. Under install-time load (multiple concurrent e2e sandboxes on one runner) the monitoring bring-up is slow across the whole block rather than on a single resource, so the shorter budgets risk failing the whole install on PRs whose branches never touched monitoring — and per-resource bumps are whack-a-mole: raising one budget just moves the risk to the next component in the block. On a calm sandbox each of these reaches ready/operational in under a second, which is what distinguishes the two cases — slow-not-stuck convergence under contention, not a product regression. This lifts the whole tenant-root monitoring wait-set to a single uniform 15m budget and wraps each wait in an on-timeout diagnostic block that dumps the resource YAML (status conditions / `updateStatus`) plus tenant-root pod state. A genuinely stuck resource still fails — now at 15m, and with legible diagnostics instead of a bare `timed out` line — so this does not weaken the gate's ability to catch real breakage. The extra budget costs near-zero wall-time on the happy path and stays well inside the E2E job's 180m cap. The remaining upstream angle is the real driver: how long the VictoriaMetrics/VictoriaLogs stack and the Grafana bring-up take to become operational under resource contention during install. This change addresses the e2e robustness side; reducing that bring-up cost is a separate, product-side improvement. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved and standardized readiness checks during tenant setup so application bring-up waits behave more consistently. * Enhanced failure handling to provide clearer diagnostics (including resource details and pod status) when readiness checks time out. * **Tests** * Standardized longer readiness wait timeouts for multiple application components to reduce flakiness during e2e runs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
cd68d5efdc |
fix(e2e): drain tenant VMs/PVCs between kubernetes tests to fix the node-join flake (#3159)
## What this PR does Fixes the dominant flake in the tenant-Kubernetes e2e suite, where a run fails on worker-node-join under sandbox memory/storage pressure. The two heavy Kamaji tenant tests — `kubernetes-latest` then `kubernetes-previous` — run back-to-back in one CI sandbox, each provisioning a tenant cluster with KubeVirt worker VMs. The inter-test cleanup hook (`cozy_cleanup` in `hack/e2e-apps/run-kubernetes.sh`) deletes the tenant `Kubernetes` CR and waits only for the CR to disappear. That wait finalizes as soon as the CR's finalizers clear, which merely *triggers* teardown of the CAPK worker VMs and their DataVolume-backed disk PVCs — it does not await it. The virt-launcher pods keep their guest RAM reserved until the VMIs are gone, so the next tenant test's worker VMs begin scheduling against a sandbox the previous tenant has not yet vacated. Under the constrained sandbox this starves memory, a worker VM misses the node-join budget, and the MachineHealthCheck cannot remediate when both workers are unhealthy — so the run flakes on worker-node-join. This adds a bounded drain barrier to `cozy_cleanup`: after deleting the CR it now blocks until the tenant's VirtualMachines, VirtualMachineInstances and disk PVCs have actually drained from the `tenant-test` namespace, so the next test starts on a freed sandbox. The wait is bounded (5m) and best-effort: cozytest already runs `cozy_cleanup` wrapped in `|| true`, and the waiter returns loudly (warning plus a leftover dump) on timeout, so a stuck teardown can never hang the job past the deadline — it leaves the sandbox no worse than before. The probe is namespace-scoped: `tenant-test` is provisioned with etcd/monitoring/seaweedfs disabled so it carries no baseline PVCs, and the e2e apps run sequentially each cleaning up after themselves, so the only VMs/VMIs/PVCs present belong to the tenant being torn down. The pure drain exit-condition (`cozy_tenant_drained`) is factored out and unit-tested in `hack/run-kubernetes-drain_test.bats` (runs under `make unit-tests`), including the invariant that a failed probe's `err` sentinel is never misread as a completed drain. ### Screenshots Not applicable — e2e harness change, no UI. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes test cleanup so tenant resources are fully drained before the next test starts, reducing leftover VMs, VM instances, and PVCs. * Added more reliable timeout handling and clearer diagnostics when teardown or node readiness checks fail, making flaky test issues easier to identify. * **Tests** * Added coverage for tenant-drain detection to verify empty, whitespace-only, multi-resource, and error scenarios behave as expected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |