mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
ci/cozystack-api-pin-securitygroup
4527
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrei KvapilandClaude
|
21ee665585 |
fix(cozystack-api): pin the image to the build that ships the SecurityGroup API
#2922 added the sdn.cozystack.io SecurityGroup API to cozystack-api, but main's committed image is still the v1.5.0 release, which predates it — the new build was only pinned transiently inside #2922's own E2E and never committed. As a result every PR that does not itself rebuild cozystack-api installs the old apiserver, so `kubectl get securitygroup` returns "the server doesn't have a resource type" and the securitygroup / serviceexposure e2e tests fail on unrelated PRs. Pin cozystack-api to the current main-HEAD build that carries the API (image mirrored to ghcr from the #2922 build). Stopgap until a release re-pins it through the normal build. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
a31a73cb87 |
fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness (#3141)
## What this PR does The tenant `etcd`, `ingress` and `monitoring` HelmReleases create VictoriaMetrics custom resources — `VMPodScrape` directly in `etcd` and `ingress`, and `VMCluster` / `VMAlert` / `VLCluster` / `VMAlertmanager` / `VMServiceScrape` via the `monitoring-system` child that `monitoring` creates. All of these are intercepted by the `victoria-metrics-operator` validating webhook, which is configured with `failurePolicy: Fail`. On a cold install the operator pod can be briefly unavailable on `:9443` (for example a transient API-server blip during startup leaves the webhook backend not yet serving). In that window the tenant releases reconcile, the webhook rejects their VM* resources with `connection refused`, and the releases churn through failed-install → rollback → retry. They carry `install.remediation.retries: -1` and recover eventually, but the churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. This adds a Flux `spec.dependsOn` from each of those three tenant releases to the `victoria-metrics-operator` HelmRelease in `cozy-victoria-metrics-operator`, so they wait for the operator to be Ready before creating any VM* resource. The system monitoring components already gate on the operator the same way (a variant-level `dependsOn` in the platform package sources); this brings the tenant releases to parity. `failurePolicy: Fail` is preserved — validation is unchanged. The tenant releases are sharded (`sharding.fluxcd.io/key`) while the operator release is not. helm-controller resolves `dependsOn` through an uncached API read that bypasses the shard's watch-label-selector cache (since v1.1.0, PR fluxcd/helm-controller#1070), so the cross-shard reference is visible to the tenant shard's controller; this repo ships helm-controller v1.5.0. A helm-unittest suite pins the gate on all three releases and asserts that the tenant releases creating no VM* resources (`seaweedfs`, `gateway`) do not carry it, documenting the intentional narrowness of the gate. ### Release note ```release-note fix(tenant): gate tenant etcd, ingress and monitoring HelmReleases on victoria-metrics-operator readiness so they no longer fail to install when the operator's admission webhook is briefly unavailable on a cold install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved tenant installation reliability by adding readiness gating: etcd, ingress, and monitoring Helm releases now wait for the Victoria Metrics operator to become Ready before applying dependent resources. * Prevents install/rollback failures caused by temporary admission webhook unavailability during cold starts. * **Tests** * Added Helm-unittest coverage to confirm the dependency gate is present only for templates that create VM* resources (and omitted for templates that don’t). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
03435dcc50 |
fix(flux): tune source-watcher concurrency + CPU to unblock platform install under load (#3163)
## What this PR does `source-watcher` is the Flux controller that turns each `PackageSource` into an `ExternalArtifact` (one `ArtifactGenerator` per source, ~100 builds on the management cluster during install). The `cozystack-platform` HelmRelease blocks until all of those artifacts are Ready, so source-watcher's throughput sits on the install's critical path. It was the only Flux controller left without concurrency/CPU tuning. In the bootstrap all-in-one pod it ran on the binary default (`--concurrent=10`) with the shared 100m CPU request; in the production `FluxInstance` it was absent from the reconciler patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. A failed run's diagnostics show the shape of this: the `cozystack-platform` HelmRelease timed out with `timeout waiting for: PackageSource/… status: 'InProgress'` (a run-to-run varying tail — e.g. `bucket-application`, `linstor-gui`, `capi-provider-core`), while its ~200 ExternalArtifacts reached Ready in a staggered trickle spread over ~60 seconds rather than together. That staggered completion of independent per-artifact builds is the signature of a throughput-limited builder, not a single all-or-nothing pull. This raises source-watcher to `--concurrent=20` in both install paths: - **Bootstrap all-in-one pod** (`internal/fluxinstall/manifests/fluxcd.yaml`, regenerated from `flux-aio.cue` + the `flux-aio` Makefile): `--concurrent=20` plus a 500m CPU request. The AIO pod sets no CPU limits, so the request is what drives source-watcher's proportional CPU share when the node is saturated. - **Production `FluxInstance`** (`packages/system/fluxcd/values.yaml`): source-watcher is added to the shared reconciler patch, so it now gets the same `--concurrent=20`, `--requeue-dependency=5s` and 2000m/2048Mi limits as the other tuned controllers. The `flux-aio` module exposes a per-controller resources override for the watcher (set via CUE) but no args override, so the bootstrap `--concurrent` bump is a guarded post-render patch in the Makefile; the embedded manifest is the product of that pipeline. source-watcher's support for both `--concurrent` and `--requeue-dependency` was confirmed against its `cmd/main.go`, and the production `op: replace` on `resources/limits` is safe because the distribution ships source-watcher with a `resources.limits` block (same shape as source-controller). Optional follow-up, intentionally left out to keep this focused on the root cause: if the artifact-build tail still brushes the 10-minute platform HelmRelease timeout under load, the installer's HelmRelease install/upgrade timeout values give a safety margin. ### Release note ```release-note fix(flux): tune source-watcher concurrency and CPU so the platform install does not time out under load ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Increased the source watcher’s concurrency to improve throughput and better handle larger workloads. * Added higher CPU and memory settings for the watcher to support more stable operation under load. * **Bug Fixes** * Updated deployment patching so the watcher receives the same configuration adjustments as other controllers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a40be5e212 |
fix(e2e): widen tenant-root monitoring bring-up waits to 15m under load (#3160)
## What this PR does The `Configure Tenant and wait for applications` install test brings up the entire root-tenant monitoring stack and waits for each component to become ready/operational. Those waits span the VictoriaMetrics operator resources (`vmalert`/`vmalertmanager`, `vlclusters/generic`, `vmcluster/shortterm`+`longterm`) and the Grafana pair (`grafana-db` CNPG cluster, `grafana-deployment`). Each converges only once its underlying workloads schedule and turn Ready, and during platform bring-up they contend for the same node resources as the rest of the install — so their convergence is load-sensitive. Only `vmalert`/`vmalertmanager` used a 15m budget; the other four used 10m, so this block carried a 10m/15m split even though all its resources contend for the same node capacity during install. Under install-time load (multiple concurrent e2e sandboxes on one runner) the monitoring bring-up is slow across the whole block rather than on a single resource, so the shorter budgets risk failing the whole install on PRs whose branches never touched monitoring — and per-resource bumps are whack-a-mole: raising one budget just moves the risk to the next component in the block. On a calm sandbox each of these reaches ready/operational in under a second, which is what distinguishes the two cases — slow-not-stuck convergence under contention, not a product regression. This lifts the whole tenant-root monitoring wait-set to a single uniform 15m budget and wraps each wait in an on-timeout diagnostic block that dumps the resource YAML (status conditions / `updateStatus`) plus tenant-root pod state. A genuinely stuck resource still fails — now at 15m, and with legible diagnostics instead of a bare `timed out` line — so this does not weaken the gate's ability to catch real breakage. The extra budget costs near-zero wall-time on the happy path and stays well inside the E2E job's 180m cap. The remaining upstream angle is the real driver: how long the VictoriaMetrics/VictoriaLogs stack and the Grafana bring-up take to become operational under resource contention during install. This change addresses the e2e robustness side; reducing that bring-up cost is a separate, product-side improvement. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved and standardized readiness checks during tenant setup so application bring-up waits behave more consistently. * Enhanced failure handling to provide clearer diagnostics (including resource details and pod status) when readiness checks time out. * **Tests** * Standardized longer readiness wait timeouts for multiple application components to reduce flakiness during e2e runs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
cd68d5efdc |
fix(e2e): drain tenant VMs/PVCs between kubernetes tests to fix the node-join flake (#3159)
## What this PR does Fixes the dominant flake in the tenant-Kubernetes e2e suite, where a run fails on worker-node-join under sandbox memory/storage pressure. The two heavy Kamaji tenant tests — `kubernetes-latest` then `kubernetes-previous` — run back-to-back in one CI sandbox, each provisioning a tenant cluster with KubeVirt worker VMs. The inter-test cleanup hook (`cozy_cleanup` in `hack/e2e-apps/run-kubernetes.sh`) deletes the tenant `Kubernetes` CR and waits only for the CR to disappear. That wait finalizes as soon as the CR's finalizers clear, which merely *triggers* teardown of the CAPK worker VMs and their DataVolume-backed disk PVCs — it does not await it. The virt-launcher pods keep their guest RAM reserved until the VMIs are gone, so the next tenant test's worker VMs begin scheduling against a sandbox the previous tenant has not yet vacated. Under the constrained sandbox this starves memory, a worker VM misses the node-join budget, and the MachineHealthCheck cannot remediate when both workers are unhealthy — so the run flakes on worker-node-join. This adds a bounded drain barrier to `cozy_cleanup`: after deleting the CR it now blocks until the tenant's VirtualMachines, VirtualMachineInstances and disk PVCs have actually drained from the `tenant-test` namespace, so the next test starts on a freed sandbox. The wait is bounded (5m) and best-effort: cozytest already runs `cozy_cleanup` wrapped in `|| true`, and the waiter returns loudly (warning plus a leftover dump) on timeout, so a stuck teardown can never hang the job past the deadline — it leaves the sandbox no worse than before. The probe is namespace-scoped: `tenant-test` is provisioned with etcd/monitoring/seaweedfs disabled so it carries no baseline PVCs, and the e2e apps run sequentially each cleaning up after themselves, so the only VMs/VMIs/PVCs present belong to the tenant being torn down. The pure drain exit-condition (`cozy_tenant_drained`) is factored out and unit-tested in `hack/run-kubernetes-drain_test.bats` (runs under `make unit-tests`), including the invariant that a failed probe's `err` sentinel is never misread as a completed drain. ### Screenshots Not applicable — e2e harness change, no UI. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes test cleanup so tenant resources are fully drained before the next test starts, reducing leftover VMs, VM instances, and PVCs. * Added more reliable timeout handling and clearer diagnostics when teardown or node readiness checks fail, making flaky test issues easier to identify. * **Tests** * Added coverage for tenant-drain detection to verify empty, whitespace-only, multi-resource, and error scenarios behave as expected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6db5ae9c6f |
fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert, VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system child of monitoring) guarded by the victoria-metrics-operator validating webhook with failurePolicy: Fail. On a cold install the operator pod can be briefly unavailable on :9443, so these releases reconcile before the webhook backend is serving and are rejected with connection refused. They carry install.remediation.retries: -1 and would eventually recover, but the failed-install -> rollback -> retry churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. Add a Flux spec.dependsOn from each of these releases to the victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator, so they wait for the operator to be Ready before creating VM* resources. The system monitoring components gate on the operator the same way, via a variant-level dependsOn. The cross-shard reference (tenant releases are sharded, the operator release is not) resolves via helm-controller's uncached dependency check (APIReader, v1.1.0+). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b25a132747 |
fix(velero): add startupProbe so slow startup does not crashloop the install gate (#3138)
## What Add a `startupProbe` to the velero server container (and supply it through the package values), so a slow server start under a heavy parallel install no longer trips the liveness probe. Pin the probe with a helm-unittest regression guard. ## Why Velero binds its metrics/health endpoint (`:8085`, `http-monitoring`) only after the server loads its plugins and connects to the API server. Under a heavy parallel platform install that can take minutes, but the upstream liveness probe starts checking `/metrics` at `initialDelaySeconds=10` and kills the container after `failureThreshold=5` (~130s total) — before `:8085` is bound. The kubelet SIGTERMs velero (exit code 0 / Completed — a graceful kill, not a crash), it BackOff-restarts, and never escapes the loop inside the install-gate window. Because `backupstrategy-controller` hard-depends on velero, this blocks the install gate. ## How While a `startupProbe` is failing, the kubelet holds off the liveness and readiness probes entirely. So the startupProbe grants the server an explicit startup budget — `failureThreshold * periodSeconds = 30 * 10s = 300s` of grace — and then hands over to the unchanged, tight upstream liveness (`10s / 30s / 5`) for steady-state crash detection. This is strictly better than loosening liveness, which would also blunt steady-state detection for the lifetime of every pod. The upstream velero chart (12.0.3) does not render a `startupProbe`, so the package patches the deployment template via `patches/add-startup-probe.patch`, re-applied after re-vendor by the Makefile `update` target — the same vendored-patch convention used by `seaweedfs`, `ingress-nginx`, and the existing `startupProbe` patches in `grafana-operator` and `kubevirt-cdi-operator`. The `startupProbe` value itself is supplied through the package values override. ## Notes Surfaced as a recurring (not flaky) install failure under heavy parallel load. Does not close a specific issue. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a configurable Velero `startupProbe` for the `/metrics` endpoint (`http-monitoring`) to better accommodate slow startup. * **Bug Fixes** * When metrics are disabled, the rendered container no longer includes `startupProbe`, `livenessProbe`, or `readinessProbe`. * **Tests** * Added invariant chart rendering tests to verify `startupProbe` behavior (including pinned timing fields) and confirm existing probe timing remains unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
41ff94eca9 |
fix(flux): tune source-watcher concurrency and CPU under load
source-watcher reconciles one ArtifactGenerator per PackageSource and builds the resulting ExternalArtifacts. On the management cluster that is ~100 artifact builds during install, and the platform HelmRelease blocks until all of them are Ready. It was the only Flux controller left untuned. In the bootstrap all-in-one pod it ran without an explicit --concurrent (binary default 10) and with the shared 100m CPU request; in the production FluxInstance it was missing from the --concurrent / CPU patch that source-, kustomize- and helm-controller already receive. Under load its per-artifact builds finish at staggered times, leaving a random tail of not-yet-Ready artifacts that times out the platform install. Raise source-watcher to --concurrent=20 in both deployments. In the bootstrap all-in-one pod also give it a 500m CPU request; the pod has no CPU limits, so the request is what drives its CPU share under contention. In the production FluxInstance add it to the shared reconciler patch, so it now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the other tuned controllers already have. The flux-aio module exposes a resources override for the watcher (set via CUE) but no args override, so the bootstrap --concurrent bump is a post-render patch in the Makefile. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
09f9d0bdec |
fix(e2e): widen Grafana bring-up waits to 15m under load
The Configure-Tenant install test completes the tenant-root monitoring bring-up by waiting for the grafana-db CNPG cluster and the grafana-deployment Deployment. Both used a 10m budget while the VictoriaMetrics waits above them use 15m. They complete the same monitoring bring-up and contend for the same node resources during install, so under install-time load (concurrent e2e sandboxes on one runner) either can be slow, and the 10m/15m split within the same contend-for-resources bring-up risks failing the whole install on a PR that never touched monitoring. Raise both Grafana waits to the same uniform 15m budget as the vm-operator waits and wrap them in an on-timeout diagnostic block that dumps the resource YAML plus tenant-root pod state, so a genuinely stuck resource still fails at 15m with legible diagnostics instead of a bare "timed out" line. The extra budget costs near-zero wall-time on the happy path and stays inside the E2E job budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1ffbb30d25 |
fix(e2e): widen VictoriaMetrics operational waits to 15m under load
vlclusters/generic and vmcluster wait for updateStatus=operational with a 10m budget, while their vmalert/vmalertmanager sibling in the same block already uses 15m. All three are vm-operator-managed monitoring resources that only become operational once their workloads schedule and turn Ready, and during platform bring-up they contend for node resources with the rest of the install. On a calm sandbox each converges in under a second, but under install-time load (concurrent e2e sandboxes on one runner) monitoring bring-up is slow, and a 10m/15m split inside a block whose resources all contend for the same node capacity risks failing the whole install on a PR that never touched monitoring. Unify vlclusters and vmcluster on the sibling's 15m budget so the whole block shares one headroom, and dump resource YAML plus pod state on timeout across all three waits so a genuine stuck-not-slow regression stays legible instead of a bare "timed out" line. The extra budget costs near-zero wall-time on the happy path and stays inside the E2E job budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f7271b8011 |
fix(velero): add startupProbe so slow startup does not crashloop the install gate
Velero binds its metrics/health endpoint (:8085, http-monitoring) only after the server loads its plugins and connects to the API server. Under a heavy parallel platform install that can take minutes, but the upstream liveness probe begins checking /metrics at initialDelaySeconds=10 and kills the container after failureThreshold=5 (~130s total) before :8085 is bound. The kubelet SIGTERMs velero (exitCode 0/Completed), it BackOff-restarts, and never escapes the loop inside the install-gate window. Because backupstrategy-controller hard-depends on velero, this blocks the gate. Add a startupProbe to the velero server container. While it is failing the kubelet holds off liveness and readiness entirely, granting failureThreshold * periodSeconds = 30 * 10s = 300s of startup grace, then hands over to the unchanged, tight upstream liveness (10s / 30s / 5) for steady-state crash detection. This is strictly better than loosening liveness, which would also blunt steady-state detection. The upstream velero chart (12.0.3) does not render a startupProbe, so the package patches the deployment template via patches/add-startup-probe.patch (re-applied after re-vendor by the Makefile update target, matching the seaweedfs/ingress-nginx convention) and supplies the startupProbe through the package values override. Pin the startupProbe budget and the unchanged liveness with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9c18260f95 |
fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What The e2e install pipeline still flakes on a loaded runner, failing at a different point each run. Root-caused via repeated runs and a live SSH breakpoint; the platform itself is fine (the components do converge), the failures are slowness + self-inflicted churn under resource pressure. ### Fixes 1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory limit was `110Mi`. The updater watches every pod in the cluster, so on a full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the HelmRelease "early due to stalled resources", cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raised to `512Mi` (request `200Mi`). 2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA `VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA recommender(s) during install and adding pod churn on an already busy cluster. Switched to `Initial`, matching the existing monitoring/etcd `VPA->Initial` fixes. 3. **Install wait budgets** — components reach readiness but overshoot tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`. (The crust-gather failure-snapshot timeout that kept whole jobs hanging to the 3h cancel is already on `main`.) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased several installation and readiness wait times to better handle slower environment startup and reduce premature failures. * Updated vertical autoscaling behavior to avoid unnecessary pod restarts during deployment. * Raised resource limits for the autoscaling updater to improve reliability under load. * **Tests** * Aligned automated checks with the new autoscaling behavior and longer readiness expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
ec42c52967 |
fix(e2e): skip cilium-agent restart for the reserved:ingress IP variant (#3142)
## What this PR does The Cilium leak healer (`hack/e2e-cilium-endpoint-leak-healer.sh`, the interim CI band-aid for the Cilium endpoint-leak class) escalates an unhealed leaked-IP reuse to a Tier-2 `cilium-agent` restart. For one variant that restart is both futile and harmful. When the disputed IP is the node's per-node Cilium ingress IP (`CiliumNode.spec.ingress.ipv4`), it is held by the `reserved:ingress` endpoint — allocated from the node pod CIDR whenever Gateway API / Envoy is enabled (`enable-envoy-config`, `daemon/infraendpoints/infra_ip_allocation.go`). Restarting the agent cannot clear it: the agent re-derives the same ingress IP from the CiliumNode spec on every boot. Worse, the restart can re-trigger the race that caused the wedge in the first place — it rebuilds the host-scope (`kubernetes` IPAM) allocation bitmap empty (`pkg/ipam/hostscope.go`; `RestoreFinished()` is a no-op), then re-reserves the ingress IP via a path decoupled from pod-endpoint restore, so a CNI ADD served in that window can be handed the ingress IP again. That is the infra-IP-vs-pod-IP double-allocation behind the "IP already in use" sandbox rejection. This is an upstream Cilium bug in host-scope IPAM with Gateway API; no released Cilium fixes it as of v1.19.5 (the deployed version). The complete fix belongs upstream — the agent must reserve `spec.ingress.ipv4` in the host-scope allocation bitmap before the CNI server serves allocations, or reconcile it with the restored `reserved:ingress` endpoint, or source it from a dedicated range. That is tracked upstream in cilium/cilium#46799. This PR makes the CI healer correct for that variant: it detects an exact match of the wedged IP against the node's `CiliumNode.spec.ingress.ipv4` and skips the Tier-2 restart — surfacing the case and rescheduling the wedged pod best-effort instead. The match is gated so an empty ingress IP (CiliumNode absent or RBAC denied) never matches, so a false positive can never suppress a restart that would clear a real in-memory leak. The change adds a `ciliumnodes` read to the healer RBAC and a unit test pinning the exact-match-and-non-empty contract. ### Screenshots N/A — CI test-infrastructure change, no UI impact. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced endpoint leak healer to skip Tier-2 agent restarts when the disputed IP exactly matches a node’s reserved per-node Cilium ingress IP. * Added lightweight unit tests for the new IP-matching behavior. * **Bug Fixes** * Prevents unnecessary remediation when the disputed IP is empty or doesn’t exactly match the node ingress IP. * When a match is detected, the healer deletes the affected wedged pod as a best-effort reschedule step. * **Chores** * Updated healer permissions to read Cilium node ingress information cluster-wide. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
93d2c30a8f |
fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public (#3137)
## What Add a `cozystack.cozystack-basics` edge to the `kubevirt-cdi` `default` variant's `dependsOn` in the platform package, plus a helm-unittest regression guard. ## Why `kubevirt-cdi` ships the `cdi-clone-dv` RoleBinding into the `cozy-public` namespace, but `cozy-public` is provisioned by `cozystack-basics`. The variant previously only depended on `cozystack.networking`, so kubevirt-cdi reconciled long before `cozystack-basics` created the namespace. On a fresh install the RoleBinding could race namespace creation and fail with `namespaces "cozy-public" not found`. Flux retries normally mask this, but under slow reconcile it surfaces as a hard `UpgradeFailed`. This mirrors the existing `backupstrategy-controller` ordering, which already depends on `cozystack-basics` for the identical reason (it writes a Bucket into `tenant-root`). There is no dependency cycle: `cozystack-basics`' own dependencies (`tenant-application`, `cozystack-engine`, `gateway-api-crds`) do not reach `kubevirt-cdi`. ## Notes Surfaced via a slow-reconcile install in #3124's E2E. Does not close a specific issue. Tradeoff: kubevirt-cdi now installs after the full `cozystack-basics` chain instead of just `networking`. Its only consumers (`vm-default-images`, `vm-disk-application`) install later still, so the added latency introduces no new ordering problem — the same correctness-over-speed tradeoff already accepted for `backupstrategy-controller`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved upgrade reliability for kubevirt-cdi by enforcing correct dependency ordering before applying its RoleBinding-related resources. * Reduced intermittent `UpgradeFailed` outcomes during slower reconciliation scenarios. * **Tests** * Added a regression test to verify kubevirt-cdi manifest `dependsOn` includes the expected prerequisites for consistent deployment. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d54e3e4b58 |
test(e2e): capture worker VM/CSR state on tenant node-join failure
When a tenant Kubernetes cluster hits the 12m node-Ready deadline with fewer than two nodes joined, the failure path dumped only `describe nodes` and `get hr`. That cannot tell apart the two node-join failure sub-modes: a worker VM that never booted (virt-launcher Pending or OOMKilled) versus a VM that booted fine but whose kubelet never registered a Node. The tenant cilium-operator HelmRelease reports "InProgress" identically in both, so the current dump is not enough to diagnose the failure or design a fix. Dump scoped, guarded diagnostics before failing fast: - worker VM/VMI/virt-launcher state on the management cluster, which is what actually splits the two sub-modes; - tenant kubelet CSRs plus the talos-csr-signer sidecar log, which reveal a boot-but-no-Node stall. In-guest Talos/kubelet capture is deliberately left out: talosctl needs a tenant-cluster talosconfig the runner does not have (the tenant workers use a separate Talos CA from the management sandbox), so the gap is documented in the failure path rather than shipped as a call that would fail mTLS and capture nothing. Every capture is guarded with `|| true` so a capture failure never masks the real exit. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei KvapilandClaude
|
d6f66d2dcd |
fix(vertical-pod-autoscaler): stop the updater OOM + VPA-for-VPA eviction churn
Two root causes of e2e install flakiness, both in the VPA package: 1. The updater memory limit was 110Mi. The updater watches every pod in the cluster, so on a full cozystack install it OOM-kills, which trips the Deployment's ProgressDeadlineExceeded and makes Flux fail the HelmRelease 'early due to stalled resources' - cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raise the limit to 512Mi (request 200Mi); the recommender is already unlimited. 2. The two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto, which evicts the VPA recommender(s) during install and adds pod churn on an already busy cluster. Switch to Initial, matching the existing monitoring/etcd VPA->Initial fixes. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei KvapilandClaude
|
433203c500 |
test(e2e): widen install waits for a loaded runner
The e2e install still flakes on a busy runner where components reach readiness but overshoot tight wait budgets: - keycloak HR chain (keycloak-db CNPG -> keycloak StatefulSet -> keycloak-configure) reaches Ready ~15-18m in: 10m -> 20m. - tenant aggregate wait (hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root): the tenant stack only starts creating pods ~9-10m in and then needs several more minutes: 10m -> 20m. - per-app tenant readiness waits (ingress, etcd, VictoriaLogs/Metrics clusters, grafana-db, grafana): 5m -> 10m. Diagnosed via a live SSH breakpoint: the HRs all do reach Ready, just outside the old budgets. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
fc862528fd |
fix(e2e): drain tenant VMs/PVCs between kubernetes tests
The inter-test cleanup hook deletes the tenant Kubernetes CR and waits only for the CR to disappear. That wait finalizes as soon as the CR's finalizers clear, which merely TRIGGERS teardown of the CAPK worker VMs and their DataVolume-backed disk PVCs -- it does not await it. The virt-launcher pods keep their guest RAM reserved until the VMIs are gone, so the next back-to-back tenant test's worker VMs begin scheduling against a sandbox the previous tenant has not yet vacated. Under the constrained CI sandbox this starves memory, a worker VM misses the node-join budget, and the run flakes on worker-node-join (the MachineHealthCheck cannot remediate when both workers are unhealthy). Block cozy_cleanup until the tenant VMs, VMIs and disk PVCs have actually drained from tenant-test before it returns, so the next test starts on a freed sandbox. The wait is bounded (5m) and best-effort: cozytest runs cozy_cleanup wrapped in `|| true` and the waiter returns loudly on timeout, so a stuck teardown can never hang the job longer than the deadline. The probe is namespace-scoped -- tenant-test is provisioned with etcd/monitoring/seaweedfs disabled so it carries no baseline PVCs, and e2e apps run sequentially each cleaning up after themselves, so the only VMs/VMIs/PVCs present belong to the tenant being torn down. The pure drain exit-condition (cozy_tenant_drained) is unit-tested in hack/run-kubernetes-drain_test.bats, including the invariant that a failed probe's "err" sentinel is never misread as a completed drain. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
20e4eab281 |
test: make negative bats assertions fail under cozytest (#3132)
## What this PR does
Every `*.bats` file in the repo runs through `hack/cozytest.sh`, not
real bats — the e2e files via `packages/core/testing/Makefile` and the
`hack/*.bats` unit tests via `make bats-unit-tests`. cozytest runs each
`@test` under `set -e`, and POSIX errexit is suppressed for a pipeline
beginning with `!`. So a bare `! cmd` must-not-happen assertion never
fails the test: when the forbidden condition actually occurs (the
command succeeds), the negated pipeline returns non-zero but errexit
ignores it and the test passes anyway. Ten such assertions across six
files were silently vacuous.
This converts each to `if cmd; then echo "<reason>"; false; fi`. The
`if` condition is exempt from errexit (so the probe itself never aborts
the test), and the `false` in the taken branch is a plain command under
`set -e`, so it aborts and fails the test with a legible message when
the forbidden condition holds. This is the must-not-happen inverse of
the `cmd || { echo …; false; }` pattern already used in the suite (e.g.
harbor.bats, per docs/agents/e2e-testing.md). Original redirections and
quoting are preserved, so happy-path behavior is unchanged — only the
failure path now actually fails.
Converted assertions:
- `bucket`: a read-only user must not be able to upload (access-control
regression guard)
- `gateway`: a child tenant must not have its own gateway HelmRelease
(inheritance)
- `kuberture`: external-dns public/internal hostname isolation
- `e2e-install-cozystack`: tenant-name validation must reject rather
than warn; the version ConfigMap must be gone after delete
- `build-matrix` (unit test): talos/installer must be excluded from the
parallel matrix
- `capture-dataplane` (unit test): kube-dns and pending rows must be
filtered out
A separate in-flight etcd test change owns two more such assertions in
`etcd.bats`; they are left out of this PR to avoid a conflicting edit.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Strengthened several end-to-end/Bats assertions to fail more
explicitly when expected cluster conditions aren’t met.
* Improved matrix validation to reliably confirm excluded components are
absent.
* Updated dataplane load balancer filtering, bucket readonly upload, and
gateway inheritance checks to avoid false/ambiguous pass cases.
* Tightened DNS hostname isolation probes and cozystack installer
validations (tenant name dashes, deletion protection) with clearer
“expected vs found” failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
925cb87af9 |
test: fix dead bats setup/teardown hooks under cozytest (#3134)
## What this PR does cozytest.sh — the runner for every `*.bats` file in the repo (e2e via `packages/core/testing/Makefile`, unit tests via `make bats-unit-tests`) — is not real bats: it never invokes `setup()`/`teardown()`. Three such hooks outside `etcd.bats` were therefore dead. Cleanup ran only via each test's success-path inline delete, so a failing test leaked its resources into the shared `tenant-test` namespace — and, for kuberture, cluster-scoped RBAC that persists across sandbox re-runs. This wires the cleanup to `cozy_cleanup()`, the hook cozytest does run (once at suite exit and on the first failing test): - **external-dns**: dead `teardown()` → `cozy_cleanup()` deleting all ExternalDNS CRs. ExternalDNS is a stateless Deployment with the `inmemory` provider, so a plain delete fully reclaims it — no drain needed. The per-test inline deletes remain fail-loud on the success path; `cozy_cleanup` is the failure-path net. - **kuberture**: dead `teardown()` → `cozy_cleanup()` routed through the existing `_cleanup()`, so the probe Pods, ServiceAccount, and cluster-scoped ClusterRole/ClusterRoleBinding are reclaimed on the failure path too. `_cleanup`'s per-delete `|| true` is also dropped (keeping `--ignore-not-found`): the success-path inline call now fails the test on a stuck teardown (e2e-testing.md §4), while the failure path stays best-effort via the runner's `cozy_cleanup || true` wrap. - **build-matrix_test**: the dead `setup()` only `cd`'d to the repo root, which cozytest already runs from, so the relative `hack/build-matrix.sh` calls resolve without it — removed. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved end-to-end test cleanup for external-dns by switching to suite-aligned cleanup that reliably removes ExternalDNS resources even on failures or early aborts. * Enhanced kuberture end-to-end cleanup to remove probe components and related RBAC/package more safely, with errors handled consistently to aid diagnosis. * Updated test setup expectations for matrix builds so tests run correctly from the repository root without relying on a custom directory hook. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
12fe3256fd |
test: make negative bats assertions fail under cozytest
These bats files are all run by hack/cozytest.sh, which executes each @test under `set -e`. POSIX errexit is suppressed for a pipeline that begins with `!`, so a bare `! cmd` must-not-happen assertion never fails the test: when the forbidden condition actually occurs (cmd succeeds) the negated pipeline returns non-zero but errexit ignores it and the test passes anyway. These assertions were silently vacuous. Replace each with `if cmd; then echo "<reason>"; false; fi`, the idiom already used and documented elsewhere in these files — the if-condition is exempt from errexit and the `false` in the taken branch fails the test loudly. The forbidden condition now fails the test as intended: - bucket: read-only user must not be able to upload (access control) - gateway: child tenant must not have its own gateway HelmRelease - kuberture: external-dns public/internal hostname isolation - e2e-install: tenant-name validation must reject (not warn); the version ConfigMap must be gone after delete - build-matrix: talos/installer excluded from the parallel matrix - capture-dataplane: kube-dns and pending rows are filtered out Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bf07b8c42b |
test: fix dead bats setup/teardown hooks under cozytest
cozytest.sh — the runner for every bats file here — is not real bats: it never invokes setup()/teardown(). So these hooks were dead. Cleanup ran only via each test's success-path inline delete, and a failing test leaked its resources into the shared tenant-test namespace (and, for kuberture, cluster-scoped RBAC that persists across sandbox re-runs). Wire the cleanup to cozy_cleanup(), the hook cozytest does run at suite exit and on the first failing test: - external-dns: dead teardown() -> cozy_cleanup() deleting all ExternalDNS CRs. ExternalDNS is a stateless Deployment with the inmemory provider, so a plain delete fully reclaims it — no drain needed. - kuberture: dead teardown() -> cozy_cleanup() routed through the existing _cleanup(), so the probe Pods, SA, and cluster-scoped ClusterRole and ClusterRoleBinding are reclaimed on the failure path too. build-matrix_test: drop the dead setup() entirely — it only cd'd to the repo root, which cozytest already runs from, so the relative hack/build-matrix.sh calls resolve without it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
cbc314a8c5 |
fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public
The kubevirt-cdi default variant only depended on cozystack.networking, which reconciles long before cozystack-basics provisions the cozy-public namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public, so on a fresh install the RoleBinding can race namespace creation and fail with `namespaces "cozy-public" not found`. Flux retries normally mask it, but under slow reconcile it surfaces as a hard UpgradeFailed. Add a cozystack.cozystack-basics edge so the namespace exists first, mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics' own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4e53f5e448 |
fix(e2e): skip cilium-agent restart for the reserved:ingress IP variant
The cilium leak healer escalates an unhealed leaked-IP reuse to a Tier-2 cilium-agent restart. For one variant that restart is both futile and harmful: when the disputed IP is the node's per-node Cilium ingress IP (CiliumNode.spec.ingress.ipv4), held by the reserved:ingress endpoint (allocated from the node pod CIDR whenever Gateway API / Envoy is on), restarting the agent cannot clear it -- the agent re-derives the same ingress IP from the CiliumNode spec on every boot -- and can re-trigger the very race that caused the wedge: the restart rebuilds the host-scope (kubernetes) IPAM bitmap empty, then re-reserves the ingress IP via a path decoupled from pod-endpoint restore, so a CNI ADD in that window can be handed the ingress IP again. This is an upstream cilium bug in host-scope IPAM with Gateway API; no released cilium fixes it as of v1.19.5. Detect that variant by an exact match of the wedged IP against the node's CiliumNode.spec.ingress.ipv4 and skip the Tier-2 restart -- surface the case and reschedule the wedged pod best-effort instead. The match is gated so an empty ingress IP (CiliumNode absent or RBAC denied) never matches, so a false positive can never suppress a restart that would clear a real in-memory leak. Add a CiliumNode read to the healer RBAC and a unit test pinning the exact-match-and-non-empty contract. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5d9f7331f3 |
ci: overlay current-main images for packages a PR didn't rebuild (#3148)
## What this PR does Conditional PR builds (the per-package matrix in `pull-requests.yaml`) only rebuild packages whose dir changed. Every other package falls back to the image refs committed in the tree — the last release's digests (currently `v1.5.0`), which go stale as `main` advances. So e2e and the installer's packages artifact ran **current-main configs against last-release images** for everything outside a PR's build matrix, and a PR installed via the `cozy-installer` chart was an incoherent mix (your changes fresh, everything else at the last release). In `finalize`, this overlays current-main image refs onto every package the PR did **not** rebuild: - Source of truth is the `cozystack-packages:main` OCI artifact that `build-main.yaml` already publishes (`make build` → installer `image-packages`) — the whole current-main packages tree, refs digest-pinned to the images `build-main` pushed. **No `build-main` change is needed.** - For each build unit not in the PR's build matrix, copy its ref-bearing files (`values.yaml` + `images/*.tag`) from that tree. Units the PR rebuilt keep their fresh `pr-<N>-<sha>` refs. - The copy is **self-validating**: a file is overlaid only when every differing line is image-reference-bearing (a full `@sha256:` ref, or a split `image`/`repository`/`registry`/`tag`/`digest` key, or a `--…-image=` arg); otherwise the unit is left on its committed ref. - A missing artifact is a no-op (prior behaviour), and the step is best-effort so it can never block a PR. Runs after the registry login and before the installer build, so both the installer's `cozystack-packages:<pr>` artifact and the `pr.patch` e2e applies reflect the overlay. Net effect: a PR is tested and installed as **current-main + its own changes**, coherent across config and image. Covered by `hack/overlay-main-images_test.bats` (auto-discovered by `make unit-tests` → `bats-unit-tests`, run in the `checks` job): single-line and split-form ref overlay, skip-rebuilt, drift self-validation, and missing-artifact no-op. ### Release note ```release-note ci: PR e2e and the installer artifact now run current-main images for packages a PR doesn't rebuild, instead of the last release's images — so a PR is tested and installed as current-main plus its own changes. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **CI / New Features** * Automatically detects which package areas were modified in the pull request and uses that to target image reference updates. * Overlays “current main” digest/image references for packages not rebuilt, while always skipping core special components. * If overlay processing fails, emits a warning and continues; installer builds still use the overlay-applied artifacts. * **Tests** * Expanded integration-style coverage for overlay behavior, touched-list skipping, split-form YAML references, drift/non-image preservation, vendored-chart exclusions, and missing-artifact no-op scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
c725a41f3a |
feat(tenant): recursive (hierarchical) resource quotas for sub-tenants (#3086)
## What this PR does Adds **hierarchical (recursive) resource quotas** for tenants so a tenant can no longer escalate its allocation by creating sub-tenants. Today `tenant.spec.resourceQuotas` becomes a plain per-namespace `ResourceQuota`, which ignores the tenant tree: a super-admin inside a quota'd tenant can create a sub-tenant with a larger quota — or no quota at all — and consume more than was allocated. A tenant's declared quota is now treated as the budget for its **whole sub-tree**: - A child that declares its own quota **carves a fixed slice** out of the parent's remaining budget (parent 10 CPU → child 4 CPU → parent has 6 left). - A child **without** a quota **shares** the parent's remaining pool rather than being unbounded. - A child quota that exceeds the parent's remaining budget is **rejected**. Two cooperating layers, mirroring OpenShift's ClusterResourceQuota split (admission + reconciliation): 1. **Declaration-time gate** in the aggregated apiserver (`apps.cozystack.io` Tenant create/update): deterministically rejects a sub-tenant whose declared quota exceeds the parent's remaining budget. 2. **Runtime enforcer** in `cozystack-controller`: maintains a per-namespace `tenant-quota-allocated` ResourceQuota that clamps each pool member to its share of the parent budget, aggregating sub-tree usage so members collectively stay within the budget. Kubernetes enforces the most restrictive ResourceQuota in a namespace, so this binds without fighting Flux over the chart-rendered `tenant-quota`. Overcommitted pools (a parent quota lowered below the sum of its children) emit a `QuotaOvercommitted` event. A `--tenant-quota-buffer-percent` flag temporarily inflates pool budgets during rollout so workloads already over a freshly-introduced quota keep running. The model and the selector-spans-namespaces / aggregated-usage approach are adapted from OpenShift's ClusterResourceQuota controllers (`openshift/api`, `library-go`, `cluster-policy-controller`, `apiserver-library-go`), Apache-2.0, Copyright Red Hat, Inc. and the OpenShift Authors; attribution is preserved in the source. The per-namespace quota arithmetic reuses the upstream Kubernetes quota helpers (`k8s.io/apiserver/pkg/quota/v1`). No API types or the tenant chart change; the feature works with the existing `tenant.spec.resourceQuotas` field. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(tenant): tenant resource quotas are now hierarchical — a sub-tenant's quota is carved out of its parent's remaining budget and can no longer exceed it; sub-tenants without a quota share the parent's pool. Set `--tenant-quota-buffer-percent` on cozystack-controller for a temporary grace buffer during upgrades. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added hierarchical tenant quota enforcement across tenant namespaces, including shared pool handling and configurable quota buffer support. * Tenant quota changes are now validated on create and update to prevent over-allocation. * **Bug Fixes** * Improved quota calculations so usage, carve-outs, and nested tenant limits are applied more consistently. * Stale quota allocations are now cleaned up when namespaces no longer belong to an active tenant pool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
297764feff |
fix(ci): create the overlay output dir before flux pull
flux pull artifact --output requires the target directory to already exist; it stats the path before any network call and aborts with "invalid output path: stat _out/mainpkgs: no such file or directory" when the dir is missing. The step deleted _out/mainpkgs with rm -rf but never recreated it, so flux failed locally on every run and the || fallback misreported the failure as "artifact not published" — leaving the current-main overlay a permanent no-op even though the artifact exists and is anonymously pullable. Add mkdir -p after the rm -rf so the directory exists before flux pull, and reword the fallback message so a genuine future pull failure is no longer mistaken for an absent artifact. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
scooby87
|
5b26ec3462 |
fix(mariadb): clean up orphaned operator-generated secrets (#3056) (#3071)
## What this PR does After deleting a `MariaDB` app CR with metrics and/or replication enabled, two Secrets created by the mariadb-operator were left orphaned in the tenant namespace: - `<release>-metrics-password` - `<release>-repl-password` These are generated by the operator (`k8s.mariadb.com/v1alpha1`) as a side effect of `metrics.enabled` / `replication.enabled`, so they are **not** Helm-owned and survive `helm uninstall`. The existing post-delete hook (`templates/hooks/cleanup-pvc.yaml`) only swept PVCs, leaving the Secrets behind. This PR extends that hook to also clean them up: - grants the cleanup `Role` RBAC on `secrets`; - deletes both password Secrets by name with `--ignore-not-found`; - adds a `helm-unittest` suite locking in the behaviour, plus a `test` Makefile target so the package's tests actually run in CI (`hack/helm-unit-tests.sh` only runs packages that define a `test` target). Fixes #3056. ### Release note ```release-note fix(mariadb): garbage-collect operator-generated metrics-password / repl-password Secrets when a MariaDB app is deleted ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved MariaDB uninstall cleanup to remove leftover operator-generated password Secrets in addition to persistent volumes/claims. * Hardened the cleanup hook Job with stricter pod/container security settings, least-privilege Secret permissions, idempotent deletions for missing resources, and correct non-zero exit on real failures. * **Tests** * Added a cleanup hook test suite validating Secret deletion behavior, failure propagation, and that cleanup RBAC grants least-privilege access only to the required Secrets. * Added a Helm unit test `test` target for the MariaDB chart. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
d988b3e8c8 |
fix(apiserver): read tenant pool quota usage via direct client and grant RBAC
parentPoolUsage listed ResourceQuotas through the aggregated apiserver's cached client, which lazily starts a cluster-wide ResourceQuota informer. cozystack-api has no RBAC to list resourcequotas, so the informer never synced and tenant create/update of a bounded child hung until the request timed out (observed on a cluster). Read quota usage through the direct (uncached) watch client instead, scoped to the pool's member namespaces, so no cluster-wide informer is started, and grant cozystack-api get/watch/list on resourcequotas for those reads. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
184324430f |
fix(controller): don't clamp pool resources the parent budget leaves unbounded
ComputePools derived a pool's Available with SubtractWithNonNegativeResult over the union of the budget and carve-out keys, so a bounded child carving a resource its parent does not bound surfaced that resource as 0 in the parent pool's Available. The controller then wrote a tenant-quota-allocated clamping the parent namespace to zero of a resource it never limited, blocking its workloads (observed on a cluster: a memory-bounded child froze its cpu-only parent's memory). Mask Available to the budget's resource names so only resources the pool root actually bounds are enforced; mirrors the fix already applied to Pool.Overcommitted. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
e3f09bab1b |
fix(apiserver): subtract parent pool usage in hierarchical tenant quota admission
The declaration-time gate only subtracted sibling carve-outs from a parent's budget, not the parent pool's current usage. A tenant already consuming most of its budget could therefore carve a full-budget child, letting the sub-tree collectively exceed the root's allocation. Sum the current usage of the parent pool (its own namespace plus its unbounded descendants, read from each member's rendered ResourceQuota.status.used) and subtract it from the remaining budget, mapping each shorthand quota key to its rendered limit key (renderedLimitKey). Boundedness is decided by declared resourceQuotas so the result is independent of whether Flux has rendered a freshly-created tenant's quota yet. This is best-effort at declaration time (a snapshot; concurrent writes can briefly overshoot and running pods are never evicted) — the tenant-quota controller remains the runtime backstop. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
b0556cc753 |
fix(controller): ignore unbounded resources in tenant quota overcommit check
Pool.Overcommitted reported a resource as overcommitted whenever a sub-tenant carved out a resource the pool root does not bound. Admission explicitly allows a child to bound a resource its parent leaves unbounded, and an unbounded budget can never be overcommitted, so this produced a spurious QuotaOvercommitted warning event on every reconcile. Restrict the check to resources present in the pool budget. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Andrei KvapilandClaude
|
55f1edda78 |
fix(controller): read tenant budget from rendered quota, skip inert pools
Take a tenant's declared budget from its chart-rendered tenant-quota ResourceQuota (.spec.hard) instead of the raw tenant.spec.resourceQuotas values. The chart expands the shorthand values into real quota resource names and applies the cluster's request/limit allocation ratios, so reading the rendered object keeps the controller-owned allocated quota in exactly the same key space — the two quotas compose correctly and the controller never replicates that cluster-configurable transformation. Also skip pools that need no tightening beyond the chart quota (a lone tenant with no sub-tenants and no carve-outs): the controller writes an allocated quota only when a budget is actually shared or carved up, so a single-tenant namespace is left exactly as the chart renders it. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
844deb170f |
feat(controller): enforce recursive tenant quotas at runtime
Add a tenant quota reconciler to cozystack-controller that realises the hierarchical (sub-tree) quota model at runtime. Each tenant's declared quota is the budget for its whole sub-tree: a child that declares its own quota carves a fixed slice out of the parent budget, while a child without a quota shares the parent's remaining pool. The controller maintains a per-namespace 'tenant-quota-allocated' ResourceQuota (Kubernetes enforces the most restrictive quota, so it binds without fighting Flux over the chart-rendered 'tenant-quota'), aggregating sub-tree usage so a pool's members collectively stay within the budget. A --tenant-quota-buffer-percent flag temporarily inflates pool budgets so workloads already over a freshly-introduced quota keep running during rollout; overcommitted pools emit a QuotaOvercommitted event. The model and the selector-spans-namespaces/aggregated-usage approach are adapted from OpenShift's ClusterResourceQuota controllers (openshift/api, library-go, cluster-policy-controller, apiserver-library-go), Apache-2.0, Copyright Red Hat, Inc. and the OpenShift Authors. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
c87612b538 |
feat(apiserver): enforce hierarchical tenant quota allocation at admission
Add a declaration-time gate in the aggregated apiserver: a child tenant's declared resourceQuotas are carved out of its parent tenant's remaining quota and may not exceed it. A child without a quota shares the parent pool and is allowed; a parent without a quota does not constrain children. This blocks quota escalation via sub-tenant creation (declaring a bigger or unbounded child) without re-implementing pod-level admission. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
3333b325a8 |
fix(ci): preserve a PR's own edits to non-rebuilt package image refs
The current-main overlay skipped only rebuilt units (plus talos/installer), so for a package the PR edited but did not rebuild — an upstream image bump in a non-build-unit package such as keycloak or cert-manager — the committed ref differed from the main artifact on image-reference lines only, the drift check passed, and the overlay reverted the PR's edit back to current-main. e2e and the installer artifact then ran against the stale image. Pass the PR's edited package dirs (derived from the plan job's git diff) into the overlay skip set so a PR's own image-ref edits win over the artifact. Add tests pinning the preserve behaviour and covering the --image= arg form and the file-absent-in-PR-tree branch. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
52f7d05786 |
ci: overlay current-main images for packages a PR didn't rebuild
Conditional PR builds (the per-package matrix in pull-requests.yaml) only rebuild packages whose dir changed. Every other package falls back to the image refs committed in the tree — the last release's digests (currently v1.5.0), which go stale as main advances. So e2e and the installer's packages artifact ran current-main configs against last-release images for everything outside the PR's build matrix, and a PR installed via the cozy-installer chart was an incoherent mix. This is the systemic gap in issue #3143 (its concrete bite: the merged objectstorage-controller requeue fix #3034 never ran in e2e, flaking the harbor BucketClaim check). In finalize, pull the cozystack-packages:main OCI artifact that build-main already publishes (the whole current-main packages tree, refs digest-pinned to the images build-main pushed) and walk every ref-bearing file in it (values.yaml + images/*.tag), overlaying the repo's copy. This is driven by the artifact, not the build matrix, so it covers the ENTIRE first-party tree — apps/system/core AND extra/, library/ — and a merged fix to any first-party image (e.g. the objectstorage-sidecar ref under extra/seaweedfs) takes effect. For a file build-main did not rebuild the artifact copy is byte-identical and is skipped. Skipped explicitly: units the PR itself rebuilt (their pr-<N>-<sha> refs win), packages/core/talos and packages/core/installer (owned by their dedicated build-talos / finalize-installer jobs), and vendored charts/ subtrees. The copy is self-validating: a file is overlaid only when every differing line is image-reference-bearing (a full @sha256: ref, or a split image/repository/registry/tag/digest key, or a --…-image= arg); otherwise it keeps its committed ref. A missing artifact is a no-op (prior behaviour) and the step is best-effort so it never blocks a PR. No build-main change is needed: cozystack-packages:main is already published by `make build`. Runs before the installer build so the flux-pushed packages artifact and the pr.patch e2e applies both reflect the overlay. Result: a PR is tested and installed as current-main + its own changes, coherent across config and image, instead of current-main configs running on last-release images. Covered by hack/overlay-main-images_test.bats (run by `make unit-tests` in the checks job): single-line and split-form ref overlay, extra/* coverage, skip-rebuilt, talos/installer skip, vendored-charts prune, drift self-validation, and missing-artifact no-op. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
eab9b034f6 |
fix(seaweedfs): enforce read-only bucket access by bumping cosi-driver to v0.3.1 (#3136)
## What this PR does The SeaweedFS COSI driver shipped at `v0.1.2` hardcodes read-write S3 actions (`Read,Write,List,Tagging`) for every BucketAccess and never reads the `accessPolicy` BucketAccessClass parameter. The chart already defines a `-readonly` BucketAccessClass carrying `parameters.accessPolicy: readonly`, and the `bucket` app routes `readonly: true` users to it — but `v0.1.2` ignores the parameter, so a credential meant to be read-only is silently issued with full read-write access and can upload and delete objects. This is an access-control bug. Upstream `seaweedfs/seaweedfs-cosi-driver` added `accessPolicy` handling in `v0.3.0`: `readonly` maps to `Read,List`, while `readwrite` and the empty default stay `Read,Write,List,Tagging`. This bumps the driver to `v0.3.1` via a `seaweedfs.cosi.image` override in the package `values.yaml`, so the override survives `make update` and the vendored chart is left untouched. The default read-write action set is unchanged, so existing read-write credentials are unaffected; only read-only grants are tightened to actually be read-only. The `extra/seaweedfs` chart already runs `v0.3.1` with the same objectstorage-sidecar, so the driver/sidecar combination is already proven in this repo. A chart unit test pins the rendered driver image so this fix cannot silently regress on a future re-vendor. This does not close #1921. That issue bundles two features — read-only bucket access (fixed here) and Object Lock / WORM (the `-lock` BucketClass), which is independent and out of scope here. Relates to #1921. ### Release note ```release-note Fix SeaweedFS read-only bucket credentials being issued with read-write access; the COSI driver now honours the read-only access policy. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Pin the SeaweedFS COSI driver image to `ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1` to keep access behavior consistent and avoid relying on a vendored default. * Ensure the rendered SeaweedFS COSI deployment uses the expected container name and the pinned driver image. * **Tests** * Add a YAML test suite to validate the generated COSI deployment template matches the expected container name and pinned image. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
bef42229d4 |
fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061) (#3077)
## What this PR does After deleting a `Harbor` app CR, two things were left orphaned in the tenant namespace: - the **jobservice / trivy PVCs** (`<release>-jobservice`, and `<release>-trivy` when Trivy is enabled), because `packages/apps/harbor/templates/harbor.yaml` sets `persistence.resourcePolicy: "keep"`, which annotates the upstream Harbor subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux deliberately never deletes them; - cert-manager's **ACME http01-solver** resources (`cm-acme-http-solver-*`), via the same stuck-challenge mechanism as the Bucket issue (#3060) when Harbor issues a non-wildcard cert. This adds a single `post-delete` cleanup hook that: - **deletes the upstream Harbor subchart's kept PVCs** — selecting by `app.kubernetes.io/instance=<release>-system` (the subchart runs as the `<release>-system` HelmRelease, so its PVCs carry that label, **not** `<release>`). A hook is required precisely because `resourcePolicy: keep` blocks normal reclaim. This intentionally covers both the jobservice and trivy PVCs: both hold only regenerable data (jobservice file logs; trivy's vulnerability-DB cache), and registry images live in S3. - **deletes this release's `Certificate`** (deterministic name `<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`, `Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via owner references. - **host-scoped belt-and-braces sweep** for any ACME http01 solver objects cert-manager may have orphaned (e.g. the Certificate was already gone, so deleting it did not cascade). The sweep is scoped **strictly to this release's Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses by `.spec.rules[0].host` — **not** a namespace-wide `acme.cert-manager.io/http01-solver=true` label sweep, because a Cozystack tenant namespace can host many apps and a blanket label sweep would delete sibling apps' in-flight challenge solvers. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, with CPU/memory requests and limits set. RBAC is scoped to exactly the resources it deletes (`persistentvolumeclaims`, `certificates`, `orders`, `challenges`, solver `ingresses`) and verified against the vendored cert-manager CRDs. Failures are logged but never block teardown. ## Cleanup scope — what is and isn't removed **Removed:** the upstream Harbor subchart's kept PVCs (jobservice + trivy) and this release's cert-manager ACME artifacts (Certificate, and host-scoped Order/Challenge/solver-Ingress). **Not touched (by design):** - **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG `Cluster` and governed by CNPG's own reclaim policy; they carry CNPG labels, not `app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with a different DB size into the same namespace can therefore leak the old DB PVCs; this is left to CNPG's lifecycle on purpose (the data is the user's). - **Redis PVCs** — managed by the redis-operator with their own labels, likewise untouched. - **Solver `Pod`/`Service`** — not deleted directly; they are removed by cert-manager's owner-ref cascade when the Certificate/Challenge is deleted, so the hook grants no RBAC for them. > [!NOTE] > The PVC sweep is eventually-consistent: while the inner `<release>-system` > HelmRelease is still being uninstalled by Flux, a PVC may briefly stay in > `Terminating` until its consumer Pods (jobservice/trivy) exit. The Job's > deletion request always succeeds; reclaim completes once the pods are gone. > [!WARNING] > **This overrides the intentional `persistence.resourcePolicy: "keep"` setting > and force-deletes the jobservice/trivy PVCs on app deletion.** Those PVCs hold > only jobservice file logs and the trivy vuln-DB cache (registry images live in > S3), so data-loss risk is low — but if retaining them across re-installs is > desired, this is a conscious product decision to revisit. The ACME cleanup > affects only cert-manager's transient artifacts. Fixes #3061. > Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and route it > through `cozy-lib.images-registry` (pre-existing debt shared with the MariaDB > cleanup hook), out of scope for this PR. ### Release note ```release-note fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched. ``` |
||
|
myasnikovdaniil
|
3e20a0cd41 |
fix(kubernetes): make node group resources optional when instanceType is set (#3121)
## What this PR does
A tenant Kubernetes node group can be sized two ways: by `instanceType`
or by an explicit `resources` block. The generated values schema,
however, marked `resources` as required on every node group, so a node
group sized purely by `instanceType` was rejected at install time:
```text
at '/nodeGroups/<name>': missing property 'resources'
```
The same failure occurs through the typed `Kubernetes` CR: an empty
`resources: {}` is dropped during the CR-to-HelmRelease translation, so
by validation time the key is absent and the required check fails — even
though sizing was already fully specified by `instanceType`.
This marks `resources` optional in the node group definition and
regenerates the dependent artifacts (`values.schema.json`, API types,
README, and the `ApplicationDefinition`). No change is needed to emit
the VM: the chart already renders `domain.cpu`/`domain.memory` only when
`resources` is present and `instancetype` only when set, so a node group
sized purely by `instanceType` now validates and installs. Node groups
that specify `resources` are unaffected.
`instanceType` and `resources` are alternative, mutually exclusive
sizing sources — KubeVirt rejects a VirtualMachine that references an
instancetype and also sets explicit `domain.cpu`/`domain.memory`, since
instancetype CPU/memory cannot be overridden. To avoid emitting a VM
that the instancetype admission webhook would later reject, the chart
now fails fast at render time with a clear message when a node group
sets both a non-empty `instanceType` and explicit `resources`.
Tests: helm-unittest coverage proves a node group with `instanceType`
and no `resources` passes schema validation and renders without explicit
`domain.cpu`/`memory`; a complementary case confirms explicit
`resources` are still honoured; and a `failedTemplate` case pins the
render-time rejection of the conflicting both-set combination. The
omit-resources case fails against the pre-fix schema with the exact
error above, so it is a real regression guard.
Addresses #3046.
### Screenshots
N/A — schema and chart change, no UI impact.
### Release note
```release-note
fix(kubernetes): allow a node group to omit `resources` when it is sized by `instanceType` (previously failed schema validation with "missing property 'resources'"); reject the unsupported combination of `instanceType` and explicit `resources` at render time
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Worker node groups can optionally size CPU/memory via `resources`, and
can omit `resources` when using `instanceType`.
* **Bug Fixes**
* Render now validates CPU/memory rules: `instanceType` with only one of
`resources.cpu`/`resources.memory` fails, and `instancetype` is omitted
when both `resources.cpu` and `resources.memory` are set (with
`instanceType` ignored).
* **Documentation**
* Updated API/Helm docs and JSON schema to clarify `resources` vs
`instanceType` precedence and the “set both CPU and memory (or neither)”
requirement.
* **Tests**
* Added helm-unittest scenarios covering omission, precedence, and
render-time failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
0892c5a38c |
fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does After deleting a `Qdrant` app CR, the StatefulSet data PVC (`qdrant-storage-<release>-0`) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default. The vendored upstream Qdrant chart does **not** support `persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml` never renders that field), so the OpenBAO-style value passthrough is not possible here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the MariaDB hook) that deletes the release's PVCs by label. The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by every Qdrant in a namespace), so to select PVCs safely per release this also passes `persistence.additionalLabels.app.kubernetes.io/instance: <release>` through the HelmRelease — giving the data PVC a release-unique label the hook selects on. The cleanup Job is hardened: runs as non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped to `persistentvolumeclaims`. A real cleanup failure is logged (not silently swallowed) but never blocks namespace teardown. > [!WARNING] > **Deleting a Qdrant app now permanently removes its data PVC and the vector data it holds.** > This is intentional and fixes the storage leak; the hook only fires on full > release deletion (not on scale-down). Back up before deleting. Fixes #3059. ### Release note ```release-note fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a post-delete cleanup hook for Qdrant to remove leftover StatefulSet data PVCs after deletion. * Included hardened Kubernetes settings for the cleanup job to improve runtime safety. * **Tests** * Added Helm chart tests that validate the post-delete cleanup job and its RBAC permissions. * Enhanced end-to-end verification to confirm the data PVC is fully removed after deleting a Qdrant instance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
bf14a73b39 |
feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922)
## What this PR does
Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.
The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.
The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.
Implements the attachment model from #1614.
### Release note
```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
* Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Алексей АртамоновandClaude Opus 4.8
|
b25f294d46 |
fix(harbor): select kept PVCs by release label so the trivy PVC is also cleaned
The post-delete cleanup hook selected PVCs by app.kubernetes.io/instance=<release>-system, a label only the jobservice PVC carries (via the chart's "harbor.labels" helper). The trivy PVC is a StatefulSet volumeClaimTemplate labeled via "harbor.legacy.labels" with a selector using "harbor.matchLabels" — neither sets app.kubernetes.io/instance — so the sweep silently missed it. Trivy is enabled by default, so a default Harbor delete left the trivy PVC orphaned, contradicting the PR's own claim. Select on release=<release>-system instead: both kept PVCs carry it, while CNPG (db) and Redis PVCs use different label schemes and the registry is on S3, so only the two intended PVCs match. The unit test now asserts the corrected selector and guards against regressing to the instance label. Also drop the by-name delete of Orders/Challenges (their hash-suffixed names make a by-name delete a no-op; the Certificate owner-ref cascade plus the host-scoped sweep handle them), narrow the hook RBAC to challenges only, and note the Gateway-mode no-op. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
11c417e258 |
test(mariadb): assert both cleanup failure branches and exact RBAC rule set
Strengthen the cleanup hook suite per review: - assert the PVC and Secret delete branches each set cleanup_failed=1, not just the final exit, so a regression in either branch is caught; - pin the Role to exactly two rules so a broader unscoped secrets rule can't slip past the subset `contains` check. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
ae7c232409 |
fix(qdrant): anchor PVC ordinal so cleanup never deletes a sibling's data
The post-delete hook selected PVCs with an unanchored name prefix (qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting release "qdrant-test" matched not only its own PVC (qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC (qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting the sibling's data (or wedging it in Terminating via pvc-protection). Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own data PVCs match. A release name that is a prefix of a sibling no longer collides. - cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$". - tests: assert the anchored selector and add a notMatchRegex guard against regressing to the unanchored prefix. - e2e: gate the post-delete wait on a NotFound result specifically, so a transient API/auth error is not misread as "PVC reclaimed". Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Алексей Артамонов
|
c2a8a3b41f |
fix(mariadb): fail cleanup hook on real delete errors
The post-delete cleanup hook used `kubectl delete ... --ignore-not-found || echo WARNING`, which swallowed genuine RBAC/API failures and let the Job succeed with orphaned password Secrets still present. Track delete failures and propagate them via a non-zero exit, while --ignore-not-found still tolerates already-absent resources. Add a helm-unittest assertion locking in the non-zero exit-on-failure behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com> |
||
|
Aleksei Sviridkin
|
a349c753ad |
fix(kubernetes): make node group resources optional and prefer them over instanceType
A node group is sized either by instanceType or by explicit resources, but the generated values schema required resources on every node group. A node group with instanceType and no resources — or an empty resources dropped during typed-CR to HelmRelease translation — failed schema validation with "missing property 'resources'" even though sizing was fully specified. Mark resources optional in the node group definition and regenerate the schema, API types, README, and ApplicationDefinition. The chart template emits domain.cpu/memory only when resources is present, so a node group sized purely by instanceType now passes validation and installs. When a node group sets both instanceType and explicit resources, explicit resources take precedence: the instancetype is omitted from the rendered VM and the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit domain.cpu/memory with no instancetype to conflict with. instanceType stays required and defaulted (no schema change), so the console's required instanceType dropdown keeps working — sending the default instanceType plus explicit resources simply sizes the VM by resources. This replaces an earlier render-time hard failure on the both-set case, which was unreachable to clear from the dashboard (the required dropdown cannot be emptied). Explicit resources are all-or-nothing alongside an instanceType: a partial spec (only cpu, or only memory) on a group that carries a non-empty instanceType would emit one domain override next to the instancetype — which KubeVirt rejects — so it now fails at render time with a clear error. Partial resources on a group without an instanceType (used to tune kubelet reservations) are unaffected. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
665e7e4609 |
test(e2e): make etcd.bats teardown run and settle under cozytest (#3131)
## What this PR does The e2e runner `hack/cozytest.sh` is not real bats — it converts `@test` blocks to shell functions and recognises a `cozy_cleanup()` hook, but never calls `setup()`/`teardown()` and never opens fd 3. So `etcd.bats`'s `setup()` never ran (etcd leaked its HelmRelease and data PVCs into the shared `tenant-test` namespace) and `dump_diagnostics` wrote to an unopened fd 3, silently dropping failure diagnostics. The teardown's `kubectl wait --for=delete` lines also passed `--ignore-not-found`, which `wait` does not accept. This moves teardown into an `etcd_drain()` helper, run three ways: a pre-clean at the start of the first scenario, inline at the end of the last scenario (under `set -e`, so a teardown that does not settle fails the suite), and as `cozy_cleanup` (the best-effort safety net cozytest runs at suite exit and on the first failing test). The drain reclaims what a plain CR/HR delete leaves behind: - the StatefulSet's retained `data-etcd-*` PVCs (not owned by the Etcd CR) — a later install otherwise reuses the same-named volumes, boots the previous member identity, never forms a fresh quorum, and the chart's `helm --wait` fails; and - the cluster-scoped Kamaji `DataStore`, whose leaked finalizer wedges the HR uninstall — cleared each pass until the Etcd CR, HR, DataStore, and PVCs are all gone, bounded by a 4-minute deadline that fails the drain on timeout. Remnants are probed by name so a transient API error is never mistaken for a successful delete. Diagnostics now go to stdout, which cozytest captures and prints on failure. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved end-to-end cleanup so leftover resources are removed more reliably between runs. * Made failure diagnostics easier to capture when tests fail. * Added stronger cleanup handling at the start and end of the test flow to reduce flaky suite behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
0c6426feaa |
fix(seaweedfs): enforce read-only bucket access by bumping cosi-driver to v0.3.1
The seaweedfs COSI driver shipped at v0.1.2 hardcodes read-write S3 actions (Read, Write, List, Tagging) for every BucketAccess and never reads the accessPolicy parameter. The chart already defines a -readonly BucketAccessClass carrying parameters.accessPolicy: readonly, but v0.1.2 ignores it, so a credential meant to be read-only can still write (upload and delete) objects. Bump the driver to v0.3.1, which honours accessPolicy: readonly by issuing Read,List-only credentials (read-write and the empty default stay Read,Write,List,Tagging). The override lives in the package values.yaml rather than the vendored chart so it survives make update; the sister extra/seaweedfs chart already runs v0.3.1. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7c67aafed5 |
fix(kubernetes): run the Talos reconcile Job in the main install phase (#3145)
## What this PR does The tenant worker `MachineDeployment` `md0` intermittently never reaches `status.replicas=2`. CAPI blocks MachineSet creation until the `TalosConfigTemplate` that the deployment's `bootstrap.configRef` points to exists, and on failing installs that template is never created. ### Root cause — an ordering deadlock The `TalosConfigTemplate` is created only by the `talos-reconcile` **post-install Helm hook** (the chart can no longer render it at template time — three of its inputs are produced asynchronously after apply). A post-install hook runs only after Helm has applied the release's main resources and, when Helm waits, after they become Ready. But the same release also emits in-tenant addon HelmReleases — notably **cilium** — whose readiness needs tenant worker nodes, and those nodes can only be created once this hook has produced the `TalosConfigTemplate`. So when the main-resource wait is in effect, the install deadlocks: it waits for cilium to be Ready, cilium waits for worker nodes, the nodes wait for the `TalosConfigTemplate`, and the template waits for the post-install hook — which never runs, because the install never finishes the wait that precedes the hook phase. The install times out and `md0` never scales. The release sets `helm-install-disable-wait` precisely to skip that main-resource wait so the hooks run, but its effect is unreliable — which is what makes the failure bimodal: when the wait is skipped the hook runs and the template lands in ~46s; when it is not, the deadlock strands the install. A timestamped poller over a failing run (both `kubernetes-latest` and `kubernetes-previous`, 640 polls) captured: `TalosConfigTemplate` absent in 640/640 polls; both post-install hook Jobs (`talos-reconcile` and `bootstrap-token`) never created — the install never reached the post-install hook phase; all main resources applied (MachineDeployment, KubevirtMachineTemplate, MachineHealthCheck, WorkloadMonitor, child HelmReleases, Talos PKI/secrets) and the four hook inputs ready early and stable; the parent HelmRelease stuck at `Running 'install' action 20m` then looping into `upgrade`, with the cilium sub-release timing out on `cilium-operator` (which has no worker node to schedule on) — downstream of the missing template. ### The fix Render the `talos-reconcile` Job and its RBAC / ServiceAccount / egress `CiliumNetworkPolicy` as ordinary **main-phase** resources instead of post-install hooks. The Job is then created during the apply itself — not after the wait — and runs on the management cluster, which has nodes (unlike the tenant `cilium-operator`). It produces the `TalosConfigTemplate`, `md0` scales, workers join, cilium goes Ready, and the main-resource wait (if any) closes within the install budget: the Job completes in tens of seconds, and node boot plus cilium readiness is a few minutes — comfortably inside the 20m install timeout. This works whether or not `helm-install-disable-wait` takes effect, so it does not depend on that annotation's reliability. Because a main-phase Job is immutable, it carries a content-hash name suffix over its whole rendered spec (mirroring the `KubevirtMachineTemplate` idiom in `cluster.yaml`): an identical render keeps the same name and Helm no-ops it; any spec change yields a new name and a fresh Job, so no change ever lands on the same name and trips an immutable-field patch error. The applied `TalosConfigTemplate` keeps its stable `<release>-<group>` name (referenced by `bootstrap.configRef`), so an unchanged render re-applies a byte-identical template — a CAPI no-op that never rolls existing workers. The Job name is truncated to stay within the 63-char DNS-label limit while always preserving the hash. The runtime input-waits, the single `kubectl apply` of the template with a `KamajiControlPlane` ownerReference for GC, and the `extraHostEntries` are all unchanged. `helm-install-disable-wait`'s unreliable effectiveness is a real latent issue (it explains the created-vs-never bimodality), but this fix breaks the deadlock independent of it, so that is a separate follow-up rather than a blocker here. Supersedes #3139 (a 10-minute wait-budget bump that treated the symptom; the deadlock is structural and a larger budget would not have helped). ### Screenshots Not applicable — no UI changes. ### Testing helm-unittest extended to pin: no `helm.sh/hook` on any of the seven talos-reconcile documents, the content-hash name, `ttlSecondsAfterFinished`, the preserved template apply + ownerReference + extraHostEntries, the hash rotating on a spec change, and the long-name truncation staying within 63 chars. Full chart suite green (128 tests). The applied template is byte-identical to the previous hook output (verified by diff), so existing tenants are upgrade-safe. ### Release note ```release-note fix(kubernetes): create the tenant worker TalosConfigTemplate from a main-phase Job instead of a post-install hook, so worker MachineDeployments reliably scale during install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes addon install behavior by running reconcile work during the main chart phase, reducing readiness deadlocks and upgrade issues. * Added more reliable reconciliation handling with stable, change-detected job recreation and updated network access for the reconcile workload. * Refined chart and test guidance to match the new install flow and Helm behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |