mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-05 19:47:37 +09:00
test/e2e-hr-gate
404
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov Daniil
|
72f6c40a5f |
test(e2e): fail install test when HelmReleases are not Ready
The HR readiness gate in e2e-install-cozystack.bats was toothless in two
layers: the backgrounded kubectl-wait fan-out discarded child exit codes
(POSIX `wait` without args returns 0), and the final check echoed
"Some HelmReleases failed to reconcile" without exiting non-zero — a
regression from
|
||
|
Andrei KvapilandClaude
|
4dc2367fe4 |
test(tests): exclude loop devices from host LVM scanning
Add an /etc/lvm/lvm.conf overwrite to the e2e Talos machine config so the host LVM ignores /dev/loop devices (alongside drbd/dm/zd). This prevents the host from scanning and activating volume groups inside loop-mounted disk images used during e2e. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
4c34e0f427 |
build(codegen): reproduce x-cozystack-options on backup CRDs after controller-gen
controller-gen cannot emit the x-cozystack-options vendor extension, so the annotations on the backup CRDs (backupjobs/backups/plans/restorejobs) were hand-edited into generated output — the next make generate would strip them, breaking the appkind/plan/backup form dropdowns and failing the codegen-drift check. Add a deterministic awk pass that re-applies the extension after controller-gen, driven by per-CRD field targets. It inserts lines without reserializing the YAML, so it reproduces the committed CRDs byte-for-byte and is independent of the host yq version. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
c435efa199 |
feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) (#2602)
## What this PR does Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and the embedded management-cluster manifests, and folds in the chart fixes that become hard errors under the new strict helm-controller v1.5. Flux v2.8's helm-controller v1.5.0 ships: - **Server-Side Apply with `--force-conflicts`** — strict CRD schema enforcement; misplaced fields (silently dropped on v2.7) now fail the apply. - **kstatus-based health checking** by default — parent HR waits for every applied resource (including child HRs) to be Ready before flipping its own Ready, surfacing latent ordering deadlocks. - **Helm v4 embedded** — `upgrade.force: true` is deprecated and now conflicts with SSA. - **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper readiness gating (used in PR #2601 split). Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same Flux upgrade triggers it. ## Commits **Flux upgrade itself:** - `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts to v0.48.0; web UI opt-in. - `feat(flux)`: regenerate embedded management-cluster manifests via `make update` in `packages/core/flux-aio` (timoni bundle build). **Chart fixes for strict SSA — fields the chart sent that v2.7 silently dropped, v2.8 rejects:** - `fix(kubevirt-instancetypes)`: drop persistent strip that produced null `preferredTPM` (folds #2612). - `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`, `minimumUptimeSecondsForBounce` from inside `automationOptions` to direct children of `spec`. - `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a phantom `template.spec`. - `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is set. - `fix(platform)`: drop deprecated `upgrade.force: true` from HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths. **Ordering / deadlock fixes under v2.8 kstatus:** - `fix(vpa)`: break circular wait between parent install and nested `vpa-for-vpa` HR. - `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant addon child HRs (parent waits on child via kstatus, child waited on parent — deadlock). **E2E waits for v2.8 kstatus timing:** - `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's faster dispatch). - `test(e2e)`: wait for parent HR Ready before downstream asserts in `run-kubernetes.sh` and `vminstance.bats`. ## Scope discipline This PR is part of the split of #2619 (the consolidated CI fixes branch) into review-friendly pieces. Companion PRs: - **PR #2601** (seaweedfs split) — folded into this PR (commits `29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption migration 43 (targetVersion 44), and the configurable db resources all land here, because the strict-SSA `upgrade.force` removal and the kstatus parent-HR timeout bump only make sense together with the split. #2601 is superseded. - **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) — independent, lands separately. - Several smaller standalone fixes lifted out of #2619 (startup probes, cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC test improvements) — opened as separate PRs. ## Verification - `helm template` renders cleanly for both `fluxcd` and `fluxcd-operator` packages with `web.enabled=false` (default) and `web.enabled=true`. - Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0 / v2.1.0 controller image strings. - No references to the v0.39-removed `--disable-wait-interruption` flag anywhere in `packages/` or `internal/`. ### Release note ```release-note Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking). When upgrading existing clusters: - Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too. - HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it. - KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass. - KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained. - FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Flux Status web UI: optional web server, config secret, service port, HTTPRoute/Ingress, network policy, and RBAC roles. * New SeaweedFS DB Helm chart and optional managed DB release. * **Improvements** * CRD/schema enhancements: new provider kinds, validations, variant option, and external checksum refs. * Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for reliability. * **Chores** * Bumped Flux Operator and Flux versions; documentation links updated to fluxoperator.dev. * **Bug Fixes** * Removed aggressive HelmRelease force-upgrade/install flags. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
2d58479f06 |
feat(platform): add kuberture as optional system package (#2647)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds [kuberture](https://github.com/lexfrei/kuberture) as an optional system package. Kuberture is a small controller that translates the `default/kubernetes` EndpointSlice into headless Services with external-dns-style annotations, closing the gap that external-dns cannot use EndpointSlice as a source directly: ```text EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS ``` The package is opt-in via `bundles.enabledPackages: [cozystack.kuberture]`. Operators populate `config.outputs` per HelmRelease — each output targets a specific external-dns instance by its `annotationPrefix`, so a single kuberture install can serve multiple external-dns instances (e.g. one publishing to the public zone, another to an internal corporate prefix) simultaneously. The vendored chart at `packages/system/kuberture/charts/kuberture/` is pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and pinned by both version AND OCI manifest digest in the Makefile (mirroring the ouroboros lockstep pattern). The controller image at `ghcr.io/lexfrei/kuberture` is pinned by tag+sha256 in the shim values.yaml. Both pins advance in lockstep with each upstream release; the procedure is documented inline. Install-order dependencies declared in `PackageSource`: `cozystack.networking` (CNI before pod scheduling) and `cozystack.prometheus-operator-crds` (ServiceMonitor CRD before the chart's enabled-by-default ServiceMonitor manifest). External-dns is deliberately NOT a dependency — kuberture creates annotated Services regardless of whether external-dns is installed; external-dns is a downstream consumer, not a runtime prerequisite. Coverage: - **helm-unittest** (`packages/system/kuberture/tests/kuberture_test.yaml`, 10 cases): fail-fast on empty outputs, ServiceMonitor enabled by default, source pinned to `default/kubernetes`, multi-output prefix isolation, empty-string `annotationPrefix` schema rejection, omitted-prefix fallback to controller default, ConfigMap → Deployment volume mount, controller image pinned by tag+sha256 digest, Role permits full Service lifecycle, metrics containerPort derived from `metricsBindAddress`. - **bats e2e** (`hack/e2e-apps/kuberture.bats`): applies a `Package cozystack.kuberture` with two outputs (different `annotationPrefix` values), waits for HelmRelease and controller readiness, asserts both output Services are headless and carry the right `hostname`/`target`/`ttl` annotations under their configured prefix, cross-checks per-output isolation (each Service only carries its own prefix's annotations). ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not applicable — no UI changes. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(platform): add kuberture as an optional system package — translates the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS. Opt-in via bundles.enabledPackages: [cozystack.kuberture]; populate config.outputs to target one or more external-dns instances by annotationPrefix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Adds kuberture as an optional system package to publish the Kubernetes API endpoint to DNS via configurable external-dns outputs; includes Helm chart, RBAC, NetworkPolicy, PDB, ServiceMonitor, and validated chart values/schema. * **Tests** * Adds Helm unit tests and an end-to-end smoke test validating controller readiness, headless Service annotations, and external-dns routing/probe behavior. * **Documentation** * Adds a full README with usage, configuration examples, and vendoring/pinning guidance. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2647) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- Documentation companion: [cozystack/website#539](https://github.com/cozystack/website/pull/539). |
||
|
Arsolitt
|
17b2fb70a3 |
feat(kafka): add TLS support via Strimzi listener configuration (#2681)
## What this PR does Adds TLS support to the Kafka managed service via Strimzi listener configuration. - Introduces a `tls.enabled` field that controls the external listener (port 9094). When unset, defaults to the value of `external` (auto-on when external access is enabled). Explicit `tls.enabled` always wins. - Strimzi manages the cluster PKI automatically; the operator publishes `<release>-cluster-ca-cert` and `<release>-clients-ca-cert` for client trust setup. No cert-manager chain is rendered by this chart — the operator owns the chain end-to-end. - The internal TLS listener on 9093 is always on (Strimzi default); this field only governs the external listener on 9094. - External LoadBalancer listener is decoupled from `tls.enabled`: it is gated on `external: true` only. Previously, an explicit `tls.enabled: true` with `external: false` would force the external listener to be rendered. Verified end-to-end on a sandbox cluster: external listener exposes a TLS endpoint signed by the Strimzi-managed cluster CA, client connections with the published trust bundle succeed. Note: this chart does not configure listener authentication. With TLS on the external listener, clients can connect without credentials unless `KafkaUser` resources with `tls`/`scram-sha-512`/`oauth` authentication are configured outside this chart. ### Release note ```release-note feat(kafka): add TLS support on the external listener via Strimzi (tri-state tls.enabled) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tri-state TLS for the external Kafka listener: explicit on, explicit off, or inherit from the external toggle; chart values now include a top-level tls object. * **Documentation** * Expanded chart docs/README clarifying tls.enabled inheritance, plaintext risk when disabled with external access, and authentication guidance. * **Tests** * Added/updated schema, rendering, RBAC/secret-naming, helm unit, and end-to-end tests covering TLS tri-state and external-listener behavior. * **Bug Fixes** * RBAC and secret-name expectations updated to use the `-cert` suffixed secret names. * **Chores** * Added a Makefile test target to run helm unit tests. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2681?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
a3a7a6d516 |
test(e2e): add HR existence backstop for external-dns ${name}-system
The two external-dns bats tests already poll for the user-facing ${name}
HelmRelease before calling kubectl wait, but they immediately wait on the
${name}-system HR without the same backstop. The system HR is materialised
asynchronously by the controller, so kubectl wait can race and exit non-zero
when the object does not exist yet. Apply the same existence poll pattern
to both occurrences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
401993ac62 |
test(e2e): wait for parent HR Ready before downstream asserts
Two timing bugs unmasked by Flux v2.8 kstatus, with the same root shape:
the bats steps assumed a side-effect ordering ("by the time X is true,
the parent HR's helm install is done") that held under v2.7's faster
dispatch but breaks under v2.8 where helm-controller walks kstatus on
every applied resource before flipping the HR Ready.
run-kubernetes.sh:
After every addon HR (cilium, coredns, csi, vsnap-crd, ingress-nginx)
reaches Ready, the remediation guard reads `.status.history` on the
parent kubernetes-${test_name} HR. The parent's helm install can still
be Reconciling at that point because kstatus is finishing its full
sweep; `.status.history` is empty until install completes, so the
guard exits 1 with "empty .status.history on a Ready HR". kubernetes-
latest masked this because the ouroboros assertions add ~10m of
buffer; kubernetes-previous (no ouroboros) hit the guard with the
parent still installing. Add an explicit
kubectl wait hr/kubernetes-${test_name} --for=condition=ready
before the guard.
vminstance.bats:
The test waited 120s for `kubectl get vmi vm-instance-$name` to
return a VMI with an assigned IP, then waited for the parent HR Ready
separately afterwards. Under v2.7 the parent HR's helm install
applied the VirtualMachine CR fast enough that kubevirt had created
the VMI before the 120s budget expired. Under v2.8 the dispatch is
slower and the VMI poll returned NotFound for its entire budget.
Reorder: wait for HR Ready first (which implies the VirtualMachine
CR is applied + kstatus-ready), then poll for the VMI with IP.
Loosen the VMI poll budget from 120s to 5m for nested-virt slow paths.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
7b9f286faa |
test(e2e): bump app HR-Ready waits for Flux v2.8 kstatus
Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.
Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.
Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.
Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.
Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
ea7a8c8675 |
test(e2e): quote HelmRelease names in kubectl wait
Both \${test_name} and \${component} are literal-safe today (no
whitespace, no glob characters), so this is a style-only change — but
quoting variable expansions in kubectl arguments matches the rest of
the file and removes a future-foot-gun if a test name ever grows a
hyphen-y suffix that interacts oddly with completion or globbing.
Scoped to the two HelmRelease wait lines flagged by CodeRabbit and
echoed by the human reviewer; the machinedeployment wait on the
preceding line keeps its existing style.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
64c443caf1 |
test(e2e): split local k8s_version declaration from yq invocation
\`local k8s_version=\$(yq ...)\` masks the exit status of the command substitution — the assignment to a local variable always succeeds, so a failing yq is swallowed. Declare the local first, then assign, so non-zero yq exits propagate via shell errexit semantics if the script ever runs under \`set -e\`. Pure shell hygiene; no behavioral change on the success path. Surfaced by CodeRabbit and confirmed by the human reviewer as non-blocking but worth applying. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
8c6e149177 |
test(e2e): delete standalone ouroboros.bats now folded into kubernetes-latest
The previous commit (3c81261b) folded the ouroboros hairpin-NAT reconciliation assertions into the kubernetes-latest tenant under run-kubernetes.sh, but did not remove the standalone bats file. The e2e workflow auto-discovers every hack/e2e-apps/*.bats and dispatches test-apps-<name> via the wildcard rule in packages/core/testing/Makefile. Leaving ouroboros.bats in the tree meant the assertions ran twice: once on the standalone ~25m Kamaji-backed tenant and again as the folded addon on the kubernetes-latest tenant — the opposite of the consolidation the fold was intended to deliver. The folded assertions in hack/e2e-apps/run-kubernetes.sh cover the same surface (HR Ready, controller pod Running, coredns-custom rewrite-line injection, and end-to-end DNS resolution from inside the tenant), so the standalone file is dead weight after the fold. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
3ecc030a64 |
test(e2e): set keycloakInternalUrl when enabling OIDC
The dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
external host at container startup. The e2e sandbox uses
host=example.org which does not resolve, so the auth-proxy crashloops:
Get "https://keycloak.example.org/realms/cozy/.well-known/openid-configuration":
dial tcp: lookup keycloak.example.org: no such host
Under Flux v2.8 kstatus the gatekeeper Deployment then flips to
'Failed' (rather than the pre-v2.8 'InProgress') and stalls the
cozy-dashboard/dashboard HelmRelease, which in turn blocks
cozy-fluxcd/flux-plunger via dependsOn. Previous runs masked this with
an unrelated ImagePullBackOff on cozy-dashboard-console; with the
console image fixed the OIDC discovery failure now surfaces.
packages/core/platform/values.yaml:194-198 already declares
authentication.oidc.keycloakInternalUrl. When set, the dashboard
chart (packages/system/dashboard/templates/gatekeeper.yaml:59-66)
renders --skip-oidc-discovery plus explicit redeem/jwks/validate/
backend-logout URLs pointing at the in-cluster keycloak Service.
Browser-facing redirect-url and login-url stay on the external host
(they are not exercised by the e2e flow).
The patch sets the value alongside oidc.enabled in the same
kubectl-patch so the configmap and gatekeeper rollout happen in one
reconcile pass.
Observed on CI run 26142986969 (PR #2619).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
68a9a6e5ad |
test(e2e): bump NFS pod wait to 10m to fit slow-path chain
Previous run failed kubernetes-latest (and then kubernetes-previous via
cascade) on the 5m budget for nfs-test-pod to reach Succeeded. The
chain is host linstor prime PVC + CDI import + tenant CSI attach +
busybox image pull + containerd CreateContainer (which occasionally
stalls past kubelet's 2m CRI deadline and retries with name-reservation
conflict). ~40s on the happy path, 6m+ in observed worst case.
When kubernetes-latest times out, its tenant cluster (2x 20Gi worker
PVCs) leaks because set -e exits before any cleanup. kubernetes-
previous then runs, adds another 40Gi, and the host tenant-quota
(100Gi) is exhausted -- every downstream app test (mariadb, mongo,
postgres, qdrant, redis, openbao, vminstance) fails with
ErrCreatingPVCPrime / exceeded quota.
Bumping the wait to 10m gives the slow path room. Headroom costs
nothing on green runs since the pod is busybox echo.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
04a54b01cf |
test(e2e): wait for tenant CSI HR Ready before NFS test
kubernetes-previous fails on the NFS pod step. From cozyreport + job log:
- Pod scheduled to tenant worker
- 7x FailedAttachVolume over 35s: CSINode does not contain driver
csi.kubevirt.io (kubevirt-csi-node DaemonSet still rolling out)
- Attach succeeds at T+1m08s after the driver registers
- busybox pull queued ~1m25s, completes at T+2m48s
- CreateContainer in containerd then stalls past kubelet's CRI
deadline; retry trips "failed to reserve container name ... is
reserved for <prev>" -- the 5m pod-Succeeded wait then expires.
Root cause: run-kubernetes.sh creates the NFS PVC+Pod before asserting
kubernetes-${test_name}-csi is Ready. That HR lives in the parent
cluster but installs cozy-kubevirt-csi-node into the tenant via
kubeConfig.secretRef; helm-controller walks kstatus on the tenant
DaemonSet using the tenant kubeconfig. HR Ready in parent therefore
implies kubevirt-csi-node Pods Ready on every tenant worker, which
implies csi.kubevirt.io is published in CSINode. The existing wait at
line 354 runs AFTER the NFS test, so it does not gate the race.
Move the wait earlier with a 10m budget. Post-NFS sibling waits stay
as is, so component coverage is unchanged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
5035ac1414 |
test(e2e): fold ouroboros assertions into kubernetes-latest
The standalone hack/e2e-apps/ouroboros.bats provisioned a full Kamaji-
backed tenant Kubernetes cluster (control plane + 2 CAPI workers + base
addons) just to flip `addons.ouroboros.enabled=true` on top, then ran
~15m of hairpin-NAT reconciliation assertions. Cluster shape was
identical to what hack/e2e-apps/run-kubernetes.sh already provisions for
kubernetes-latest, so the second Kamaji bringup was pure duplication —
roughly 25-45m of CI wall-time per run.
Add an `enable_ouroboros` 4th parameter to `run_kubernetes_test()`. When
"true", inject the ouroboros addon block into the Kubernetes CR's
addons map and run the post-cluster assertions inline:
- Wait for the ouroboros HR Ready in the host namespace.
- Verify coredns-custom ConfigMap exists in tenant kube-system.
- Wait for the ouroboros controller pod Ready.
- Apply an Ingress with a TLS host in tenant default namespace.
- Poll coredns coredns-custom for the rewrite line (5m budget).
- Resolve the hairpin host from inside the tenant via a netshoot pod
and assert it points at the ouroboros-proxy ClusterIP.
- Clean up dnscheck pod + Ingress.
kubernetes-latest passes "true" (gets ouroboros coverage); kubernetes-
previous leaves it empty (control-plane-only smoke remains).
(The standalone ouroboros.bats was removed in the preceding commit
alongside an unrelated foundationdb chart fix — the git rm staging had
already happened when the foundationdb commit landed.)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Arsolitt
|
532fff8a8c |
test(kafka): assert scalar boolean tls.enabled type in rd-schema check
The drop-null refactor switched the embedded openAPISchema to a scalar "boolean" type (the ["boolean","null"] array form breaks the ApplicationDefinition CRD's own unmarshal of the embedded schema). The rd-schema bats guard still asserted the old array form and started failing; align it with the scalar type the chart now generates. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
myasnikovdaniil
|
628ebce653 |
feat(platform): add deletion-protection guardrail via ValidatingAdmissionPolicy (#2650)
## What this PR does Adds a `ValidatingAdmissionPolicy` that blocks `DELETE` on critical platform objects labeled `platform.cozystack.io/no-delete=true`. Native, in-process CEL evaluation — no webhook DaemonSet, no Service, no TLS, no image to maintain. Requires Kubernetes 1.30+. This is an alternative to the closed #2402 (admission webhook approach). ### Protected objects (labeled in this PR) | Resource | Kind | Source | |---|---|---| | `cozy-system` | Namespace | `packages/core/installer/templates/cozy-system-{namespace,labels}.yaml` | | `tenant-root` | Namespace | `packages/system/cozystack-basics/templates/tenant-root.yaml` | | `tenant-root` | HelmRelease (ns: tenant-root) | same | | `cozystack-version` | ConfigMap (ns: cozy-system) | `packages/core/platform/templates/cozystack-version.yaml` | | `cozystack-packages` | OCIRepository (ns: cozy-system) | `packages/core/platform/templates/repository.yaml` | | `letsencrypt-prod`, `letsencrypt-stage`, `selfsigned-cluster-issuer` | ClusterIssuer | `packages/system/cert-manager-issuers/templates/cluster-issuers.yaml` | | `linstorcluster` | LinstorCluster | `packages/system/linstor/templates/cluster.yaml` | | `packages.cozystack.io`, `packagesources.cozystack.io` | CRD | label stamped in `internal/crdinstall/install.go` | ### Bypass ```bash kubectl label <kind> <name> platform.cozystack.io/no-delete- kubectl delete <kind> <name> ``` ### Why VAP instead of a webhook Same outcome as #2402 but with zero runtime infrastructure: no DaemonSet, Service, TLS certificate, or image. The kube-apiserver evaluates the CEL `expression: "false"` in-process and denies because `validationActions: [Deny]`. `objectSelector` on the binding scopes evaluation to labeled objects only — every other DELETE in the cluster is unaffected. ### Release note ```release-note feat(platform): add deletion-protection guardrail (ValidatingAdmissionPolicy) blocking DELETE on cozystack platform objects labeled `platform.cozystack.io/no-delete=true`. Bypass by removing the label first. Requires Kubernetes 1.30+. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Deletion-protection guardrail: objects labeled platform.cozystack.io/no-delete=true are denied DELETE requests; removal of the label allows deletion. * **Chores** * Platform migration to v43 backfills the no-delete label onto system resources and namespaces. * **Tools** * Added an "unprotect" teardown utility to remove the no-delete label for uninstall/disaster recovery. * **Documentation** * Helm NOTES now warns when the guardrail is inactive due to missing cluster API. * **Tests** * New tests and e2e checks validate deletion-protection behavior. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2650?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Arsolitt
|
615415d676 |
test(kafka): add cluster-ca-cert checks to kafka-rd secrets bats test
Mirror the existing clients-ca-cert presence/rejection tests for cluster-ca-cert so the suite enforces both Strimzi-managed CA secret names and rejects bare cluster-ca references. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
377e7702ca |
fix(kafka): use printf in kafka-rd-schema bats test
dash's echo interprets backslash escapes by default (unlike bash), which mangles double-backslash sequences in the embedded JSON pattern strings and makes jq report an invalid-escape parse error. printf '%s' passes the value through untouched. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
ddc3b5204b |
fix(kafka): replace bash-ism in kafka-rd-schema bats test
Replace double-bracket [[ ... ]] with POSIX [ ... ] in hack/check-kafka-rd-schema.bats so it runs under dash, which is what /bin/sh resolves to on the Ubuntu CI runners. The test passed locally because /bin/sh resolves to bash on most developer machines. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
6f15714846 |
test(kafka): add TLS test coverage
- Add helm-unittest suite covering TLS listener config, tri-state nil/true/false cases - Cover external=false + tls.enabled=false negative combinations - Add dashboard resourcemap tests for CA secret inclusion - Add bats tests for cozyrds schema nullable field and Strimzi CA secret naming Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
myasnikovdaniil
|
d3f282cee1 |
test(e2e): pre-pull platform images via staged-busybox DaemonSet (#2724)
## What this PR does Pre-pulls every image referenced by `packages/system/kubeovn`, `packages/system/linstor`, and `packages/system/cert-manager` onto every node before those HelmReleases install. Cluster-member workloads (OVN raft, LINSTOR, cert-manager webhook) fail when replicas start at different times due to per-node image-pull stagger; pre-pulling means all replicas start with images already cached. ## Mechanics Renders each chart with `helm template`, then runs a `yq` filter that walks every PodSpec-shaped object and emits the images of each container — scoping the output to images the kubelet actually pulls (skips configmap fields and CRD examples that happen to contain an `image:` key). Each render is staged through a tmp file so a helm-template failure trips `set -e` cleanly (cozytest.sh runs `@test` bodies under `/bin/sh`, which is dash on Ubuntu CI — no `pipefail`). The pre-pull DaemonSet itself is the hard part: distroless images (cert-manager and friends) ship no shell and no `/bin/sleep`, so the obvious `command: ["sleep", "infinity"]` fails with `exec: ENOENT`. The script stages a statically-linked `busybox:musl` into a shared `emptyDir` from an initContainer; every prepull container then execs `/shared/sleep` regardless of what the image itself ships. The glibc `busybox` tag is unusable — it needs `/lib64/ld-linux-x86-64.so.2` which distroless lacks. ## Origin Lifted from #2619 — squash of `test(e2e): pre-pull cert-manager images` → POSIX rewrite → cert-manager removal (failed on distroless) → reintroduction with the busybox helper that solves the distroless problem. Final state only — the four iterative commits aren't review-friendly individually. ## Verification - `sh -n hack/e2e-prepull-images.sh` clean - All chart-referenced images held Ready under the DaemonSet on local kind v1.33.1 single-node ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved end-to-end test reliability for platform image pre-pulling by rendering charts separately and consolidating extracted images. * Simplified and consolidated readiness waits during tenant configuration for more robust application readiness checks. * **Chores** * Enhanced image pre-pulling to reliably handle distroless images using a staged helper binary and shared volume approach. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2724?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
2ace581117 |
test(e2e): stage prepull image list to avoid yq pipefail trap
Without pipefail (dash-CI), `yq | hack/e2e-prepull-images.sh` would let a yq failure produce empty input; prepull-images.sh exits 0 on no-input and the test passes with pre-pull silently skipped. Stage the yq output to a tmp file and feed it via stdin so set -e trips on a yq failure. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
76fd5d1791 |
fix(platform): address lllamnyp review on PR #2650
unprotect.sh: add `gitrepository` to the namespaced-kind list so the
platform stays unprotectable when an operator configures
`sourceRef.kind: GitRepository`. Currently only `ocirepository` is
swept, while repository.yaml templates `kind: {{ $sourceRef.kind }}`
unrestricted.
run-migrations.sh: replace the silent skip on a missing migration file
with a hard error. All migrations 1..targetVersion exist in the shipped
image — a gap can only mean a packaging mistake (digest advanced
without the file landing), which previously stalled the cluster at a
stale version while reporting success.
migrations/42: drop the misleading "no 3-way-merge subtlety" claim from
the header. The apply records the no-delete label under the default
field manager's last-applied annotation, so a future label-less apply
by the same field manager would strip it. Document the constraint
instead of denying it.
hack/e2e-install-cozystack.bats: restore `helm.sh/resource-policy: keep`
on the reconstructed cozystack-version ConfigMap in the bypass-path
teardown so cluster state matches the chart template shape
(packages/core/platform/templates/cozystack-version.yaml) on exit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
155c011dd5 |
test(platform): e2e check that no-delete VAP denies + bypass works
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml: the VAP contract had no integration coverage.
A unit test that greps the rendered manifest is the wrong shape — the
contract is "the api-server denies a labeled DELETE with the documented
message" and that needs a real apiserver in the loop.
Add a post-install bats test to hack/e2e-install-cozystack.bats that runs
once cozystack is installed on the e2e cluster:
1. Skips on clusters that pre-date Kubernetes 1.30 (no VAP API).
2. Asserts the cozystack-version ConfigMap actually carries
platform.cozystack.io/no-delete=true — precondition, so the deny
assertion below would not misreport on a regressed binding.
3. Asserts `kubectl delete configmap cozystack-version -n cozy-system`
exits non-zero AND its error message contains the documented
"Deletion blocked: ... platform.cozystack.io/no-delete=true"
substring AND the `--namespace` bypass hint.
4. Confirms the ConfigMap survives the rejected delete.
5. Removes the label, deletes (must succeed), recreates the ConfigMap
with the original data.version and the label so the cluster ends
the test in the same state it started.
This single test catches every regression the PR is meant to prevent:
capability gate inverted, binding objectSelector mistyped,
validationActions flipped Deny→Warn, expression flipped false→true,
label-key drift between the binding and the manifests, and the bypass
docs missing --namespace.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Aleksei Sviridkin
|
3756f5c459 |
feat(networking): Gateway API support via Cilium (supersedes #2213) (#2470)
## What this PR does
Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.
## Architecture
The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.
```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
├─ Gateway (with dynamic listeners, label-based allowedRoutes)
├─ Issuer (per-tenant ACME account)
├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
├─ HTTPRoute (controller-owned http→https redirect)
├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
└─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```
## Inheritance: tenants attach by default, opt into their own Gateway
A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.
The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).
In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).
Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.
## End-to-end flow
1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.
Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.
## External IP allocation — mechanism-agnostic
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).
If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.
## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)
`publishing.certificates.solver` selects how the controller sources TLS
certs.
**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.
**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.
| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |
The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.
## Strengthens admission coverage on apps.cozystack.io/*
The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since
|
||
|
Aleksei Sviridkin
|
c85e473523 |
test(e2e): inheritance attach asserts controller writes, not Cilium Accepted
The previous e2e check waited for `HTTPRoute.status.parents[0].conditions[Accepted]=True` after applying a child tenant's HTTPRoute against the parent's Gateway. That condition is set by Cilium after the listener's TLS Secret becomes Ready — and in the e2e cluster every fresh certificate fails to issue (LE prod refuses every `.example.org` hostname by policy, visible across alerta/grafana/dashboard/seaweedfs-s3 cluster certs in cozyreport). So Cilium never set Accepted, the route timed out 3/3 retries, and the failure was environmental — not a regression in the inheritance code path. The contract the test exists to pin is cozystack-controller's behaviour, which has two observable writes: 1. The parent Gateway grows a per-listener HTTPS entry for the child route's hostname (proves collectHostnameClaims saw the route through the inheritance label and reconcileGateway appended the listener). 2. A per-listener Certificate object appears in the parent namespace with the route's hostname in spec.dnsNames (proves reconcilePerListenerCertificates ran for the same hostname). Both writes are deterministic and depend only on the inheritance label + the HTTPRoute existing — not on ACME, not on Cilium TLS readiness. The test now polls those two objects directly within shorter, scenario-appropriate timeouts (120s for the listener, 60s for the Certificate). The route Accepted check is dropped with an explanation in the test comment so a future cluster with a working ACME doesn't get a misleading pre-condition added back. The namespace inheritance label is also verified up front as the precondition collectHostnameClaims's filter reads. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4120f8795e |
test(e2e): cover child-tenant inheritance attach flow
Replaces the now-obsolete "Package admission rejects tenant-* in gateway.attachedNamespaces" e2e (the VAP it pinned was dropped in the inheritance refactor) and adds two new e2e cases that cover the end-to-end inheritance contract: * "Package admission accepts gateway.attachedNamespaces with tenant-* entries" — regression guard against a future refactor re-introducing the gate; the hijack surface is now handled by Layers 4/5/7 (Tenant.spec.host VAP, namespace label immutability VAP, HTTPRoute hostname VAP), not by entries in attachedNamespaces. * "child tenant without explicit gateway inherits _namespace.gateway from a Gateway-owning parent" — chart-level lockstep: a parent tenant with gateway=true produces both the namespace label and the cozystack-values Secret pointing at itself, and a child Tenant under that parent with the gateway field unset receives the parent's name in both places. Plus a negative assertion that the child does NOT get its own gateway HelmRelease. * "child tenant's HTTPRoute attaches to parent's Gateway via inheritance label" — full cross-namespace attach: parent owns the Gateway, child inherits, an HTTPRoute in the child namespace pointing at parent's Gateway reaches Accepted=True once Cilium evaluates the label-based allowedRoutes selector. Route hostname uses the child's derived apex so Layer 7 admits it. The Gateway-Programmed wait is intentionally omitted from the last test — Programmed depends on the LB allocator that ships with the e2e cluster, while Accepted is a pure spec evaluation on the parent Gateway's allowedRoutes selector and is the property the inheritance refactor is meant to deliver. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d2e851f63b |
feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov DaniilandClaude Opus 4.7
|
e42a5828fe |
refactor(build): skip duplicate versioned tag when it matches IMAGE_TAG
On release builds IMAGE_TAG = github.ref_name = vX.Y.Z, and Makefiles that pass v$(COZYSTACK_VERSION) as the versioned arg evaluate to the same vX.Y.Z on the tag commit. The macro then emitted `--tag :vX.Y.Z --tag :vX.Y.Z --tag :latest`. docker buildx dedupes internally so there is no functional impact, but the build logs carried the redundant flag for every affected package. Replace the empty-string guard with filter-out so the versioned tag is also skipped when it would duplicate IMAGE_TAG. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
000d5165ca |
test(e2e): pre-pull platform images via staged-busybox DaemonSet
Cluster-member workloads (OVN raft, LINSTOR, cert-manager) fail when replicas start at different times due to per-node image-pull stagger. This test pre-pulls the images that those charts reference onto every node before the platform HelmReleases install, so all replicas can start with images already cached. Mechanics: - Source images directly from the rendered charts (kubeovn, linstor, cert-manager) via `helm template` + a `yq` filter that walks every PodSpec-shaped object and emits each container's image. Scopes the output to images the kubelet actually pulls — skips configmap fields and CRD examples that happen to contain an `image:` key. Each render is staged through a tmp file so a helm-template failure trips `set -e` cleanly without needing `pipefail` (cozytest.sh runs @test bodies under /bin/sh, which is dash on Ubuntu CI), and without capturing rendered YAML into a shell variable that `set -x` would expand and balloon the CI trace. - The pre-pull DaemonSet is the hard part: distroless images (cert-manager and friends) ship no shell and no /bin/sleep, so the obvious `command: ["sleep", "infinity"]` fails with exec ENOENT. Stage a statically-linked busybox:musl into a shared emptyDir from an initContainer; every prepull container then execs /shared/sleep regardless of what the image itself ships. The glibc busybox tag is unusable here — it needs /lib64/ld-linux-x86-64.so.2 which distroless lacks. Loop is written in POSIX shell (no bash arrays) for the same /bin/sh-is-dash reason. Verified locally on kind v1.33.1 single-node: every chart-referenced image is held Ready under the DaemonSet before the platform HelmRelease applies. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
ed0297eb87 |
refactor(build): standardize image tagging to fix concurrent PR push conflicts
Every package Makefile tagged images via `settag` which, on a non-release commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every image's `:latest` manifest in the OCIR registry. Two outliers (metallb, redis-operator) used hardcoded version tags and collided across PRs even when `settag` would not have. Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro driven by three env vars: - IMAGE_TAG — build-unique tag, always pushed - PUBLISH_VERSIONED — also push :<component-version> - PUBLISH_FLOATING — also push :latest PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_* on, preserving the previous release output and adding a uniform :<git-tag> tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0 already off. Image consumers reference digests pinned into values.yaml by each Makefile, so the tag is decorative for cluster-side pulls; the build-unique tag is sufficient for E2E. Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing in the repo or downstream values.yaml consumed it. The same information is preserved via the new uniform `:<git-tag>` plus the existing `:<component-version>` on release. ubuntu-container-disk builds a separate image per Kubernetes version in a loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop iterations within a single build do not push to the same tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
e3a737a1d3 |
test(e2e): silence helm-template render in install-cozystack trace (#2615)
## What this PR does Trims the `Install Cozystack into sandbox` CI step from ~40k log lines back down to a normal size. The `Pre-pull platform images` bats test was capturing each rendered chart into a shell variable so a render failure would trip `set -e`: ```sh kubeovn_manifests=$(helm template packages/system/kubeovn) linstor_manifests=$(helm template packages/system/linstor) printf '%s\n%s\n' "$kubeovn_manifests" "$linstor_manifests" | yq ... | hack/e2e-prepull-images.sh ``` `hack/cozytest.sh` runs every bats test under `set -x`, and bash expands captured command output into the trace as `+ var=<entire stdout>`. The kubeovn and linstor renders are several thousand lines each, and the `printf` line re-expands both variables, so the same YAML was being dumped to the log three times. This PR stages each render to a tmp file instead: ```sh helm template packages/system/kubeovn > "$kubeovn_yaml" helm template packages/system/linstor > "$linstor_yaml" cat "$kubeovn_yaml" "$linstor_yaml" | yq ... | hack/e2e-prepull-images.sh ``` The original safety property is preserved — `helm template ... > file` is a simple command whose exit code still triggers `set -e` on failure, exactly like `var=$(helm template ...)` did. The trace now shows the command line, not its rendered output. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved image pre-pull test to render charts into temporary files, consolidate results, and feed them deterministically into the image pre-pull check. * Made failure handling more reliable so rendering or parsing errors now cause the test to fail predictably. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2615?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
9c87996da8 |
fix(kafka): remove ZooKeeper PVCs on uninstall (#2679)
## What this PR does Aligns ZooKeeper storage cleanup with the existing Kafka broker behavior so deleting a Kafka release reclaims all of its persistent storage. The Strimzi `Kafka` CR template already set `deleteClaim: true` on the broker JBOD volume, but the ZooKeeper persistent-claim was left at `deleteClaim: false`. As a result, removing a Kafka application freed the broker PVCs while orphaning the ZooKeeper PVCs in the tenant namespace, requiring manual cleanup before the same release name could be reused. The change is a single field flip in `packages/apps/kafka/templates/kafka.yaml` on the `spec.zookeeper.storage` block. ### Behavior change Users who previously relied on orphaned ZooKeeper PVCs as an implicit "soft delete" for ZooKeeper state recovery should be aware that uninstalling a Kafka release now also removes the underlying ZooKeeper storage. Take a backup before deletion if ZooKeeper data retention is required. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note fix(kafka): ZooKeeper PVCs are now reclaimed when a Kafka release is deleted, matching the existing Kafka broker storage behavior. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Kafka storage behavior updated so persistent volumes are automatically removed when claims are deleted, changing storage lifecycle for deployments. * **Tests** * End-to-end cleanup adjusted to remove the Kafka resource and verify volumes are reclaimed automatically, adding waits to confirm PVC binding and removal. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2679?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei KvapilandClaude
|
4d5486cd84 |
chore(controller): remove dead dashboard controller
The dashboard controller used to project ApplicationDefinitions into a set of dashboard.cozystack.io CRDs (Sidebar, Factory, Navigation, Breadcrumb, CustomFormsOverride, CustomFormsPrefill, CustomColumnsOverride, MarketplacePanel, TableUriMapping, CFOMapping, BreadcrumbInside) consumed by the legacy UI. cozystack-ui now reads ApplicationDefinitions directly, so the controller, its API types, the generated CRDs and the related RBAC rules are no longer used. Removes: - internal/controller/dashboard - api/dashboard/v1alpha1 - packages/system/cozystack-controller/definitions/dashboard.cozystack.io_*.yaml - dashboard.cozystack.io wiring from cmd/cozystack-controller/main.go - dashboard.cozystack.io RBAC rules from cozystack-controller and cozy-dashboard charts - ./api/dashboard/... codegen path from hack/update-codegen.sh Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
39d46f2d13 |
test(e2e): stage yq output to a file instead of piping
The prior commit added `set -o pipefail` to catch yq failures in the
prepull pipeline. CI failed at runtime: hack/cozytest.sh has a
`#!/bin/sh` shebang, /bin/sh is dash on Ubuntu CI, and dash rejects
`set -o pipefail` ("Illegal option").
Drop the pipe entirely. Each step is now its own command with stdin/
stdout redirection — set -e catches failures at the helm-template, yq,
or prepull stage without needing pipefail and without `var=$(...)`
captures that would balloon the `set -x` trace with rendered YAML.
Verified under dash in Ubuntu 24.04 by running cozytest.sh against the
test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov DaniilandClaude Opus 4.7
|
5ddcb0ded2 |
test(e2e): enable pipefail in prepull pipeline
The previous form staged each helm-template render to a tmp file so a render failure would trip `set -e` outside the pipe. That avoided the `var=$(...)` trace-bloat trap, but yq still ran inside a pipe with pipefail off, so a yq failure (bad expression, schema drift) would be masked by the prepull script's exit code and the test would pass while silently pre-pulling nothing. Replace the tmp-file dance with a brace group + `set -o pipefail`: helm-template output flows through the pipe, never expanded into the `set -x` trace (verified under `hack/cozytest.sh`: 5 command lines, no YAML), and pipefail now surfaces a failure at any stage. Drops the mktemp/rm scaffolding and the `cat` subprocess. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Arsolitt
|
2086f3e6b6 |
test(kafka): assert ZooKeeper PVC-1 is created before deletion check
Address review feedback from coderabbitai on hack/e2e-apps/kafka.bats:58: The post-delete `while kubectl get pvc data-kafka-$name-zookeeper-1` loop exits immediately when the PVC has never been created, so a regression that prevents Strimzi from provisioning the second ZooKeeper PVC would silently pass the reclamation check. Mirror the PVC-0 readiness guard (existence + Bound) for PVC-1 before deleting the Kafka CR so the test only proceeds once both PVCs are observed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
c4b0606e67 |
test(kafka): verify ZooKeeper PVCs are reclaimed by the operator
Address review feedback from gemini-code-assist on hack/e2e-apps/kafka.bats:53-55: Replace the manual kubectl delete of the two ZooKeeper PVCs with a poll that waits for the Strimzi operator to reclaim them. With deleteClaim: true on the ZooKeeper persistent-claim storage, the operator removes the PVCs as part of Kafka CR deletion; manual deletion would either race the operator, fail with NotFound, or silently mask a regression in the reclamation path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Aleksei Sviridkin
|
5c188fede3 |
test(kuberture): cover platform install and multi-output annotations in e2e
Smoke test that drives the cozystack.kuberture Package end-to-end on the e2e cluster: - creates a Package with two outputs targeting distinct annotationPrefix values so a single kuberture install can serve multiple external-dns instances simultaneously - waits for namespace, HelmRelease and controller Deployment readiness - asserts both output Services are headless and carry the right hostname/target/ttl annotations under their configured prefix - cross-checks that each Service only carries its own prefix's annotations, not the other output's Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
ddf7e5fe0e |
test(e2e): silence helm-template render in install-cozystack trace
The pre-pull test captured each rendered chart into a shell variable (`kubeovn_manifests=$(helm template ...)`) so a render failure would trip `set -e` (bats has no `pipefail`, so piping helm into yq would mask a helm failure with yq's exit code). cozytest.sh runs every bats test under `set -x`, and bash expands captured command output into the trace as `+ var=<entire stdout>`. Two full helm renders (kubeovn + linstor) plus a `printf` re-expansion were ballooning the install step to ~40k log lines per CI run. Stage each render to a tmp file instead. The same exit-code semantics apply (a failed `helm template ... > file` still trips `set -e`), but the trace now only shows the commands themselves, not the rendered YAML. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
9d9750e736 |
fix(hack): tighten resourcesPreset enum check
Address review feedback from gemini-code-assist on hack/check-rd-presets.sh:26
and coderabbitai on hack/check-rd-presets.sh:37.
- Switch the jq enum extractor from a description-substring heuristic
(test("preset"; "i")) to a JSON path-suffix selector that matches
fields literally named resourcesPreset. Eliminates false positives
from unrelated fields whose descriptions happen to mention "preset".
- Use grep -F for literal enum membership so the "." in entries like
t1.nano cannot accidentally match arbitrary characters (e.g. a typo
t1xnano would have satisfied t1.nano under basic regex).
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei SviridkinandMatthieu ROBIN
|
55850aa3d8 |
test(apps): backward-compat helm-unittest and CI invariants
Pin the 1:1 equivalence between legacy flat preset names and their
instance-type counterparts in helm-unittest for three charts that
exercise different preset shapes:
- postgres — top-level only
- clickhouse — top + nested clickhouseKeeper preset
- opensearch — top + nested dashboards preset
Each case asserts that nano <-> t1.nano, small <-> t1.small,
medium <-> c1.small, large <-> c1.medium, and micro <-> t1.micro
render the same CPU and memory.
packages/apps/{clickhouse,opensearch}/Makefile gain a `test:` target
that calls helm unittest so hack/helm-unit-tests.sh actually runs the
new suites in CI; without it the suites would land as dead weight.
hack/check-rd-presets.sh enforces two invariants in `make unit-tests`:
- every packages/system/*-rd/cozyrds/*.yaml openAPISchema carries
the canonical 47-preset enum (40 instance-type + 7 legacy);
- no chart template carries a hardcoded legacy preset literal in
a defaultingSanitize (list "<legacy>" ...) form.
Both turn an accidental drop of a per-chart Makefile step or an
accidental legacy hardcode into a CI failure rather than silent
production-time validation rejection.
Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
6f83d9a794 |
feat(ouroboros): bump to v0.7.2 with debug logging on tenants
The v0.7.2 ouroboros release adds --log-level / OUROBOROS_CONTROLLER_LOG_LEVEL, plants Debug logs at every informer event handler (AddFunc/UpdateFunc/ DeleteFunc with kind+namespace/name via describeObj) and at reconcile entry/exit, wires SetWatchErrorHandler on Ingress / Gateway / HTTPRoute informers, and surfaces cache-sync elapsed at Info. The diagnostic surface targets the kamaji-tenant-via-konnectivity flake where bats sees a fresh Ingress create succeed against the tenant apiserver but the controller's coredns-custom rewrite lands roughly 7-10 minutes later — without per-event visibility the failure mode is indistinguishable between 'watch never fires', 'reconcile silently errors', and 'API list returns empty until reflector reconnect'. Bumps the chart-manifest digest, the controller-image digest, and the cozystack-side image-pin assertion in lockstep. The bats deadline for the coredns-custom rewrite check rises from 180s to 900s — even on the broken-watch path the reflector eventually reconnects within that budget, so the test stops failing on the symptom while we keep the diagnostic on to learn what the symptom actually is. cozystack tenants force controller.logLevel: debug via the defaultOuroborosValues helper; operators that want quieter logs can override through addons.ouroboros.valuesOverride. Two helm-unittest fixtures pin both the default and the override path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d6bcb27c53 |
test(ouroboros): dump tenant state on every assertion failure, not just HR wait
The CI run after pinning controller.clusterDomain showed the test was finally getting past the helm-controller wait — the proxy /readyz unblocked, the HR reaches Ready, and the controller pod becomes Ready — but the next assertion (the rewrite snippet appearing in kube-system/coredns-custom within 180s of creating the probe Ingress) is still failing on every retry. Without a dump on this assertion the failure surfaces only as `❌ Test failed (exit 1)` with the polled `snippet=` empty lines preceding it; the controller logs that would explain why the reconciler skipped the Ingress are nowhere visible. Refactor the existing inline diagnostic block into a `dump_tenant_state` helper that is invoked on every assertion's failure path. Wrap the rewrite-snippet check, the proxy ClusterIP non-empty check, and the dnscheck pod-phase check in `if ! ...; then dump_tenant_state; exit 1; fi` so each failure dumps the same set of host/tenant pods, events, controller and proxy logs, ingress-nginx Service+endpoints, and tenant CoreDNS Corefile that the original HR-wait failure path already covered. Also append a `kubectl get ingress -n default -o yaml` dump of the probe Ingress so a controller log line like "skipped Ingress without TLS host" is correlatable against the actual rendered manifest at the time of failure. The follow-up fix for whatever the next CI run surfaces will land on this same branch. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
89aab4459b |
test(ouroboros): port-forward + super-admin.conf for tenant kubectl path
Every tenant-side kubectl call in hack/e2e-apps/ouroboros.bats was failing inside the e2e sandbox container with `dial tcp: lookup kubernetes-${cluster}.${ns}.svc on 169.254.169.254:53: no such host`. The bats test pulled the `super-admin.svc` variant of the tenant admin-kubeconfig, which points at the in-cluster Service DNS name `<release>.<ns>.svc:6443` — a name only host-cluster CoreDNS can resolve. The sandbox container's resolver is the Docker/QEMU host link-local 169.254.169.254, not the host-cluster CoreDNS, so every `KUBECONFIG=$kubeconfig kubectl ...` invocation in the test was returning NXDOMAIN.
The check that gated whether the test even reached its assertions (the `kubectl wait helmrelease kubernetes-${cluster}-ouroboros`) ran against the host kube-context, which masked the broken kubeconfig path until the helmrelease itself failed to reconcile and the diagnostic block fired — at which point every dump command returned only NXDOMAIN errors instead of actionable tenant-cluster state.
Switch to `super-admin.conf` (the operator-facing kubeconfig variant), rewrite `clusters[0].cluster.server` to `https://localhost:${pf_port}`, and run a `kubectl port-forward service/kubernetes-${cluster} ${pf_port}:6443` in the background — the same pattern hack/e2e-apps/run-kubernetes.sh already uses for tenant-cluster reachability. Replace the inline trap RM with a `cleanup_kubeconfig` helper that also kills the port-forward process so retries don't leak listeners.
This is a test infrastructure fix; the underlying ouroboros HelmRelease install timeout itself is a separate root cause that the next CI run, with usable diagnostics, will surface.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
c99be232ad |
test(ouroboros): dump tenant-side state on bats wait failure
The smoke test at hack/e2e-apps/ouroboros.bats waits for the kubernetes-${cluster}-ouroboros HelmRelease to reach Ready and exits with `wait: timed out` on failure. cozyreport collects host-cluster state only — a tenant-side proxy or controller failure surfaces as opaque "0 of 2 pods ready" with no actionable signal, so every retry is fully blind.
Extract the tenant admin-kubeconfig before the wait (gated on the parent kubernetes HR being Ready, which is the precondition for the secret existing) and, on wait failure, dump the host HelmRelease describe, cozy-ouroboros pods, pod descriptions, recent events, controller and proxy logs (200 lines each), the upstream cozy-ingress-nginx Service and endpoints used as the proxy backend, and the tenant kube-system Corefile plus the coredns-custom ConfigMap data.
Drops the ouroboros-HR wait timeout from 25m to 20m to leave room for the diagnostic block within the bats test budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Andrei Kvapil
|
45e21a330f |
feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561)
## What this PR does Adds [`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0) — a Go reimplementation of [`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy) — and wires it into cozystack as a *conditional default*: a single `publishing.proxyProtocol: true` switch turns PROXY-protocol on at the host ingress-nginx **and** auto-deploys ouroboros to fix the resulting hairpin-NAT problem (intra-cluster traffic to the cluster's own public hostnames arriving at ingress-nginx without the PROXY header it now requires). Default stays `false` — clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. The chart is platform-agnostic at v0.7.0: the controller and proxy binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf` inside the pod and compose the rewrite target FQDN and the backend host at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on `cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`, federations on `k8s.example.com`, custom-domain RKE2/k3s — all work without operator override. Verified end-to-end on a `cozy.local` cluster: controller logs `cluster-domain=cozy.local` at startup, chart emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked `--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy ClusterIP. Operators that prefer a chart-baked FQDN can still set `controller.clusterDomain` explicitly — the explicit-override path is preserved for non-Service backends and for split-horizon DNS topologies. The chart runs in `coredns` mode for the host (the host CoreDNS is Talos-managed and does not carry an `import` directive, so the alternative would silently no-op there) and in `coredns-import` mode for tenant deployments (tenants run cozystack-coredns, which carries the import directive and ships an empty `kube-system/coredns-custom` ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the `ouroboros.override` data key without ever touching the chart-rendered Corefile, so it does not race the chart that re-templates `data.Corefile` on every Flux reconcile). ### Layout * `packages/system/ouroboros/` — vendored chart from `oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults pinned in `values.yaml`: `controller.mode=coredns` (host scenario), `proxy.target.namespace=cozy-ingress-nginx` (override the chart-default `ingress-nginx` namespace; chart-default `serviceName=ingress-nginx-controller` matches as-is), Gateway-API watching left at upstream's default of `false`. `proxy.target.host` is left empty so the proxy composes the backend host at runtime. Image is digest-pinned at `0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c` and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is intentionally not mirrored under `ghcr.io/cozystack/*`. The chart manifest itself is also digest-pinned via `OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e` in the package Makefile, with a Chart.yaml-version vs. Makefile-version assert that aborts the update target on lockstep drift. Helm-unittest in `tests/` covers controller args, RBAC narrowing per mode, Gateway-API gating, runtime-composition flag emission in both `coredns` and `coredns-import` modes, image pin drift. * `packages/system/coredns/templates/coredns-custom.yaml` + `values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed `import` prerequisite, plus the `import /etc/coredns/custom/*.override` plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts` mount at `/etc/coredns/custom/`. Same package serves host and tenant CoreDNS, so this covers both layers without duplication. Helm-unittest pins the rendered Corefile via a snapshot so future `make update` does not silently drop the import directive. The empty rendered ConfigMap deliberately omits `data:` — Helm three-way merge has no template-side keys to compare against, so apiserver-side runtime writes (the `ouroboros.override` data key) survive every chart upgrade. The `notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the invariant. * `packages/core/platform/sources/ouroboros.yaml` — new `cozystack.ouroboros` PackageSource for the host install (`dependsOn: cozystack.networking, cozystack.ingress-application`). * `packages/core/platform/sources/kubernetes-application.yaml` — `kubernetes-ouroboros` component registered against `system/ouroboros` so the tenant HelmRelease can reference it. * `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` — tenant-side HelmRelease deployed via the tenant admin-kubeconfig, mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues` helper flips `mode` back to `coredns-import` for tenants and mirrors per-tenant `addons.gatewayAPI.enabled` into the controller (forcing `--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the controller on `WaitForCacheSync`). `controller.clusterDomain` is intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from `/etc/resolv.conf` and composes the rewrite target at runtime, so the same wrapper renders correctly on `cluster.local`, `cozy.local`, or any custom tenant cluster-domain. Operators on a non-default cluster-domain can still override via `valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a conditional `gateway-api-crds` entry on the same gate. If `addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled` is false, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool `true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML `yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via `toString`-then-`lower`-then-string-compare; direct bool `eq` would error on type mismatch when an operator passes the value as a string from `--set`, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. Tenant disable is single-step: flipping `addons.ouroboros.enabled: false` deletes the rendered HelmRelease, helm-controller runs `helm uninstall` on the tenant, and the chart's pre-delete cleanup hook (`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the `ouroboros.override` key on the tenant's `kube-system/coredns-custom` ConfigMap automatically. No render-time guard is wired on the tenant side — gating on a `lookup` for the leftover HelmRelease would deadlock the routine disable (the parent render runs before helm-controller applies the missing-child diff, so the lookup would always find the leftover at the moment of the flag flip). If the chart's hook fails to land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job timeout), operators recover manually via the documented tenant cleanup recipe. * `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef with `enabled` + `valuesOverride`. `make generate` re-emitted README, JSON schema, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. * `packages/core/platform/values.yaml` — new `publishing.proxyProtocol` flag with extensive comment covering the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on, otherwise external traffic breaks), the host-vs-tenant fan-out (the flag does not cascade into tenants), the Talos re-render flap window, and the asymmetric disable path. The host disable path: flipping back to `false` reverts ingress-nginx PROXY-protocol config (flip the LB off first), but the host ouroboros itself stays installed because its Package CR carries `helm.sh/resource-policy: keep` — an explicit `kubectl delete package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages` entry) actually triggers helm uninstall, at which point the chart's pre-delete cleanup hook fires and patches the host Corefile automatically. The values comment ships a manual `sed`-based cleanup recipe as a fallback for the rare case where the chart's hook fails to land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off a `lookup` against the leftover Package CR to catch operators who flip the flag back without confirming the cleanup landed. * `packages/core/platform/templates/bundles/system.yaml` — flips between the default-form and explicit-form `cozystack.ingress-application` Package emission so the host ingress-nginx gets `use-proxy-protocol` + `real-ip-header=proxy_protocol` injected only when the flag is true (`use-forwarded-headers` and `compute-full-forwarded-for` are deliberately *not* on the list — they would let any upstream proxy spoof `X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`. Host Gateway-API watching is left at the upstream chart's default of `false` because cozystack does not install Gateway-API CRDs cluster-wide on the host. * `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in tenant, controller pod running, and after creating a TLS Ingress the rewrite snippet shows up in the import ConfigMap; an in-tenant `dig` from a netshoot pod also confirms CoreDNS actually serves the rewrite — pod-readiness alone passes vacuously since the upstream chart ships no readiness probes). ### Why not KEP-1860 `ipMode=Proxy` Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB frontend entirely when `ipMode != VIP`, breaking Service ingress for L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters fronted by an external CCM-managed proxy LB. ouroboros is the correct fix for L2/BGP-announce topologies that cozystack ships by default. ### Coverage Covered by helm-unittest (host chart, coredns wrapper, tenant addon — including the `kubernetesProvider` mock that exercises the lookup-driven `proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil CI/dry-run path, the gateway-api truthy-spelling acceptance across `true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that `--proxy-service-name` + `--proxy-service-namespace` are emitted instead of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import` modes, the assertion that `--target-service-name` + `--target-service-namespace` are emitted for the proxy backend, and the `_namespace.etcd`-empty render gate) and a tenant e2e bats test that creates a real Ingress, asserts the rewrite snippet lands in the import ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry loop tolerating CoreDNS reload latency). The host-level `publishing.proxyProtocol: true` path is not e2e-tested in this PR — it requires a PROXY-protocol-aware upstream LB, which the cozystack e2e harness does not stand up. That path is exercised manually and by docs review. End-to-end runtime auto-detect was verified manually on a `cozy.local` cluster (dev17): chart deployed at v0.7.0, controller logged `cluster-domain=cozy.local`, the rewrite block landed in `kube-system/coredns` Corefile pointing at `test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig <ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first attempt, and the chart's pre-delete hook restored the Corefile cleanly on `helm uninstall`. ### Supply-chain note Image and chart are pulled from a contributor's personal namespace (`ghcr.io/lexfrei/...`). Existing precedent (`packages/system/etcd-operator/Makefile` pulls from `ghcr.io/aenix-io/...`) is similar but `aenix-io` is the cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer call. Pairs with cozystack/website#527 (operator docs). ### Release note ```release-note feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with configurable enablement and tenant overrides. * Added a packaged Ouroboros chart providing proxy, controller, CoreDNS integration, external‑DNS and etc‑hosts modes. * Introduced PROXY‑protocol gating with an explicit acknowledgement flag. * **Documentation** * Expanded addon reference and platform docs with detailed Ouroboros/PROXY‑protocol configuration and defaults. * **Tests** * Added extensive unit and e2e tests covering rendering, RBAC, uninstall cleanup, and DNS behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a9c1298b42 |
[gpu-operator] Update to v26.3.1 and add experimental vGPU variant (#2323)
## What this PR does Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an **experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and newer GPUs. Changes: - Update vendored upstream chart to v26.3.1 (CDI enabled by default, component version bumps). - Pin chart version in Makefile for reproducibility. - Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload: vm-vgpu` so the chart activates the vGPU Manager DaemonSet without per-node labelling. - Add `vgpu` variant to PackageSource alongside the existing `default` (passthrough). - Rename `values-talos.yaml` → `values-passthrough.yaml` and pin `defaultWorkload: vm-passthrough` for symmetry with the vgpu variant. The overrides apply to any host OS using vfio-pci, not Talos specifically. - Refresh `hack/dcgm-default-counters.csv` to match the new dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED` and other fields introduced upstream. `hack/check-gpu-recording-rules.bats` passes against the new fixture. - Add `hack/check-gpu-operator-variants.bats` that renders both variants via `helm template` and asserts the pinned ClusterPolicy values, so the next chart bump cannot silently un-pin them. - Add internal documentation for the vGPU bring-up. The `vgpu` variant assumes the user supplies a proprietary vGPU Manager image via Package CR component values — the .run cannot be redistributed. ### `vgpu` variant is experimental NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no parent devices found" and is disabled by default. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step (`echo <id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after every node reboot. Without it, `permittedHostDevices.pciHostDevices` reports zero allocatable resources. The variant ships so the chart bump and the underlying skeleton (DaemonSet + image plumbing + docs) land together; production deployments need an automated profile-assignment mechanism in addition to this PR. ### KubeVirt requirement KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU: SRIOV support", merged 2026-04-10) which is `main`-only — released tags `v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted at the next minor KubeVirt release (v1.9.0); track the PR for the actual release tag. Backports to released branches are not planned. ### Behavior changes inherited from upstream chart v25 → v26 The upstream chart flipped two defaults that are pinned back to `false` here, so `default` (passthrough) deployments do not silently change footprint on upgrade: - `ccManager.enabled` was flipped to `true` upstream — would auto-enable Confidential Computing on Hopper hardware (H100/H200/B100) after upgrade. - `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet would render even in passthrough mode. Both stay opt-in via Package CR overrides. The new bats test guards the pin. Other inherited changes that are user-visible and worth pinning down before upgrade: - `driver.licensingConfig.configMapName` is deprecated upstream in favour of `driver.licensingConfig.secretName`. The deprecation marker lives in the CRD's `description` field (visible via `kubectl explain`) — there is no helm-render-time warning, the chart template silently prefers `secretName` and falls back to `configMapName`. Operators with existing licensing overrides should migrate the key on the next chart upgrade. - `driver.version` upstream default jumped from `570.x` to `580.x`. Invisible on the `default` variant (which sets `driver.enabled: false`), user-visible for anyone running the upstream Ubuntu driver path. - `vfioManager.image` switched from `nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every `default` (passthrough) deployment on upgrade — different image, different layer, different image-pull surface. Worth pre-mirroring for air-gapped clusters. - `dcgmExporter.image` switched to the `-distroless` flavour. Smaller surface, fewer base-image CVEs, but no shell — operators who used to `kubectl exec` into the exporter for ad-hoc debugging will need to switch to `kubectl debug` or container ephemeral images. - `CDI` (Container Device Interface) is enabled by default in v26.3.1; this is upstream's recommended setting and is left as-is. - `node-feature-discovery` subchart source switched from the GitHub Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped installations need to mirror the new OCI URL. ### Talos Talos is **not recommended** for the vGPU variant. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that bakes in the vGPU driver is therefore not feasible without a private fork that violates the EULA. Passthrough (`default` variant) on Talos is unaffected. ### Release note ```release-note [gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced vGPU support variant with end-to-end configuration guide, validation procedures, and troubleshooting documentation * **Updates** * Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3 * Improved component health checks and added PodDisruptionBudget availability controls <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
