mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
ci/parallel-container-disk
18
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
0240dc128f |
fix(platform): order backupstrategy-controller after cozystack-basics
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
6e219c241b |
feat(platform,gateway): distribute operator wildcard secret name
Add publishing.certificates.wildcardSecretName to the platform chart and propagate it through the cozystack-values channel as _cluster.wildcard-secret-name. Only the Secret NAME travels — the cert/key material is never broadcast, unlike the public kube-root-ca that already rides this channel. When the key is set, the gateway chart switches the rendered TenantGateway to certMode=existingSecret with a wildcardSecretRef pointing at the operator-supplied Secret, and skips the ACME solver/provider/issuer validations that no longer apply. Wildcard mode takes precedence over solver=dns01. TLS-passthrough services and attached namespaces continue to render unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1d747b3d8d |
feat(platform): auto-wire KubeVirt permittedHostDevices and HostDevices feature gate when gpu-operator is enabled (#2768)
Closes #2765. ## What this PR does Makes the passthrough and vGPU variants of `cozystack.gpu-operator` produce a KubeVirt CR that is immediately usable, without an out-of-band `kubectl patch`. Today the gpu-operator stack reports green, `nvidia.com/<model>` shows up under node `.status.allocatable`, but every `VirtualMachine` referencing the resource via `domain.devices.hostDevices` is rejected by the admission webhook because `HostDevices` is not in `developerConfiguration.featureGates` and the resource is not listed under `permittedHostDevices`. That manual reconciliation is now driven by the platform. The change has three parts. `packages/system/kubevirt` now parameterizes the KubeVirt CR template. `developerConfiguration.featureGates` keeps its existing static defaults and appends `.Values.extraFeatureGates`; `spec.configuration.permittedHostDevices` renders verbatim from `.Values.permittedHostDevices` when set; `spec.configuration.mediatedDevicesConfiguration` does the same. Defaults stay empty so the rendered CR is byte-identical to the previous template when no overrides are supplied. The chart's `make update` target re-applies the four sed transforms after fetching the upstream CR, so refreshing the KubeVirt release version stays a single command. `packages/core/platform` adds two defaults tables under `files/`. `gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName` slugs follow the `<arch>_<model>_<form>_<mem>` convention `nvidia-sandbox-device-plugin` generates as of v25.x, and `externalResourceProvider: true` is set everywhere because those resources are advertised by the sandbox plugin rather than KubeVirt's in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family span the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and Blackwell SR-IOV vGPU is intentionally out of scope for the default table and is expected to be advertised via the user-override hook described below. `templates/bundles/iaas.yaml` now selects between the two tables via a new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough, `vgpu` for mediated; default is passthrough). The same variant flows through to the gpu-operator Package CR, replacing the previous always-default call. When the package is enabled, the platform injects `HostDevices` into `extraFeatureGates` plus the chosen defaults into `permittedHostDevices` for the kubevirt component's values. Operators extend or replace via top-level platform values: ```yaml gpu: replaceDefaults: false # append (default) vs. wipe-and-replace permittedHostDevices: pciHostDevices: [] # cluster-specific PCI passthrough entries mediatedDevices: [] # cluster-specific vGPU entries mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings ``` `docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step and describe the auto-wiring contract, the per-variant defaults, the escape hatches, and the sandbox-device-plugin `resourceName` slug convention so an operator whose node publishes a different name knows where to override. ## Test coverage `helm-unittest` covers eight branches in `packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`: gpu-operator disabled emits no extras; gpu-operator enabled at default variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user overrides append by default; `replaceDefaults: true` wipes the platform table; an unknown variant fails fast with a descriptive error. `packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the empty-values render and the three populated shapes at the CR template level. `make helm-unit-tests` passes locally on the full repo. ### Release note ```release-note feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults. **Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Bundle-managed GPU Operator with selectable passthrough vs vGPU variants; platform now injects GPU settings into KubeVirt and forwards vGPU manager overrides when enabled. * Default platform mappings for PCI passthrough and vGPU mediated-device resources added. * **Documentation** * Expanded GPU/vGPU deployment guide with bundle-managed vs manual guidance, wiring/precedence rules, resource-name conventions, and verification steps. * **Tests** * New and expanded tests for bundle→KubeVirt wiring, variant behaviors, default/override merging, and idempotent template parameterization. * **Chores** * Added Helm test target and hardened KubeVirt template update/idempotency checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
931f286b9f |
fix(ingress,platform): deliver publishing.proxyProtocol to host ingress-nginx (#2799)
## What this PR does `publishing.proxyProtocol: true` is documented to enable PROXY-protocol on the host ingress-nginx, but the three nginx config keys (`use-proxy-protocol`, `real-ip-header`, `enable-real-ip`) never reached the controller. They were written into the `ingress-system` component values of the `cozystack.ingress-application` Package, but that component has no `install:` block, so the package controller never builds a HelmRelease for it and the values are dropped. External traffic behind a PROXY-protocol L4 load balancer kept breaking. This routes the switch through `_cluster.proxy-protocol` in the `cozystack-values` Secret — the same path `expose-external-ips` already travels to reach the host ingress release — and the `extra/ingress` chart applies it to the external-facing ingress (the namespace named by `expose-ingress`). The dead component-values injection is removed. The `extra/ingress` chart also gains a first-class `proxyProtocol` value, so PROXY-protocol can be enabled per Ingress application (a per-app boolean knob like `cloudflareProxy`). Two `fail` guards keep the new knobs safe: the per-app `proxyProtocol` value is rejected on the external-facing (host) ingress — that path must go through `publishing.proxyProtocol`, which also installs ouroboros (hairpin handling) and arms the disable-hazard guard — and `cloudflareProxy` + `proxyProtocol` together are refused, since Cloudflare's HTTP proxy does not speak PROXY-protocol to origins and its `real_ip_header` server-snippet would override `real-ip-header: proxy_protocol`. `use-forwarded-headers` / `compute-full-forwarded-for` are deliberately not set: without a paired `proxy-real-ip-cidr` they would let clients spoof `X-Forwarded-For`, and the PROXY-protocol frame is already the source of truth for the client address. helm-unittest coverage pins runtime delivery on the host path, the per-app path, and that non-host tenants stay untouched; the previous test that only checked template emission into the (non-consumed) Package component is replaced. Affected versions: v1.4.0–v1.4.2 and current main (introduced in #2561). **Upgrade note (action required).** Because the flag was a silent no-op before this fix, a v1.4.x cluster that set `publishing.proxyProtocol: true` aspirationally — without its L4 load balancer actually injecting PROXY frames — kept working by accident. After this fix the flag really enables `use-proxy-protocol`, which makes ingress-nginx require a PROXY frame on every connection; if the LB does not send one, all external ingress breaks. Before upgrading, confirm the upstream LB injects PROXY-protocol v1 frames, or unset `publishing.proxyProtocol`. Closes #2796 ### Release note ```release-note fix(ingress): publishing.proxyProtocol now enables PROXY-protocol on the host ingress-nginx (the config was silently dropped before reaching the controller); the Ingress application also gains a proxyProtocol value to enable PROXY-protocol per-app. ACTION REQUIRED on upgrade: the flag was previously a no-op, so a cluster that set it without the upstream LB injecting PROXY-protocol v1 frames will lose all external ingress once it takes effect — verify the LB, or unset publishing.proxyProtocol before upgrading. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tenant ingress support for PROXY-protocol via a new proxyProtocol option (default: false). * Platform-level publishing.proxyProtocol now controls external/host ingress behavior. * **Bug Fixes / Validation** * Rejects enabling per-app proxyProtocol on external-facing host ingresses. * Rejects using proxyProtocol together with cloudflareProxy to prevent conflicting real‑IP handling. * **Documentation** * Updated docs describing proxyProtocol usage, defaults, and safety requirements. * **Tests** * Added tests covering proxyProtocol wiring, config emission, and failure cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
479929ebfb |
ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown (#2558)
## What this PR does
Drops the 3× retry loop on `Run E2E tests` and `Install Cozystack into
sandbox`. `Prepare environment` keeps its 3× retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network) where
transient runner hiccups warrant a retry.
On failure, the test step now captures `kubectl get hr -A -o wide` and
`kubectl get events -A` under a collapsible group so triage starts with
the actual broken-state snapshot.
> [!NOTE]
> An earlier revision of this PR also doubled every bats timeout. That
commit was dropped in a rebase and is intentionally **not restored**:
the timeout class that actually matters (per-app HR-Ready waits) has
since been standardized at 5m on `main` (
|
||
|
Aleksei Sviridkin
|
c32812ccf9 |
fix(platform): deliver publishing.proxyProtocol to host ingress-nginx
The host PROXY-protocol switch wrote the nginx config keys into the ingress-system component values of the cozystack.ingress-application Package. That component has no install block, so the package controller never built a HelmRelease for it and the values were dropped — the host ingress-nginx never received use-proxy-protocol and external traffic behind a PROXY-protocol L4 load balancer kept breaking. Route the switch through _cluster.proxy-protocol in the cozystack-values Secret instead, the same path expose-external-ips travels to reach the host ingress release, and drop the dead component-values injection. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
117d317fd8 |
fix(platform): drop fragile numeric vGPU mediatedDeviceTypes defaults
The vgpu defaults shipped a mediatedDevicesConfiguration block whose mediatedDeviceTypes were hardcoded nvidia-NNN ids. Those ids are not stable identifiers — they are the per-(SKU, driver) indices under which a host enumerates mdev types in /sys/.../mdev_supported_types/, so the same profile gets a different number on a different card or driver branch. Auto-injecting them made KubeVirt try to create mdev types that usually do not exist on the operator's nodes, and duplicated work the GPU Operator's vGPU Device Manager already does. Delegate mdev creation to the vGPU Device Manager, which is name-based and stable, and keep only the name-based permittedHostDevices.mediatedDevices exposure list (mdevNameSelector) in the defaults. KubeVirt-driven creation stays available through the .gpu.mediatedDevicesConfiguration override for operators who supply host-verified type ids; document the lookup recipe. Update the bundle wiring, the gpu/vgpu values comments, the vGPU guide and the tests accordingly (the default vgpu path now emits no mediatedDevicesConfiguration; the override path is covered explicitly). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
919073601d |
fix(platform): match NVIDIA passthrough defaults to the resourceName the plugin advertises
The auto-injected permittedHostDevices only works if each resourceName equals what nvidia-sandbox-device-plugin advertises on the node; otherwise the admission webhook rejects the VM — the very manual step this bundle removes. Several defaults did not match the slug the plugin (nvcr.io/nvidia/kubevirt-gpu-device-plugin) derives from the PCI-IDs database name, which uppercases the name and keeps the die suffix, the Tesla brand and the form-factor/memory tokens: 10DE:1EB8 TU104GL_T4 -> TU104GL_TESLA_T4 10DE:20B7 GA100_A30 -> GA100GL_A30_PCIE 10DE:2331 GH100_H100_PCIE_80GB-> GH100_H100_PCIE 10DE:2321 GH100_H100_NVL_94GB -> GH100_H100L_94GB 10DE:1DB1/1DB4/1DF6 + new 1DB5 -> GV100GL_TESLA_V100_* family Device 10DE:1DB1 was also mislabelled: it is the SXM2 16GB part, not the 32GB one — the 32GB SXM2 V100 is a distinct device (10DE:1DB5) that was missing entirely. Relabel 1DB1 as 16GB and add 1DB5. Restate the header comment as the actual generation rule and pin the corrected slugs (T4, A30, both V100 SXM2 parts) with a regression test that also asserts the tidy/short forms never reappear. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
da0dcf9edf |
test(platform): pin nil-safe GPU bundle wiring against null gpu values
The .Values.gpu accesses in the iaas bundle are fully parenthesized
((( .Values.gpu ).x).y) so the field walk stays nil-safe when an operator
nulls the whole gpu block or leaves a sub-key blank (an empty YAML key
parses to null, which Helm then drops from .Values). A bare dotted walk
would panic with "nil pointer evaluating interface {}.<field>" on the
first nil hop.
Add regression cases covering gpu: null in both passthrough and vgpu
variants plus blank sub-keys, and a comment warning against flattening
the parenthesized form. A mutation check confirms the cases fail with the
nil-pointer panic the moment the parentheses are removed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
91efa1f4eb |
feat(platform): auto-wire KubeVirt host-device support when gpu-operator is enabled
When cozystack.gpu-operator is in bundles.enabledPackages, the platform
now mirrors the chosen GPU variant into the KubeVirt CR — the HostDevices
feature gate is appended to developerConfiguration.featureGates, and a
starter permittedHostDevices block is filled from a per-variant defaults
table. Without this, a passthrough-variant GPU shows up under
.status.allocatable on the node but the KubeVirt admission webhook
rejects every VM that references it as a hostDevices.deviceName.
Two defaults tables ship under packages/core/platform/files/:
- gpu-passthrough-defaults.yaml covers Hopper (H100/H200), Ada
Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10),
Turing (T4), Volta (V100/V100S). PCI vendor:device pairs are
stable; resourceName slugs mirror nvidia-sandbox-device-plugin's
<arch>_<model>_<form>_<mem> convention. externalResourceProvider
is set on every entry because the resources are advertised by
the sandbox-device-plugin DaemonSet, not by KubeVirt itself.
- gpu-vgpu-defaults.yaml covers the Pascal-Ampere mdev profile
range the upstream vgpu-device-manager walks /sys/class/mdev_bus/
for. Ada Lovelace and Blackwell SR-IOV vGPU is out of scope for
the chart's default list (NVIDIA driver path differs) and is
expected to be advertised via the user-override hook below.
bundles.iaas.gpuOperatorVariant picks which table applies (default =
passthrough, vgpu = mediated). The same variant flows through to the
gpu-operator Package CR, replacing the previous always-default call.
Operators extend or replace the defaults via .gpu in platform values:
gpu:
replaceDefaults: false
permittedHostDevices:
pciHostDevices: [...]
mediatedDevices: [...]
mediatedDevicesConfiguration: {...}
replaceDefaults: false (the default) appends the user list to the
NVIDIA table; true drops the table entirely for clusters with only
non-NVIDIA accelerators or a strict allowlist requirement.
Companion helm-unittest suite covers the eight branches: no gpu
package emits no extras; gpu-operator at default variant emits the
HostDevices gate plus pciHostDevices; vgpu variant emits the
HostDevices gate plus mediatedDevices and mediatedDevicesConfiguration;
user overrides append by default; replaceDefaults wipes; unknown
variant fails fast.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
80fbfa1ed8 |
fix(networking): point host ouroboros proxy at the root-tenant ingress (#2800)
## What this PR does With `publishing.proxyProtocol: true`, the host emits a `cozystack.ouroboros` Package using the no-override Package macro, so the proxy inherits the wrapper chart default backend `ingress-nginx-controller.cozy-ingress-nginx`. That layout describes a managed Kubernetes tenant cluster, not the host: on the host, `extra/ingress` deploys ingress-nginx into the root tenant namespace as `root-ingress-controller` in `tenant-root`. The composed backend FQDN never resolves, so `ouroboros-proxy` never reaches Ready. This emits the host ouroboros Package with a `proxy.target` override derived from `publishing.ingressName` — namespace plus `<short>-ingress-controller` service name — mirroring the `trimPrefix "tenant-"` transform `extra/ingress` applies to its release namespace. The derivation reuses the existing platform invariant that `publishing.ingressName` is the namespace where the host ingress controller runs. Adds bundle helm-unittest coverage for the default ingress name and a non-default one, and corrects the misleading wrapper `values.yaml` comment that claimed the host uses `cozy-ingress-nginx`. Scope: this fixes the ouroboros readiness defect only. The separate defect where the injected nginx PROXY-protocol config keys are not consumed by the host ingress is tracked independently and not touched here. Closes #2797 ### Release note ```release-note fix(networking): point the host ouroboros PROXY-protocol proxy at the root-tenant ingress so it becomes Ready when publishing.proxyProtocol is enabled ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced proxy protocol configuration to automatically route through the appropriate ingress-nginx service based on your ingress name settings. * **Tests** * Added test coverage for proxy protocol bundle wiring and ingress service routing. * **Documentation** * Clarified proxy target configuration behavior and fallback mechanisms. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
56014f1fdc |
fix(platform): order cozystack-basics after the APIs its admission policies type-check
cozystack-basics ships cluster-wide ValidatingAdmissionPolicies matching
tenants.apps.cozystack.io and the Gateway API types. KCM's VAP status
type-checker panics on a nil schema when it type-checks a policy against a
matched type that isn't resolvable yet; during a cold install this crashes
KCM, which stalls cert/token issuance, so cozystack-api never starts and the
tenants schema never resolves — a bootstrap deadlock.
Add dependsOn edges so the package installs only after cozystack-engine
(cozystack-api) and gateway-api-crds are Ready. Drift detection is off on
operator-generated HelmReleases, so a template capability gate would drop the
policy at first install and never re-add it; ordering is the reliable fix.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
2e23198b44 |
fix(platform): install velero by default
backupstrategy-controller is installed by default since |
||
|
Aleksei Sviridkin
|
f90411da7a |
fix(networking): point host ouroboros proxy at the root-tenant ingress
With publishing.proxyProtocol enabled the host emits a cozystack.ouroboros Package via the no-override Package macro, so the proxy inherits the wrapper chart default backend ingress-nginx-controller.cozy-ingress-nginx. That layout describes a managed Kubernetes tenant cluster; on the host, extra/ingress deploys ingress-nginx into the root tenant namespace as root-ingress-controller in tenant-root. The composed backend FQDN never resolves, so ouroboros-proxy never reaches Ready. Emit the host ouroboros Package with a proxy.target override derived from publishing.ingressName (namespace plus <short>-ingress-controller service name), mirroring the trimPrefix transform extra/ingress applies. Add bundle helm-unittest coverage for the default and a non-default ingress name, and correct the misleading wrapper values.yaml comment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
91aae36da3 |
feat(platform): NOTES.txt warns when no-delete VAP is silently skipped
Address review feedback from @lexfrei on packages/core/platform/templates/ deletion-protection.yaml: on a pre-1.30 cluster the capability gate (`.Capabilities.APIVersions.Has "admissionregistration.k8s.io/v1/ ValidatingAdmissionPolicy"`) evaluates to false and the policy + Binding render to nothing. There was no signal in `helm install`/`helm upgrade` output that the guardrail had been silently skipped, so operators on older clusters lost the protection without any way to discover that from the install log. Compounded the integration-test gap @lllamnyp called out in blocker 3. Add packages/core/platform/templates/NOTES.txt that renders a multi-line WARNING whenever the capability is absent: states the guardrail is NOT active, explains why (1.30+ required), notes that the label is still stamped on every protected object so the guardrail becomes active automatically on the next upgrade after a cluster bump, no migration step required. Add packages/core/platform/tests/notes_capability_gate_test.yaml: positive case asserts the warning text + the 1.30 hint + the label string render on a chart-level `capabilities.apiVersions: []`. Negative case asserts the warning is absent when admissionregistration.k8s.io/v1/ ValidatingAdmissionPolicy is in the capability list. Verified the positive case fails when the warning text is removed from NOTES.txt. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e34eeb0ee7 |
fix(platform): document --namespace bypass and add VAP unit tests
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml:
The bypass instructions in both the template-author-facing comment and the
user-facing CEL message did `kubectl label <kind> <name>` with no
--namespace flag. Several protected objects are namespaced (cozystack-version
ConfigMap, cozystack-packages Repository, tenant-root HelmRelease), so an
operator who copies the documented command verbatim against their default
context gets a "not found" error and concludes that the bypass mechanism is
broken. Updated both call sites to show --namespace <ns> with an explicit
"(omit for cluster-scoped resources)" caveat.
The reviewer also flagged that the VAP contract is otherwise unasserted by
unit tests. Add packages/core/platform/tests/deletion_protection_test.yaml
covering:
- both the VAP and the Binding carry platform.cozystack.io/no-delete=true
on themselves (the self-protection contract)
- failurePolicy is intentionally absent on the VAP
- the deny message contains the bypass command WITH --namespace and the
cluster-scoped caveat (regression catch for this fix; verified failing
when --namespace is dropped from the template)
- the Binding's objectSelector + validationActions: [Deny] still match
the no-delete label
Wire packages/core/platform/Makefile's test target to helm unittest so
hack/helm-unit-tests.sh actually runs the new suite in CI (the existing
tests/ files were orphaned without it).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Aleksei Sviridkin
|
d2e851f63b |
feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
7f01c7eb32 |
feat(platform): single switch for host PROXY-protocol + ouroboros
Adds publishing.proxyProtocol on the platform values: a single flag that both turns PROXY-protocol on at the host ingress-nginx (via the ingress-application Package CR's component values dict) and emits the new cozystack.ouroboros Package CR for the host install. The two pieces are coupled by design — turning PROXY-protocol on at ingress-nginx without ouroboros breaks intra-cluster traffic to the cluster's own public hostnames (kube-proxy / Cilium-KPR short-circuits the LB IP, the connection arrives at ingress-nginx without the PROXY header it now requires, and ingress-nginx closes the connection). The new sources/ouroboros.yaml PackageSource declares the host chart with dependsOn: cozystack.networking + cozystack.ingress-application so install ordering is deterministic. The bundle template emits the host ouroboros Package via the standard helper — the parent chart's Gateway-API watching default of false stays in place, because cozystack does not install Gateway-API CRDs cluster-wide on the host (system/ gateway-api-crds is referenced only from the kubernetes-application PackageSource, tenant scope). Forcing --gateway-api on a host without the CRDs would CrashLoop the controller on WaitForCacheSync against missing GVRs. The flag default is false. Clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. Disabling: flipping publishing.proxyProtocol back to false stops emitting the cozystack.ouroboros Package CR, but every Package CR carries helm.sh/resource-policy: keep — Helm leaves the existing Package on the cluster so ouroboros stays installed and the live Corefile rewrite block keeps pointing at the still-running ouroboros-proxy Service. Flag-flip alone is idempotent and safe. To actually uninstall ouroboros the operator removes the Package CR explicitly (or adds the package name to bundles.disabledPackages) and runs the documented Corefile cleanup recipe in the same window — otherwise stale rewrites point at a deleted Service and in-cluster DNS for hairpinned hostnames returns connection-refused. The values comment ships the cleanup recipe (a JSON merge-patch on data.Corefile that strips the BEGIN/END block while preserving labels, annotations, and other data keys including any Talos-managed metadata) and documents the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on), the host-vs-tenant fan-out (the flag does not cascade into tenant clusters; tenants opt in per-Kubernetes-CR via addons.ouroboros.enabled), and the Talos re-render flap window (machine-config pushes wipe the BEGIN/END markers; the controller re-applies on next reconcile, ≤10 minutes by default). Also drops the stale "distro-full" entry from the bundles.system.variant options comment in core/platform/values.yaml — the variant was removed in v1.0.1 and was never restored. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
