mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-06 20:13:43 +09:00
Aleksei Sviridkin
3364290761
## What this PR does Adds an opt-in `publishing.exposure=loadBalancer` mode for the ingress-nginx Service as a migration path away from `Service.spec.externalIPs`, which is deprecated upstream in Kubernetes v1.36 ([KEP-5707](https://github.com/kubernetes/enhancements/issues/5707), [kubernetes#137293](https://github.com/kubernetes/kubernetes/pull/137293)). The `AllowServiceExternalIPs` feature gate is expected to default to off around v1.40 and the implementation to be removed around v1.43. Stacked on top of #2464 (cilium v1.19.3 bump) — depends on it for the `CiliumLoadBalancerIPPool` at `cilium.io/v2`. ### Behavior - New platform value `publishing.exposure` — enum `externalIPs | loadBalancer`, default `externalIPs` (current behavior unchanged on upgrade). - Plumbed through `cozystack-values` into each tenant's ingress HelmRelease via the new `expose-mode` key. - When `exposure=loadBalancer` and the current namespace matches `publishing.ingressName`, the Service becomes `type: LoadBalancer` with `externalTrafficPolicy: Local` and a `CiliumLoadBalancerIPPool` announces the addresses from `publishing.externalIPs` via Cilium LB IPAM. - The pool uses a namespace-only `serviceSelector` (`io.kubernetes.service.namespace: <ns>`) — any LoadBalancer Service in the tenant namespace draws from it. See "Pool ownership" below. - IPv4 addresses get `/32` CIDRs, IPv6 addresses get `/128`. Mixed families supported. Pre-CIDR entries (`192.0.2.10/32`) are accepted without double-suffixing. - Unknown values, `loadBalancer` with empty externalIPs, and stray empty entries from `publishing.externalIPs` are rejected at render time with explicit error messages. ### Pool ownership The `CiliumLoadBalancerIPPool` is rendered from `packages/apps/tenant/templates/cilium-lb-pool.yaml`, not from the ingress chart. The tenant chart is the per-tenant owner of cross-cutting resources (Namespace, `cozystack-values` Secret, HelmReleases for ingress and gateway), so one pool per tenant lives there. Cilium LB IPAM rejects overlapping CIDRs across pools regardless of `serviceSelector` — the last-added pool gets `cilium.io/PoolConflict` and stops allocating. Keeping the pool in the ingress chart would collide with the Gateway-API PR (#2470), which materialises its own LoadBalancer Service from the same `publishing.externalIPs` range when `tenant.spec.gateway=true`. Moving the pool to the tenant chart with a namespace-only selector lets a single pool back both services (ingress-nginx today, a Cilium Gateway Service once #2470 lands). Only the ingress-loadBalancer signal is wired here (`_cluster.expose-mode=loadBalancer` + `.Values.ingress=true` + publishing tenant). #2470 rebases on top of this PR, drops its own `packages/extra/gateway/templates/cilium-lb-pool.yaml`, and adds an OR branch for `.Values.gateway` in the tenant template. ### Scope Only the ingress-nginx Service is migrated by this setting. Other cozystack components that still write `Service.spec.externalIPs` directly (notably the `vpn` app at `packages/apps/vpn/templates/service.yaml`) need separate follow-up before Kubernetes v1.40. ### Tests - `packages/extra/ingress/tests/exposure_test.yaml` — 10 helm-unittest cases on the ingress-nginx Service: type / externalTrafficPolicy / externalIPs assertion for both modes, unknown-mode rejection (case-sensitive enum), empty-IPs failure, empty-entry filtering, non-publishing-tenant fallback. - `packages/apps/tenant/tests/exposure_test.yaml` — 9 cases on the pool itself: IPv4, IPv6, mixed, pre-CIDR input, empty-entry filtering, `ingress=false` in publishing tenant (no pool), non-publishing tenant (no pool), empty externalIPs (no pool). - Both suites are auto-discovered by `hack/helm-unit-tests.sh` via the `test:` target in each package's Makefile. ### Caveats (copied from the inline `values.yaml` comment) - `loadBalancer` mode uses `externalTrafficPolicy: Local`. The external IP must already be routed to a node that hosts an ingress pod (floating IP / upstream router / podAntiAffinity). - Cilium does not announce the IP on its own unless L2 announcements or BGP are enabled in the Cilium values (disabled by default in cozystack). - Switching the value on a running cluster causes the ingress-nginx Service to be recreated (`upgrade.force: true` on the HelmRelease + Service kind change); expect a brief interruption of ingress traffic. ### Release note ```release-note feat(ingress): add opt-in publishing.exposure=loadBalancer mode that uses type: LoadBalancer + CiliumLoadBalancerIPPool instead of the deprecated Service.spec.externalIPs. Default (externalIPs) preserves existing behavior. ```