Files
Andrei KvapilandClaude f919afe15f revert(ingress)!: remove broken publishing.exposure=loadBalancer mode
Reverts the ingress loadBalancer exposure mode added in #2468.

The mode rendered the ingress-nginx Service as type: LoadBalancer and a
CiliumLoadBalancerIPPool over publishing.externalIPs, as a migration path
off Service.spec.externalIPs (deprecated upstream in Kubernetes v1.36 per
KEP-5707). In default Cozystack the implementation produces two failure
modes:

- publishing.externalIPs is the operator-supplied list of node public IPs
  (addresses the cluster nodes already hold and listen on), not a
  virtual-VIP pool. Treating it as one and DNATing those IPs via Cilium
  LB-IPAM collides with the existing ingress-nginx DNAT on the same IPs
  (Service.spec.externalIPs path), so traffic is forwarded
  non-deterministically.
- Cilium in default Cozystack runs with ipam.mode: kubernetes and no
  L2/BGP announcer, so a pool over addresses outside the node range is
  allocated but never advertised — the Service stays unreachable.

The KEP-5707 deprecation is real, but the right migration is a
mechanism-agnostic tenant API knob translated by a Cozystack-side
controller into whichever backend the cluster admin has configured
(MetalLB pool, Cilium LB-IPAM, robotlb, externalIPs pin) — not a
chart-side IPPool baked into the tenant chart. That follow-up belongs in
its own PR.

Removed:
- publishing.exposure value (and the inline comment) from
  core/platform/values.yaml
- expose-mode plumbing line in core/platform/templates/apps.yaml
- CiliumLoadBalancerIPPool template at
  apps/tenant/templates/cilium-lb-pool.yaml
- type: LoadBalancer + externalTrafficPolicy: Local branch in
  extra/ingress/templates/nginx-ingress.yaml (Service renders back as
  ClusterIP + spec.externalIPs when externalIPs are set, falling through
  to type: LoadBalancer when they are not — pre-#2468 shape)
- Exposure mode section from extra/ingress/README.md
- exposure_test.yaml fixtures in both charts

Operators who explicitly set publishing.exposure=loadBalancer will see
their ingress Service revert to ClusterIP + spec.externalIPs on upgrade.
Anyone on the default value (externalIPs) is unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-05-11 10:42:29 +02:00
..
2024-08-12 14:47:11 +02:00
2024-07-16 17:31:52 +02:00
2025-09-24 12:32:37 +02:00
2026-03-25 15:57:25 +05:00

Tenant

A tenant is the main unit of security on the platform. The closest analogy would be Linux kernel namespaces.

Tenants can be created recursively and are subject to the following rules:

Tenant naming

Tenant names must be alphanumeric:

  • Lowercase letters (a-z) and digits (0-9) only
  • Must start with a lowercase letter
  • Dashes (-) are not allowed, unlike with other services
  • Maximum length depends on the cluster configuration (Helm release prefix and root domain)

This restriction exists to keep consistent naming in tenants, nested tenants, and services deployed in them. A tenant cannot be named foo-bar because parsing internal resource names like tenant-foo-bar would be ambiguous.

For example:

  • The root tenant is named root, but internally it's referenced as tenant-root.
  • A nested tenant could be named foo, which would result in tenant-foo in service names and URLs.

Unique domains

Each tenant has its own domain. By default, (unless otherwise specified), it inherits the domain of its parent with a prefix of its name. For example, if the parent had the domain example.org, then tenant-foo would get the domain foo.example.org by default.

Kubernetes clusters created in this tenant namespace would get domains like: kubernetes-cluster.foo.example.org

Example:

tenant-root (example.org)
└── tenant-foo (foo.example.org)
    └── kubernetes-cluster1 (kubernetes-cluster1.foo.example.org)

Nesting tenants and reusing parent services

Tenants can be nested. A tenant administrator can create nested tenants using the "Tenant" application from the catalogue.

Higher-level tenants can view and manage the applications of all their children tenants. If a tenant does not run their own cluster services, it can access ones of its parent.

For example, you create:

  • Tenant tenant-u1 with a set of services like etcd, ingress, monitoring.
  • Tenant tenant-u2 nested in tenant-u1.

Let's see what will happen when you run Kubernetes and Postgres under tenant-u2 namespace.

Since tenant-u2 does not have its own cluster services like etcd, ingress, and monitoring, the applications running in tenant-u2 will use the cluster services of the parent tenant.

This in turn means:

  • The Kubernetes cluster data will be stored in etcd for tenant-u1.
  • Access to the cluster will be through the common ingress of tenant-u1.
  • Essentially, all metrics will be collected in the monitoring from tenant-u1, and only that tenant will have access to them.

Example:

tenant-u1
├── etcd
├── ingress
├── monitoring
└── tenant-u2
    ├── kubernetes-cluster1
    └── postgres-db1

Parameters

Common parameters

Name Description Type Value
host The hostname used to access tenant services (defaults to using the tenant name as a subdomain for its parent tenant host). string ""
etcd Deploy own Etcd cluster. bool false
monitoring Deploy own Monitoring Stack. bool false
ingress Deploy own Ingress Controller. bool false
seaweedfs Deploy own SeaweedFS. bool false
schedulingClass The name of a SchedulingClass CR to apply scheduling constraints for this tenant's workloads. string ""
resourceQuotas Define resource quotas for the tenant. map[string]quantity {}

Configuration

Resource Quotas

The resourceQuotas parameter allows you to limit resources available to the tenant. Supported keys include:

Compute resources (converted to requests.X and limits.X):

  • cpu - Total CPU cores (e.g., "4" or "500m")
  • memory - Total memory (e.g., "4Gi" or "512Mi")
  • ephemeral-storage - Ephemeral storage limit (e.g., "10Gi")
  • storage - Total persistent storage (e.g., "100Gi")

Object count quotas (passed as-is):

  • pods - Maximum number of pods
  • services - Maximum number of services
  • services.loadbalancers - Maximum number of LoadBalancer services
  • services.nodeports - Maximum number of NodePort services
  • configmaps - Maximum number of ConfigMaps
  • secrets - Maximum number of Secrets
  • persistentvolumeclaims - Maximum number of PVCs

Example:

resourceQuotas:
  cpu: 4
  memory: 4Gi
  storage: 10Gi
  services.loadbalancers: "3"
  pods: "50"