fix(controller, platform): give unbounded pool members the root's LimitRange (#5265)

## What this PR does

Since tenant quotas became hierarchical (#3086), an unbounded sub-tenant
(no `resourceQuotas` of its own) shares its parent's pool and receives a
controller-owned `tenant-quota-allocated` ResourceQuota. The tenant
chart, however, renders the `tenant-range-limits` LimitRange only next
to the `tenant-quota` of a tenant that declares `resourceQuotas`, so
these namespaces end up with a quota on `limits.cpu`/`limits.memory` and
no container defaults. Admission then rejects every container that omits
one of those limits.

Pods that predate the quota keep running and fail as soon as they are
recreated. In production this took down CNPG postgres instances with
backups enabled: the `plugin-barman-cloud` sidecar injected by the
barman-cloud plugin sets CPU and memory requests and a memory limit but
no CPU limit, so the instance pod is refused on its next restart. The
same happened to the tenant `kubernetes` app: the `wait-for-kubeconfig`
init container of `cluster-autoscaler`, `kcsi-controller` and the
`bootstrap-token` jobs has no resources, so their ReplicaSets and Jobs
were refused with `failed quota: tenant-quota-allocated: must specify
limits.cpu` in every unbounded sub-tenant hosting a tenant cluster. On a
production cluster, 131 tenant pods currently get their CPU or memory
from the tenant LimitRange (VictoriaMetrics
`config-init`/`config-reloader`, Harbor exporter, MariaDB agent, Strimzi
entity operator, among others), so the defect is not specific to one
chart: any of them breaks once it lands in such a namespace. Workloads
that set every limit explicitly are unaffected.

The controller already decides which namespaces it clamps, so it now
copies the pool root's chart LimitRange into each unbounded member whose
allocated quota bounds `limits.cpu` or `limits.memory` as
`tenant-range-limits-allocated`, keeps it in step with the root, and
deletes it once the namespace leaves the pool (for instance when the
sub-tenant declares its own quota and the chart renders its own
LimitRange). The chart renders its LimitRange under any non-empty
`resourceQuotas`, including one that bounds no compute (#4930), so the
copy is limited to members whose quota bounds a compute limit: elsewhere
it would impose a 128Mi default memory limit the quota never asked for
and refuse pods, virt-launcher among them, that are admitted today. This
keeps the change independent of #4954. The copy has its own name so that
Helm can still create the chart-owned object in that case without an
ownership conflict. The root is the single source of truth for the
defaults, so the values stay defined in
`packages/apps/tenant/templates/quota.yaml` only. The controller gains
`limitranges` RBAC and watches the two LimitRange names so a change
propagates without waiting for the 60 s resync.

Tests: five reconciler cases cover a member under a non-compute quota
(no copy, and a stale one removed), the copy into an unbounded member
(and not into the root), propagation of a change on the root, garbage
collection once the member is bounded (the chart-owned object is left
alone), and a root without a LimitRange. Disabling the copy makes the
first two fail.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. The code touches only
`internal/controller/tenantquota/` and the `cozystack-controller` RBAC
template, but the website's `system-memory-limits.md` states that tenant
namespaces get defaults only from the chart's `tenant-range-limits`,
which this change makes untrue; the follow-up is an issue, since the
page should change once this is merged.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#806
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(controller, platform): sub-tenants without their own resourceQuotas now receive their parent pool's container defaults (LimitRange) alongside the allocated quota, when that quota bounds a CPU or memory limit, so pods that omit such a limit, such as the postgres barman-cloud backup sidecar, are no longer rejected at admission.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Unbounded pool members now receive their pool’s CPU and memory limit
settings when their allocated quota sets those limits. Pool setting
updates are reflected in members during reconciliation.
* When a member has its own quota, inherited limit settings are removed
while settings configured directly for that member remain unchanged.
* No inherited settings are created when the pool has no limit settings
or the allocated quota does not set CPU or memory limits.
* Existing inherited settings are retained if an update temporarily
fails.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-15 02:33:34 +03:00
commit 46cb99df5b
3 files changed
+269 -5

No files matched your search

+95 -2
View File
@@ -52,6 +52,17 @@ const (
// whenever it is below the chart quota, without the controller fighting Flux
// over the chart-owned object.
allocatedQuotaName = "tenant-quota-allocated"
// chartLimitRangeName is the LimitRange the tenant chart renders next to
// chartQuotaName, giving containers default requests and limits. The chart
// renders both only for a tenant that declares resourceQuotas.
chartLimitRangeName = "tenant-range-limits"
// allocatedLimitRangeName is the controller-owned copy of the pool root's
// chart LimitRange, written into every unbounded pool member. Those members
// render no chart LimitRange, yet carry an allocated quota; without
// container defaults, any pod omitting a limit the quota bounds is rejected
// at admission. A distinct name keeps the copy from colliding with the
// chart-owned object if the member later declares its own quota.
allocatedLimitRangeName = "tenant-range-limits-allocated"
// managedByLabel marks the controller-owned ResourceQuotas so they can be
// listed and garbage-collected.
managedByLabel = "quota.cozystack.io/managed-by"
@@ -102,6 +113,9 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result
pools := ComputePools(tenants)
desired := map[string]corev1.ResourceList{}
// limitRangeSource maps each unbounded pool member whose quota bounds a
// compute limit to the pool root whose chart LimitRange it must inherit.
limitRangeSource := map[string]string{}
for _, p := range pools {
if over := p.Overcommitted(); len(over) > 0 {
logger.Info("tenant quota pool is overcommitted by sub-tenant carve-outs", "pool", p.Root, "overcommit", over)
@@ -120,6 +134,9 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result
for _, ns := range p.Members {
if existing[ns] {
desired[ns] = buffered.EnforcedHard(ns, usedByNS)
if ns != p.Root && boundsComputeLimits(desired[ns]) {
limitRangeSource[ns] = p.Root
}
}
}
}
@@ -134,6 +151,22 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result
logger.Error(err, "failed to garbage-collect stale allocated quotas")
}
written := map[string]bool{}
for ns, root := range limitRangeSource {
ok, err := r.upsertAllocatedLimitRange(ctx, ns, root)
if err != nil {
logger.Error(err, "failed to apply allocated limit range", "namespace", ns, "poolRoot", root)
}
// Keep the existing copy on a transient error rather than dropping the
// defaults the namespace's pods rely on.
if ok || err != nil {
written[ns] = true
}
}
if err := r.gcAllocatedLimitRanges(ctx, written); err != nil {
logger.Error(err, "failed to garbage-collect stale allocated limit ranges")
}
return ctrl.Result{RequeueAfter: resyncInterval}, nil
}
@@ -224,6 +257,50 @@ func (r *Reconciler) gcAllocatedQuotas(ctx context.Context, desired map[string]c
return nil
}
// upsertAllocatedLimitRange copies the pool root's chart LimitRange into an
// unbounded member namespace. It reports false, without error, when the root
// has no chart LimitRange to copy.
func (r *Reconciler) upsertAllocatedLimitRange(ctx context.Context, namespace, poolRoot string) (bool, error) {
source := &corev1.LimitRange{}
if err := r.Get(ctx, types.NamespacedName{Namespace: poolRoot, Name: chartLimitRangeName}, source); err != nil {
if apierrors.IsNotFound(err) {
return false, nil
}
return false, err
}
lr := &corev1.LimitRange{ObjectMeta: metav1.ObjectMeta{Name: allocatedLimitRangeName, Namespace: namespace}}
_, err := controllerutil.CreateOrUpdate(ctx, r.Client, lr, func() error {
if lr.Labels == nil {
lr.Labels = map[string]string{}
}
lr.Labels[managedByLabel] = managedByValue
lr.Labels["internal.cozystack.io/managed-by-cozystack"] = ""
lr.Spec = *source.Spec.DeepCopy()
return nil
})
return err == nil, err
}
// gcAllocatedLimitRanges deletes controller-owned LimitRanges in namespaces
// that are no longer unbounded pool members (e.g. the tenant gained its own
// quota and now renders the chart LimitRange, or was deleted).
func (r *Reconciler) gcAllocatedLimitRanges(ctx context.Context, keep map[string]bool) error {
managed := &corev1.LimitRangeList{}
if err := r.List(ctx, managed, client.MatchingLabels{managedByLabel: managedByValue}); err != nil {
return err
}
for i := range managed.Items {
m := &managed.Items[i]
if keep[m.Namespace] {
continue
}
if err := r.Delete(ctx, m); err != nil && !apierrors.IsNotFound(err) {
return err
}
}
return nil
}
func (r *Reconciler) recordOvercommit(ctx context.Context, rootNamespace string, over corev1.ResourceList) {
if r.Recorder == nil {
return
@@ -237,8 +314,8 @@ func (r *Reconciler) recordOvercommit(ctx context.Context, rootNamespace string,
}
// SetupWithManager wires the controller. Every relevant change (a tenant
// HelmRelease, a namespace, or a tenant ResourceQuota whose usage moved)
// coalesces into one full-tree sweep.
// HelmRelease, a namespace, a tenant ResourceQuota whose usage moved, or a
// tenant LimitRange) coalesces into one full-tree sweep.
func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error {
toSweep := handler.EnqueueRequestsFromMapFunc(func(context.Context, client.Object) []reconcile.Request {
return []reconcile.Request{sweepKey}
@@ -249,11 +326,15 @@ func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error {
tenantQuotas := predicate.NewPredicateFuncs(func(o client.Object) bool {
return o.GetName() == chartQuotaName || o.GetName() == allocatedQuotaName
})
tenantLimitRanges := predicate.NewPredicateFuncs(func(o client.Object) bool {
return o.GetName() == chartLimitRangeName || o.GetName() == allocatedLimitRangeName
})
return ctrl.NewControllerManagedBy(mgr).
Named("tenantquota-controller").
Watches(&helmv2.HelmRelease{}, toSweep, builder.WithPredicates(tenantReleases)).
Watches(&corev1.Namespace{}, toSweep).
Watches(&corev1.ResourceQuota{}, toSweep, builder.WithPredicates(tenantQuotas)).
Watches(&corev1.LimitRange{}, toSweep, builder.WithPredicates(tenantLimitRanges)).
Complete(r)
}
@@ -271,6 +352,18 @@ func ownedNamespace(hrNamespace, appName string) string {
}
}
// boundsComputeLimits reports whether a quota bounds limits.cpu or
// limits.memory, the only case in which admission rejects a container that
// omits a limit. The chart renders its LimitRange under any non-empty
// resourceQuotas, so the root may carry one under a quota that bounds no
// compute at all; copying it then would impose default limits the quota
// never asked for.
func boundsComputeLimits(hard corev1.ResourceList) bool {
_, cpu := hard[corev1.ResourceLimitsCPU]
_, memory := hard[corev1.ResourceLimitsMemory]
return cpu || memory
}
// maxResourceList returns the per-resource maximum of a and b.
func maxResourceList(a, b corev1.ResourceList) corev1.ResourceList {
out := corev1.ResourceList{}
@@ -227,3 +227,172 @@ func TestReconcile_GarbageCollect(t *testing.T) {
t.Fatalf("stale allocated quota in tenant-gone should have been garbage-collected")
}
}
// chartLimitRange builds the chart-rendered "tenant-range-limits" LimitRange,
// which the tenant chart emits alongside its "tenant-quota".
func chartLimitRange(namespace, defaultCPU string) *corev1.LimitRange {
return &corev1.LimitRange{
ObjectMeta: metav1.ObjectMeta{Name: chartLimitRangeName, Namespace: namespace},
Spec: corev1.LimitRangeSpec{Limits: []corev1.LimitRangeItem{{
Type: corev1.LimitTypeContainer,
Default: resourceList(map[string]string{"cpu": defaultCPU, "memory": "128Mi"}),
DefaultRequest: resourceList(map[string]string{"cpu": "25m", "memory": "128Mi"}),
}}},
}
}
func getLimitRange(c client.Client, namespace, name string) (*corev1.LimitRange, bool) {
lr := &corev1.LimitRange{}
if err := c.Get(context.Background(), types.NamespacedName{Namespace: namespace, Name: name}, lr); err != nil {
return nil, false
}
return lr, true
}
// TestReconcile_UnboundedMemberGetsPoolLimitRange: an unbounded child renders
// no chart quota and therefore no chart LimitRange, yet the controller clamps
// it with an allocated quota. Without container defaults, every pod that omits
// a limit the quota bounds (e.g. a sidecar without a CPU limit) is rejected at
// admission, so the controller must copy the pool root's LimitRange into it.
func TestReconcile_UnboundedMemberGetsPoolLimitRange(t *testing.T) {
r, c := newReconciler(t,
tenantHR("foo", "tenant-root"),
tenantHR("bar", "tenant-foo"),
ns("tenant-foo"), ns("tenant-foo-bar"),
chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil),
chartLimitRange("tenant-foo", "250m"),
)
if _, err := r.Reconcile(context.Background(), sweepKey); err != nil {
t.Fatalf("reconcile: %v", err)
}
lr, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName)
if !ok {
t.Fatalf("unbounded member tenant-foo-bar has an allocated quota but no LimitRange")
}
if len(lr.Spec.Limits) != 1 {
t.Fatalf("tenant-foo-bar LimitRange limits = %d, want 1", len(lr.Spec.Limits))
}
if got := lr.Spec.Limits[0].Default[corev1.ResourceCPU]; got.String() != "250m" {
t.Fatalf("tenant-foo-bar default cpu = %q, want 250m copied from the pool root", got.String())
}
if lr.Labels[managedByLabel] != managedByValue {
t.Fatalf("tenant-foo-bar LimitRange is not marked as controller-owned")
}
if _, ok := getLimitRange(c, "tenant-foo", allocatedLimitRangeName); ok {
t.Fatalf("pool root tenant-foo already has the chart LimitRange and must not get a second one")
}
}
// TestReconcile_PoolLimitRangeFollowsRoot: a change to the root's chart
// LimitRange propagates to the members on the next sweep.
func TestReconcile_PoolLimitRangeFollowsRoot(t *testing.T) {
root := chartLimitRange("tenant-foo", "250m")
r, c := newReconciler(t,
tenantHR("foo", "tenant-root"),
tenantHR("bar", "tenant-foo"),
ns("tenant-foo"), ns("tenant-foo-bar"),
chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil),
root,
)
if _, err := r.Reconcile(context.Background(), sweepKey); err != nil {
t.Fatalf("reconcile: %v", err)
}
root.Spec.Limits[0].Default[corev1.ResourceCPU] = resource.MustParse("500m")
if err := c.Update(context.Background(), root); err != nil {
t.Fatalf("update root LimitRange: %v", err)
}
if _, err := r.Reconcile(context.Background(), sweepKey); err != nil {
t.Fatalf("reconcile: %v", err)
}
lr, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName)
if !ok {
t.Fatalf("tenant-foo-bar lost its LimitRange")
}
if got := lr.Spec.Limits[0].Default[corev1.ResourceCPU]; got.String() != "500m" {
t.Fatalf("tenant-foo-bar default cpu = %q, want 500m after the root changed", got.String())
}
}
// TestReconcile_PoolLimitRangeGarbageCollected: once the child declares its own
// quota, the chart renders its LimitRange and the controller-owned copy goes.
func TestReconcile_PoolLimitRangeGarbageCollected(t *testing.T) {
stale := &corev1.LimitRange{
ObjectMeta: metav1.ObjectMeta{
Name: allocatedLimitRangeName,
Namespace: "tenant-foo-bar",
Labels: map[string]string{managedByLabel: managedByValue},
},
}
r, c := newReconciler(t,
tenantHR("foo", "tenant-root"),
tenantHR("bar", "tenant-foo"),
ns("tenant-foo"), ns("tenant-foo-bar"),
chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil),
chartLimitRange("tenant-foo", "250m"),
chartQuota("tenant-foo-bar", map[string]string{"limits.cpu": "4"}, nil),
chartLimitRange("tenant-foo-bar", "250m"),
stale,
)
if _, err := r.Reconcile(context.Background(), sweepKey); err != nil {
t.Fatalf("reconcile: %v", err)
}
if _, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName); ok {
t.Fatalf("bounded child tenant-foo-bar renders its own LimitRange; the controller copy must be removed")
}
if _, ok := getLimitRange(c, "tenant-foo-bar", chartLimitRangeName); !ok {
t.Fatalf("the chart-owned LimitRange must never be touched")
}
}
// TestReconcile_PoolRootWithoutLimitRange: if the root has no chart LimitRange
// there is nothing to copy; the quota is still enforced and the sweep succeeds.
func TestReconcile_PoolRootWithoutLimitRange(t *testing.T) {
r, c := newReconciler(t,
tenantHR("foo", "tenant-root"),
tenantHR("bar", "tenant-foo"),
ns("tenant-foo"), ns("tenant-foo-bar"),
chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil),
)
if _, err := r.Reconcile(context.Background(), sweepKey); err != nil {
t.Fatalf("reconcile: %v", err)
}
if _, ok := allocatedHard(t, c, "tenant-foo-bar", "limits.cpu"); !ok {
t.Fatalf("tenant-foo-bar must still get its allocated quota")
}
if _, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName); ok {
t.Fatalf("no LimitRange can be copied from a root that has none")
}
}
// TestReconcile_NoLimitRangeWithoutComputeLimitQuota: the chart renders its
// LimitRange for any non-empty resourceQuotas, compute or not. A member whose
// allocated quota bounds neither limits.cpu nor limits.memory is admitted
// without limits today; copying the root's defaults there would cap or refuse
// pods (one requesting more than the default memory limit, for instance)
// that the quota itself never constrains.
func TestReconcile_NoLimitRangeWithoutComputeLimitQuota(t *testing.T) {
stale := &corev1.LimitRange{
ObjectMeta: metav1.ObjectMeta{
Name: allocatedLimitRangeName,
Namespace: "tenant-foo-bar",
Labels: map[string]string{managedByLabel: managedByValue},
},
}
r, c := newReconciler(t,
tenantHR("foo", "tenant-root"),
tenantHR("bar", "tenant-foo"),
ns("tenant-foo"), ns("tenant-foo-bar"),
chartQuota("tenant-foo", map[string]string{"services.loadbalancers": "1"}, nil),
chartLimitRange("tenant-foo", "250m"),
stale,
)
if _, err := r.Reconcile(context.Background(), sweepKey); err != nil {
t.Fatalf("reconcile: %v", err)
}
if _, ok := allocatedHard(t, c, "tenant-foo-bar", "services.loadbalancers"); !ok {
t.Fatalf("tenant-foo-bar must still get its allocated quota")
}
if _, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName); ok {
t.Fatalf("tenant-foo-bar's quota bounds no compute limit; it must carry no LimitRange copy")
}
}
@@ -49,10 +49,12 @@ rules:
resources: ["namespaces"]
verbs: ["get", "list", "watch", "patch", "update"]
# TenantQuotaReconciler manages the per-namespace "tenant-quota-allocated"
# ResourceQuota that enforces each tenant's share of its parent pool, and emits
# events on the tenant namespace when sub-tenant quotas overcommit the budget.
# ResourceQuota that enforces each tenant's share of its parent pool, copies the
# pool root's "tenant-range-limits" LimitRange into unbounded members as
# "tenant-range-limits-allocated", and emits events on the tenant namespace when
# sub-tenant quotas overcommit the budget.
- apiGroups: [""]
resources: ["resourcequotas"]
resources: ["resourcequotas", "limitranges"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["events"]