mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-29 16:17:20 +09:00
fix(rabbitmq): right-size the default resources preset to s1.nano (#3936)
## What this PR does
Raises the RabbitMQ default resource preset from `t1.nano` (128Mi
memory) to `s1.nano` (250m CPU / 512Mi memory).
A default-sized RabbitMQ 4.2 broker on Cozystack 1.6.1 was OOMKilled
with exit 137, reached 10 restarts, and never became Ready. A canary on
a larger preset reached 3/3 Ready with zero restarts; its working set
peaked at 125.8Mi. Against the 128Mi `t1.nano` ceiling that is 98% of
the limit in steady state, and cgroup accounting (working set plus page
cache) crosses it under load. The canary and its storage were removed
after validation.
### Sizing
`s1.nano` keeps the CPU of `t1.nano` (250m) and quadruples memory to
512Mi — a 4x margin over the observed 125.8Mi peak.
An earlier revision of this PR proposed `u1.nano` (1Gi). Review
correctly objected that 1Gi reserves 8x the observed working set with no
measured peak to justify it, and asked to either report that peak or
right-size the default. The peak is reported above, and the default is
right-sized to the smallest preset that clears the ceiling with a real
margin rather than to the first one that is comfortably large.
### Upgrade impact
cozystack-api applies OpenAPI schema defaults on the read path and
stores what the client sent: `Get` runs `applySpecDefaults` on its way
out (`pkg/registry/apps/application/rest.go:1232`) and the write puts
`app.Spec` into `HelmRelease.spec.values` (`rest.go:1652`). That splits
existing brokers in two.
**Brokers with no stored `resourcesPreset`** pick up the new chart
default on the next Helm reconcile. For them this upgrade is:
- a rolling restart of every replica, and
- a rise in guaranteed memory reservation from 128Mi to 512Mi per
replica, because the sanitize helper sets memory request equal to limit
— roughly 1.1Gi extra reserved per 3-replica instance.
**Brokers whose values went through a read-modify-write keep `t1.nano`
and are not resized.** `Update` opens with `r.Get` (`rest.go:489`), so a
GET-then-PUT — `kubectl edit` is the unambiguous case, and any client
that reads before writing behaves the same — freezes the schema default
of the moment into the stored values, and a stored value beats the chart
default. Those brokers are not restarted, not resized, and keep hitting
the ceiling this PR raises.
Nothing downstream can separate a deliberate `t1.nano` from one a round
trip baked in, so this is documented rather than migrated. After the
upgrade the affected brokers are the ones that still report the old
preset:
```
kubectl get rabbitmq -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRESET:.spec.resourcesPreset
```
The same command before the upgrade tells you nothing: the cozyrd
default is still `t1.nano` then, so the read path fills it in for both
groups. Afterwards, anything reading `t1.nano` carries a stored value;
setting `resourcesPreset: s1.nano` explicitly on the ones that should
move is the fix.
PVCs are untouched, so there is no data loss, but node headroom should
be checked before upgrading.
Also worth knowing when reading the parameter table: the preset does not
only apply to a fully omitted `resources`.
`cozy-lib.resources.defaultingSanitize` merges it key by key, so a
broker pinned to `resources: {cpu: 700m}` keeps its CPU and moves from
128Mi to 512Mi with everyone else. The field descriptions now say so,
and a test pins it.
The values, generated schema, ApplicationDefinition, typed API, and
README all carry the same default. Tests cover the new default, an
explicit legacy override, explicit resources, and a partially set
`resources` map inheriting the preset for what it leaves out.
Validation:
- `helm template` with the chart's `cozy-lib` dependency renders `limits
250m / 512Mi` and `requests 0.025 / 536870912` for the default; an
explicit `t1.nano` renders `128Mi / 134217728`; explicit `700m / 2Gi`
renders `0.07 / 2147483648`. The unit test asserts exactly these values.
- Provider counterpart: cozystack/terraform-provider-cozystack#34.
### Screenshots
Not a UI change.
### Downstream repositories
The default is restated by the Terraform provider, so the linked
provider PR updates it and documents the state-plan implication. No
other trigger-map entry matches this diff.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/pull/34
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(rabbitmq): raise the default resource preset from t1.nano to s1.nano so default-sized brokers are no longer OOMKilled. Brokers with no stored `resourcesPreset` are restarted on upgrade and their guaranteed memory reservation grows from 128Mi to 512Mi per replica; check node headroom before upgrading. Brokers whose values already pin `resourcesPreset`, including ones where a read-modify-write through the API froze the old default into them, keep their old size: list them after the upgrade with `kubectl get rabbitmq -A -o custom-columns=NAME:.metadata.name,PRESET:.spec.resourcesPreset` and set `resourcesPreset: s1.nano` explicitly on the ones that should move. The preset also fills any resource a partially set `resources` leaves out, so a broker pinned to cpu alone moves too. A tenant that opted into `resourceQuotas` and sits near its memory ceiling has the extra reservation rejected by quota admission, which shows up as a StatefulSet that never scales rather than as pods left Pending.
```
This commit is contained in:
6 files changed
+92
-21
No files matched your search
Generated
+3
-3
@@ -20,11 +20,11 @@ type ConfigSpec struct {
|
||||
// Number of RabbitMQ replicas.
|
||||
// +kubebuilder:default:=3
|
||||
Replicas int `json:"replicas"`
|
||||
// Explicit CPU and memory configuration for each RabbitMQ replica. When omitted, the preset defined in `resourcesPreset` is applied.
|
||||
// Explicit CPU and memory configuration for each RabbitMQ replica. Every resource left unset here is taken from `resourcesPreset`.
|
||||
// +kubebuilder:default:={}
|
||||
Resources Resources `json:"resources,omitempty"`
|
||||
// Default sizing preset used when `resources` is omitted.
|
||||
// +kubebuilder:default:="t1.nano"
|
||||
// Default sizing preset. It supplies every resource `resources` does not set, not only a `resources` left empty entirely.
|
||||
// +kubebuilder:default:="s1.nano"
|
||||
ResourcesPreset ResourcesPreset `json:"resourcesPreset"`
|
||||
// Persistent Volume Claim size available for application data.
|
||||
// +kubebuilder:default:="10Gi"
|
||||
|
||||
@@ -24,10 +24,10 @@ The scope is **definitions only** — vhosts, users, permissions, queues, exchan
|
||||
| Name | Description | Type | Value |
|
||||
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------- |
|
||||
| `replicas` | Number of RabbitMQ replicas. | `int` | `3` |
|
||||
| `resources` | Explicit CPU and memory configuration for each RabbitMQ replica. When omitted, the preset defined in `resourcesPreset` is applied. | `object` | `{}` |
|
||||
| `resources` | Explicit CPU and memory configuration for each RabbitMQ replica. Every resource left unset here is taken from `resourcesPreset`. | `object` | `{}` |
|
||||
| `resources.cpu` | CPU available to each replica. | `quantity` | `""` |
|
||||
| `resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` |
|
||||
| `resourcesPreset` | Default sizing preset used when `resources` is omitted. | `string` | `t1.nano` |
|
||||
| `resourcesPreset` | Default sizing preset. It supplies every resource `resources` does not set, not only a `resources` left empty entirely. | `string` | `s1.nano` |
|
||||
| `size` | Persistent Volume Claim size available for application data. | `quantity` | `10Gi` |
|
||||
| `storageClass` | StorageClass used to store the data. | `string` | `""` |
|
||||
| `external` | Enable external access from outside the cluster. | `bool` | `false` |
|
||||
@@ -139,13 +139,7 @@ resources:
|
||||
`resourcesPreset` sets named CPU and memory configurations for each replica.
|
||||
This setting is ignored if the corresponding `resources` value is set.
|
||||
|
||||
| Preset name | CPU | memory |
|
||||
|-------------|--------|---------|
|
||||
| `nano` | `100m` | `128Mi` |
|
||||
| `micro` | `250m` | `256Mi` |
|
||||
| `small` | `500m` | `512Mi` |
|
||||
| `medium` | `500m` | `1Gi` |
|
||||
| `large` | `1` | `2Gi` |
|
||||
| `xlarge` | `2` | `4Gi` |
|
||||
| `2xlarge` | `4` | `8Gi` |
|
||||
Presets follow a cloud-style `<series>.<size>` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases of their 1:1 instance-type equivalents.
|
||||
|
||||
See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping.
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
suite: RabbitMQ default resources
|
||||
|
||||
templates:
|
||||
- templates/rabbitmq.yaml
|
||||
|
||||
tests:
|
||||
- it: defaults each RabbitMQ replica to the s1.nano preset
|
||||
release:
|
||||
name: rabbitmq-test
|
||||
namespace: tenant-test
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.resources.limits.cpu
|
||||
value: 250m
|
||||
- equal:
|
||||
path: spec.resources.limits.memory
|
||||
value: 512Mi
|
||||
- equal:
|
||||
path: spec.resources.requests.cpu
|
||||
value: "0.025"
|
||||
- equal:
|
||||
path: spec.resources.requests.memory
|
||||
value: "536870912"
|
||||
|
||||
- it: preserves t1.nano as an explicit backward-compatible override
|
||||
release:
|
||||
name: rabbitmq-test
|
||||
namespace: tenant-test
|
||||
set:
|
||||
resourcesPreset: t1.nano
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.resources.limits.cpu
|
||||
value: 250m
|
||||
- equal:
|
||||
path: spec.resources.limits.memory
|
||||
value: 128Mi
|
||||
|
||||
- it: prefers explicit resources over the default preset
|
||||
release:
|
||||
name: rabbitmq-test
|
||||
namespace: tenant-test
|
||||
set:
|
||||
resources:
|
||||
cpu: 700m
|
||||
memory: 2Gi
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.resources.limits.cpu
|
||||
value: 700m
|
||||
- equal:
|
||||
path: spec.resources.limits.memory
|
||||
value: 2Gi
|
||||
- equal:
|
||||
path: spec.resources.requests.cpu
|
||||
value: "0.07"
|
||||
- equal:
|
||||
path: spec.resources.requests.memory
|
||||
value: "2147483648"
|
||||
|
||||
- it: fills the preset into every resource a partial resources map leaves unset
|
||||
release:
|
||||
name: rabbitmq-test
|
||||
namespace: tenant-test
|
||||
set:
|
||||
resources:
|
||||
cpu: 700m
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.resources.limits.cpu
|
||||
value: 700m
|
||||
- equal:
|
||||
path: spec.resources.limits.memory
|
||||
value: 512Mi
|
||||
- equal:
|
||||
path: spec.resources.requests.memory
|
||||
value: "536870912"
|
||||
@@ -8,7 +8,7 @@
|
||||
"default": 3
|
||||
},
|
||||
"resources": {
|
||||
"description": "Explicit CPU and memory configuration for each RabbitMQ replica. When omitted, the preset defined in `resourcesPreset` is applied.",
|
||||
"description": "Explicit CPU and memory configuration for each RabbitMQ replica. Every resource left unset here is taken from `resourcesPreset`.",
|
||||
"type": "object",
|
||||
"default": {},
|
||||
"properties": {
|
||||
@@ -41,9 +41,9 @@
|
||||
}
|
||||
},
|
||||
"resourcesPreset": {
|
||||
"description": "Default sizing preset used when `resources` is omitted.",
|
||||
"description": "Default sizing preset. It supplies every resource `resources` does not set, not only a `resources` left empty entirely.",
|
||||
"type": "string",
|
||||
"default": "t1.nano",
|
||||
"default": "s1.nano",
|
||||
"enum": [
|
||||
"t1.nano",
|
||||
"t1.micro",
|
||||
|
||||
@@ -58,11 +58,11 @@
|
||||
## @param {int} replicas - Number of RabbitMQ replicas.
|
||||
replicas: 3
|
||||
|
||||
## @param {Resources} [resources] - Explicit CPU and memory configuration for each RabbitMQ replica. When omitted, the preset defined in `resourcesPreset` is applied.
|
||||
## @param {Resources} [resources] - Explicit CPU and memory configuration for each RabbitMQ replica. Every resource left unset here is taken from `resourcesPreset`.
|
||||
resources: {}
|
||||
|
||||
## @param {ResourcesPreset} resourcesPreset="t1.nano" - Default sizing preset used when `resources` is omitted.
|
||||
resourcesPreset: "t1.nano"
|
||||
## @param {ResourcesPreset} resourcesPreset="s1.nano" - Default sizing preset. It supplies every resource `resources` does not set, not only a `resources` left empty entirely.
|
||||
resourcesPreset: "s1.nano"
|
||||
|
||||
## @param {quantity} size - Persistent Volume Claim size available for application data.
|
||||
size: 10Gi
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ spec:
|
||||
plural: rabbitmqs
|
||||
singular: rabbitmq
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of RabbitMQ replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration for each RabbitMQ replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"tls":{"description":"TLS configuration. TLS is off unless `tls.enabled` is set to true.","type":"object","default":{},"properties":{"disableNonTLSListeners":{"description":"Close the plaintext AMQP (5672) and management (15672) listeners. This is a broker-level switch: it drops in-cluster clients as well as external ones, and it cannot be combined with `users` or `vhosts`. Erlang distribution (25672) and epmd (4369) stay plaintext regardless. Defaults to false.","type":"boolean"},"enabled":{"description":"Enable TLS for AMQPS (5671) and Management HTTPS (15671). Applies whether or not `external` is set. Plaintext AMQP (5672) and management (15672) keep serving unless `disableNonTLSListeners` is set as well. Moves the `port` key of the `<release>-default-user` Secret from 5672 to 5671, and the metrics port from 15692 to 15691. Defaults to false.","type":"boolean"}}},"version":{"description":"RabbitMQ major.minor version to deploy","type":"string","default":"v4.2","enum":["v4.2","v4.1","v4.0","v3.13"]},"users":{"description":"Users configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"password":{"description":"Password for the user.","type":"string"}}}},"vhosts":{"description":"Virtual hosts configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","required":["roles"],"properties":{"roles":{"description":"Virtual host roles list.","type":"object","properties":{"admin":{"description":"List of admin users.","type":"array","items":{"type":"string"}},"readonly":{"description":"List of readonly users.","type":"array","items":{"type":"string"}}}}}}}}}
|
||||
{"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of RabbitMQ replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration for each RabbitMQ replica. Every resource left unset here is taken from `resourcesPreset`.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset. It supplies every resource `resources` does not set, not only a `resources` left empty entirely.","type":"string","default":"s1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"tls":{"description":"TLS configuration. TLS is off unless `tls.enabled` is set to true.","type":"object","default":{},"properties":{"disableNonTLSListeners":{"description":"Close the plaintext AMQP (5672) and management (15672) listeners. This is a broker-level switch: it drops in-cluster clients as well as external ones, and it cannot be combined with `users` or `vhosts`. Erlang distribution (25672) and epmd (4369) stay plaintext regardless. Defaults to false.","type":"boolean"},"enabled":{"description":"Enable TLS for AMQPS (5671) and Management HTTPS (15671). Applies whether or not `external` is set. Plaintext AMQP (5672) and management (15672) keep serving unless `disableNonTLSListeners` is set as well. Moves the `port` key of the `<release>-default-user` Secret from 5672 to 5671, and the metrics port from 15692 to 15691. Defaults to false.","type":"boolean"}}},"version":{"description":"RabbitMQ major.minor version to deploy","type":"string","default":"v4.2","enum":["v4.2","v4.1","v4.0","v3.13"]},"users":{"description":"Users configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"password":{"description":"Password for the user.","type":"string"}}}},"vhosts":{"description":"Virtual hosts configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","required":["roles"],"properties":{"roles":{"description":"Virtual host roles list.","type":"object","properties":{"admin":{"description":"List of admin users.","type":"array","items":{"type":"string"}},"readonly":{"description":"List of readonly users.","type":"array","items":{"type":"string"}}}}}}}}}
|
||||
release:
|
||||
prefix: rabbitmq-
|
||||
labels:
|
||||
|
||||
Reference in new issue
Block a user