From 48779cc03b23b7debb6dc6e4f39958fe9c165e0a Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Thu, 29 Oct 2026 10:00:55 +0100 Subject: [PATCH 1/5] fix(redis): run one Sentinel when Redis runs a single pod The chart pinned three Sentinels whatever the replica count, so a single-pod instance paid for three pods that have nothing to fail over to. One Sentinel is enough for the operator to locate the master: its quorum is replicas/2+1, which is 1. The operator gives the Sentinel Deployment a PodDisruptionBudget with minAvailable 1 whenever Redis has two replicas or fewer. On a single Sentinel that budget refuses every eviction and would block node drains, so it is disabled in that case. From two Redis pods up nothing changes: three Sentinels, budget kept. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/redis/types.go | 4 +- packages/apps/redis/README.md | 16 +-- .../apps/redis/templates/redisfailover.yaml | 17 ++- .../redis/tests/sentinel_replicas_test.yaml | 103 ++++++++++++++++++ packages/apps/redis/values.schema.json | 4 +- packages/apps/redis/values.yaml | 4 +- packages/system/redis-rd/cozyrds/redis.yaml | 2 +- 7 files changed, 132 insertions(+), 18 deletions(-) create mode 100644 packages/apps/redis/tests/sentinel_replicas_test.yaml diff --git a/api/apps/v1alpha1/redis/types.go b/api/apps/v1alpha1/redis/types.go index 509c1e6ce..c2363749c 100644 --- a/api/apps/v1alpha1/redis/types.go +++ b/api/apps/v1alpha1/redis/types.go @@ -17,7 +17,7 @@ type Config struct { } type ConfigSpec struct { - // Number of Redis replicas. + // Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to. // +kubebuilder:default:=2 Replicas int `json:"replicas"` // Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied. @@ -51,7 +51,7 @@ type ConfigSpec struct { // Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it. // +kubebuilder:validation:Minimum=0 MinReplicasMaxLag *int `json:"minReplicasMaxLag,omitempty"` - // Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. + // Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. // +kubebuilder:default:={} Sentinel Sentinel `json:"sentinel,omitempty"` } diff --git a/packages/apps/redis/README.md b/packages/apps/redis/README.md index 0922bf821..28152d9f9 100644 --- a/packages/apps/redis/README.md +++ b/packages/apps/redis/README.md @@ -17,7 +17,7 @@ Service utilizes the freshworks-oss Redis Operator (a maintained fork of the arc | Name | Description | Type | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | --------- | -| `replicas` | Number of Redis replicas. | `int` | `2` | +| `replicas` | Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to. | `int` | `2` | | `resources` | Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied. | `object` | `{}` | | `resources.cpu` | CPU available to each replica. | `quantity` | `""` | | `resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` | @@ -48,13 +48,13 @@ Service utilizes the freshworks-oss Redis Operator (a maintained fork of the arc ### Sentinel parameters -| Name | Description | Type | Value | -| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----- | -| `sentinel` | Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. | `object` | `{}` | -| `sentinel.resources` | Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`. | `object` | `{}` | -| `sentinel.resources.cpu` | CPU available to each replica. | `quantity` | `""` | -| `sentinel.resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` | -| `sentinel.resourcesPreset` | Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`. | `string` | `{}` | +| Name | Description | Type | Value | +| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----- | +| `sentinel` | Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. | `object` | `{}` | +| `sentinel.resources` | Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`. | `object` | `{}` | +| `sentinel.resources.cpu` | CPU available to each replica. | `quantity` | `""` | +| `sentinel.resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` | +| `sentinel.resourcesPreset` | Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`. | `string` | `{}` | ## Parameter examples and reference diff --git a/packages/apps/redis/templates/redisfailover.yaml b/packages/apps/redis/templates/redisfailover.yaml index 27c7e9047..2dd1fecaf 100644 --- a/packages/apps/redis/templates/redisfailover.yaml +++ b/packages/apps/redis/templates/redisfailover.yaml @@ -9,6 +9,14 @@ top-level resources falls back to resourcesPreset. */}} {{- $sentinel := .Values.sentinel | default dict -}} +{{- /* + A lone Redis pod has no replica to promote, so three Sentinels would only + watch it. One Sentinel still lets the operator find the master (its quorum is + replicas/2+1, so 1). The operator gives the Sentinel Deployment a PDB with + minAvailable 1 whenever Redis has two replicas or fewer, which on a single + Sentinel would block every node drain, so that PDB is dropped with it. +*/}} +{{- $sentinelReplicas := ternary 1 3 (le (int .Values.replicas) 1) -}} {{- $sentinelPreset := $sentinel.resourcesPreset | default .Values.resourcesPreset -}} {{- $sentinelResources := .Values.resources -}} {{- if or $sentinel.resources $sentinel.resourcesPreset -}} @@ -65,7 +73,10 @@ spec: # pods they keep no RDB, only a sentinel.conf the init container rewrites # on every start, so a chart rollback to an older engine is safe here. image: {{ $redisImage | quote }} - replicas: 3 + replicas: {{ $sentinelReplicas }} + {{- if eq $sentinelReplicas 1 }} + disablePodDisruptionBudget: true + {{- end }} resources: {{- include "cozy-lib.resources.defaultingSanitize" (list $sentinelPreset $sentinelResources $) | nindent 6 }} redis: image: {{ $redisImage | quote }} @@ -152,8 +163,8 @@ metadata: labels: workloads.cozystack.io/resource-preset: {{ $sentinelPreset | quote }} spec: - minReplicas: 2 - replicas: 3 + minReplicas: {{ ternary 1 2 (eq $sentinelReplicas 1) }} + replicas: {{ $sentinelReplicas }} kind: redis type: sentinel selector: diff --git a/packages/apps/redis/tests/sentinel_replicas_test.yaml b/packages/apps/redis/tests/sentinel_replicas_test.yaml new file mode 100644 index 000000000..012f58247 --- /dev/null +++ b/packages/apps/redis/tests/sentinel_replicas_test.yaml @@ -0,0 +1,103 @@ +suite: Sentinel count follows the Redis replicas + +release: + name: my-redis + namespace: tenant-test + +set: + _cluster: + cluster-domain: cluster.local + _namespace: + host: example.com + +templates: + - templates/redisfailover.yaml + +tests: + - it: "runs a single Sentinel for a single Redis pod" + set: + replicas: 1 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 1 + - equal: + path: spec.redis.replicas + value: 1 + + # The operator's Sentinel PDB is minAvailable 1 at two Redis replicas or + # fewer; on one Sentinel it would refuse every eviction and block node drains. + - it: "drops the Sentinel PDB when only one Sentinel runs" + set: + replicas: 1 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.disablePodDisruptionBudget + value: true + + - it: "keeps three Sentinels and their PDB from two Redis pods up" + set: + replicas: 2 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + - notExists: + path: spec.sentinel.disablePodDisruptionBudget + + - it: "keeps three Sentinels for three Redis pods" + set: + replicas: 3 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + + - it: "keeps three Sentinels at the chart default" + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + + - it: "reports one expected Sentinel pod to the workload monitor" + set: + replicas: 1 + documentSelector: + path: metadata.name + value: my-redis-sentinel + asserts: + - equal: + path: spec.replicas + value: 1 + - equal: + path: spec.minReplicas + value: 1 + + - it: "reports three expected Sentinel pods to the workload monitor" + set: + replicas: 2 + documentSelector: + path: metadata.name + value: my-redis-sentinel + asserts: + - equal: + path: spec.replicas + value: 3 + - equal: + path: spec.minReplicas + value: 2 diff --git a/packages/apps/redis/values.schema.json b/packages/apps/redis/values.schema.json index 4bf3e0964..1a301615f 100644 --- a/packages/apps/redis/values.schema.json +++ b/packages/apps/redis/values.schema.json @@ -3,7 +3,7 @@ "type": "object", "properties": { "replicas": { - "description": "Number of Redis replicas.", + "description": "Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.", "type": "integer", "default": 2 }, @@ -172,7 +172,7 @@ "minimum": 0 }, "sentinel": { - "description": "Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.", + "description": "Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.", "type": "object", "default": {}, "properties": { diff --git a/packages/apps/redis/values.yaml b/packages/apps/redis/values.yaml index 2a580c8e6..7917ca34d 100644 --- a/packages/apps/redis/values.yaml +++ b/packages/apps/redis/values.yaml @@ -55,7 +55,7 @@ ## @value xlarge ## @value 2xlarge -## @param {int} replicas - Number of Redis replicas. +## @param {int} replicas - Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to. replicas: 2 ## @param {Resources} [resources] - Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied. @@ -127,5 +127,5 @@ authEnabled: true ## @field {Resources} [resources] - Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`. ## @field {ResourcesPreset} [resourcesPreset] - Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`. -## @param {Sentinel} [sentinel] - Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. +## @param {Sentinel} [sentinel] - Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. sentinel: {} diff --git a/packages/system/redis-rd/cozyrds/redis.yaml b/packages/system/redis-rd/cozyrds/redis.yaml index a1d619c34..cfcaeb01d 100644 --- a/packages/system/redis-rd/cozyrds/redis.yaml +++ b/packages/system/redis-rd/cozyrds/redis.yaml @@ -8,7 +8,7 @@ spec: plural: redises singular: redis openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} + {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} release: prefix: redis- labels: From 7c1bd33b08a6cb690cf574de7fe392f15cae771d Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 2 Nov 2026 11:23:46 +0100 Subject: [PATCH 2/5] fix(redis): delete the Sentinel PDB left behind on scale-down Setting sentinel.disablePodDisruptionBudget only stops the operator from ensuring the PDB; it never deletes the one it created while three Sentinels ran. An existing instance scaled to one Redis pod therefore kept minAvailable 1 on its single Sentinel, which refuses every eviction. A post-upgrade hook deletes rfs- in that case. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- packages/apps/redis/README.md | 2 +- packages/apps/redis/templates/_resources.tpl | 4 + .../redis/templates/hooks/sentinel-pdb.yaml | 119 ++++++++++++++++++ .../apps/redis/templates/redisfailover.yaml | 6 +- .../redis/tests/sentinel_pdb_hook_test.yaml | 78 ++++++++++++ 5 files changed, 206 insertions(+), 3 deletions(-) create mode 100644 packages/apps/redis/templates/hooks/sentinel-pdb.yaml create mode 100644 packages/apps/redis/tests/sentinel_pdb_hook_test.yaml diff --git a/packages/apps/redis/README.md b/packages/apps/redis/README.md index 28152d9f9..478f75986 100644 --- a/packages/apps/redis/README.md +++ b/packages/apps/redis/README.md @@ -79,7 +79,7 @@ See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-pr ### sentinel -`sentinel.resources` and `sentinel.resourcesPreset` size the three Sentinel pods separately from the Redis pods. A Sentinel holds only its configuration and the failover state, so it needs a small fraction of what a Redis replica holding the dataset does. When the `sentinel` block sets neither, the Sentinel pods are sized like the Redis pods, which is how every instance was sized before these parameters existed. +`sentinel.resources` and `sentinel.resourcesPreset` size the Sentinel pods separately from the Redis pods. A Sentinel holds only its configuration and the failover state, so it needs a small fraction of what a Redis replica holding the dataset does. When the `sentinel` block sets neither, the Sentinel pods are sized like the Redis pods, which is how every instance was sized before these parameters existed. ```yaml resourcesPreset: u1.small diff --git a/packages/apps/redis/templates/_resources.tpl b/packages/apps/redis/templates/_resources.tpl index 6539c99a1..98119dbb2 100644 --- a/packages/apps/redis/templates/_resources.tpl +++ b/packages/apps/redis/templates/_resources.tpl @@ -47,3 +47,7 @@ These presets are for basic testing and not meant to be used in production {{- printf "ERROR: Preset key '%s' invalid. Allowed values are %s" .type (join "," (keys $presets)) | fail -}} {{- end -}} {{- end -}} + +{{- define "redis.sentinelReplicas" -}} +{{- ternary 1 3 (le (int .Values.replicas) 1) -}} +{{- end -}} diff --git a/packages/apps/redis/templates/hooks/sentinel-pdb.yaml b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml new file mode 100644 index 000000000..e29613c8a --- /dev/null +++ b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml @@ -0,0 +1,119 @@ +{{- /* + An instance scaled down to one Redis pod keeps the Sentinel PDB the operator + created while it ran three Sentinels: setting disablePodDisruptionBudget only + stops the operator from ensuring it, never deletes it, and minAvailable 1 on + a single Sentinel refuses every eviction. A fresh install with one replica + never gets that PDB, so this runs on upgrade only. +*/}} +{{- if and .Release.IsUpgrade (eq (include "redis.sentinelReplicas" . | int) 1) }} +{{- $name := printf "%s-sentinel-pdb" .Release.Name }} +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +rules: + - apiGroups: ["policy"] + resources: ["poddisruptionbudgets"] + resourceNames: ["rfs-{{ .Release.Name }}"] + verbs: ["get", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ $name }} +subjects: + - kind: ServiceAccount + name: {{ $name }} + namespace: {{ .Release.Namespace }} +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "5" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + ttlSecondsAfterFinished: 3600 + # Bounds the kubectl call too: a --request-timeout would make kubectl skip + # its in-cluster credentials (hack/check-hook-request-timeout.bats). + activeDeadlineSeconds: 120 + backoffLimit: 2 + template: + metadata: + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + policy.cozystack.io/allow-to-apiserver: "true" + spec: + serviceAccountName: {{ $name }} + restartPolicy: Never + terminationGracePeriodSeconds: 10 + securityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + containers: + - name: delete-pdb + image: '{{ include "cozy-lib.image" (list "clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c" $) }}' + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + cpu: 200m + memory: 128Mi + env: + - name: HOME + value: /tmp + command: + - kubectl + - --namespace={{ .Release.Namespace }} + - delete + - poddisruptionbudget + - rfs-{{ .Release.Name }} + - --ignore-not-found=true + volumeMounts: + - name: tmp + mountPath: /tmp + volumes: + - name: tmp + emptyDir: {} +{{- end }} diff --git a/packages/apps/redis/templates/redisfailover.yaml b/packages/apps/redis/templates/redisfailover.yaml index 2dd1fecaf..05a96e3cc 100644 --- a/packages/apps/redis/templates/redisfailover.yaml +++ b/packages/apps/redis/templates/redisfailover.yaml @@ -14,9 +14,11 @@ watch it. One Sentinel still lets the operator find the master (its quorum is replicas/2+1, so 1). The operator gives the Sentinel Deployment a PDB with minAvailable 1 whenever Redis has two replicas or fewer, which on a single - Sentinel would block every node drain, so that PDB is dropped with it. + Sentinel would block every node drain, so that PDB is dropped with it; the + operator does not delete a PDB it stops managing, hence + hooks/sentinel-pdb.yaml. */}} -{{- $sentinelReplicas := ternary 1 3 (le (int .Values.replicas) 1) -}} +{{- $sentinelReplicas := include "redis.sentinelReplicas" . | int -}} {{- $sentinelPreset := $sentinel.resourcesPreset | default .Values.resourcesPreset -}} {{- $sentinelResources := .Values.resources -}} {{- if or $sentinel.resources $sentinel.resourcesPreset -}} diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml new file mode 100644 index 000000000..71ef30e23 --- /dev/null +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -0,0 +1,78 @@ +suite: stale Sentinel PDB removed on scale-down to one Redis pod + +# The operator only stops ensuring the Sentinel PDB once +# disablePodDisruptionBudget is set; it never deletes the one it created while +# three Sentinels ran. Left in place, its minAvailable 1 refuses every eviction +# of the single Sentinel and blocks node drains. + +templates: + - templates/hooks/sentinel-pdb.yaml + +release: + name: my-redis + namespace: tenant-test + +set: + _cluster: + cluster-domain: cluster.local + _namespace: + host: example.com + +tests: + - it: deletes the operator's Sentinel PDB on an upgrade to one replica + release: + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.annotations["helm.sh/hook"] + value: post-upgrade + - equal: + path: spec.template.spec.containers[0].command + value: + - kubectl + - --namespace=tenant-test + - delete + - poddisruptionbudget + - rfs-my-redis + - --ignore-not-found=true + - notMatchRegex: + path: spec.template.spec.containers[0].command[4] + pattern: request-timeout + + - it: lets the hook delete only that PDB + release: + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Role + asserts: + - equal: + path: rules + value: + - apiGroups: ["policy"] + resources: ["poddisruptionbudgets"] + resourceNames: ["rfs-my-redis"] + verbs: ["get", "delete"] + + - it: renders nothing on install, where no such PDB exists + set: + replicas: 1 + asserts: + - hasDocuments: + count: 0 + + - it: renders nothing while three Sentinels run + release: + upgrade: true + set: + replicas: 2 + asserts: + - hasDocuments: + count: 0 From 2541aec62550e9b5cee12ec6e15cc5b58bc5adc9 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 2 Nov 2026 11:32:58 +0100 Subject: [PATCH 3/5] fix(redis): keep the Sentinel PDB hook Job name within 63 characters The API server copies a Job's name into its pods' job-name label, which takes 63 characters at most, and a release name may take 53, so -sentinel-pdb could be refused and leave the stale PDB in place. The Job name is truncated; the ServiceAccount, Role and RoleBinding keep the full name. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../redis/templates/hooks/sentinel-pdb.yaml | 5 ++- .../redis/tests/sentinel_pdb_hook_test.yaml | 36 +++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/packages/apps/redis/templates/hooks/sentinel-pdb.yaml b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml index e29613c8a..1649caa79 100644 --- a/packages/apps/redis/templates/hooks/sentinel-pdb.yaml +++ b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml @@ -57,7 +57,10 @@ subjects: apiVersion: batch/v1 kind: Job metadata: - name: {{ $name }} + {{- /* The API server copies a Job's name into its pods' job-name label, + which takes 63 characters at most, while a release name may take 53. + The objects above have no such limit and keep the full name. */}} + name: {{ $name | trunc 63 | trimSuffix "-" }} labels: app.kubernetes.io/instance: {{ .Release.Name }} annotations: diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml index 71ef30e23..bc6aea706 100644 --- a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -76,3 +76,39 @@ tests: asserts: - hasDocuments: count: 0 + + # The pods' job-name label takes 63 characters at most; a 53-character + # release name plus -sentinel-pdb would pass it and Kubernetes would refuse + # the Job, leaving the stale PDB in place. + - it: keeps the Job name within 63 characters for the longest release name + release: + name: redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Job + asserts: + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9-]{1,63}$ + - matchRegex: + path: metadata.name + pattern: '[a-z0-9]$' + - equal: + path: spec.template.spec.containers[0].command[4] + value: rfs-redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + + - it: keeps the full Job name for a short release name + release: + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: my-redis-sentinel-pdb From a7f84242d1b82398f5dd478452bc8e0ae8f891cd Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Sun, 8 Nov 2026 19:36:44 +0100 Subject: [PATCH 4/5] fix(redis): keep three Sentinels for a zero or negative replica count The operator runs three Redis pods when the count is zero or negative, but the chart treated any count up to 1 as the single-pod case. Such a cluster got one Sentinel and no Sentinel PDB, while three Redis pods ran under it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- packages/apps/redis/templates/_resources.tpl | 2 +- .../redis/tests/sentinel_pdb_hook_test.yaml | 20 +++++++++++++ .../redis/tests/sentinel_replicas_test.yaml | 28 +++++++++++++++++++ 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/packages/apps/redis/templates/_resources.tpl b/packages/apps/redis/templates/_resources.tpl index 98119dbb2..45399c958 100644 --- a/packages/apps/redis/templates/_resources.tpl +++ b/packages/apps/redis/templates/_resources.tpl @@ -49,5 +49,5 @@ These presets are for basic testing and not meant to be used in production {{- end -}} {{- define "redis.sentinelReplicas" -}} -{{- ternary 1 3 (le (int .Values.replicas) 1) -}} +{{- ternary 1 3 (eq (int .Values.replicas) 1) -}} {{- end -}} diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml index bc6aea706..5676f8d31 100644 --- a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -77,6 +77,26 @@ tests: - hasDocuments: count: 0 + # The operator runs three Redis pods for a zero or negative count, so their + # Sentinel PDB stays. + - it: renders nothing for a zero replica count + release: + upgrade: true + set: + replicas: 0 + asserts: + - hasDocuments: + count: 0 + + - it: renders nothing for a negative replica count + release: + upgrade: true + set: + replicas: -1 + asserts: + - hasDocuments: + count: 0 + # The pods' job-name label takes 63 characters at most; a 53-character # release name plus -sentinel-pdb would pass it and Kubernetes would refuse # the Job, leaving the stale PDB in place. diff --git a/packages/apps/redis/tests/sentinel_replicas_test.yaml b/packages/apps/redis/tests/sentinel_replicas_test.yaml index 012f58247..fc60d52b0 100644 --- a/packages/apps/redis/tests/sentinel_replicas_test.yaml +++ b/packages/apps/redis/tests/sentinel_replicas_test.yaml @@ -54,6 +54,34 @@ tests: - notExists: path: spec.sentinel.disablePodDisruptionBudget + # The operator turns a zero or negative count into three Redis pods, so these + # keep three Sentinels and their PDB rather than the single-pod case. + - it: "keeps three Sentinels and their PDB for a zero replica count" + set: + replicas: 0 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + - notExists: + path: spec.sentinel.disablePodDisruptionBudget + + - it: "keeps three Sentinels and their PDB for a negative replica count" + set: + replicas: -1 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + - notExists: + path: spec.sentinel.disablePodDisruptionBudget + - it: "keeps three Sentinels for three Redis pods" set: replicas: 3 From 9fba8af69cd2083fbfbeb21e14551f21ad47a73d Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Sun, 8 Nov 2026 19:45:18 +0100 Subject: [PATCH 5/5] test(redis): pin the release name in the Sentinel PDB hook for the longest release The long-name case checked only that the Job name fits in 63 characters and that the Job targets the right PDB. A Job name that dropped the release, letting two releases' hooks replace each other, or a Role granting a different PDB would both have passed. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../redis/tests/sentinel_pdb_hook_test.yaml | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml index 5676f8d31..03f3da02d 100644 --- a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -116,10 +116,30 @@ tests: - matchRegex: path: metadata.name pattern: '[a-z0-9]$' + # The full release name stays in the Job name, so the hooks of two + # releases in one namespace never replace each other's Job. + - matchRegex: + path: metadata.name + pattern: ^redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa- - equal: path: spec.template.spec.containers[0].command[4] value: rfs-redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + - it: lets the hook delete the Sentinel PDB of the longest release name + release: + name: redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Role + asserts: + - equal: + path: rules[0].resourceNames + value: + - rfs-redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + - it: keeps the full Job name for a short release name release: upgrade: true